Jexer : About | Downloads | Screenshots | Sixel Tests | API Docs | Wiki | GitLab Project Page | SourceForge Project Page

TL;DR


Introduction

My very first computer program was written in BASIC on an Apple IIe in elementary school, sometime in 1985 or 1986. Naturally it was the famous 10 PRINT "HELLO WORLD" 20 GOTO 10 that everyone was instructed to do. In those days, in the beginning, all was the command line.

In 1988 I was fortunate enough to obtain my first computer of my own: an IBM PC XT clone made by DTK, almost identical to the one pictured here. Mine ran MS-DOS 3.3, had a full 640k RAM, two 360k floppy drives, and a color (CGA) monitor. I quickly learned how to make small programs in GWBASIC, then moved on to Turbo Pascal, Turbo Assembler, and Turbo C++. Along the way I got a 2400 bps modem, 30MB hard drive, and started dialing local bulletin board systems (BBSes). Those were amazing years to have a computer: the world was shrinking rapidly, and the text interfaces of the BBS world were a great equalizer flattening the gap between those like me with a 16-bit CPU and CGA, and the mainstream users with a 32-bit CPU and VGA.

In the thirty years between the fall of DOS and the rise of vast cloud computing infrastructures, text interfaces have made a bit of a comeback. My largest personal projects have been in text-based programs, recreating the best parts of the DOS world for modern systems. This document outlines my progression through the years, with some minor dips into the technical bits.


Stage 1: ANSI.SYS

"ANSI colors", also called "escape sequences" and "VT100 codes", go a very long way to making the command line nicer. I was first exposed to ANSI.SYS (later replaced with PC Magazine's ANSI.COM driver) by the BBS world. Connecting to sites and seeing glorious color was amazing!

Despite writing lots of console programs, I did not really use ANSI sequences until I moved to Linux in the 90's. Prior to that, when I programmed in DOS using assembly language, I would just write directly to video memory at 0xB800:0000 where I could set color, blinking, and so on. This was the same method used by the excellent DOS mouse-supported text user interfaces (TUI) like Turbo Vision:

I switched full-time to Linux around 1999, and at that time I started using ANSI sequences, first to colorize logger messages (errors in red), and later to draw boxes on the screen that vaguely looked like the TUI windows of the DOS era. Below is the earliest version of ANSI sequences I used, dating from about November 2000: 

// ANSI Console Sequences
#define ANSI_CLEAR      "\033[2J"

#define ANSI_UP         "\033[1A"
#define ANSI_DOWN       "\033[1B"
#define ANSI_FORWARD    "\033[1C"
#define ANSI_BACKWARD   "\033[1D"

#define ANSI_F_BLACK    "\033[30m"
#define ANSI_F_RED      "\033[31m"
#define ANSI_F_GREEN    "\033[32m"
#define ANSI_F_YELLOW   "\033[33m"
#define ANSI_F_BLUE     "\033[34m"
#define ANSI_F_MAGENTA  "\033[35m"
#define ANSI_F_CYAN     "\033[36m"
#define ANSI_F_WHITE    "\033[37m"

#define ANSI_B_BLACK    "\033[40m"
#define ANSI_B_RED      "\033[41m"
#define ANSI_B_GREEN    "\033[42m"
#define ANSI_B_YELLOW   "\033[43m"
#define ANSI_B_BLUE     "\033[44m"
#define ANSI_B_MAGENTA  "\033[45m"
#define ANSI_B_CYAN     "\033[46m"
#define ANSI_B_WHITE    "\033[47m"

#define ANSI_NORMAL     "\033[0m"
#define ANSI_BOLD       "\033[1m"
#define ANSI_UNDERLINE  "\033[4m"
#define ANSI_BLINK      "\033[5m"
#define ANSI_REVERSE    "\033[7m"
#define ANSI_CONCEALED  "\033[8m"

The header above shows the basics of ANSI.SYS. One can clear the screen (which also homes the cursor), move up/down/left/right, set attributes, and set foreground and background color. This is enough to draw the entire screen. One only needs to add code to figure out the screen size (usually ioctl(TIOCGWINSZ)), and now you have what many people call a "TUI" application.

A lot of people stop here. The have colorized output, can respond to basic keystrokes (not special keys like arrows and function keys), and users are happy.


Stage 2: ncurses

In 2003 I got fed up with the poor state of the raw Linux console. Sure we had colors, but every time I switched virtual terminals I lost my scrollback, could not capture what I saw to file, and could not use serial file transfer protocols. So I started writing a program called Qodem that I thought would be a matter of a few weeks, but instead took about 14 years before I called it done.

Following this thread of my terminal programming evolution: those ANSI colors from way back in November 2000 became the basis of Qodem's color handling. Qodem used a logo screen I made with TheDraw, which saved the screen as a pair of attribute/CP437 bytes just like CGA. For each attribute, I needed to convert it to a curses COLOR macro. Below is the map of VGA to curses colors: 

/*
 * convert_thedraw_screen() defines its colors in terms of the CGA bitmask.
 * This maps those bits to a curses color number.
 */
static short pc_to_curses_map[] = {
    COLOR_BLACK,
    COLOR_BLUE,
    COLOR_GREEN,
    COLOR_CYAN,
    COLOR_RED,
    COLOR_MAGENTA,

    /*
     * This is really brown
     */
    COLOR_YELLOW,

    /*
     * Really light gray
     */
    COLOR_WHITE

    /*
     * The bold colors are:
     *
     * dark gray
     * light blue
     * light green
     * light cyan
     * light red
     * light magenta
     * yellow
     * white
     */
};

Qodem went far beyond just emitting ANSI sequences for color and position. It used the ncurses library to handle screen and keyboard, allowing it to run similarly on Linux console, Xterm, and many, many more. ncurses gives you lots of capabilities, including but not limited to:

Using these features, Qodem was eventually able to get very pretty:

Given the list above, one would think that ncurses gives you everything you need for a TUI as good as Turbo Vision. With sufficient attention to detail, one could in fact pull it off. But there are some limitations:

Qodem was also a very unusual program in that it was a terminal emulator for several different terminal types: Avatar, ANSI.SYS, VT52, VT100/102, VT220, Linux, and a subset of Xterm. Because Qodem had to be "both sides" of the terminal, I had to learn how to get true VT100/102 output on to the ncurses screen. This led to some pretty interesting hacks as seen below:

By the time I was finished with Qodem, I had implemented several workarounds that most ncurses applications do not need:

Almost everyone else stops here. The have a nearly complete TUI experience that works on lots of platforms, and users are very happy.

Yet I continued on...


Stage 3: Directly Speaking To Xterm

In 2013 I wanted to write my own BBS. But as I thought through the mini-BBS I wrote for Qodem (what it calls "host mode"), I was unsatisfied with trying a similar solution to Qodem's host mode. Qodem uses a basic state machine: show screen, get input, switch screen, continue. I wanted something more general-purpose. In fact, I wanted something like ... Turbo Vision.

A full mouse-driven TUI like Turbo Vision. How would I create that? It was a pretty bold undertaking. But as they say about climbing mountains: one step at a time. I even had my first widget in mind: Qodem's text field library.

At that time I was experimenting with the D programming language, and called this project D-TUI. My first step was taking Qodem's C source and making some objects out of it:

Following the thread of evolution, the original ANSI sequences that became Qodem's CGA to curses color map morphed into the basis of D-TUI's colors: 

enum Color {

    /// Black.  Bold + black = dark grey
    BLACK   = 0,

    /// Red
    RED     = 1,

    /// Green
    GREEN   = 2,

    /// Yellow.  Sometimes not-bold yellow is brown
    YELLOW  = 3,

    /// Blue
    BLUE    = 4,

    /// Magenta (purple)
    MAGENTA = 5,

    /// Cyan (blue-green)
    CYAN    = 6,

    /// White
    WHITE   = 7,
}

    /**
     * Create a SGR parameter sequence for both foreground and
     * background color change.
     *
     * Params:
     *    foreColor = one of the Color.WHITE, Color.BLUE, etc. constants
     *    backColor = one of the Color.WHITE, Color.BLUE, etc. constants
     *    header = if true, make the full header, otherwise just emit
     *    the color parameter e.g. "31;42;"
     *
     * Returns:
     *    the string to emit to an ANSI / ECMA-style terminal, e.g. "\033[31;42m"
     */
    public static string color(Color foreColor, Color backColor,
        bool header = true) {

        uint ecmaForeColor = foreColor;
        uint ecmaBackColor = backColor;

        // Convert Color.* values to SGR numerics
        ecmaBackColor += 40;
        ecmaForeColor += 30;

        auto writer = appender!string();
        if (header) {
            formattedWrite(writer, "\033[%d;%dm", ecmaForeColor, ecmaBackColor);
        } else {
            formattedWrite(writer, "%d;%d;", ecmaForeColor, ecmaBackColor);
        }
        return writer.data;
    }

With bits from Qodem, I had my initial guide for making D-TUI's ECMA-48 backend. Wait, backend? What?

A TUI is just like a GUI, a full windowing system. The abstractions at its heart are the window, widget, and event. (Java's Swing GUI calls these pieces JFrame, JComponent, and AWTEvent.) User input gets turned into events, events are routed to widgets that are contained in windows, and windows are drawn to the screen. Turning X10 mouse protocol into a TMouseEvent is the easy part, getting those mouse actions turned into draggable resizable windows is much harder. Once the core windowing system is running, the user-facing side can be anything.

It was a goal of mine to have something that could run on Windows and Mac, as well as Linux. Thus ncurses was out. But I knew plenty enough about Xterm from my Qodem experience that I figured I could just assume I was speaking directly to an Xterm and save a lot of hassle. By 2013 the network effect had wiped out the non-Xterm non-VT102/220 terminals anyway. So I set about making a TUI that spoke Xterm and Win32 console.

Getting D-TUI running only took a few months, but getting it running well took several years. By then it had been transliterated into Java and become Jexer, with backends for Xterm and Swing, Qodem's VT102/VT220/Xterm emulator rebuilt as an object-oriented terminal widget, and lots of corner cases related to threading and window management worked out. Most importantly, the Xterm backend was fully capable of matching the DOS-era TUI systems.

Jexer has menus, windows, status bars, and lots of working widgets: text fields (taken from Qodem's text field code), checkboxes, radiobuttons, message boxes, treeviews, scrollbars, and many more. It runs glass smooth on Swing and Xterm, can replicate its screen to multiple windows, and can be instantiated multiple times in a single Swing application. It is almost everything the fixed-size text-cell metaphor can be.

Yet there was one more frontier waiting to be tackled. Xterm is much more than DOS, and it was time to clear the final boss.


Stage 4: Let Me Show You What A Real Xterm Can Do

Xterm goes very deep. It slices and dices text, can do horizontal and vertical scrolling regions, and has three different methods of putting graphics on the screen: Tektronix 4014, ReGIS, and sixel. The first two are vector based, and not well suited to what I need. But sixel, well sixel can be made to do wonders! I had a working TUI. It ran on multiple platforms, with multiple backends. What would it take to get bitmap images to work? For the Swing backend, it was a matter of one afternoon. But for Xterm, it was quite a bit more.

The primary trick to sixel isn't encoding bitmap data, it's doing it quick enough to be worthwhile. Sixel support for a general-purpose TUI has several constraints to work within:

Along the way to implementing sixel support, I also discovered a critical Xterm-specific implementation detail: if I overwrote a part of a continuous sixel image with text, the screen areas to the right and below that part of the image got corrupted. That meant I could not just put the mouse cursor where I wanted at the very end of the master drawing loop as normal: if the mouse was on top of an image cell, it would have to either be drawn as an image cell itself, or the entire row of image cells would have be drawn as at minimum three parts: left of mouse, mouse, then right of mouse.

Sixel evolved through many iterations:

  1. I generated a basic palette using a hue / saturation / luminance model, with defined number of bits for each parameter. This provided a wide-ranging rainbow of colors.
  2. I converted the 24-bit ("true color") RGB image data associated with a cell to an indexed image using Floyd-Steinberg dithering.
  3. I got a basic image on the screen. Colors were initially very wrong because I did not set the palette on every sixel output sequence.
  4. I set the colors on each image cell, and now the image appeared correctly.
  5. I got the sixel output limited to only the affected text cells, i.e. fixed it so that it did not create artifacts outside the TImage area.
  6. I changed the palette matching code from using Euclidean distance to the nearest color in the palette to converting the RGB to HSL and choosing from only the nearest neighbors in HSL space. This eliminated about 97% of the search time per pixel.
  7. I grouped adjacent cells with image data into a single larger sixel image, which prevented duplicating the palette information a lot.
  8. I cached the previously generated sixel output: if the same sequence of cells came in, just pass back what was calculated before.

The most interesting change was the switch from Euclidean to HSL search in SixelPalette.matchColor(). The resulting color space dithers like the color wheel below:

Even though Euclidean search should in theory be more accurate, the color wheel with Euclidean search looked much worse: blotchy with weird banding. The HSL match results in banding that follows the circles and approximate radii of the palette. In this case worse seemed to be better: much faster, less accurate, but more in line with what people see in other media like newspapers and pulp magazines. I ultimately ended up with a 1024-color general-purpose palette that works quite well for a number of different kinds of images: faces, high color objects, and nature. See for yourself:

It may not look like much, but the picture above represents a world first: overlapping images mixed with text cells on the same Xterm screen. See how nature.jpg image and its window border covers part of the Gemstones.jpg image: to my knowledge, that has never been done before. What has been done before is:

(Ed March 9, 2021: notcurses is another TUI system that has successfully integrated text and images. It's really great, and is pushing the envelope past anything that has come before. Please check it out.)

Let me take a sidebar: Why is Jexer's 100% Java design important to me? Because in accepting that limitation, I have figured out techniques to get Xterm to do what I want that can work in any language, not just those that have easy access to C libraries. Someone could port Jexer to Common Lisp, JavaScript, VB.NET, Python, Perl, or really any language that can read and write to stdin/stdout and spawn programs. Because it turned out that with just the ability to spawn 'stty' and 'script', I could get a working system. You don't need direct access to C's termios or forkpty(), so the hurdle to figuring out a language's foreign function interface is gone.

Getting back to the palette discussion, I was happily surprised to see that even light skin tones and hair look pretty good:

Alas, my palette can't win them all. :-) Below you can clearly see where some of the text cell borders are in the dark areas.

Still, with sixel support in Jexer, I feel that I have complete control of what I want my Xterm to do. The final boss is down, and I can put my focus now on the wrap up: fixing bugs, further performance improvements, and polish. A few months later, I added double-width / double-height to Jexer's terminal window, and over about 8 hours got sixel input parsing working. Below is yet another world first: Jexer running inside itself on xterm, and inside that displaying images and double-width/double-height:

I feel now that I have mastered the terminal.


Conclusion

Understanding and pushing the VT100/Xterm interface has been an interesting time in my programming life. Going from colorized log output to image colorspace matching feels a bit like the journey from Twinkle Twinkle Little Star to Tocotta And Fugue In D Minor.

My personal life has changed a lot through these years. I do not expect that I will be able to walk a path this complex again. But I have generally enjoyed my time doing this, and all of this code is available to the world under MIT and public domain / CC0 licenses for any use. Here are direct links my text-based projects:

It has been a long path, but I finally have the terminal interface I have been wanting to use.


Post-Script - End Of My Open-Source Hobby

On December 25, 2019, I ended the "open-source" part of my programming hobby. Most projects are archived read-only (GitHub/GitLab) or abandoned (SourceForge), all notifications are disabled, and I have severed what links from open-source to my real life that I can find.

I will continue to write code that I need/want, but for the foreseeable future will just be putting it "out there" with none of the project management stuff that often comes with the open-source label. Maybe I write documentation; maybe I test on other platforms, distros, and compilers; maybe I build a deb/rpm package. Or maybe I don't do any of that.

For now, issues are just for me, and merges are disabled. I have put some additional thoughts on the future of terminals here.

Please do not contact me regarding these projects. Any such mail will most likely be deleted without being read.


Post-Post-Script - I Am Transgender

In summer 2020 I accepted myself as a trans woman, and am now transitioning. If you are a programmer, or otherwise connected to open-source, please keep in mind the following:


The Epilogue That Never Ends - Pandemic Edition

2021 was really something else, huh? I spent nearly all of it working from home while starting up puberty number two. Most of this year I have been almost entirely unable to code, even when I wanted to. It seems HRT often increases ADHD symptoms (NSFW warning). That's a fun list, I've experienced most of the effects so far. My eyesight got slightly-less-bad around month 6 requiring new glasses (thank you Arlene at Lenscrafters, I love them!), more freckles all over, and effectively no body odor now. But not quite everything: cute shoes are never in my size...sigh....

I finally cleared some projects at work, took vacation, got some rest, and found myself able to actually code again. I don't know how long it will last, but for a while I am having fun. There are some really great terminals out there, it is exciting seeing some of the less-than-gigantic projects become their own unique gems. I credit the almost-supernatural developer Nick Black behind the amazing notcurses project for inspiring me for sure, and I hope others too. Look for yourself! This isn't my picture, it's from notcurses, and when I saw it the first time it really blew me away:

See that? Image over text, text over image, alpha blending, just wow!

It took a while for my terminal widget to minimally handle the notcurses demo, mainly due to performance -- the Java GC is both blessing and curse. So far I have had to: iron out thread contention between the terminal reading thread and screen rendering thread, implement more terminal reports so that notcurses can detect the features, and most importantly not cache every text cell being rendered by the Swing backend. (On my system, the memory pressure of that unbounded cache killed my X server. Several times. :-( ) There are still some things around fonts and Unicode sextant-type "glyphs" to clean up...

...But after all that, now I have bitmaps over text within the same cell, and that really did open a new world for terminals for me.

(Of course, this being multiheaded Jexer I can't help but show off a little :-) ...)

Performance was a huge drag, and a ton of crash bugs were in store. But here are some things I learned:

My last thing this year was building out custom bitmap layers which can be added both under and over widgets on the windows, and also over the whole screen. I call this Tackboard, because conceptually it looks like this:

You just stick stuff on it, and Tackboard will slice-and-dice it onto the logical screen and layer transparent image pieces along the way. Tackboard knows nothing of Jexer's windowing system: it is not a widget or window. You could take Jexer's packages for backend, bits, event, io, net, and tackboard, and then build an entirely new system out of it that looks and behaves however you wish. You could even build an AWT Toolkit on top of Tackboard and run AWT/Swing applications directly from an Xterm. (Which if no one gets to in a couple years, I just might do myself.)

I added SGR-Pixel mode (1016) for the mouse, and then had some fun with custom bitmap mice pointers, which can be over a widget or over the whole screen. My final screenshot shows images under and over text, under and over each other, and with smooth mouse motion. At this point I could feasibly abandon the text cell metaphor entirely and just go all-out pixel-based -- but that would be sooooooo slow. (So knowing me...I'll have to try that too someday. ;-) )

A couple weeks later Nick inspired me yet again to tackle better sixel encoding. I ended up putting together a few different strategies -- none of them nearly as good as notcurses, but overall they are a lot better and faster than what I had before -- and...well... can you tell that I remember the Geocities era fondly? ;-)

So I wound down 2021 with new things to play with. Thank you to the terminal and application authors out there continually banging away, inspiring me to better things. And for being kind and accepting of this most-probably-autistic-and-very-likely-ADHD fangirl of y'all.



Fin.

Why I Do Not Post My Projects Anymore

I am finished, again. All projects are archived, all notifications are disabled, and I have logged out.

These are dark times for all trans people, no matter where we live. I live in a "safe" state. My family does not. I do not know when, or if, I will ever see them in person again. With so many states and countries now openly hostile to all trans people -- telling us in no uncertain terms that they want us to die -- well, I have a lot of reflecting to do. No energy remains for open-source anything.

At this time I am seeking to avoid an outcome like this, or like this. You can click the trans heart in the footer to find a few memes and writings if you are so inclined.

We only want to live in peace.

Fix society. Please.

MIT Jexer is available to all under the MIT license. SourceForge.net Logo