Support for True Color (16 millions colors)

[akochkov]

Now more terminals bring support for 16 million colors (see references below).

Here's a test case for terminal:

printf "\x1b[38;2;255;100;0mTRUECOLOR\x1b[0m\n"

It's a common confusion about terminal colors... Actually we have this:

• plain ascii
• ansi escape codes (16 color codes with bold/italic and background)
• 256 color palette (216 colors+16gray + ansi) (colors are 24bit)
• 24bit true color (8*8*8 colors (aka 16 milion)

The 256 color palete is configured at start, and it's a 6*6*6 cube of colors, each of them defined as a 24bit (8*8*8 rgb) color.

This means that current support can only display 256 *different* colors in the terminal, while truecolor means that you can display 16 milion different colors at the same time.

Truecolor escape codes doesnt uses a color palete. It just specifies the color itself.

[1] ​https://en.wikipedia.org/wiki/ANSI_color

Here is terminals discussions:

Now supporting truecolor:

• st (from suckless) - ​http://lists.suckless.org/dev/1307/16688.html
• konsole (already fixed) ​https://bugs.kde.org/show_bug.cgi?id=138740
• all libvte based terminals: ​https://bugzilla.gnome.org/show_bug.cgi?id=704449
• sakura ​https://bugs.launchpad.net/sakura/+bug/1202564
• Also iterm2 have support for truecolor.

Not supporting truecolor:

• urxvt - ​http://lists.schmorp.de/pipermail/rxvt-unicode/2013q3/001826.html

P.S. Also sent email to the S-Lang library developers

[akochkov]

Here is another console programs discussion:

• mutt - ​http://dev.mutt.org/trac/ticket/3674
• mcabber - ​https://bitbucket.org/McKael/mcabber-crew/issue/126/support-for-true-color-16-millions-colors
• emacs - ​http://emacs.1067599.n5.nabble.com/RFC-Add-tty-True-Color-support-td299962.html
• tig - ​https://github.com/jonas/tig/issues/227
• vim - ​https://bitbucket.org/ZyX_I/vim/commits/5e3877f

[egmont]

Slang thread: ​http://lists.jedsoft.org/lists/slang-users/2014/0000002.html
Ncurses thread: ​https://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00007.html

I'd be happy to work on true color support (btw I was the one adding 256 colors in #2169) if I knew how to do in slang and ncurses - but it seems to me that at this moment neither of them provides support.

[egmont]

Note: I have a feeling that the 256 color support isn't as popular as it could be. Probably skins are not as popular in general, most people just use the default. This might be due to the lack of a runtime skin selector (#2165).

We've had 256 color support for about 3 years. In addition to the demo skin I created (which I still use every day, but haven't heard from anyone else :)) we ship only 2 more (one of them having a few minor variants), and I couldn't find other ones on the Internet either.

True colors could give a boost, it surely would be a cool feature, but it won't save the world :)

[ossi]

i would say the reason is that 256 colors just aren't all that useful in a TUI. by extension, TrueColor? seems like the most useless feature one could add to a terminal ever.

[akochkov]

Replying to ossi:

i would say the reason is that 256 colors just aren't all that useful in a TUI. by extension, TrueColor? seems like the most useless feature one could add to a terminal ever.

I can't agree with you. Especially if you're working mostly in console programs (like I am e.g.), you will need to make it beautiful (from your point of view of course), as much as possible.

[egmont]

ossi: Let me try to challenge you. In enlightenment's terminal you can - from command line - specify a background video to play. Do you still think true color is the most useless feature? :D :D :D

I think I get your point, true color (opposed to 256) is on one hand "why", on the other hand "why not"... For me, "why not" is slightly stronger.

[ossi]

that background video thingie doesn't need the true color for the actual terminal function. just sayin'. ;)

the thing is that the factual resolution of a terminal (and even more the applications it offers) does in no way enable you to make use of 16.7 million colors at the same time (it's quite unlikely that you have much more than 20k cells available to start with). even 256 colors at the same time is a bit of a challenge (i guess a different shade for every file type can get you there).
given that true color doesn't come for free (assuming a linear frame buffer of aligned structures, it doubles space requirements from 8 to 16 bytes per cell), and i *really* like my virtually unlimited scrollback, the "why not" seems pretty obvious to me.

[egmont]

I didn't imply any correlation between true colors and bg video, just mentioned something for fun that is IMO way more useless ;)

There's no point comparing the number of character cells to the number of available colors. My whole desktop is less than 16M pixels, still I have access to 16M colors. The point in terminal is not to use all of them at the same time, but to use any of them at any time.

I personally started using Linux and terminal in '96 (using X window of color depth 8, always changing colors so that only the focused window was displayed correctly, probably the worst user-unfriendly compromise I've ever seen). Since then, CPUs became at least 20-fold faster (and that's just the clock speed, so I'd bet 100-fold or similar), RAM installed became almost 1000-fold larger. Way back then, storing a character cell required probably 3 or 4 bytes. (8-bit encoding, but additional line drawing characters were already available, so that's more than 8 bits for the character, plus 9 fg + 9 bg colors (8 + default), bold, probably reverse, underline and blink were supported too, I'm not sure). Now the super wasteful way of storing would be 16 bytes, meaning an at most 5.33-fold (but probably much smaller) growth during this time period.

I see no reason why any terminal would store a cell in 16 bytes (4 unused) instad of 12, any access to the scrollback content is followed by drawing on the screen, compared to that the price of reading from nonaligned offset is negligible. As a matter of fact, using 12 bytes per cell is also a total waste. I don't know how terminals other than VTE store their scrollback content, I only know about VTE (gnome-terminal). For the onscreen part it uses the obvious way: records of 12 bytes; whereas for the scrollback it uses UTF-8 character stream, and a runlength encoding of identical attributes (that is, a record of 16 bytes consisting of an offset (8 bytes) where the attribute changes, and 8 bytes encoding the true colors and other attributes). In the worst case scenario (attribute changes in every cell, and 4-byte UTF-8) it's 20 bytes for a cell, but typically much less than that, usually (English text and no color change) it's just 1 byte. Other approaches might include compression of the scrollback, or whatnot. Software engineers have solved way tougher problems than this. If any terminal emulator is using 12+ bytes per cell and you think it's too much, please file them an enhancement request.

That being said, this discussion here is pretty pointless. If you have serious concerns, it should be brought up with terminal developers. Quite a few terminals already support true colors, probably some others will follow too. Once ncurses/slang makes this feature available for mc, I'll be happy to add support, it won't hurt anyone and this step definitely won't increase the terminal's memory usage. My current skin (sand256) was a heavy compromise between what I had imagined and what I could reach using the 256 colors. With true colors I'll be able to implement exactly what I imagine - this doesn't sound like a useless feature to me after all :)

[egmont]

I attached demo patches that let you use true colors in MC (using a patched Slang).

On 32 bit machines the Slang patch breaks the ABI compatibility. This means that you shouldn't replace the system wide Slang, or else I'm afraid all applications using Slang will break. You should also take care to compile MC against the patched Slang header files.

On 64 bit machines there's no such issue, Slang remains ABI compatible with the unpatched version, and it's okay to compile MC against the system-wide Slang headers.

In either cases I recommend to install the library under some experimental location, and use LD_LIBRARY_PATH or other similar mechanism to make the patched MC use that version instead of the system wide.

There's no terminfo support yet, so the Slang patch doesn't even try to verify if the terminal has support. It just uses hardcoded escape sequences. (Well, maybe it verifies that more than 8 colors are available, so set TERM=xterm-256color or similar.)

The MC patch is really minimal, it's basically just that it doesn't reject the new color names, and knows how to create a canonical string representation that is used as a key in a table. The actual interface to Slang uses color names (strings), and didn't change at all.

After applying the patches, you can start using standard #rrggbb-like color names in addition to the currently available ones. Short #rgb names are also accepted (e.g. #abc is the same as #aabbcc). Additional color names, such as the ones available in HTML or in X Window, are not supported.

Anton, if you take time to create a beatiful truecolor skin, I'd be happy to see it :)

[dickey]

I overlooked this item yesterday, but added it to my summary.

​http://invisible-island.net/ncurses/ncurses.faq.html#xterm_16MegaColors

[akochkov]

So, slang library now integrated True Color support, please integrate the support in the Midnight Commander as well ​http://lists.jedsoft.org/lists/slang-users/2015/0000020.html

Also, after years of filling that bug most of the terminals are now support the 24bit colors - see the current status of them at my gist ​https://gist.github.com/XVilka/8346728
And for those terminals, that are declined to add support of it, forks or patches are available.

[egmont]

The patch is work in progress (proof of concept), there's a fixme inside that needs to be addressed.

Ideally the feature should be submitted along with at least 1 brand new beautiful truecolor skin.

Anton, do you feel like working on either of these?

[egmont]

Also we'd need a proper error message if the user chooses a truecolor skin while the terminal doesn't support it; just as we do now with 256 color skins on an 8 color terminal. Checking for slang (as opposed to ncurses) and duplicating slang's check for COLORTERM sounds like a nice start. Plus check for slang's version if we can.

[akochkov]

@egmont: yes, I'll do the skin.

[egmont]

truecolor support v2

[egmont]

Here's the second version of the patch.

Memleak fixed.

Since it's a brand new and not widespread feature, bring up informative error box guiding the user towards a solution to have true colors.

Note that for now you have to change the "SLANG_VERSION < 20301" check to "SLANG_VERSION < 20300" to be able to try out the feature on locally on your computer; although the version committed should have 20301 just as in the patch.

[egmont]

Just for fun: There are at least 3 solarized skins for mc out there in the wild. They use the ancient 16-color palette and look good if you configure your terminal's colors. I've attached a script that converts these skins to use direct RGB so that no prior terminal setup is required for a nice looking color scheme.

[egmont]

Converts 16-color solarized skins to rgb

[god12]

Looking forward to see zenburn in true colors :)

[timofonic]

Any news?

Please...

I need True Color! :D

[egmont]

We're waiting for:

• an official Slang release (not prerelease) with true color support,

• someone to create a great true color skin (do you feel like creating one?).

[egmont]

FYI: S-Lang 2.3.1 (with true color support) has just been released. Announcement: ​http://lists.jedsoft.org/lists/slang-users/2016/0000014.html

Anyone feels like creating a beautiful true color skin (with its dominant colors being "far away" from the 256-color palette)?

[egmont]

I incorrectly stated that slang doesn't expose its runtime version and hence relied on the compile time version to see if it already supports true colors. This is now fixed, v3 checks the runtime version.

[egmont]

truecolor support v3

[egmont]

Guys (devs),

Could you please review this patch?

I'd prefer to make it into next mc, even if we're not ready with a nice demo skin yet.

In the mean time I have a (hopefully amazing) idea for a set of true color skins. It'll take quite some time to implement them, but we'll see if I manage to find the time and patience, and more importantly, manage to actually implement them given my close to zero graphical skills :)

[andrew_b]

Branch: 3145_true_color.
changeset:2cf9cd59f06e2cd09ebb33ca9a040ef5ec41fcc1

Looks OK. It doesn't break the current color handling.

[egmont]

Let's keep this ticket for the code change only, and have a separate ticket for the creation of some true color skins: #3724.

[andrew_b]

Merged to master: [682a5116edd20b8ba81743a1f7495c883b0ce644].

git log --pretty=oneline 3cf0d55..682a511

[egmont]

FYI: Expecting 256 colors available as a prerequisite for true color support may not be a good thing to have.

Let's keep an eye on ​https://bugs.kde.org/show_bug.cgi?id=371919 (comment 5 and whatever response it'll receive).

[egmont]

Okay, luckily konsole changed its default to TERM=xterm-26color, so there's nothing left to do on our side.

[mooffie]

I wrote a wiki page explaining how to compile Slang/MC on a 32-bits system to get 16M colors: doc-devel/slang-16Mcolors-32bits.

[egmont]

Nice idea, thanks a lot mooffie! I've updated the skins to point to that wiki.

Two nitpicks:

• I guess it would be nicer to point users to slang 2.3.1 (or 2.3.1a), and of course update the patch accordingly. Obviously and unfortunately it requires some more work.

• Once the skins will be committed (in just a few days probably), the "download a sample skin" section will no longer be needed.

Thanks!

[egmont]

... oh and if the slang patch is updated, it could incorporate the perl magic as well. I'm not sure if we should address irrelevant warnings, though.

[mooffie]

(I updated that wiki page to show how to use a static S-Lang library, not shared one, so I removed the LD_LIBRARY_PATH trick (but mentioned it in a comment).)

Replying to egmont:

• I guess it would be nicer to point users to slang 2.3.1 (or 2.3.1a), and of course update the patch accordingly. Obviously and unfortunately it requires some more work.

I don't think it's worth the bother. It'd require work. I'm not familiar with Slang's codebase.

Is there any word on when development on Slang 3.0 starts? There isn't any trace in their public repository for this.

• Once the skins will be committed (in just a few days probably), the "download a sample skin" section will no longer be needed.

Right, I updated the comment there.

... oh and if the slang patch is updated, it could incorporate the perl magic as well.

Good idea.

I'm not sure if we should address irrelevant warnings, though.

We should. They dominate the output and make it hard to develop. Now there are only two (or three) warnings (EDIT: no, I believe those two are emitted when compiling slang, not mc, so, indeed, they're utterly irrelevant to us.)

[egmont]

Replying to mooffie:

I don't think it's worth the bother. It'd require work. I'm not familiar with Slang's codebase.

I might do it one day.

Is there any word on when development on Slang 3.0 starts? There isn't any trace in their public repository for this.

I don't think there's any. We could ask the author. I'd guess it's very very far away.