Use own screen drawing method rather than slang/ncurses

[egmont]

MC currently uses either slang or ncurses for display.

It's one thing that I firmly disagree with having two choices. It's a really minor benefit for a very few users to choose from two. On the other hand, it's a significant developer overhead, double testing work, and source of many discrepancies between the two versions (e.g. #2164, #2291, #3114, #3158, #3254, and many more already closed bugs). I believe developers should pick one and offer only that. Bugs would no longer go unnoticed just because a developer used the other library, which would lead to better overall quality.

My recommendation is to not use any of these two, but to write an extremely simple basic screen handling component in mc. (This is what quite a few apps, e.g. vim, emacs, less do. They do rely on libtinfo [part of ncurses] for querying termcap/terminfo, but not on ncurses to draw the screen. Sounds the most reasonable approach to me.)

Ncurses is actively maintained but is a giant bloat, containinig bugs that effect mc's behavior (e.g. the just fixed ​http://lists.gnu.org/archive/html/bug-ncurses/2014-08/msg00020.html, although I'm afraid it still has similar bugs).

Slang is not really maintained and haven't seen a stable release in 3+ years.

Switching to our own drawing of the screen would allow to address these issues:

• #1849: mcview understanding ANSI color escapes, work like "less -R". This requires to get rid of the ancient brain-damaged palette support of both ncurses and slang, and instead have each cell store its fg and bg color.

• #3145 True color support. Neither ncurses nor slang seems to want to ever address this.

• #3065 Italic text. Supported by pre-versions of both libraries, but we'll have to wait for stable ncurses-5.10 and slang-2.3 to get it "official" and widespread. No idea when these are going to happen.

• Strikethrough and other possible text attributes: Ncurses will not add this because the bitfield is already fully packed. No info about slang.

• #3229 Pasting with proper EOL. Neither slang nor ncurses allows to set whether at the end of the line there's an explicit newline (to be copy-pasted) or an overflow (copy-pasting shouldn't add a newline). Terminal emulators, however, do remember which of the two happened. Our own layer could easily do the requested thing for every line.

• Similarly, spaces at the end of the line, or TAB characters could also be reserved for copy-paste purposes, this isn't offered either by these libraries.

[andrew_b]

Replying to egmont:

My recommendation is to not use any of these two, but to write an extremely simple basic screen handling component in mc.

Let write own slang/ncurses. Let write own glib (#157). What's than? (Looking to systemd) Let write own glibc (#125)?

I strongly disagree with this way.

Switching to our own drawing of the screen would allow to address these issues:

• #1849: mcview understanding ANSI color escapes, work like "less -R". This requires to get rid of the ancient brain-damaged palette support of both ncurses and slang, and instead have each cell store its fg and bg color.

• #3145 True color support. Neither ncurses nor slang seems to want to ever address this.

• #3065 Italic text. Supported by pre-versions of both libraries, but we'll have to wait for stable ncurses-5.10 and slang-2.3 to get it "official" and widespread. No idea when these are going to happen.

• Strikethrough and other possible text attributes: Ncurses will not add this because the bitfield is already fully packed. No info about slang.

I don't think that all these embellishments are really need in TUI program. Really. mc is the program for old users. Young Linux users use various GUI filemanagers and don't use mc. The transformation of mc to quasi-GUI program is a wrong way.

[egmont]

Replying to andrew_b:

Let write own slang/ncurses.

No. Only a very small subset of them.

To continue my list, at least the text viewer "less", the text editors "vim", "emacs", "joe", the web browser "w3m", the mail client "alpine", the irc client "irssi", the monitoring "top" use libtinfo but not libncurses. If they all did it (and I bet the list is much longer), it really can't be that hard.

It's just one buffering layer that imagines what should be on the screen, marks the dirty regions, and when requested prints in on the screen using cursor positioning and color changing escapes. Not rocket science.

It wouldn't use all the bells and whistles ncurses has, like heavily optimizing screen redrawals by all those horizontal/vertical scrolling escape sequences and such, figuring out if it can save a byte or two in the network traffic which was probably relevant 20 years ago but not anymore. It would just repaint everything that changed.

Let write own glib (#157). What's than? (Looking to systemd) Let write own glibc (#125)?

There's a huge difference. Glib, glibc offer tons of handy functions, but if you don't need one or cannot use one, you just don't use. Slang/curses not only offer handy functions, but also expose some notable limitations.

I don't think that all these embellishments are really need in TUI program. Really. mc is the program for old users. Young Linux users use various GUI filemanagers and don't use mc. The transformation of mc to quasi-GUI program is a wrong way.

They are not "needed", but were requested from our users, some are quite insignificant, some might noticably increase mc's usability for some people.

I basically filed this ticket to point out a common factor from several other bug reports. I'm not planning to work on this and probably nobody's going to in the foreseeable future, but it's an idea worth considering in the long run.

[ossi]

my take (some points re-iterated from other issues):

• the true color stuff is utterly useless. seriously, who cares?
• the other attributes would be nice, but whatever.
• the EOL handling is an actual issue. i regularly cat files to be able to c&p them, because the hard-wrapped mcview output is useless.
• the scrolling optimizations *are* important. i notice that whenever i use less in a remote terminal that doesn't properly scroll back.
• i would have gotten rid of slang years ago

[gotar]

Replying to egmont:

Replying to andrew_b:

Let write own slang/ncurses.

No. Only a very small subset of them.

Small subset including various termcaps, truecolor, italic, bold, underline, strike, blink, dynamic resizing, transparency, unicode and arabic languages 'only'. Nice joke man!:)

To continue my list, at least the text viewer "less", the text editors "vim", "emacs", "joe", the web browser "w3m", the mail client "alpine", the irc client "irssi", the monitoring "top" use libtinfo but not libncurses.

They either must be as small as possible or don't use windows or were written before ncurses or had NIH syndrome or anything else that could be a reason then or now.

If they all did it (and I bet the list is much longer), it really can't be that hard.

OTOH such simple tools as mtr, cmake, bwm-ng, aumix, iptraf, telnet(!), iftop, gdb, hexedit, minicom, statserial(!), dialog, mutt, htop, tig use ncurses. Comparing top to htop or alpine to mutt shows which approach is superior.

It's just one buffering layer that imagines what should be on the screen, marks the dirty regions, and when requested prints in on the screen using cursor positioning and color changing escapes. Not rocket science.

Yep, nothing complicated. Have you posted the patches already? I'd like to try if your code works properly over minicom connected by telnet from my switch.

It wouldn't use all the bells and whistles ncurses has, like heavily optimizing screen redrawals by all those horizontal/vertical scrolling escape sequences and such,

So it would be worse and unusable over slow links?

figuring out if it can save a byte or two in the network traffic which was probably relevant 20 years ago but not anymore. It would just repaint everything that changed.

You mean entire screen (without these 'fancy' scrolling - I'm not sure minicom supports cursor positioning)? Moving file selector by 50 files in panel on 238x78 terminal I use would require 238x78x4(at least - unicode+attributes)*50=3.7 MB of data. Over 9600 bps line that's about 52 minutes.

Sure, I'm exaggerating, but only to show you some corner cases that must be handled. It's not that trivial as handling someone's putty.

There's a huge difference. Glib, glibc offer tons of handy functions, but if you don't need one or cannot use one, you just don't use. Slang/curses not only offer handy functions, but also expose some notable limitations.

All you need is:

1. create libsuperterm (no need to make it a part of mc, let others use it and benefit),
2. show it's superiority.

They are not "needed", but were requested from our users, some are quite insignificant, some might noticably increase mc's usability for some people.

The first one you've mentioned is mine. If you want to classify this, it only makes syntax highlighting possible in viewer - the function that is already available for editor, so definitely possible using current libraries.

I basically filed this ticket to point out a common factor from several other bug reports. I'm not planning to work on this and probably nobody's going to in the foreseeable future, but it's an idea worth considering in the long run.

It's an idea that must be abandoned, as it is the most common mistake ("let's write better libX, this shouldn't be so complicated").

You write own library, when existing ones MISS some features, not HAVE TOO MUCH of them.

[egmont]

Replying to gotar:

Small subset including various termcaps, truecolor, italic, bold, underline, strike, blink, dynamic resizing, transparency, unicode and arabic languages 'only'. Nice joke man!:)

Termcaps: Yup indeed.

Truecolor: Yes that'd be one of the easy points.

All the attributes you list are pretty much the same, implementing all of them is hardly any more work than implementing one.

Transparency: What do you mean by that?

Unicode: Is that really a problem in 2014? It's the default, no matter what you do.

Arabic: No proper support for that in terminals or in any terminals apps or toolkits, and wouldn't be a goal here either. See ​https://bugzilla.gnome.org/show_bug.cgi?id=321490 where with a former developer of gnome-terminal, current maintainer of fribidi and bicon, native speaker of a RTL language we pretty much come to the conclusion that there's no currently known way to get it right in terminals. (I myself also have half year of experience with RTL/BiDi on the web.) If you can contribute anything worthwile in this topic, you're more than welcome. Otherwise, this remains a non-goal.

They either must be as small as possible or don't use windows

mc doesn't use windows in this sense either. Remember: mc can work with either ncurses or slang, it uses the common subset of these two. Implementing an own library wouldn't be any harder than reimplementing slang's tty handling, which is really not that huge piece of code.

OTOH such simple tools as mtr, cmake, bwm-ng, aumix, iptraf, telnet(!), iftop, gdb, hexedit, minicom, statserial(!), dialog, mutt, htop, tig use ncurses.

Some are indeed good points (some might even use ncurses's UI toolkits which is a good reason to choose ncurses, we don't use that), I'm not saying that ever choosing ncurses is necessarily a bad decision. Some I guess depend on the system, e.g. for me no sign of ncurses in cmake or telnet. Mentioning dialog was a beauty: It's written by the same guy as ncurses, of course it uses that :)

Yep, nothing complicated. Have you posted the patches already?

Patches don't appear out of the blue. Even if it's simple and straightforward, it requires noticeable amount of work (that is, time). Free time doesn't appear out of the blue, and I'm obviously not going to do it after a main developer clearly stated that he'd reject it. And I'm not going to do it if other pieces of significant work that I've put in mc (e.g. #3250) sit here endlessly waiting for valuable feedback or acceptance by developers.

I'd like to try if your code works properly over minicom connected by telnet from my switch.

If I did it, I'd be more than happy to take your feedback from your system and improve until it's perfect for you.

It wouldn't use all the bells and whistles ncurses has, like heavily optimizing screen redrawals by all those horizontal/vertical scrolling escape sequences and such,

So it would be worse and unusable over slow links?

Currently mc defaults to using slang (not ncurses), and slang doesn't do these optimizations either. Guess what, press the Down arrow in mcview, you get a repaint of the whole view area.

With some minimal optimizations I could easily do better, I've been thinking about it already. Probably it wouldn't be as good as ncurses's optimization, but hopefully would be good enough for everyone, and definitely way better than slang.

You write own library, when existing ones MISS some features, not HAVE TOO MUCH of them.

That was exactly why I proposed this approach. Both existing libraries MISS the features of direct (non palette) colors and true colors [and no willingness from either of the two libs to ever support these two], controlling trailing spaces and controlling hard/soft linewraps [I haven't talked to them about these, but I guess this would become the very last entries on their todo lists].

[god12]

The S-Lang now have TrueColor? support in master. Not sure if there are any API changes required. So I think it might make sense to focus on S-Lang support and deprecate ncurses (with the option to eventually dropping it). Maybe writing our own API layer for drawing/palettes on top of slang - the way libnewt does for example: ​https://fedorahosted.org/newt/

[andrew_b]

(In #3145) Branch: 3145_true_color.
changeset:2cf9cd59f06e2cd09ebb33ca9a040ef5ec41fcc1

Looks OK. It doesn't break the current color handling.

[egmont]

Looks like there are a few new screen libraries emerging:

• termbox: ​https://github.com/nsf/termbox
• termbox-go: ​https://github.com/nsf/termbox-go
• tcell: ​https://github.com/gdamore/tcell
• notcurses: ​https://github.com/dankamongmen/notcurses ​https://notcurses.com/

Let's keep an eye on them :)