mouse clicks don't work on ncurses-6.2, generate garbage to stdout

[slyfox]

It's forward of downstream ​https://bugs.gentoo.org/753578 report.

When mc is built with ncurses-6.2 mouse keypresses in xterm don't work and generate garbage output. The following workaround seems to be enough to fix it locally:

lib/tty/key.c

@@ -2124 +2124 @@
         gboolean extended = c == MCKEY_EXTENDED_MOUSE;
 
 #ifdef KEY_MOUSE
-        extended = extended || (c == KEY_MOUSE && xmouse_seq == NULL
-                                && xmouse_extended_seq != NULL);
+        extended = extended || (c == KEY_MOUSE && xmouse_extended_seq != NULL);
 #endif /* KEY_MOUSE */
 
         xmouse_get_event (event, extended);

$ LANG=C ./src/mc --version
GNU Midnight Commander 4.8.25-59-ga2bccac62
Built with GLib 2.64.5
Built with ncursesw 6.2
With builtin Editor
With subshell support as default
With support for background operations
With mouse support on xterm
With support for X11 events
With internationalization support
With multiple codepages support
With ext2fs attributes support
Virtual File Systems:
 cpiofs, tarfs, sfs, extfs, ftpfs, sftpfs, fish
Data types:
 char: 8; int: 32; long: 64; void *: 64; size_t: 64; off_t: 64;

Some details:

$ tput kmous | cat -v
^[[<
$ echo $TERM
xterm-256color

Actual generated sequences by xterm are in form of "\e[<0;10;28M", but mc expects-non-extended form of "\e[bxy".

[andrew_b]

Related to #3954.

[slyfox]

Posted initially from the wrong terminal. Updated to expected:

$ tput kmous | cat -v
^[[<

[slyfox]

I think man user_caps describes XM / xm usercaps meaning of how to enable and parse extended mouse events: ​https://www.man7.org/linux/man-pages/man5/user_caps.5.html

"XM" string, override ncurses's built-in string which enables/disables xterm mouse mode.

  ...

  The XM capability has a single parameter.  If nonzero, the
  mouse protocol should be enabled.  If zero, the mouse protocol
  should be disabled.  ncurses inspects this capability if it is
  present, to see whether the 1006 protocol is used.  If so, it
  expects the responses to use the SGR 1006 xterm mouse protocol.

  ...

  The terminal database building blocks for this mouse feature
  also have an experimental capability xm.  The “xm” capability
  describes the mouse response.  Currently there is no
  interpreter which would use this information to make the mouse
  support completely data-driven.

  ...

Thus it looks like mere presence of XM capability should enable extended mouse event format mode.

​https://invisible-island.net/xterm/ctlseqs/ctlseqs.html describes "Ps = 1 0 0 6 ⇒ Enable SGR Mouse Mode, xterm."

$ tput XM 1 |& cat -v
^[[?1006;1000h

[slyfox]

Thus it looks like mere presence of XM capability should enable extended mouse event format mode.

I misinterreted it. The XM it specfically checked for SGR mode:

​https://github.com/mirror/ncurses/blob/master/ncurses/base/lib_mouse.c#L387

[andrew_b]

Unfortunately, proposed patch breaks compatibility with ncurses-5: mouse clicks don't work and generate garbage to stdout.

[Polynomial-C]

So mouse input is either broken with ncurses5 or with ncurses6. That's quite unfortunate and I wonder if it is really better to retain support for ncurses5 when most distros already have switched over to ncurses6.

[andrew_b]

Replying to Polynomial-C:

So mouse input is either broken with ncurses5 or with ncurses6.

I can't say so. Mouse with ncurses5 works fine.

That's quite unfortunate and I wonder if it is really better to retain support for ncurses5 when most distros already have switched over to ncurses6.

Distros of what? Linux? What about LTS Linux distros? What about other (including proprietary) OSes where mc works? Should we drop them all and support only modern Linuxes?

[Polynomial-C]

Replying to andrew_b:

Replying to Polynomial-C:

So mouse input is either broken with ncurses5 or with ncurses6.

I can't say so. Mouse with ncurses5 works fine.

Yeah but like you said, fixing it for ncurses6 breaks it for ncurses5 which is really... unfortunate.

That's quite unfortunate and I wonder if it is really better to retain support for ncurses5 when most distros already have switched over to ncurses6.

Distros of what? Linux? What about LTS Linux distros? What about other (including proprietary) OSes where mc works? Should we drop them all and support only modern Linuxes?

I doubt any LTS distro will ship a recent mc (although I would like to see that happen of course).
I don't know about proprietary OSses of course but how many of these other OSses still use ncurses5? ncurses-6.0 had been released in August 2015. So even many recent LTS distros most likely already ship ncurses6.

But sorry for raising such a fruitless discussion. I kicked it loose merely out of some frustration about this situation. If you want to retain ncurses5 support you have all rights to do so of course.
I just hope this issue will get resolved at some point.

[egmont]

I was the one doing the initial work for the extended mouse, plus one followup fix for an ncurses change, also I was cc'd here, so I think I should say something.

On the other hand, I stopped volunteering to open source projects, I just simply lost motivation and decided to devote my time to other things, so I won't work on a fix to this problem, sorry.

Taking a huge step backwards from the concrete details (which I haven't looked at), here's my take on the story.

---

Any simple project can do the following: Ask the terminal to enable mouse reporting (1000), and ask it to enable the xterm extension (1006) too. Then wait for input. If the terminal doesn't support mouse, the app just won't receive mouse events. If the terminal supports, the app will receive mouse events. These will be either in the old or in the new format, depending on whether the terminal supports the extension or not. The two have totally distinguishable prefixes, the input is parseable unambiguously. This is all what you need to get perfect mouse handling, it's as simple as this. Note that in this story does not mention slang, ncurses, terminfo etc.

Added to this story is a quite complex logic in mc trying to respect terminfo, plus ncurses and/or terminfo that has already for the second time changed in a way that broke mc's complex logic, even though the underlying mouse protocol did not change, hence any app going with the trivial manual parsing approach would have just kept working perfectly.

You might want to fix just this current problem, but it comes with a nontrivial implementation cost (apparently none of you understands yet what exactly should be done), quite large testing cost (at least 3 different ncurses/terminfo versions), and who guarantees it's not going to break again with ncurses 6.3, 7.0 etc.?

I have so many problems with terminfo anyway, it's a terrible system. Half of the mouse protocol, i.e. whether the feature is supported, and if so then what the escape sequence's prefix is, is stored in terminfo. The second half of the escape sequence, i.e. what raw bytes or decimals specify what properties, are hardcoded in the app anyway (or handled via ncurses methods, but if mc used them then it wouldn't have broken recently). What's the point in this knowledge split? Makes no sense to me. Are there any terminals out there which support the legacy 1000 or the extended 1006 modes, *except* that the prefix is different? I highly doubt so. Yet mc aims to supports that.

What we see here is how to unnecessarily overcomplicate something inherently simple (both in ncurses/terminfo and in mc), to the extent that it keeps breaking and it's unmaintainable.

As I already mentioned in #3954, mc's approach of partially using ncurses/terminfo for mouse handling, and partially using its own approach, is probably the worst. Either relying fully on ncurses methods, or doing it all manually would be probably much simpler to implement, simpler to test, and would work more robustly. Since mc also supports slang, which as far as I recall doesn't offer mouse handling, I recommend that you throw out every sign of terminfo/ncurses from the mouse handling and do it all manually. It's simple and robust.

---

On an irrelevant note, I fully agree with Polynomial-C. LTS distros won't update mc. Any other OS that ships an old ncurses for whatever reason can also stick to an older mc. The user base who manually upgrade their mc on an old system is negligible. I never understood the demand of using the newest apps on top of an old system, everyone who's happy with a 5 year old base system should be happy with a 5 year old mc. It's pointless to put any effort into supporting 5+ year old systems, it's the worst use of the scarce engineering resources, and potentially harms a multiple magnitudes larger user base. You should focus on the future, not the past (if supporting both is expensive or unreasonable).

Hope I could help.

[zaytsev]

Hi Egmont,

Thank you very much for your input! Well, I guess we need wait for a hero to make things simple again since you've decided to use your time in more meaningful ways :-) (I envy you ;-) )...

Regarding LTS distros, well... it all depends on how you work and what you need to do. I'm the kind of guy, who uses RHEL, Ubuntu LTS and such on servers & desktops for years, but need a few tools current beyond an extremely stable and bugless base system which I basically just use as a launcher for these few tools - one of those is mc (the other one is browser, obviously). Apparently such people are not very numerous, but there are quite a few of us.

Yury

[zaytsev]

Since mc also supports slang, which as far as I recall doesn't offer mouse handling, I recommend that you throw out every sign of terminfo/ncurses from the mouse handling and do it all manually. It's simple and robust.

Any chance you can be enticed to offer a patch :) ? Upon re-reading your comment years later, I'm afraid that this is indeed the way to go...