[Patch] Problems displaying UTF-8 manual pages

[egmont]

Formatted manual pages using UTF-8 charset are handled incorrectly by mc-4.7-pre2.

Problem 1: Accented characters in bold (yellow) text are displayed incorrectly: the letter, followed by a dot, then the letter once again, all in white.

Problem 2: Search does not find text that are bold or underlined (yellow or red) and contain accented letters. Exception: it finds if only the first character is accented, but highlights the match incorrectly.

I attach such a formatted manpage (emitted by groff-utf8) for your convenience - you might have to press F9 in mcview.

[egmont]

Formatted UTF-8 manual page

[egmont]

Same in .zip so that you can download it

[egmont]

I attach a proof of concept patch. This patch fixes displaying UTF-8 nroff pages, but probably break non-utf8 ones. It doesn't address the issues with search either.

So this patch is not to be applied in its current form, it needs more work. It just shows where the bugs are: "from - 2" obviously assumes single byte locales, so does mcview_get_byte to get c_next, as well as g_ascii_isprint (there's a fixme entry about it somewhere else in that file).

[egmont]

proof of concept only - not a ready patch

[slavazanko]

created branch 1629_nroff_utf8 (parent: master)

changeset:7adcdc570a07353ad64bc017352eb245dff3753b

Please, test it. I'm tested on:

*) with option --enable-charset in utf-8 codepage
*) with option --enable-charset in one-byte codepage
*) with option --disable-charset in utf-8 codepage (old behavior like no patch)
*) with option --disable-charset one-byte codepage (old behavior like no patch)

With '--disable-charset' option mans looks as no patch present.

Keep here your comments about this, please.

[egmont]

Haven't tested, but the patch looks good. One question, though:

What's the point in reverting c_prev's assignment for the "no charset" case? I think my version is cleaner here: the source is shorter (you save one mcview_get_byte() call and one #ifdef), plus, the charset and the no charset code becomes much more similar to each other. I think having two different code paths for setting c_prev makes the code harder to understand and maintain.

[slavazanko]

Branch rebased. Also, fixed c_prev's assignment

initial changeset:95311b5e5a0cc8d0c4b79a93dabe906319fe533c

Review, Please.

[egmont]

I cannot compile&run right now, but thoroughly looking at the patch it looks good. If you've tested, please go ahead and apply, thanks!

Please don't forget that it only addresses the first issue in this bug, not the second one. That one's still open.

[slavazanko]

yep, i forgot about second part of bugreport :)

[egmont]

Friendly ping... What happened to the changesets created by slavazanko? They are no longer accessible.

For your convenience I attach an updated version of my patch (rebased on 4.7.0.2).

Even though this patch only fixes the first part of the bug, I'd appreciate if you went ahead and submitted the fix. Later we can build on top of this and fix searching in manpages too.

[egmont]

Ping...

You nearly submitted the patch 6 months ago, and since then nothing happened. Moreover, your changesets are no longer available - why?

I'd be happy to work on the second part of the story, but only if the first part is already submitted - I don't have time to waste for creating patches if they are ignored by the developers. Thanks for your understanding.

[slavazanko]

Recreated branch changeset:1629_nroff_utf8.

Initial changeset:f5055c78684694a7a81575c30ee9815927b81d4c

Branch have resolutions of two issues:

• show UTF-8 symbols in nroff'ed text and
• UTF-8 search over nroff'ed text.

Review, please

[slavazanko]

Merge changeset:d7fd84507c063adb79b02b544d07203b73518831

Commits:

git log --pretty=oneline 7d52669..79660a7

[slavazanko]

cherry-picked in stable branch:

git log --pretty=oneline 84cb81e5..52047442