mcview hex edit: UTF-8 chars not updated

[egmont]

mcview hex edit mode, in fully UTF-8 environment

UTF-8 characters show up nicely in the right column (followed by a space to align with the number of bytes).

However, as soon as you edit the file (either by the hex codes, or by moving to the right column and entering an UTF-8 character), that character doesn't show up there.

Expected: during editing the file, the right hand side should continuously be updated to look exactly as if that was the original contents of the file.

[egmont]

hex.c mcview_display_hex() does the following: It takes the next Unicode character from the original file, then overwrites the bytes from the edit buffer and composes the Unicode again. This works as long as character boundaries don't change, but fail when they do.

The logic is incorrect: handling whether the given byte should be taken from the original file or from the edit buffer should be underneath any call that assembles UTF-8 bytes into Unicode.

[egmont]

Quick fix

[egmont]

Quick fix attached.

The intermediate bytes of UTF-8 are skipped because the UTF-8 parser returns -1 and tty_print_anychar() does nothing. This in turn means that the correct attributes (background color or underline) are not applied. This would be nice to fix (although a bit problematic with CJK).

Also, it would be nice to make a difference between continuation bytes of UTF-8 (drawn as spaces, as currently) and actual invalid UTF-8 in the file (should be displayed with dots). This would require to look back by a few bytes from the beginning of the viewport.

Anyway, my patch is already a significant improvement over the current code.

[egmont]

Fix v2

[egmont]

Second fix attached. This one knows if a byte is a contination byte of a previously started UTF-8 sequence (prints space, with a correct attribute if the byte was modified) or invalid UTF-8 (prints dot, again with the correct attribute if the byte was modified). It takes care not to print a space over the second byte of a CJK, as that would wipe out the CJK char.

[egmont]

Fix v3

[egmont]

One more change. If you changed only a single byte of a UTF-8 (e.g. C3 A9 -> C3 AD), only the corresponding cell on the right was marked as modified (in this case "é " -> "í ", whereas only the space was marked as modified [underlined, different color (not really visible in case of a space :)) etc. according to the skin], the preceding letter which changed from é to í was not marked so).

Also, there was no way of marking on the right side whether the second byte of a CJK changed, since the corresponding cell is occupied by the wide character, marking if the first byte was changed.

I modified the code to mark the whole string representing a UTF-8 character (e.g. "í ") as modified if any of its bytes were modified. I think this makes more sense if you're requesting UTF-8 view anyways, and is also consistent in CJK.

If this is not the behavior someone's looking for, they probably don't want UTF-8 either but an 8-bit charset (which they get with Alt+E). Even in UTF-8 mode, the left hand hex view part still shows on a per-byte basis if they were modified. In UTF-8 mode, the right hand side shows characters (rather than bytes), and hence also shows for the characters (rather than bytes) if they were modified.

[egmont]

Friendly ping... a patch has been attached for 6 months now. Could you please take a look? It would be nice to release this fix along with the mcviewer rewrite.

[andrew_b]

Thanks! Applied with some trivial modifications: indentation and reduce variable scope.

Branch: 3259_hexedit_utf8
changeset:a7d326d30b2ff2a16ddb0fdedae7bcea771d29b0

[andrew_b]

Merged to master: [a7d326d30b2ff2a16ddb0fdedae7bcea771d29b0].