Compressed man pages are shown unformatted

[ZlatkO]

While working on #4271, I noticed that in 4.8.27, compressed man pages are shown unformatted (ie. "as-is" roff code) both with <F3> (view) and with <ENTER> (execute). Switching back to 4.8.26 makes things work as expected again, ie. viewed formatted with mcview on <F3>, and processed through nroff/man/less on <ENTER>.

To reproduce: gzip mc.1, then hit <F3> or <ENTER> on the resulting mc.1.gz.

[andrew_b]

Please show the output of

head -n 3 mc.1

and

file mc.1

[ZlatkO]

Note: mc.1 was only an example, this actually affects all compressed man pages on my system. Anyway, here we go:

[root@disclosure:~/usrlocal/stow/mc-4.8.27/share/man/man1]# head -n 3 mc.1
.\" -*- mode: troff; coding: UTF-8 -*-
.\"TOPICS "Topics:"
.TH MC 1 "August 2021" "MC Version 4.8.27" "GNU Midnight Commander"

Please let me know if you need any further details.

[andrew_b]

Please show the output of

file mc.1

file -z -L -S mc.1.gz

[ZlatkO]

Sure, no problem:

[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# ls -la
total 168
drwxr-xr-x 2 root root   4096 Aug 18 10:51 .
drwxr-xr-x 9 root root   4096 Aug 18 06:41 ..
-rw-r--r-- 1 root root 133514 Aug 18 06:41 mc.1
-rw-r--r-- 1 root root  20890 Aug 18 06:41 mcedit.1
-rw-r--r-- 1 root root   2544 Aug 18 06:41 mcview.1
[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# file mc.1
mc.1: troff or preprocessor input, ASCII text
[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# gzip mc.1
[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# file mc.1.gz 
mc.1.gz: gzip compressed data, was "mc.1", last modified: Wed Aug 18 04:41:29 2021, from Unix, original size modulo 2^32 133514
[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# 

[andrew_b]

file -z -L -S mc.1.gz

[ZlatkO]

Replying to andrew_b:

file -z -L -S mc.1.gz

Oops, sorry, my mistake:

[root@disclosure:/usr/local/stow/mc-4.8.27/share/man/man1]# file -z -L -S mc.1.gz 
mc.1.gz: troff or preprocessor input, ASCII text (gzip compressed data, was "mc.1", last modified: Wed Aug 18 04:41:29 2021, from Unix)

[andrew_b]

In [8857423e4ebb770b6f0ea3103abf5d35c85fcbe8] we changed the method of man page recognition from regex to type. type method means that we used the output of file utility. Now in the mc.ext we have:

    471 # Manual page - compressed
    472 type/^(ASCII )?troff.*gzip compressed
    473         Open=@EXTHELPERSDIR@/text.sh open man.gz %var{PAGER:more}
    474         View=%view{ascii,nroff} @EXTHELPERSDIR@/text.sh view man.gz %var{PAGER:more}
    475 
    476 type/^(ASCII )?troff.*bzip compressed
    477         Open=@EXTHELPERSDIR@/text.sh open man.bz %var{PAGER:more}
    478         View=%view{ascii,nroff} @EXTHELPERSDIR@/text.sh view man.bz %var{PAGER:more}
    479 
    480 type/^(ASCII )?troff.*bzip2 compressed
    481         Open=@EXTHELPERSDIR@/text.sh open man.bz2 %var{PAGER:more}
    482         View=%view{ascii,nroff} @EXTHELPERSDIR@/text.sh view man.bz2 %var{PAGER:more}
    483 
    484 # Manual page
    485 type/^(ASCII )?troff
    486         Open=@EXTHELPERSDIR@/text.sh open man %var{PAGER:more}
    487         View=%view{ascii,nroff} @EXTHELPERSDIR@/text.sh view man %var{PAGER:more}

Let's look at the output of file. You case:

# file mc.1
mc.1: troff or preprocessor input, ASCII text
# file -z -L -S mc.1.gz 
mc.1.gz: troff or preprocessor input, ASCII text (gzip compressed data, was "mc.1", last modified: Wed Aug 18 04:41:29 2021, from Unix)

And mine one:

$ file mc.1 
mc.1: ASCII troff or preprocessor input text
$ file -z -L mc.1.gz 
mc.1.gz: ASCII troff or preprocessor input text (gzip compressed data, was "mc.1", last modified: Tue Aug 24 08:17:47 2021, from Unix)

Or

$ file mc.1 
mc.1: UTF-8 Unicode troff or preprocessor input text
$ file -z -L mc.1.gz 
mc.1.gz: UTF-8 Unicode troff or preprocessor input text (gzip compressed data, was "mc.1", last modified: Tue Aug  3 11:09:10 2021, from Unix)

$ file --version
file-4.26
magic file from /etc/magic:/usr/share/file/magic

I suppose that your file is newer than mine and it's output isn't matched the regexp from mc.ext anymore.
The second issue, mc.ext isn't support unicode troff text.

[ossi]

relaxing the regexp to an unanchored "\<troff\>.*<foo> compressed" should do (i'm assuming this is using gnu extended regexp style).

[ZlatkO]

Replying to andrew_b:

[...]

$ file --version
file-4.26
magic file from /etc/magic:/usr/share/file/magic

I suppose that your file is newer than mine and it's output isn't matched the regexp from mc.ext anymore.

Yes, indeed:

[zlatko@disclosure:~]$ file --version
file-5.40
magic file from /etc/file/magic
seccomp support included

I only recently upgraded from Slackware-14.2's original file-5.36 to Slackware-current's file-5.40. 5.36 had its own set of problems, coincidentally also mainly related to mcview & man pages. :-)

The second issue, mc.ext isn't support unicode troff text.

Is this relevant to the problem at hand, or only a general observation?

[ZlatkO]

Replying to ossi:

relaxing the regexp to an unanchored "\<troff\>.*<foo> compressed" should do (i'm assuming this is using gnu extended regexp style).

No, unfortunately not. The mcviewer call still looks like this:

[zlatko@disclosure:~]$ pgrep -af mc/ext.d
8355 /bin/sh /usr/local/libexec/mc/ext.d/archive.sh view gz

I assume this is because of first match wins in mc.ext, and ...

# gzip
type/\(gzip compressed
        Open=/usr/local/libexec/mc/ext.d/archive.sh view gz %var{PAGER:more}
        View=%view{ascii} /usr/local/libexec/mc/ext.d/archive.sh view gz

..., which also matches, comes before ...

# Manual page - compressed
type/(ASCII )?troff.*gzip compressed
        Open=/usr/local/libexec/mc/ext.d/text.sh open man.gz %var{PAGER:more}
        View=%view{ascii,nroff} /usr/local/libexec/mc/ext.d/text.sh view man.gz %var{PAGER:more}

type/(ASCII )?troff.*bzip compressed
        Open=/usr/local/libexec/mc/ext.d/text.sh open man.bz %var{PAGER:more}
        View=%view{ascii,nroff} /usr/local/libexec/mc/ext.d/text.sh view man.bz %var{PAGER:more}

type/(ASCII )?troff.*bzip2 compressed
        Open=/usr/local/libexec/mc/ext.d/text.sh open man.bz2 %var{PAGER:more}
        View=%view{ascii,nroff} /usr/local/libexec/mc/ext.d/text.sh view man.bz2 %var{PAGER:more}

... (I already removed the ^ anchor for testing).

[andrew_b]

Branch: 4272_mc.ext_man_pages
changeset:53b1bb63acaa93a1c41a22a5c6ac1204b6140aed

Please test.

[ZlatkO]

Replying to andrew_b:

Please test.

No, still doesn't work, sorry. Same problem as ossi's quick-fix to remove the "^" anchor, ie. the match for generic gzipped data in lines 344-346 comes before the matches for compressed man pages later on and wins. It only works if I comment out these lines in mc.ext, ie. the "type/\(gzip compressed" line and the two corresponding "Open=" and "View=" entries below.

Maybe moving (all of the?) the generic type/<something> matches to the end of mc.ext would help?

[andrew_b]

Replying to ZlatkO:

generic gzipped data in lines 344-346

There is no any data in those lines. Please test the branch, not the commit as patch on top of some state of codebase.

[ZlatkO]

Replying to andrew_b:

Replying to ZlatkO:

generic gzipped data in lines 344-346

There is no any data in those lines. Please test the branch, not the commit as patch on top of some state of codebase.

Oops, sorry, my bad. What I did was to download the changeset as a diff, and apply that on top of 4.8.27. It didn't occur to me that I had to checkout and build the whole branch.

[ZlatkO]

Okay, I rebuilt mc from the 4272_mc.ext_man_pages branch and the problem seems to be gone now indeed. Compressed man pages are now viewed properly formatted again. Thanks a lot! :-)

[andrew_b]

Thanks for the feedback!

[andrew_b]

Merged to master: [6a855b1d1f444ea8dbc3d77e42f3ada372716087].