[PATCH] copying files: timestamps with nanosecond precision not preserved

[ag]

Reproducer:
$ touch dummy.txt
$ mc
(copy with shift+f5 to dummy-copied-with-mc.txt)
$ ls --full-time dummy*
-rw-r--r-- 1 andrey andrey 0 2015-12-06 21:48:49.000000000 +0100 dummy-copied-with-mc.txt
-rw-r--r-- 1 andrey andrey 0 2015-12-06 21:48:49.242551392 +0100 dummy.txt
$

The Linux kernel has utimensat() since 2.6.22, and since 2.6.26 (7+ years) it is conform to POSIX.1-2008. Even NTFS supports 100-ns precision.

[ag]

Proposed fix

[zaytsev]

FWIW, I've read the patch and it looks good to me, but I haven't tried it myself.

[zaytsev]

:-(

[and]

Loosing meta object information by object copy/move is always bad.
Let it fix this early than later.

What ":-(" means in this case? Re-send as single patches?

[zaytsev]

:-( means that I wanted to review it for 4.7.17, but no chance. Either we do it next week, or in some months, so I again moved it to 4.7.18.

Can you review it for 4.7.18? It looks good, but there are no tests and I'm not sure if all places are covered, there will be no problems on non-Linux, etc. so I feel unsafe just committing it.

[andrew_b]

Branch: 3575_nanoseconds
changeset:4c10154d5092501e8f95958df5862dee77292c1f

[andrew_b]

As I can see, now local file copying preserves nanoseconds, but fish copying doesn't.

[ag]

Andrew,

thanks for moving this forward.

Regarding fish. Yes, it doesn't support this now, especially if perl is in use. Unfortunately perl seems not to have the function utimensat per default. And since perl is always already installed, it's not likely it gets to the touch case. I asked about this already [1], as I was working on this issue.

If you would agree to make src/vfs/fish/helpers/utime

#UTIME $FISH_FILEATIME $FISH_FILEMTIME $FISH_FILENAME
if [ -n "$FISH_HAVE_PERL" ] &&
   perl -e 'utime '$FISH_FILEATIME','$FISH_FILEMTIME',@ARGV;' "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
elif TZ=UTC touch -m -t $FISH_TOUCHMTIME "/${FISH_FILENAME}" 2>/dev/null &&
     TZ=UTC touch -a -t $FISH_TOUCHATIME "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
else
  echo "### 500"
fi

more complex, adding something like this:

eval {
  require File::lchown;
  1
};
if ($@) { 
  utime '$FISH_FILEATIME','$FISH_FILEMTIME','$FISH_FILENAME';
} else {
  File::lchown::lutimes '$FISH_FILEATIME','$FISH_FILEMTIME','$FISH_FILENAME';
}

...Hm. But even with this and even if File::lchown turns out to be pre-installed (and lutimes would have been improved to support nanosecond precision), I doubt about the true nanosecond precision. Time values are stored in floats, which cannot hold the full range (seconds with nanoseconds at once), resulting only in correct microseconds. This would only introduce confusion. With zeroed subseconds, you know they're likely just zeroed, not really different.

If you believe there are enough systems without perl and with touch honoring nanoseconds in timestamps, I'll fix this. BTW, is it possible to properly configure MC to not use perl only for one helper script? Or maybe to revert the default only for this script?

[1] ​https://www.midnight-commander.org/ticket/2625#comment:9

[zaytsev]

BTW, is it possible to properly configure MC to not use perl only for one helper script?

This depends on what do you mean by "configure". Users can override helper scripts per host, but there is no way to tell mc to not to use Perl in ways other than make perl return 1 or so as far as I know.

Or maybe to revert the default only for this script?

This is possible, but these evil scripts are such a can of worms... Do you have access to various systems to test how different touches behave? My understanding is that -t doesn't support nanoseconds, and -d is not available at least on macOS and HP-UX, whereas NetBSD apparently ignores the nanosecond part. Maybe reverse touch & perl for utime, first touch with -t and then with -d (it seems that busybox doesn't have -t o_O), and if neither worked then use perl?

By the way, what about all other VFSes?

[ag]

Do you have access to various systems to test how different touches behave?

No.

This is from coreutils' NEWS file:

• Noteworthy changes in release 6.12 (2008-05-31) [stable]

New features

cp, install, mv, and touch now preserve nanosecond resolution on
file timestamps, on platforms that have the 'utimensat' and
'futimens' system calls.

Thus the support is there since 8+ years. Long enough. Or MacOS doesn't use coreutils? Is HP-UX supported by MC? When was the last time a MC developer compiled and used MC on HP-UX (otherwise I doubt one can call this "supported".) If you're not against, I'd suggest to continue this part of discussion on the mailing list [1] (though it's not mc-devel, but mc, so CC: mc-devel, if it is important).

Maybe reverse touch & perl for utime, first touch with -t and then with -d (it seems that busybox doesn't have -t o_O), and if neither worked then use perl?

You mean first try with -d (for nanoseconds), then with -t and then perl? This seems to be a good solution. For unknown reason the following script

#UTIME $FISH_FILEATIME $FISH_FILEMTIME $FISH_TOUCHMTIME $FISH_TOUCHATIME $FISH_TOUCHMTIME_W_NSEC $FISH_TOUCHATIME_W_NSEC $FISH_FILENAME
if TZ=UTC touch -m -d "$FISH_TOUCHMTIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null &&
   TZ=UTC touch -a -d "$FISH_TOUCHATIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
elif TZ=UTC touch -m -t $FISH_TOUCHMTIME "/${FISH_FILENAME}" 2>/dev/null &&
     TZ=UTC touch -a -t $FISH_TOUCHATIME "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
elif [ -n "$FISH_HAVE_PERL" ] &&
   perl -e 'utime '$FISH_FILEATIME','$FISH_FILEMTIME',@ARGV;' "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
else
  echo "### 500"
fi

makes mc to hang after a file is copied:

^C
Program received signal SIGINT, Interrupt.
0x00007f33fe721230 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007f33fe721230 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000047e5d7 in vfs_s_get_line (me=0x786be0 <vfs_fish_ops>, sock=11, buf=0x7ffccca4a750 "", buf_len=1024, term=10 '\n')
    at ../../../mc-3575.git/lib/vfs/direntry.c:1495
#2  0x000000000046754a in fish_get_reply (me=0x786be0 <vfs_fish_ops>, sock=11, string_buf=0x0, string_len=79) at ../../../../mc-3575.git/src/vfs/fish/fish.c:220
#3  0x00000000004677d1 in fish_command (me=0x786be0 <vfs_fish_ops>, super=0xf68b50, wait_reply=1, fmt=0x513437 "%s")
    at ../../../../mc-3575.git/src/vfs/fish/fish.c:269
#4  0x0000000000469a30 in fish_send_command (me=0x786be0 <vfs_fish_ops>, super=0xf68b50, 
    cmd=0x7ffccca4ad90 "FISH_HAVE_HEAD=1 export FISH_HAVE_HEAD; FISH_HAVE_SED=1 export FISH_HAVE_SED; FISH_HAVE_AWK=1 export FISH_HAVE_AWK; FISH_HAVE_PERL=1 export FISH_HAVE_PERL; FISH_HAVE_LSQ=1 export FISH_HAVE_LSQ; FISH_H"..., flags=1) at ../../../../mc-3575.git/src/vfs/fish/fish.c:1142
#5  0x000000000046a77d in fish_utime (vpath=0xfa1650, times=0x7ffccca4b2e0) at ../../../../mc-3575.git/src/vfs/fish/fish.c:1403
#6  0x000000000042d79b in mc_utime (vpath=0xfa1650, times=0x7ffccca4b2e0) at ../../../mc-3575.git/lib/vfs/interface.c:254
#7  0x00000000004c6c7f in copy_file_file (tctx=0xf7f090, ctx=0xf7ef90, src_path=0xf7e3b0 "/media/portable1/progs/file.managers/mc/install/etc/mc/sfs.ini", 
    dst_path=0xfa1200 "/sh://127.0.0.1/home/andrey/tmp/sfs.ini") at ../../../mc-3575.git/src/filemanager/file.c:2011
#8  0x00000000004c8fa0 in panel_operate (source_panel=0xf53030, operation=OP_COPY, force_single=0) at ../../../mc-3575.git/src/filemanager/file.c:2909
#9  0x00000000004becb1 in copy_cmd () at ../../../mc-3575.git/src/filemanager/cmd.c:802
#10 0x000000000043ad39 in midnight_execute_cmd (sender=0xf58f30, command=22) at ../../../mc-3575.git/src/filemanager/midnight.c:1142
#11 0x000000000043b735 in midnight_callback (w=0xf4fa10, sender=0xf58f30, msg=MSG_ACTION, parm=22, data=0x0) at ../../../mc-3575.git/src/filemanager/midnight.c:1561
#12 0x00000000004148b3 in send_message (w=0xf4fa10, sender=0xf58f30, msg=MSG_ACTION, parm=22, data=0x0) at ../../../mc-3575.git/lib/widget/widget-common.h:209
#13 0x0000000000414beb in buttonbar_call (bb=0xf58f30, i=4) at ../../../mc-3575.git/lib/widget/buttonbar.c:155
#14 0x0000000000414c57 in buttonbar_callback (w=0xf58f30, sender=0x0, msg=MSG_HOTKEY, parm=1005, data=0x0) at ../../../mc-3575.git/lib/widget/buttonbar.c:171
#15 0x0000000000418be4 in send_message (w=0xf58f30, sender=0x0, msg=MSG_HOTKEY, parm=1005, data=0x0) at ../../../mc-3575.git/lib/widget/widget-common.h:209
#16 0x000000000041958c in dlg_try_hotkey (h=0xf4fa10, d_key=1005) at ../../../mc-3575.git/lib/widget/dialog.c:434
#17 0x00000000004196a0 in dlg_key_event (h=0xf4fa10, d_key=1005) at ../../../mc-3575.git/lib/widget/dialog.c:479
#18 0x000000000041ad07 in dlg_process_event (h=0xf4fa10, key=1005, event=0x7ffccca4bb20) at ../../../mc-3575.git/lib/widget/dialog.c:1165
#19 0x00000000004198c4 in frontend_dlg_run (h=0xf4fa10) at ../../../mc-3575.git/lib/widget/dialog.c:541
#20 0x000000000041adc3 in dlg_run (h=0xf4fa10) at ../../../mc-3575.git/lib/widget/dialog.c:1196
#21 0x000000000043a84c in create_panels_and_run_mc () at ../../../mc-3575.git/src/filemanager/midnight.c:954
#22 0x000000000043becf in do_nc () at ../../../mc-3575.git/src/filemanager/midnight.c:1770
#23 0x000000000040b7bf in main (argc=1, argv=0x7ffccca4bd08) at ../../mc-3575.git/src/main.c:403

Thus I haven't yet included this change into patch.

Though running it on the command line works just fine:

$ FISH_FILENAME=/home/andrey/tmp/sfs.ini FISH_TOUCHMTIME=201610290102.03 FISH_TOUCHATIME=201610290102.03 FISH_TOUCHMTIME_W_NSEC="2016-10-29 01:02:03.456789123 +0200" FISH_TOUCHATIME_W_NSEC="2016-10-29 01:02:03.456789123 +0200" bash /media/portable1/progs/file.managers/mc/install/libexec/mc/fish/utime
### 000
$

By the way, what about all other VFSes?

I believe that local operations are being used more often than VFSes ones. Thus it is important to fix the first case in first place. Then one can move forward and look into others. And this one could be also someone else(*). So, my goal is to improve one thing while not breaking others. Also I can easily test "Shell link" locally, since sshd is a pretty default daemon. How would you setup a test for other VFSes?

[1] ​https://mail.gnome.org/archives/mc/2016-October/msg00029.html

(*) I experience problems even with archive VFS and try to avoid them. E.g. looking into an archive results into its unpacking(!) and can take too much time. Looking into an archive which turns out to be password protected results in hanging MC.

[ag]

Hi,

just to ensure we're not deadlocked: I'm standby awaiting feedback on the last patch.

I'm wondering, where to document two different defaults for utime script: "touch -d"/perl for communicating to GNU/Linux systems (with good touch from coreutils) and perl/"touch -t" for others?

Ideally, MC should inspect the remote system once connected to choose the best script, but that would be another story. And in this case each script should drop alternatives, since every alternative will be in a different script.

[andrew_b]

Replying to ag:

For unknown reason the following script

#UTIME $FISH_FILEATIME $FISH_FILEMTIME $FISH_TOUCHMTIME $FISH_TOUCHATIME $FISH_TOUCHMTIME_W_NSEC $FISH_TOUCHATIME_W_NSEC $FISH_FILENAME
if TZ=UTC touch -m -d "$FISH_TOUCHMTIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null &&
   TZ=UTC touch -a -d "$FISH_TOUCHATIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
elif TZ=UTC touch -m -t $FISH_TOUCHMTIME "/${FISH_FILENAME}" 2>/dev/null &&
     TZ=UTC touch -a -t $FISH_TOUCHATIME "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
elif [ -n "$FISH_HAVE_PERL" ] &&
   perl -e 'utime '$FISH_FILEATIME','$FISH_FILEMTIME',@ARGV;' "/${FISH_FILENAME}" 2>/dev/null; then
  echo "### 000"
else
  echo "### 500"
fi

makes mc to hang after a file is copied:

The reason is buffer overflow. buf[1024] in fish_utime() is too small to store entire updated FISH utime command. Now command is built as dynamycally allocated string not static buffer.

Branch: 3575_nanoseconds
Initial changeset:95f86611ca5b20ace3ec56abbe722996ec818f0f

Please review.

P.S. The audit is required for all other FISH commands. Those buffers might be overflowed too.

[andrew_b]

Replying to andrew_b:

The reason is buffer overflow. buf[1024] in fish_utime() is too small to store entire updated FISH utime command. Now command is built as dynamycally allocated string not static buffer.

Sorry, buffer was not overflowed, sure. But command was just truncated at buffer length and server received only 1023 bytes and waited for remain part of command.

[ag]

Andrew,

such truncation (in opposite to segfault or reading out-of-bounds memory) would be really hard to track without being very familiar with the code base. Thanks for figuring this out and fixing! If there would be a convenient assert for truncations...

And I'm very appreciating, you've already taken care of rebasing the patch and replacing the helper script and its default definition in fishdef.h.

The only minor thing, I've noticed, is:

diff --git a/src/vfs/fish/fishdef.h b/src/vfs/fish/fishdef.h
index 1bb836b..7fd49ff 100644
--- a/src/vfs/fish/fishdef.h
+++ b/src/vfs/fish/fishdef.h
@@ -76,7 +76,7 @@
 
 /* default 'utime' script */
 #define FISH_UTIME_DEF_CONTENT ""                                                                         \
-"#UTIME \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n"                         \
+"#UTIME \"$FISH_TOUCHATIME\" \"$FISH_TOUCHMTIME\" \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n" \
 "if TZ=UTC touch -m -d \"$FISH_TOUCHMTIME_W_NSEC\" \"/${FISH_FILENAME}\" 2>/dev/null && \\\n"             \
 "   TZ=UTC touch -a -d \"$FISH_TOUCHATIME_W_NSEC\" \"/${FISH_FILENAME}\" 2>/dev/null; then\n"             \
 "  echo \"### 000\"\n"                                                                                    \
diff --git a/src/vfs/fish/helpers/utime b/src/vfs/fish/helpers/utime
index bdd7a60..f100144 100644
--- a/src/vfs/fish/helpers/utime
+++ b/src/vfs/fish/helpers/utime
@@ -1,4 +1,4 @@
-#UTIME "$FISH_TOUCHATIME_W_NSEC" "$FISH_TOUCHMTIME_W_NSEC" "$FISH_FILENAME"
+#UTIME "$FISH_TOUCHATIME" "$FISH_TOUCHMTIME" "$FISH_TOUCHATIME_W_NSEC" "$FISH_TOUCHMTIME_W_NSEC" "$FISH_FILENAME"
 if TZ=UTC touch -m -d "$FISH_TOUCHMTIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null && \
    TZ=UTC touch -a -d "$FISH_TOUCHATIME_W_NSEC" "/${FISH_FILENAME}" 2>/dev/null; then
   echo "### 000"

P.S. The audit is required for all other FISH commands. Those buffers might be overflowed too.

Since the helper scripts can be altered by users, all such buffers must be allocated dynamically (but with some reasonable maximal size)? And if truncation could not be avoided, instead of calling fish_send_command() -1 must be returned immediately to prevent hanging. On the other hand, if user made some mistake in a script, fish_send_command() could hang. I'm wondering, whether this could be avoided generally.

If this is not an one-liner fix, such as:

fish_{utime,chmod,...}()
    fish_send_command()
        fish_command()
            fish_get_reply()
                vfs_s_get_line() // replace this with vfs_s_get_line_interruptible() ?

would it be better to track this in a new bug?

BTW, is vfs_s_get_line() not generally risky or there were some reasons to use it?

[andrew_b]

Replying to ag:

Andrew,
The only minor thing, I've noticed, is:

-"#UTIME \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n"                         \
+"#UTIME \"$FISH_TOUCHATIME\" \"$FISH_TOUCHMTIME\" \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n" \

}}}
As I understand, the 1st line is command for FISH server. As I understand again, the utime command has 3 arguments: atime, mtime, filename. Thus we can't send 5 arguments.
On the other hand, is there any implementation of native FISH server? I can't find any of them. All references of "files transferred over shell protocol" come to MC. In almost all cases, SSH server (is RSH is still used anywhere?) is used as FISH one and it execute shell command, and FISH command is ignored.

And if truncation could not be avoided, instead of calling fish_send_command() -1 must be returned immediately to prevent hanging. On the other hand, if user made some mistake in a script, fish_send_command() could hang. I'm wondering, whether this could be avoided generally.

If this is not an one-liner fix, such as:

fish_{utime,chmod,...}()
    fish_send_command()
        fish_command()
            fish_get_reply()
                vfs_s_get_line() // replace this with vfs_s_get_line_interruptible() ?

would it be better to track this in a new bug?

BTW, is vfs_s_get_line() not generally risky or there were some reasons to use it?

You can avoid hang of client, but what about server side?

You send a shell script to server. Then you go to read server's answer. You don't know that you send truncated script. Server still wait for script end. Now we get the following state: you are reading socket to get the server's answer, but server is reading socket to get the end of your script. What happens in server if you terminate your read?

[ag]

Replying to andrew_b:

Replying to ag:

Andrew,
The only minor thing, I've noticed, is:

-"#UTIME \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n"                         \
+"#UTIME \"$FISH_TOUCHATIME\" \"$FISH_TOUCHMTIME\" \"$FISH_TOUCHATIME_W_NSEC\" \"$FISH_TOUCHMTIME_W_NSEC\" $FISH_FILENAME\n" \

As I understand, the 1st line is command for FISH server. As I understand again, the utime command has 3 arguments: atime, mtime, filename. Thus we can't send 5 arguments.
On the other hand, is there any implementation of native FISH server? I can't find any of them. All references of "files transferred over shell protocol" come to MC. In almost all cases, SSH server (is RSH is still used anywhere?) is used as FISH one and it execute shell command, and FISH command is ignored.

Once you pointed out an issue with FISH, I've been searching on the FISH protocol and found either FISH shell or the fact, that FISH was invented for and in MC. Now you confirmed this. Looking how it works: just sends script and runs it over SSH (good question about RSH), I assumed the first commented line is a very short manual what environment variables gets the script. Sorry, I was wrong about this. So this seems to be done now.

And if truncation could not be avoided, instead of calling fish_send_command() -1 must be returned immediately to prevent hanging. On the other hand, if user made some mistake in a script, fish_send_command() could hang. I'm wondering, whether this could be avoided generally.

If this is not an one-liner fix, such as:

fish_{utime,chmod,...}()
    fish_send_command()
        fish_command()
            fish_get_reply()
                vfs_s_get_line() // replace this with vfs_s_get_line_interruptible() ?

would it be better to track this in a new bug?

BTW, is vfs_s_get_line() not generally risky or there were some reasons to use it?

You can avoid hang of client, but what about server side?

You send a shell script to server. Then you go to read server's answer. You don't know that you send truncated script. Server still wait for script end. Now we get the following state: you are reading socket to get the server's answer, but server is reading socket to get the end of your script. What happens in server if you terminate your read?

As I described above, I assumed we can detect truncation and by immediately return -1 we can avoid sending, which would block. But even if we couldn't detect the truncation, we can append/overwrite the last character with EOL (this is enough to signal the end of a script?). So the question remains, what could happen if the script itself is buggy. It can just miss an echo. Then if the read is terminated, we terminate the connection and return error? Or in another case if the script doesn't finish at all, would it be possible to terminate the connection either? I have not yet the necessary overview of the related code to be sure, how the connection establishment and finishing are handled.

[andrew_b]

Replying to ag:

As I described above, I assumed we can detect truncation and by immediately return -1 we can avoid sending, which would block. But even if we couldn't detect the truncation, we can append/overwrite the last character with EOL (this is enough to signal the end of a script?). So the question remains, what could happen if the script itself is buggy. It can just miss an echo. Then if the read is terminated, we terminate the connection and return error? Or in another case if the script doesn't finish at all, would it be possible to terminate the connection either? I have not yet the necessary overview of the related code to be sure, how the connection establishment and finishing are handled.

These questions are off topic of nanosecond precision and topic for another ticket.

So, if there are no objections here, I'm going to merge the 3575_nanoseconds branch into master one and close this thicket.

[andrew_b]

Merged to master: [4a6fb3b5343932b8a7b628959ceffdfcc15fe44c].

git log --pretty=oneline 9f47eec..4a6fb3b

[zaytsev]

New ticket: https://midnight-commander.org/ticket/4542

[zaytsev]

Was followed up by #3821