Multithreaded files copying/moving/deleting

[birdie]

In a perfect world, MC should use at least three threads for copying/moving files:

One thread to write to a ring buffer;
One thread to read from a ring buffer;
One thread to show progress every X seconds (for instance, 0.3 seconds).

Right now MC can be slow at copying for a different reason: because it spends too much time updating the screen.

Copy 10 thousand small files from tmpfs to tmpfs in MC and using cp -a, the latter might be up to 50 times faster.

Copy a single 4GB file from tmpfs to tmpfs, cp -a will be significantly faster.

[zaytsev]

I'm afraid multithreading ain't gonna happen any time soon :-/ Currently, background copying is implemented via forking.

[zaytsev]

Which one should be marked as a dupe, this one or #2300?

[mooffie]

Replying to zaytsev:

Which one should be marked as a dupe, this one or #2300?

Besides wishing for multithreading, he also made a claim about MC being slow because of screen updating. So perhaps this ticket (or a new one) should be made a task, "Ascertain the reason for copy/move/delete slowness": whether it's indeed because of screen updates or perhaps because of checking for user input.

(Later, some food for thought perhaps could be found in #2136 (which introduces a timer), #3247 and #43 which it links to, or by "git log --grep count src/filemanager/filegui.c".)

---

UPDATE: Forget my suggestion to rename this to "ascertain blah blah blah". We should mark this as a dupe of #43, or #2300. But I do wonder if anybody verified that the screen updating is indeed the main culprit.

[slyfox]

I'm sure 'cp -a' does not use threads.

As with #2193 there is no evidence kernel (or mc) is bad at actually reading/writing/caching data.
Not even a single reproducible script I can run and see any difference of strategy
copying change.

For me it's hard to imagine adding another userspace caching layer involving more memory
allocation and thread synchonization for such a simple workload will improve things.
What was described is a memcpy() test with mc's UI overhead.

That's why I keep asking for a runnable reproducer. It will be easy to inspect, explain
and mitigate observed issues.

Operating system can already see basic access patterns and prefetch data for you.
If it's not enough 'posix_fadvise(2)' is one of the ways to give operating system hints.

Unless you are on a FUSE filesystem which is an opposite to what is described in the ticket.

Having a way to ask mc from commandline to perform a certain user action might be a step
in the right direction. Like 'mc --eval='cp foo bar; exit'

[andrew_b]

Closed as duplicate of #2300.