Ticket #3630 (closed enhancement: duplicate)

Opened 3 years ago

Last modified 3 years ago

Multithreaded files copying/moving/deleting

Reported by: birdie Owned by:
Priority: minor Milestone:
Component: mc-core Version: master
Keywords: Cc:
Blocked By: Blocking:
Branch state: no branch Votes for changeset:

Description

In a perfect world, MC should use at least three threads for copying/moving files:

One thread to write to a ring buffer;
One thread to read from a ring buffer;
One thread to show progress every X seconds (for instance, 0.3 seconds).

Right now MC can be slow at copying for a different reason: because it spends too much time updating the screen.

Copy 10 thousand small files from tmpfs to tmpfs in MC and using cp -a, the latter might be up to 50 times faster.

Copy a single 4GB file from tmpfs to tmpfs, cp -a will be significantly faster.

Change History

comment:1 Changed 3 years ago by birdie

  • Summary changed from Multithreaded files copying/moving to Multithreaded files copying/moving/deleting

comment:2 Changed 3 years ago by zaytsev

I'm afraid multithreading ain't gonna happen any time soon :-/ Currently, background copying is implemented via forking.

comment:3 follow-up: ↓ 4 Changed 3 years ago by zaytsev

Which one should be marked as a dupe, this one or #2300?

comment:4 in reply to: ↑ 3 Changed 3 years ago by mooffie

Replying to zaytsev:

Which one should be marked as a dupe, this one or #2300?


Besides wishing for multithreading, he also made a claim about MC being slow because of screen updating. So perhaps this ticket (or a new one) should be made a task, "Ascertain the reason for copy/move/delete slowness": whether it's indeed because of screen updates or perhaps because of checking for user input.

(Later, some food for thought perhaps could be found in #2136 (which introduces a timer), #3247 and #43 which it links to, or by "git log --grep count src/filemanager/filegui.c".)


UPDATE: Forget my suggestion to rename this to "ascertain blah blah blah". We should mark this as a dupe of #43, or #2300. But I do wonder if anybody verified that the screen updating is indeed the main culprit.

Last edited 3 years ago by mooffie (previous) (diff)

comment:5 Changed 3 years ago by slyfox

I'm sure 'cp -a' does not use threads.

As with #2193 there is no evidence kernel (or mc) is bad at actually reading/writing/caching data.
Not even a single reproducible script I can run and see any difference of strategy
copying change.

For me it's hard to imagine adding another userspace caching layer involving more memory
allocation and thread synchonization for such a simple workload will improve things.
What was described is a memcpy() test with mc's UI overhead.

That's why I keep asking for a runnable reproducer. It will be easy to inspect, explain
and mitigate observed issues.

Operating system can already see basic access patterns and prefetch data for you.
If it's not enough 'posix_fadvise(2)' is one of the ways to give operating system hints.

Unless you are on a FUSE filesystem which is an opposite to what is described in the ticket.

Having a way to ask mc from commandline to perform a certain user action might be a step
in the right direction. Like 'mc --eval='cp foo bar; exit'

comment:6 Changed 3 years ago by andrew_b

  • Status changed from new to closed
  • Resolution set to duplicate
  • Milestone Future Releases deleted

Closed as duplicate of #2300.

Note: See TracTickets for help on using tickets.