high cpu usage during directory navigation and UTF-8 locale

[LwarX]

Probably this is not an MC fault, but anyway this is what I see while entering to/from deeply nested directory (residing on local filesystem):

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
11675 lwarx     20   0 20980 14256  3780 S  0.0  1.4  0:02.79  `- urxvtd -f -q -o
12244 lwarx     20   0  6180  1764  1428 S  0.0  0.2  0:00.39  |   `- -bash
12544 lwarx     20   0  9132  3060  2176 S  1.0  0.3  0:00.77  |   |   `- /usr/bin/mc -P /tmp/mc-lwarx/mc.pwd.12244
12546 lwarx     20   0  6164  1672  1352 R 68.0  0.2  0:28.90  |   |       `- bash -rcfile .bashrc

The more nested directory is, the higher CPU usage. Delay is also noticeable - on my 1200Mhz machine it can be up to one second. With LANG=C this slowdown goes away. I'm on Linux, bash version 3.2.

[LwarX]

I've narrowed this issue down to the underlying shell. First, I upgraded bash to 4.0.35 (with no success). Then I tried zsh and there is no slowdown at all!
Probably this is something similar to grep issue with multibyte locales: ​https://bugs.launchpad.net/ubuntu/+source/grep/+bug/7906 ​https://bugzilla.redhat.com/show_bug.cgi?id=194471 ​http://savannah.gnu.org/bugs/?14472

My guess is that MC triggers some bash operations which are slow on multibyte strings. Maybe it is somehow related to cwd length. Details are up to you, guys!

P.S. Funny thing: this is 2010 now and I am (while being reluctant and conservative for years) finally decided to switch my system to UTF-8. Nah! It is still not yet ready!

[angel_il]

please do:
ls -lR > test.list
and attach this file.

[LwarX]

I'm not sure it is useful:

$ pwd
/mnt/500/160G-hdd-onecopy/My Music/CD's/Fashionweek-fall winter20032004

$ ls -lR > /tmp/test.list
... see attached file

If I launch MC with LANG=C, filenames are unreadable but directory entering/quitting is fast.

[LwarX]

result of ls -lR

[angel_il]

mc -V

[LwarX]

$ mc -V
GNU Midnight Commander, версия 4.7.1
Виртуальная файловая система: tarfs, extfs, cpiofs, ftpfs, fish, undelfs
Со встроенным редактором
С установленной в системе библиотекой S-Lang с базой данных terminfo
C поддержкой внутренней командной оболочки
С поддержкой фоновых операций
С поддержкой мыши в xterm и консоли Linux
С поддержкой интернационализации
С поддержкой многих кодировок
Data types: char 8 int 32 long 32 void * 32 off_t 64 ecs_char 8

[angel_il]

i'm execute script

#!/bin/sh
for i in `seq 5000`; do
touch "2009 01 Дорожка $i.wma"
done

i dont have trouble...

[angel_il]

$locale

[LwarX]

$ locale
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

I launched your script and the time it takes to complete does not depend on locale settings.
Can you please elaborate a bit more on what MC does internally when user enters/leaves some directory? Is there any bash operations invoked in subshell?

[LwarX]

New fact: without subshell support (-u switch) directory switching is fast!

My guess: when MC enters any directory, it sends "cd" command to underlying subshell and probably this slowdown caused by escaping or something similar (unicode character space is huge). Can we construct benchmark to verify this?

[onlyjob]

This is more serious than it looks.

Create nested folder structure with the following command:

  for n in $(seq 100); do DN="${DN}/notsolongdirnametotestnested_$n"; done; mkdir -p "./${DN}"

and then enter as deep as possible using mc. Around No.25 I feel slowdown, but entering one between No.34 and No.40 breaks the panels. Then MC hang when I press "Ctrl+o" to hide/restore panels.

Sometimes it print out "Warning: Cannot change to..."

[egmont]

About the performance: I have the same problems with bash 3.2.48, tracing mc shows it spends way too much time in feed_subshell(). I cannot reproduce the speed problem with bash 4.0.38, 4.1 or 4.2, they are all very fast.

The "cannot change to..." is indeed another problem, I guess we're hitting the OS limit on path length or something like that.

[onlyjob]

If you try to enter long nested directories with MC with subshell enabled (bash 4.2) you will feel slowdown. Perhaps it was worse with older bash but most recent bash didn't make this problem go away.

It has nothing to do with OS limit and apparently only a problem with bash. (I couldn't reproduce with other shells).
I think partially this problem because MC changing current working directory by passing full path to subshell.

[onlyjob]

setting version to 4.8.2, since latest release is affected as well.

I can not change to 33rd directory if hierarchy created as described in comment 11.
Hiding panels hang MC.

Just a thought - perhaps 'cd' algorithm can be optimised to change only from current working directory rather than from root (which implies using full path for every 'cd' operation).

[pytnik89]

I have the same problem with version 4.8.15. MC enters in directories very slow, but without subshell (mc -u) all works perfectly. Is the way to solve it?

[zaytsev]

Try zsh? Use mc -u?

[pytnik89]

Replying to zaytsev:

Try zsh? Use mc -u?

Of course I did. I use zsh. And with 'mc -u' all is perfect.

[zaytsev]

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

[pytnik89]

Replying to zaytsev:

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

Thx for u reply, Yes it's a problem of zsh, and specialy with it zsh-suggestion plugin.

[balta2ar]

I'm not sure what we can do about that at mc side

@zaytsev It is true that zsh plugins like zsh-autosuggestions and z are contributing to the delay here. But on the other hand, I think that there IS something we could do on mc side.

zsh has this bracketed paste magic mode (​https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-magic) which can be used to tell zsh to temporarily stop processing every incoming character. I believe this could vastly improve the responsiveness of mc+zsh.

I tried to do that by my own. I described my attempts here: ​https://github.com/zsh-users/zsh-autosuggestions/issues/136#issuecomment-220615998. Also see this commit: ​https://github.com/balta2ar/mc/commit/f17ebb89791ce5ff64608929f62658fa04f4556d. Unfortunately, it didn't help in this form and I believe it's just my lack of understanding how mc handles interaction with a shell and related IO code.

As you can see, this issue has been attempted to be fixed by several parties including zsh-autosuggestions (​https://github.com/zsh-users/zsh-autosuggestions/issues/136) and z (​https://github.com/rupa/z/issues/191). They mitigated the issue to some extent but it's still there. On top of that, other shells seem to be experiencing the same problem: ​https://github.com/zeit/hyper/issues/1276

Taking all that into account, could you please consider using bracket paste magic with zsh as a special case?

[zaytsev]

I'm not quite sure what do you want us to do. It seems that you've tried bracketed paste mode for zsh, and it didn't make any difference, or did I get it wrong? Besides, I'm unconvinced that we should even be doing something like that at mc side in the first place.

How about resource hungry plugins trying to detect that cd command comes from mc and disable themselves ;-) ? It is actually very easy, the cds we send are always preceded with a space, and that's very rarely done by humans. Maybe introduce additional flag, like disable suggestions on a cd preceded with space...