Ticket #2072 (new defect)

Opened 14 years ago

Last modified 7 years ago

high cpu usage during directory navigation and UTF-8 locale

Reported by: LwarX Owned by:
Priority: major Milestone: Future Releases
Component: mc-core Version: 4.8.2
Keywords: Cc: onlyjob@…, baltazar.bz@…
Blocked By: Blocking:
Branch state: no branch Votes for changeset:

Description

Probably this is not an MC fault, but anyway this is what I see while entering to/from deeply nested directory (residing on local filesystem):

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
11675 lwarx     20   0 20980 14256  3780 S  0.0  1.4  0:02.79  `- urxvtd -f -q -o
12244 lwarx     20   0  6180  1764  1428 S  0.0  0.2  0:00.39  |   `- -bash
12544 lwarx     20   0  9132  3060  2176 S  1.0  0.3  0:00.77  |   |   `- /usr/bin/mc -P /tmp/mc-lwarx/mc.pwd.12244
12546 lwarx     20   0  6164  1672  1352 R 68.0  0.2  0:28.90  |   |       `- bash -rcfile .bashrc

The more nested directory is, the higher CPU usage. Delay is also noticeable - on my 1200Mhz machine it can be up to one second. With LANG=C this slowdown goes away. I'm on Linux, bash version 3.2.

Attachments

test.list (2.3 KB) - added by LwarX 14 years ago.
result of ls -lR

Change History

comment:1 Changed 14 years ago by LwarX

  • Priority changed from major to minor

I've narrowed this issue down to the underlying shell. First, I upgraded bash to 4.0.35 (with no success). Then I tried zsh and there is no slowdown at all!
Probably this is something similar to grep issue with multibyte locales: https://bugs.launchpad.net/ubuntu/+source/grep/+bug/7906 https://bugzilla.redhat.com/show_bug.cgi?id=194471 http://savannah.gnu.org/bugs/?14472

My guess is that MC triggers some bash operations which are slow on multibyte strings. Maybe it is somehow related to cwd length. Details are up to you, guys!

P.S. Funny thing: this is 2010 now and I am (while being reluctant and conservative for years) finally decided to switch my system to UTF-8. Nah! It is still not yet ready!

comment:2 Changed 14 years ago by angel_il

please do:
ls -lR > test.list
and attach this file.

comment:3 Changed 14 years ago by LwarX

I'm not sure it is useful:

$ pwd
/mnt/500/160G-hdd-onecopy/My Music/CD's/Fashionweek-fall winter20032004

$ ls -lR > /tmp/test.list
... see attached file

If I launch MC with LANG=C, filenames are unreadable but directory entering/quitting is fast.

Changed 14 years ago by LwarX

result of ls -lR

comment:4 Changed 14 years ago by angel_il

mc -V

comment:5 Changed 14 years ago by LwarX

$ mc -V
GNU Midnight Commander, версия 4.7.1
Виртуальная файловая система: tarfs, extfs, cpiofs, ftpfs, fish, undelfs
Со встроенным редактором
С установленной в системе библиотекой S-Lang с базой данных terminfo
C поддержкой внутренней командной оболочки
С поддержкой фоновых операций
С поддержкой мыши в xterm и консоли Linux
С поддержкой интернационализации
С поддержкой многих кодировок
Data types: char 8 int 32 long 32 void * 32 off_t 64 ecs_char 8

comment:6 Changed 14 years ago by angel_il

i'm execute script

#!/bin/sh
for i in `seq 5000`; do
touch "2009 01 Дорожка $i.wma"
done

i dont have trouble...

comment:7 Changed 14 years ago by angel_il

$locale

comment:8 Changed 14 years ago by LwarX

$ locale
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

I launched your script and the time it takes to complete does not depend on locale settings.
Can you please elaborate a bit more on what MC does internally when user enters/leaves some directory? Is there any bash operations invoked in subshell?

comment:9 Changed 14 years ago by LwarX

New fact: without subshell support (-u switch) directory switching is fast!

My guess: when MC enters any directory, it sends "cd" command to underlying subshell and probably this slowdown caused by escaping or something similar (unicode character space is huge). Can we construct benchmark to verify this?

comment:10 Changed 12 years ago by andrew_b

  • Branch state set to no branch
  • Milestone changed from 4.7 to Future Releases

comment:11 Changed 12 years ago by onlyjob

  • Priority changed from minor to major
  • Cc onlyjob@… added

This is more serious than it looks.

Create nested folder structure with the following command:

  for n in $(seq 100); do DN="${DN}/notsolongdirnametotestnested_$n"; done; mkdir -p "./${DN}"

and then enter as deep as possible using mc. Around No.25 I feel slowdown, but entering one between No.34 and No.40 breaks the panels. Then MC hang when I press "Ctrl+o" to hide/restore panels.

Sometimes it print out "Warning: Cannot change to..."

Last edited 12 years ago by onlyjob (previous) (diff)

comment:12 Changed 12 years ago by egmont

About the performance: I have the same problems with bash 3.2.48, tracing mc shows it spends way too much time in feed_subshell(). I cannot reproduce the speed problem with bash 4.0.38, 4.1 or 4.2, they are all very fast.

The "cannot change to..." is indeed another problem, I guess we're hitting the OS limit on path length or something like that.

comment:13 Changed 12 years ago by onlyjob

If you try to enter long nested directories with MC with subshell enabled (bash 4.2) you will feel slowdown. Perhaps it was worse with older bash but most recent bash didn't make this problem go away.

It has nothing to do with OS limit and apparently only a problem with bash. (I couldn't reproduce with other shells).
I think partially this problem because MC changing current working directory by passing full path to subshell.

comment:14 Changed 12 years ago by onlyjob

  • Version changed from 4.7.1 to 4.8.2

setting version to 4.8.2, since latest release is affected as well.

I can not change to 33rd directory if hierarchy created as described in comment 11.
Hiding panels hang MC.

Just a thought - perhaps 'cd' algorithm can be optimised to change only from current working directory rather than from root (which implies using full path for every 'cd' operation).

Last edited 12 years ago by onlyjob (previous) (diff)

comment:15 Changed 8 years ago by pytnik89

I have the same problem with version 4.8.15. MC enters in directories very slow, but without subshell (mc -u) all works perfectly. Is the way to solve it?

comment:16 follow-up: ↓ 17 Changed 8 years ago by zaytsev

Try zsh? Use mc -u?

comment:17 in reply to: ↑ 16 Changed 8 years ago by pytnik89

Replying to zaytsev:

Try zsh? Use mc -u?

Of course I did. I use zsh. And with 'mc -u' all is perfect.

comment:18 follow-up: ↓ 19 Changed 8 years ago by zaytsev

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

comment:19 in reply to: ↑ 18 Changed 8 years ago by pytnik89

Replying to zaytsev:

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

Thx for u reply, Yes it's a problem of zsh, and specialy with it zsh-suggestion plugin.

comment:20 Changed 8 years ago by balta2ar

  • Cc baltazar.bz@… added

comment:21 Changed 7 years ago by balta2ar

I'm not sure what we can do about that at mc side

@zaytsev It is true that zsh plugins like zsh-autosuggestions and z are contributing to the delay here. But on the other hand, I think that there IS something we could do on mc side.

zsh has this bracketed paste magic mode (https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-magic) which can be used to tell zsh to temporarily stop processing every incoming character. I believe this could vastly improve the responsiveness of mc+zsh.

I tried to do that by my own. I described my attempts here: https://github.com/zsh-users/zsh-autosuggestions/issues/136#issuecomment-220615998. Also see this commit: https://github.com/balta2ar/mc/commit/f17ebb89791ce5ff64608929f62658fa04f4556d. Unfortunately, it didn't help in this form and I believe it's just my lack of understanding how mc handles interaction with a shell and related IO code.

As you can see, this issue has been attempted to be fixed by several parties including zsh-autosuggestions (https://github.com/zsh-users/zsh-autosuggestions/issues/136) and z (https://github.com/rupa/z/issues/191). They mitigated the issue to some extent but it's still there. On top of that, other shells seem to be experiencing the same problem: https://github.com/zeit/hyper/issues/1276

Taking all that into account, could you please consider using bracket paste magic with zsh as a special case?

comment:22 Changed 7 years ago by zaytsev

I'm not quite sure what do you want us to do. It seems that you've tried bracketed paste mode for zsh, and it didn't make any difference, or did I get it wrong? Besides, I'm unconvinced that we should even be doing something like that at mc side in the first place.

How about resource hungry plugins trying to detect that cd command comes from mc and disable themselves ;-) ? It is actually very easy, the cds we send are always preceded with a space, and that's very rarely done by humans. Maybe introduce additional flag, like disable suggestions on a cd preceded with space...

Note: See TracTickets for help on using tickets.