savannah: VFS optimization request

[slavazanko]

Original: ​http://savannah.gnu.org/bugs/?8136

Submitted by:	None	Submitted on:	Fri 12 Mar 2004 11:00:45 PM UTC
Category:	VFS	Severity:	3 - Normal
Status:	None	Privacy:	Public
Assigned to:	None	Open/Closed:	Open
Release:	4.6.0	Operating System:	GNU/Linux

Original submission:

Not a bug, but feature request.

I am using VFS (extfs). I have written program (backup extractor) 
which handle "copyout" and "list" commands from mc. But if I want to
 copyout a whole directory (or a lot of files from archive), it is 
called FOR EACH extracted file. In my case, extractor (duplicity 
backupper, http://www.nongnu.org/duplicity/ ) starts very long time.
 But it can extract whole directory quickly.

I think if mc would pass to extfs script environment variable 
containing names of ALL the selected files (AND DIRECTORIES) on the 
panel before extracting, it would get ability to optimize extract 
script. For example, I can fully extract all needed files in 
temporary directory by ONE call to duplicity, and then just copy 
these files from temp directory to pathes needed by mc.

Comment 1 by Anonymous at Sat 13 Mar 2004 12:19:29 AM UTC:

Here is a simple patch (also attached). Hope you will apply it in
next release.

--- mc-4.6.1-pre1/src/file.c 2003-10-27 20:36:19.000000000 +0300
+++ mc-4.6.1-pre1.patched/src/file.c 2004-03-12 03:13:05.000000000 +0300
@@ -1865,4 +1865,27 @@
}
}
+
+ /*
+ * Dmitry Koteroff <d[at]koteroff.ru>
+ * Patch to save selected items in environment variable.
+ * This variable would be passed to extfs scripts and may be used
+ * to optimize depacking performance.
+ */
+ {{{{
+ char* env = g_strconcat("", NULL);
+ for (i = 0; i < panel->count; i++) {
+ char prevEnv, p;
+ if (!panel->dir.list[i].f.marked) continue;
+ p = concat_dir_and_file (panel->cwd, panel->dir.list[i].fname);
+ prevEnv = env;
+ env = g_strconcat(prevEnv, p, "\n", NULL);
+ g_free(p);
+ g_free(prevEnv);
+ }
+ setenv("MC_SELECTION", env, 1);
+ g_free(env);
+ }}}}
+
+
#ifdef WITH_BACKGROUND
/* Did the user select to do a background operation? */

Comment 2 by Anonymous at Sat 13 Mar 2004 06:40:31 PM UTC:

Here is the better patch, it also consider current cursor position.

P.S.
"return 0" in background operation instead of "goto glean_up" isn't a bug?..

--- mc-4.6.0/src/file.c 2002-12-26 22:04:10.000000000 +0300
+++ mc-4.6.0.patched/src/file.c 2004-03-12 21:22:17.000000000 +0300
@@ -1895,4 +1895,30 @@
}
}
+
+ /*
+ * Dmitry Koteroff <d[at]koteroff.ru>
+ * Patch to save selected items in environment variable.
+ * This variable would be passed to extfs scripts and may be used
+ * to optimize depacking performance.
+ */
+ {{{{
+ char* env = g_strconcat("", NULL);
+ for (i = 0; i < panel->count; i++) {
+ char prevEnv, p;
+ if (!panel->dir.list[i].f.marked) continue;
+ p = concat_dir_and_file (panel->cwd, panel->dir.list[i].fname);
+ prevEnv = env;
+ env = g_strconcat(prevEnv, p, "\n", NULL);
+ g_free(p);
+ g_free(prevEnv);
+ }
+ setenv("MC_SELECTION", env, 1);
+ g_free(env);
+ char *cur = concat_dir_and_file (panel->cwd, source);
+ setenv("MC_CURRENT", cur, 1);
+ g_free(cur);
+ }}}}
+
+
#ifdef WITH_BACKGROUND
/* Did the user select to do a background operation? */
@@ -1913,4 +1939,5 @@
mc_setctl (dest, MCCTL_FORGET_ABOUT, NULL);
/* file_op_context_destroy (ctx); */
+ setenv("MC_SELECTION", "", 1); setenv("MC_CURRENT", "", 1);
return 0;
}
@@ -2123,4 +2150,5 @@
} /* Many files */
clean_up:
+ setenv("MC_SELECTION", "", 1); setenv("MC_CURRENT", "", 1);
/* Clean up */

[metux]

Wouldnt it be better to introduce some new vfs operation
(bulk_copy?) for that ?

[birdie]

Also I have a firm belief that VFS plugins must be rewritten using a faster (than shell) language. It's already done for CPIO and TAR but other archivers need this treatment too. However I have no idea which language can suit this task because C/C++ language is difficult to code and patch quickly, while script languages are not really as fast as they need to be.

Try to enter an archive containing over 50 thousand files and you'll instantly notice how much time MC spends parsing a VFS plugin output.

[slavazanko]

(In #1952) Ticket frozen until ticket:3 unfixed.

[TiGR]

Are there any news on this ticket? Currently, unpacking archives is a real pain, while it is a very common use case.

[andrew_b]

A lot of changes in VFS layer are required for this issue. That will be possible only after 4.8.0 release.