Add LFTP-like HTTP filesystem to VFS

[harvie]

Hello, LFTP client have nice feature which i like to see in MC. LFTP is capable of transfering files (or directories) over almost any protocol... especialy i found usefull feature which can browse, download or recursively mirror links on some HTTP URL. In case of apache filelisting (or similar sites) this means browsing and mirroring directory tree in same way as read-only ftp.

I think such VFS can't be too hard to implement (just parse all links on page, handle relative/absolute links well and add make sure it will work well when recursively downloading whole directories).

another good feature is optional continuing of interrupted downloads...

maybe you can use LFTP binary for this (not sure if it comply with your rules).

here is example of how it looks in lftp for ​http://temp.harvie.cz/ listing:

[11:33:12] 0 ;) harvie@harvie-ntb ~ $ lftp http://temp.harvie.cz/
cd uspělo, cwd=/          
lftp temp.harvie.cz:/> ls
-rw-r--r--  --  icons/blank.png            
-rw-r--r--  --  icons/folder.png
drwxr-xr-x            -  2009-04-29 23:17  autedosk-acedomya
drwxr-xr-x            -  2008-03-04 22:04  deb
drwxr-xr-x            -  2007-10-21 22:10  doh
drwxr-xr-x            -  2008-02-26 17:17  firmwhore
-rw-r--r--         8.3K  2008-08-02 00:33  skillz.html
-rw-r--r--          313  2008-08-02 00:33  skoda.txt
-rw-r--r--         1.2M  2009-04-14 21:25  tablo.jpg
-rw-r--r--         4.0M  2009-04-19 19:58  tablo.svg
-rw-r--r--         6.8M  2009-04-19 20:40  tablo.zip
-rw-r--r--         4.1M  2009-05-12 22:46  tablo2.svg
-rw-r--r--          435  2010-03-24 11:10  zaver.txt
lftp temp.harvie.cz:/> exit

[harvie]

BTW LFTP It self provides great VFS very similar to one in mc... and there is also lftp filesystem for mounting remote servers based on fuse and using original lftp binary.

i think it can be really good idea to (re-)implement whole VFS in mc using LFTP. this can add support for multiple protocols at once... if you don't like 3rd party binaries you can fork lftp for yourself. only caveat is that right now lftp can't handle SMB which is one of really usefull features in mc when you don't have any samba client with GUI installed on your box...

see man lftp:

lftp is a file transfer program that allows sophisticated ftp, http and other connections to other hosts. If site is specified then lftp will connect to that site otherwise a connection has to be established with the open command.

lftp can handle several file access methods - ftp, ftps, http, https, hftp, fish, sftp and file (https and ftps are only available when lftp is compiled with GNU TLS or OpenSSL library). You can specify the method to use in open URL' command, e.g. open ​http://www.us.kernel.org/pub/linux'. hftp is ftp-over-http-proxy protocol. It can be used automatically instead of ftp if ftp:proxy is set to `​http://proxy[:port]'. Fish is a protocol working over an ssh connection to a unix account. SFtp is a protocol implemented in ssh2 as sftp subsystem.

[andrew_b]

Replying to harvie:

i think it can be really good idea to (re-)implement whole VFS in mc using LFTP.

Patches are welcome!

you can fork lftp for yourself.

No, thanks.

[harvie]

Replying to andrew_b:

Patches are welcome!

i wanted to add http protocol to extfs.d script, but i've found that such script needs to know whole tree which is ok for archives, tarballs, packages and similar stuff, but unusable for implementing remote filesystems like http(s)...

http can be simply implemented if the script will be passed everything after it's name. right now extfs.d scripts have no support for paths like this one:

cd /#http:user:password@domain.tld/directory
my script will be passed only
http list /