A Better FTP?

cppgodjavademigod asks: "I used to work for a company that sold a file transfer product for datacenters. It supported checkpoint/restart, encyrpted password transmition, asynchonous job procesing, etc. Is there an Open Source project that aims to provide a better FTP? I'm looking for something that makes use of multiple paths (for machines connected via more than one network), job restart, job control, secure transmission (over internet), maybe even tunneling over HTTP and redundant servers (via some kind of private P2P protocol)."

Comments

FSP, anyone?

[Chelloveck]

Remember FSP, the "File Server Protocol". It was introduced about 10 years ago and was supposed to be the FTP-killer. Technically it probably was superior, but good ol' FTP was available everywhere and was good enough. Today you'd be hard-pressed to find any FSP sites at all. The last published version of the FSP FAQ appears to be dated 1996-08-19. It seems there's really no demand for a better FTP.

rsync efficient secure file transfers

[Wills]

The rsync algorithm meets most of your requirements. rsync was proposed in 1998 by Andrew Tridgell for efficient secure file transfers. The main points are:

• For efficiency rsync skips any previously received parts of files, a process based on transmitting small checksums instead of large file chunks.
• For security you can tunnel rsync over any secure protocol such as ssh/openssh. If you don't want or need protocol-level security you can tunnel it over http.

The detailed description is here (http://samba.anu.edu.au/rsync/tech_report/), and open-source software is here (http://samba.anu.edu.au/rsync/download.html).

Overall rsync is often much (10x) faster than using compressed file transfers. It is most useful for users who frequently download new versions of packages with significant similarities between successive versions.

Grid computing pushing this issue

[rw2]

I work in grid computing and we have some needs that push this idea forward. Over at Argonne labs the Globus team has put forward this draft of extensions for some of what you talk about (i.e. it's secure and multi-path). Code exists under yet another open source license the "Globus Toolkit Public License".

Re:HTTP/1.1

[Slynkie]

I'll be honest up front: I don't have a good comparison of either the features or performance of HTTP/1.1 vs. FTP.

That said, I have to wonder whether HTTP/1.1 could be a true solution, for the simple fact that HTTP was not created specifically for the purpose suggested. In addition, for future development purposes, would we really want to bog down HTTP with features not used in everyday web transactions?

*shrug*, just my initial thought, I might not have a clue what I'm talking about =)

Bittorrent

[rafa]

bittorrent has some of what you're looking for. It automaticaally mirrors when you download, helping ease the load on the server for poular downloads. Worth checking out. It could probably be run over ipsec if you wanted to.