A Better FTP?

cppgodjavademigod asks: "I used to work for a company that sold a file transfer product for datacenters. It supported checkpoint/restart, encyrpted password transmition, asynchonous job procesing, etc. Is there an Open Source project that aims to provide a better FTP? I'm looking for something that makes use of multiple paths (for machines connected via more than one network), job restart, job control, secure transmission (over internet), maybe even tunneling over HTTP and redundant servers (via some kind of private P2P protocol)."

Comments

This is a great idea - but it won't catch on.

[Anton Anatopopov]

There are a couple of problems. The main one is that most of the servers out there understand the FTP protocol, it has critical mass, and it is 'good enough'.

This means that there is really no incentive to change to it.

The second problem is that for most people, most of the time, their Internet connection is pretty reliable. This is also improving all the time as more and more people move to DSL and cable modems instead of dialling up.

Swarmcast?

[RossyB]

Sounds a lot like Swarmcast to me...

FSP, anyone?

[Chelloveck]

Remember FSP, the "File Server Protocol". It was introduced about 10 years ago and was supposed to be the FTP-killer. Technically it probably was superior, but good ol' FTP was available everywhere and was good enough. Today you'd be hard-pressed to find any FSP sites at all. The last published version of the FSP FAQ appears to be dated 1996-08-19. It seems there's really no demand for a better FTP.

Re:FSP, anyone?

[rw2]

I hand't heard of FSP, so I read that doc. Doesn't look to me like it was supposed to be an FTP killer, but an anonymous FTP killer. Different thing. It also doesn't handle any of the important points that the poster asked about.

FSP seems to have died for lack of new or interesting things, rather than because FTP was too entrenched.

rsync has some features..

[slashkitty]

not exactly FTP, but it does tranfer files. It tunnels over ssh, and can copy vast directory trees. And for slow connections, it can both compress the data, and only transfer the files and parts of files that are needed (sorta binary diff).

rsync efficient secure file transfers

[Wills]

The rsync algorithm meets most of your requirements. rsync was proposed in 1998 by Andrew Tridgell for efficient secure file transfers. The main points are:

• For efficiency rsync skips any previously received parts of files, a process based on transmitting small checksums instead of large file chunks.
• For security you can tunnel rsync over any secure protocol such as ssh/openssh. If you don't want or need protocol-level security you can tunnel it over http.

The detailed description is here (http://samba.anu.edu.au/rsync/tech_report/), and open-source software is here (http://samba.anu.edu.au/rsync/download.html).

Overall rsync is often much (10x) faster than using compressed file transfers. It is most useful for users who frequently download new versions of packages with significant similarities between successive versions.

Re:rsync efficient secure file transfers

[gopherdata]

rsync is a great for transferring entire directory structures. I use it to keep our development, backup, and webservers up to date. We have about 4 gigs of data on our sites because rsync only transfers the files that have been updated syncing a day's updates across our entire network usually takes only a few seconds. However, as useful as rsync is, it is not a replacement for ftp.

Grid computing pushing this issue

[rw2]

I work in grid computing and we have some needs that push this idea forward. Over at Argonne labs the Globus team has put forward this draft of extensions for some of what you talk about (i.e. it's secure and multi-path). Code exists under yet another open source license the "Globus Toolkit Public License".

HTTP/1.1

[Matts]

HTTP 1.1 has most of what you're looking for. And with the DAV extensions you can get the equivalent of directory listings too.

Re:HTTP/1.1

[Slynkie]

I'll be honest up front: I don't have a good comparison of either the features or performance of HTTP/1.1 vs. FTP.

That said, I have to wonder whether HTTP/1.1 could be a true solution, for the simple fact that HTTP was not created specifically for the purpose suggested. In addition, for future development purposes, would we really want to bog down HTTP with features not used in everyday web transactions?

*shrug*, just my initial thought, I might not have a clue what I'm talking about =)

SFTP

[Anthanos]

How about SFTP? It is an FTP like protocol layered ontop of SSH. While it may not have ALL the features you were looking for, it has the most important - security.

Re:SFTP

[rw2]

While it may not have ALL the features you were looking for, it has the most important - security.

FYI, security isn't the most important for everyone.

I'm much more concerned these days with bandwidth utilization, which would be kind of hosed by a scheme that encrypts the data stream. I probably want encrypted authentication, but that's it.

Re:SFTP

[vrmlguy]

Why would encryption reduce the bandwidth? Encrypted data is generally the same size as the original, depending on whether a block or stream cypher is used and how much, if any, header data is attached to the message.

Re:SFTP

[rw2]

Encrypted data is generally the same size as the original, depending on whether a block or stream cypher is used and how much, if any, header data is attached to the message.

True that the data going down the pipe is the same size, but unless you have multiple machines sending pieces of the same data down different paths to the endpoint, you find that more often than not your CPU cannot encrypt the data quickly enough to keep up with the pipe that is available to it.
So in quick tests we find that SCP takes about twice as long as native FTP. I've never tested SFTP, but imagine similar results. Do you know?

Re:SFTP

[rw2]

FWIW, our data is nearly uncompressable so that doesn't apply to us. In the case of compressable data, the issues of encryption and compression are orthoganal. If you compress the unencrypted data (which of course you must do in either case as encrypted data isn't compressable) then you can frequently save more time in transmission than you spend on the compression step.

If you then choose to encrypt it, your crypto algorythm will at some point not be able to keep up with the pipe available.

If you choose not to encrypt it you will have saved an almost unimaginable number of cycles to use pushing data down the pipe instead.

Remember what started this thread though. I don't disagree that encryption is useful, just that the naked assertion that security is the most important thing isn't nearly always true.

sendfile

[Tumbleweed]

I doubt it does all you want, but on a related topic, I always wondered why Sendfile never made it big. It would solve a lot of other problems. I used to use it on Bitnet back in my college days. Amazingly, it's apparently an available spell for Sorcerer GNU Linux.

LFTP?

[psychosis]

LFTP is an excellent command-line and scriptable tool. Check out the fm.net page for more info.

Not sure if it does the encrypted password part, but it has almost every other bell and whistle out there. My fave is the 'mirror' and 'mirror -R' commands - does a comparison with the local file timestamps/sizes and only "get"s or "put"s the required files.

Bittorrent

[rafa]

bittorrent has some of what you're looking for. It automaticaally mirrors when you download, helping ease the load on the server for poular downloads. Worth checking out. It could probably be run over ipsec if you wanted to.

Content-Addressable Web

[Orasis]

Hello, I am the creator of Swarmcast and have just written a new paper entitled "HTTP Extensions for the Content-Addressable Web" that is available at onionnetworks.com.

The Content-Addressable Web provides all of the asked-for features, including multi-source/parallel downloads, and the ability to safely retrieve content from untrusted mirrors.

Please read the paper and tell me what you think.