Slashdot Mirror
FTP: Better Than HTTP, Or Obsolete?
An anonymous reader asks "Looking to serve files for downloading (typically 1MB-6MB), I'm confused about whether I should provide an FTP server instead of / as well as HTTP. According to a rapid Google search, the experts say 1) HTTP is slower and less reliable than FTP and 2) HTTP is amateur and will make you look a wimp. But a) FTP is full of security holes. and b) FTP is a crumbling legacy protocol and will make you look a dinosaur. Surely some contradiction... Should I make the effort to implement FTP or take desperate steps to avoid it?"
I haven't really noticed any reliability issues with http anymore. If it starts loading it usually finishes, and I haven't run into any corruption problems. Maybe if you were serving huge files ftp would be a good idea, but for 1-6 mb it's probably not worth it.
Its encrypted it is easy to get through firewalls, and it lets you run rsync (which lets you synchronise two machines automatically).
Ftp is a major pain for firewalls because unless you use it in passive mode it opens another port back after the initial connection.
Actually, http is faster. FTP keeps open another connection to control the file transfer. This connection also uses bandwith, just a little bit, but enough to matter.
My god i wish there was! Every time i need it, i have to go look for putty, and download psftp. Maybe we should write a patch for mozilla?
If you're only offering files to a group of users who you can give passwords to, you could even use SCP. (Secure copy...uses sshd on the server side)
It all depends on the application. I only use SCP to move files around if I have the choice, just because I like better security if I can have it.
But if you want to offer files to the public, I'd recommend offering both FTP and HTTP so people can use the most convenient.
What I don't care for with FTP is the continuous setup/teardown of data connections. What is even worse with active FTP is that the client side of the data connection establishes server ports, and the server becomes the client (I'd like to be able to use plug-gw from the TIS FWTK for FTP, but this is not possible for the data connections). However, even when enabling passive FTP, the data connections are too prone to lockup. The difficulty of implementing all of this in C probably contributes to the FTP server vulnerabilities.
Still, if you want both (optionally anonymous) upload ability and access from a web browser, FTP is the only game in town.
From the network perspective, the rsh/rcp mechanism is cleaner (in that there is only one connection), but it still has the problem of either passing cleartext authentication or establishing unreasonable levels of trust with trivial authentication. In addition, with rcp syntax you must know much more about the path to a file, and there is no real "browsing."
Many say that SSH is the panacea for these problems, but sometimes I am not concerned about encryption and I just want to quickly transfer a file. The SSH man pages indicate that encryption can be disabled, but I have never been able to make this work. SCP also has never been implemented in a browser for file transfers. I should also say that I've never used sftp, because it has so little support.
Someday, we will have a good, encrypted file transfer protocol (and reliable implementations of that protocol). Sorry to say, but ftp, rcp, and scp are not it. What will this new protocol support?
Boy, I never thought that I could rant about file transfer software for so long!
Use HTTP and don't make your server names unique (use round-robin DNS instead). That way broadband ISPs that use caching will only need to download your file once in a while. This is good because (a) you save money on transfer charges, and there is more bandwidth available for you, and (b) the broadband provider saves money on bandwidth AND delivers the file faster to its customers. It's win/win. There are algorythms to make ensure 'freshness' (or use a unique filename for each version if you have frequent changes.) Seriously... go HTTP only.
Of course, since these days most networks use switches instead of hubs, one has to wonder how much of a security risk sniffers are anyway. I guess there's always the possibility of ARP poisoning or some rogue router between the source and destination, but honestly, how likely is that? I guess it just depends on how secure you need your data to be.
Everyone disables the only unique thing FTP can do (third-party transfers) to prevent a variety of attacks by proxy. It might be worth accomodating decrepit machines that lack HTTP clients, but there are no actual advantages to using FTP.
We had a client a few years ago (about 1998), that we needed to set up data transfer for.
IBMGSA were running their IT systems, and they decided that FTAM (over OSI) was the way to go, citing FTP's supposed lack of reliability, and lack of features such as resume...
They were absolutely serious when they said that too.....
We eventually managed to get the whole setup working (FTAM, OSI, X.25 over an ISDN link)...but it was the most unreliable, cryptic, complicated system I've ever had to deal with...
A couple of years later, the client ditched IBMGSA, and the company that took over their systems took one look at the FTAM / OSI mess, and threw it out and replaced it with FTP over TCP/IP, and we've never had another problem.
Advanced users are users too!
Also, you forgot the first and biggest site with MAGNET links. Still, an excellent tutorial, thanks for writing it!
I would say go with http.. my reason is simple.. my college blocks ftp! There are two sides of the story, first off, the complete blocking of ftp started a week ago when they switched firewalls.. they couldn't get the ftp to work, and they decided it wasn't a high priority so they didn't do anything to fix it... After yelling at the IT department for half the day I got the priority up. But besides that.. Even before, when they allowed ftp, it was only active ftp.. I was unable to go to a website, click on a file that is served from ftp:// and download it! I'd have to use flashfxp or another client to d/l files.. I was able to figure that out quickly, but what about the other 99% of the school that wants to download say shareware from download.com or a driver from somewhere.. they are denied! FTP is a superior protocol for file transfer in my opinion, but administrators don't seem to care about it as much as they do http.
They serve different purposes, and each was designed to meet those needs. Each one has its own positives and negatives.
True, but ftp has far more negatives. The connection model ftp uses, in which the server connects back to the client for data transfers, is a horrible idea. Surely, there must be a benefit to this method (or else why would ftp have become such a standard?), but I've yet to hear it. Passive ftp (PASV) is supposed to compensate for the fact that clients behind NAT can't use ftp, and passive is pretty much supported everywhere now. But passive doesn't solve the problem of many-connections, it just shifts it: With passive ftp, there is still a seperate "data" connection for every file transfer, or transfer of directory contents. It's just that with passive ftp the data connection is initiated by the client to some high-numbered port on the server. This allows clients behind NAT to do ftp, but it makes firewalling the server a pain in the ass, especially if the server is (god forbid) behind nat itself. It's relatively easy to setup a mailserver, or webserver, or ssh server, on a machine behind nat (assuming you can forward ports on the nat gateway). But try doing the same with ftp. Passive connections won't work, unless the ftpd is configured to only use a certain range of high ports for it's data connection, and the firewall forwards that entire range. You'd think with all these connections, the protocol might support multiple connections from one login, ie list another directory while downloading. Well, thats not the case at all, you have to login again if you want simultaneous transactions.
And then if you want tunneled connections... oh boy, what a pain in the ass:
I set this all up for someone recently, and it DOES finally work, with tunneled passive ftp connections to a proftpd server behind a nat gateway, but it was a big pain. Due to the way proftpd works (not sure about other daemons...) the passive connections are directed to the interface the client is connected to. So, with an ssh tunneled connection, proftpd sees the client connection TO 127.0.0.1 (because thats where sshd is). So it tells the client, connect back to me for the data connection, my IP is 127.0.0.1. And the client tries to connect to itself, and fails saying "can't open data connection". The solution was to point the tunnel not to 127.0.0.1, but to the external IP, and have the nat gateway forward internal traffic back to itself. So traffic comes in over ssh, over to the firewallwall (where the nat rewriting happens), and then back to the same machine for ftp.
If people could just use sFTP clients, this wouldn't be an issue _at_all_ (One client connection, one tcp connection! Hey, what a concept!). But due to the integrated ftp clients in programs like bbedit and dreamweaver, I suspect we'll be dealing with that bastard protocol known as ftp for quite a long time....
Damn, reading back over this, I'm kind of ranting. BUT ITS ALL TRUE!
Sorry bout that.
[X] Post Anonymously
Because you get a larger share of the bandwidth pie. This applies both on the server end, and your ISP end. If you have bandwidth to spare, you can get 6 (or 8, or 10, depends on the client) users worth of bandwidth because each connection is treated by the server as seperate. On the other hand if you have a slow ISP, you can get a larger share of your ISP's bandwidth.
I have 3.5mbit DSL, but my ISP's performance is flaky. However, I have no problem pulling 300-350KB/s with a download accelerator.
But the question brings up another point: FTP is a rather cumbersome protocol for transferring files. Ok, so you have two separate connections for entering commands and for sending data. Why then can I not enter any commands while data is being transferred?? Some FTP clients simulate this capability by open multiple control connections, but it really should be built into the protocol, IMHO.
HTTP was designed for small text file transfers, and not really for bigger files (although it does usually handle them pretty well).
I propose an "FTP2" protocol. This protocol will have the following features over and above the original FTP protocol:
Control connection is fully functional even when data is being transferred
Multiple data connections are allowed for each control connection (configurable by server operator)
Ability to queue files for download
Separate server-wide limits on control and data connections (for example, so I can log in and look around even if all data connections are used up. If I try to download something, it is queued until a data connection frees up. This puts an end to hammering on anon-ftp servers.)
Modular authentication -- ability to support authentication methods other than clear-text passwords
Support for encrypted connections
No server connecting to the client to establish a data connection, eliminating most firewall and NAT problems
MIME-type support, rather than clunky BIN/ASCII modes.
I'm not suggesting this is the answer to your problem, I haven't seen any GUI-based fsp clients (or a unix cli one for ages for that matter).
But it did seem very useful for what it was designed to do, serve out large files to anonymous outsiders.
You're being awfully pedantic about what I said, but I suppose I can be equally pedantic in return...
My layers are not "messed up" in any way. HTTP and FTP have everything to do with firewalls because the protocols have recommended ports they run on, and HTTP sticks to one port and, thus, can be easily allowed in firewalls. FTP's use of ports depends heavily on the protocol since it specifies the use of both incoming and outgoing connections on various ports for the whole schebang unless you use passive, which is a setting, and by far not the default. Even with passive, the protocol embeds ports and is generally firewall unfriendly to firewalls, hence me saysing FTP is not firewall friendly.
Also, directory listings, as talked about elsewhere in this article, have a lot to do with HTTP as it does not have a specification for them whereas FTP does. It's a protocol thing.
Oh, and you would turn on passive if you had firewall problems, but I'm assuming that to be a typo. Long day for me, too.
Rsync is the way to go!
Rsync is great in theory, but the implementation has one major problem that makes it less than ideal for many cases: It puts a huge burden on the server, because the server has to calculate the MD5 sums on each block of each file it serves up, which is a CPU-intensive task. A machine which could easily handle a few dozen HTTP downloads at a time would choke with only a few rsync downloads.
This is a problem with the implementation, not with the theory, because it wouldn't be that difficult for the rsync server to cache the MD5 sums so that it only had to calculate them once for each file (assuming it's downloading static content -- for dynamic content rsync will probably never make sense, particularly since we can probably expect bandwidth to increase faster than processing power). The server could even take advantage of 'idle' times to precalculate sums. Once it had all of the sums cached, serving files via rsync wouldn't be that much more costly in terms of CPU power than HTTP or FTP, and it would often be *much* more efficient in terms of bandwidth.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
In FTP the sender connects to the receiver, but client issuing commands does not have to be either of those two machines. As designed, FTP should be able to allow a client to send data between two arbitrary machines. Its an infrequently used feature, and I'm not sure if it is implemented by all clients, but it is, technically, a feature that HTTP doesn't have.
Hmm.. There's no reason why you couldn't just perform the download via HTTP if your target customer or audience has the bandwidth.
If you're concerned about the customer losing connectivity, you could always just break the file up into small chunks for download.
Both FTP and HTTP, depending on clients and servers, supports continuation-downloading(ie, recovering from where you left off.) So if one connection flakes out, you can always recover.
The long and short of it?
HTTP(S) or FTP(s/FTP), it doesn't matter. With both, you can restrict access and/or grant access to specific files. You can recover from some point or offer split files. With HTTP, you can have the information piped through a back-end piece of code so that you can monitor who does what when, but something similar can be done with FTP as well.
If you've already got a webserver going and secured, you're probably better off offering the files via HTTP. Nothing unprofessional about it.
Fact is, if your target audience is the general public, you'll fare better with HTTP. Why? Because while most people who are online will probably have access to a web browser and know how to download HTTP files, they may not have an FTP client or necessarily know how to use their browser as an FTP client.
Me? I offer photos and docs online via HTTP. It's easier and I can control the visual aspect of the "download area" to create a more professional look than that of the "industrial" FTP interface.
Nothing wrong with FTP or sFTP. I use them at work and at home. But then again, when I find a site which offers HTTP or FTP? I pick HTTP to download first. It's only when I'm still looking for something that I go with FTP... which, by the way, becomes difficult to navigate on some clients. A nicely generated webpage featuring the item via HTTP would be much more elegant.
Just my two cents.
Winged Power Photography
Any serious network security professional who's not just faking it considers the FTP protocol an *evil* *abomination*. This protocol MUST DIE! Someone please publish an RFC that "officially" deprecates it.
Note I'm talking *protocol*, not any particular client or server implementation. The protocol is fundamentally flawed.
After hosting an HTTP file transfer area for some time for my company, we decided to move to an FTP setup that was a bit more sophisticated.
So far, it's been a failure for two reasons:
1. IE blows as an FTP client, and users aren't comfortable dropping into the (somewhat crappy) DOS FTP client.
2. Firewall setups at the fortune 500 companies that we deal with normally seem to keep FTP access off-site restricted.
Why are you letting these clowns ruin our country?
> For getting (possibly multiple) files whose location you don't know in advance, FTP is more flexible and efficient.
:>
Ever heard of web page directories that almost every common web server uses, and most web servers automatically do when no index file is present?
I find lynx+ web page directories a lot faster to get files across my network than ftp.
The only real advantage I see with ftp is uploading files quickly and easily... but that's not for me
In Unix, you can background this operation fairly simply...
This will start downloading your file in the background, will autoresume if something happens, waiting 15 seconds between retries, and never give up until the file is completed or the source no longer has it. You'll get an email when it completes one way or another.
Help us build a better map!
So I should abandon encryption and just be free to let whomever listen to traffic coming from my box? If the the images are locally referenced, they'll be transferred securely as well (talking https here). And modern browsers keep connections open for reuse, so that only thing that can be gleaned is when a spike happens, which means that you are loading content from the same site, but not how big it is, since a web page is a bundle of different elements that need to be retrieved.
Also, if you think that just knowing when data packets are sent and received tells you the contents of the data, then why hasn't SSH been broken yet? With those assumptions, it should be easy to use a statistical model of people's keystrokes and break into a session. Realistically? Not possible.
You've got to start somewhere, especially if you're going to get the world to transfer to using encryption everywhere. The mail server I use for private mail among friends has no unsecure channel going out of it. All of it's protocols are protected with SSL/TLS, including SMTP inbound and outbound. And besides, the argument I was making was for using encryption to connect to a public mirror site. Since the majority of mirrors have HUGE numbers of different files, and the amount of control data sent is never constant, knowing how much data is sent in either direction will NOT be a guarantee that you know what was transferred.
AND, I was talking about legal prosecution (or persecution, depending on how you see it), and a document that says you downloaded approximately enough bytes to be a suspect program isn't enough to stand up in court...
If you want to avoid sucking CPU and aren't concerned about sucking bandwidth you can use "--whole-file" to just transfer... wait for it... whole files without doing incremental checks.
But if the recipient doesn't have the file already rsync wouldn't need to do incremental checksums anyway.
(I love rsync, I mirror changes on a 250 gig file server every night, runs in 20-30 minutes generally (5-10 gigs of new stuff))
FTP, in my opinion is antiquated.
In the uploads arena, I could transfer any file faster on rsync or scp, or even cvs -z5. And thats over 56k.
FTP isn't so secure, it's pratically text on port 23 (wonder why the warez sites still use it, aren't they trying to avoid goverment agencies? and M$ and all that crap?).
HTTP is faster for smaller files as FTP eates up a whole lot of time doing the connection procedure.
kazaa sucks try OpenFT much better then kazaa
We have a cheezy VB client that can saturate 100MBs with HTTP PUTs, far faster than SMB or TLS. There's nothing wrong with the protocol at all -- the client and server tools just aren't built for it.
I have never thought of FTP as a crumbling legacy protocol full of security holes. Each protocol has its flaws, security flaws or not. HTTP isn't free from security holes.
They are two different protocols designed for two different purposes. If you plan on transfering large amounts of data, and want it to be more reliable, I would most definetely recomment FTP. FTP is made for transferring files, and that is what it is best at doing.
If you are not concearned with data integrity and plan on transferring decent sized files, I would wish you the best of luck with HTTP.
As far as security holes go? I have no idea, but I imagine both protocols have had their fair share of security issues.
For more reliable transfer of larger files, I would really recommend FTP.
---
Mike
I'm going to kick the next person that I see with their karma rating in their sig.
ftp includes mget. So I can download all the pictures with:
mget *.jpg
Without messing around clicking and saving lots of things. Sometimes a command line rocks.
Andrew
Serve up over P2P protocols, and tell people what filename to look for. If the file becomes very popular, you'll only need to serve up the page giving directions on how to download it :)
(yes, yes, I know, requiring people to use spyware-laden software is not nice, but it's a good idea in an ideal world)
Stop the brainwash
Instead of banning download managers that can do segmented downloading, why not just limit the number of connexions from a given IP? That solves both the segmenting and the "tons of files at once" problems.
I can make a dozen or more connexions to your FTP server with nothing more exotic than Netscape. Why pick on download managers when they use the same number of connexions? (BTW, Getright says right in its configuration that "some servers regard segmenting as rude" and recommends against it.) Better to limit connexions to x-many per IP address, and let the user spend them any way they wish.
BTW, if you do limit connexions, please remember that it usually takes one for browsing the site (using Netscape or whatever) PLUS one for the download manager to fetch the file. Otherwise the user who was looking with a browser has to leave the server, then wait for the browser connexion to close (which can take a while) then finally paste the link into the DLManager. So a limit of two connexions from a given IP is a nice practical minimum, and surely not a hard load for anything outside of home servers operating over dialup.
PS. I love FTP's convenience, and I always try to be extra-polite to small servers (and not rude to big ones). I do use Getright, and have segmenting disabled (which BTW is the default).
~REZ~ #43301. Who'd fake being me anyway?
"No standard directory listing"
Neither has HTTP.
"Bad error recovery, many differing standards on restart"
Erm, yeah, it's so non-standard that prozilla doesn't exist - exploiting the `resume' potential of most FTP servers to parallelize downloads starting 0, 25, 50, 75% through the file. Yeah right.
"Designed for human readable command lines not machine readability"
Oh, come *on*. When there's no standard output format at all in HTML, how can an `ls -l' output not be vaguely standard? Are you unaware of the SIZE FTP command?
"Password sent in plain text, not even concealed"
And somehow HTTP with its sending of usernames and passwords also in plaintext is "better"?? Does it look flourescent pink on your screen, or something?
"HTTP: none of the above"
Poster: no clue what he's on about. HTTP: all of the above. Readers: much annoyance.
HTTP is more easily integrated with SSL in the form of HTTPS.
Almost everyone has an ftp client. You have to install curl or wget or lftp yourself consciously in most distributions.
HTTP: no standard for file uploads, you rely on server-side configs and handlers for the purpose. How mad is that??
~Tim
--
Rushing on down to the circle of the turn
alot of companys only have one external ip.
that'd mean only one user in the entire company can use the server - bad.