Slashdot Mirror
Fast TCP To Increase Speed Of File Transfers?
Wrighter writes "There's a new story at Yahoo about a new version of TCP called Fast TCP that might help increase the speed of file transfers. Sounds like it basically estimates the maximum efficient speed of your network, and then goes for it, dumping a lot of time-consuming error checking." There's also an article at the New Scientist with some additional information.
Let's see. Transmission without error-checking is called UDP, isn't it?
This would be badass when combined with BitTorrent!
THERE IS NO DATA. THERE IS O
I remember back in my BBS days what a big deal zmodem was when it started getting used all over the place. As I recall, it changed the block size that you would receive dynamically based on line quality.
So when you sent a block of 2k and got no errors, the frame size increased to 4k...8k... etc etc... Sounds like a similar approach.
Case
P.S. That was a long time ago in a FidoNet far far away, so my terms may be off.
"Caltech is already in talks with Microsoft and Disney about using it for video on demand," the magazine added. "Hey! Let's take a technology that's potentially revolutionary, and give it to Microsoft!" Yay for Caltech!
Thats what TCP does, just in a passive fashion. The sender queues all sent packets for resend, (on a delay) and packets are deleted from the queue when an acknowledgement for a range of packets is recived from the remote computer.
If you're curious: RFC 793
Who needs erro>*H~@}&)aA=cking anyway?
That's not a soda... it's a caffeine delivery device!
Looking at the information on their web page at caltech, the FAST network project is working with alternate TCP window sizing schemes.
Namely, instead of reducing window size in the case of packet loss, window size is changed based on round trip latency. The problem being that reducing the window size in response to loss works well on most networks, but has a serious problem when dealing with very high-bandwidth links.
In such a case, the conventional TCP windowing will shrink greatly in response to even one or two lost packets, which when you are sending a LOT of data, will occur.
The real work (and it seems to be somewhat covered in their web pages) is how to use latency for congestion detection/control, but I haven't read it in enough detail to quite understand this, NOR how this scheme will interact with conventional TCP streams.
Test your net with Netalyzr
"When the researchers tested 10 Fast TCP systems together it boosted the speed to more than 6,000 times the capacity of the ordinary broadband links.
6,000 times? The tests done in labs are usually stripped-down and the results overstated just for statistical pleasure. In the real world, however, such figures are rarely achieved.
Measuring the round-trip time for packets and using this to information to predict the bandwidth delay product is nothing new. This is essentially one of the effects achieved with existing TCP congestion control algorithms such as TCP Tahoe, TCP Vegas and TCP Reno. The article is light on details and doesn't lead me to believe that they've done anything signicantly different from these three. Furthermore, if it *is* doing something different, how can it still obey the existing congestion control algorithms without thrashing? After all, we can all boost the speed of our TCP connections by simply turning off congestion control, as long as nobody else does it either. ;) [UDP's lack of congestion control is precisely why a few streaming video users can clog up an entire pipe for themselves, screwing everyone else who's using it.]
Does that mean TCP has 99.99% (humor me) overhead ?
But seriously, you can probably use large windows to send streams of packets such that a single ack is required for a bunch of them, but it's impossible to achieve 6000x more throughput just by "optimizing" the TCP protocol. Even over Internet (I'm not even talking LANs since there is obviously not that much room for improvement due to the low latency).
It's called congestion collapse and the condition is described by RFC 896 by John Nagle.
Just firing packets into the network willy-nilly is very bad; it's the "tragedy of the commons" all over again...
This protocol figures out ahead of time if it needs to slow down so its always getting acks back instead of waiting for timeouts. Also it avoids the binary backoff time that happens with timeouts.
So in response to many of the previous posts it loses none of the robustness of TCP. In the worst case its as slow as TCP and in the best case it should be equally as fast as TCP. In the average case, however, it shows a huge performance increase. Most of the time on the network is the average case so this is a good thing.
"You can now flame me, I am full of love,"
Looks like this this
SCTP is a reliable transport protocol operating on top of a connectionless packet network such as IP. It offers the following services to its users:
-- acknowledged error-free non-duplicated transfer of user data,
-- data fragmentation to conform to discovered path MTU size,
-- sequenced delivery of user messages within multiple streams,
with an option for order-of-arrival delivery of individual user
messages,
-- optional bundling of multiple user messages into a single SCTP
packet, and
-- network-level fault tolerance through supporting of multi-
homing at either or both ends of an association.
The design of SCTP includes appropriate congestion avoidance behavior
and resistance to flooding and masquerade attacks.
We substituted the coffee Slashdot normally drinks with "Sandoz Crystals", Lets see if they notice the difference
... to get a good name for this technology.
With Microsoft's, it would be ActiveTCP.
With Intel's, it would be HyperTCP.
And so on, and so forth.
This is part of a whole bunch of TCP and networking related work at CalTech.
I hate to do this to them, but the Caltech Networking Lab site has more info.
From what I see, the improvement here is to use packet delay instead of packet loss for congestion control. They claim this has a bunch of advantages for both speed and quality.
Here is a Google cached copy of their paper from March 2003.
If only I were using Fast TCP, this could have been first post!
As near as I can tell from the popular articles, and the web page referenced in the New Scientist article, "Fast TCP" is not a new protocol, but rather better congestion control for standard TCP. I'm not a network guru by any means, so please take the comments below with a grain of salt.
Currently, TCP implementations use a standard trick to play nice with small router queues. Using precise timing would be better. I hassled Mike Karels over it about 10-15 years ago, but the consensus at the time was that the hardware wasn't up to it. Now it is. Also, modern routers have gotten clever about queue management, which screws up the trick.
The new proposal is to take advantage of modern HW to measure latencies. Existing TCP could thus be used more efficiently, by allowing larger amounts of data to be outstanding on the network without trashing routers.
It is not widely understood that in 1988 the Internet DOSed itself because of a protocol design issue, and Van Jacobsen got everybody to fix it by a consensus change to the reference implementation of TCP. These articles appear to report (badly) ongoing research into that issue.
See caltech's press release on FAST for an article that actually makes sense.
Also, could someone please explain to me why boringly predictable stereotypical slashdot feedback is being modded up?
"Whoa! Faster pr0n!"
"Imagine a beowolf cluster of these!"
-Insert completely unrelated Microsoft bashing post here-
-Insert completely unrelated technobabble from some geek posting out of their ass (without reading the article first)-
News for nerds. Stuff that matters. Discussion that doesn't.
Second, it's intended for use for single big flows on gigabit networks with long latency. You have to be pumping a few hundred megabits per second on a single TCP connection over a link with 100ms latency before it really pays off. It won't do a thing for your DSL connection. It won't do a thing for your LAN. It won't do a thing for a site with a thousand TCP connections on a gigabit pipe.
Third, unlike some previously hokey attempts to modify TCP, this one has what looks, at first glance, like sound theory behind it. There's a stability criterion for window size adjustment. That's a major step forward.
(I first addressed these issues in RFC 896 and RFC 970, back in 1984-1985. Those are the RFCs that first addressed how a TCP should behave so as not to overload the network, and what to do if it misbehaves. So I know something about this.)
It sounds like they are working on a replacement for the Nagle algorithm. Nagle works well on clean connections, even if badly tuned slow-start gets annoying when you have gigabit connection and it still takes a minute to ramp up to full speed on an ISO download. Where "fast tcp" would really help is on a dirty connection. I had to connect to a supercomputer a few years ago over a 100Mbps link that corrupted or lost 30% of all packets and I had to use my own streaming on top of UDP to avoid getting hammered by shrinking windows (I still needed congestion control.) On this type of connection I'd expect their "fast tcp" might give a 10x speedup. On a normal non-noisy and relatively slow DSL connection, like I have at home, I'd be surprised by a 10% speedup.
In other words the story is all wrong, but what they are doing is actually worthwhile. You sometimes have noisy networks, especially when they are wireless or in an industrial environment. The big long haul telecoms lines are better off doing error correction on line, but in the last mile you never really know the noise characteristics so this should be handled better on the TCP level. I would probably do something like FEC with the number of recoverable errors per packet and per lost packet per logical block, tuned to the error characteristics of the network. Then call it TCP2 and release an RFC and some BSD licensed source code.. (I thought of doing this as part of building an ISP friendly P2P protocol but decided I didn't have the time..) Their solution has the advantage that it works just great with regular old TCP implementations.
First of all, the New Scientist article doesn't mention anything about TCP sliding windows or congestion control.
The whole "driving a car while looking 10 meters ahead" analogy ignores a lot of the work TCP does to keep things moving fast. The "trasnmists, waits, then sends the next" packet paragraph is almost deliberately misleading.
It tosses about a "6000 times faster" statistic without explaining 6000 times faster than what. Is my dad's 28.8 modem going to suddenly be getting throughput of 172Mbps? Of course not, but what difference is it going to make to him? I think maybe none at all, and FastTCP is only for very large network hauls, but the article has claims about me downloading a movie in 5 seconds.
My DSL line is 768kbps; I get downloads of large files through it of around 85kBps, which is a data throughput rate of 680kbps. That means that all the layers of the OSI burrito, including resends, checksums, and routining information, add up to about 12% overhead. Not the best, but not that bad, either. How much improvement is FastTCP going to get me?
The numbers for their practical test of Fast TCP connected two computers that got 925Mbps, and the "ordinary" TCP got just 266Mbps. Even that's pretty unbelievable to me; I find it hard to believe that TCP was running at about 25% efficiency.
Extraordinary claims demand extraordinary evidence. Like I said before, without further technical details, this doesn't actually sound all that different than the claims of Pixelon, which also had an eye towards video on demand.
Maybe they've got something; someone linked to the actual caltech article, which I haven't had a chance to read in detail (and wasn't linked to at the time I started my post). Caltech certainly is a cool place, so there is probably something interesting going on. But the New Scientist article is a fluff piece, pure and simple, and if calliing shennanigans on it makes me a troll, so be it.
I don't know why they don't just get nVidia to design it. That way the sending machine will only send the packets it thinks you're actually going to be able to see instead of the entire datastream.
nTCP = instant speed increase.
Error checking hasn't been removed from TCP; we've removed it from Slashdot story summaries instead, speeding up the posting by nearly 6000x.
--
Linux-Universe
Freedom Is Universal
Linux-Universe
Congestion control based on roundtrip times is old news but is uncommon AFAIK. What really happens is direct feedback from routers along a transmission's path. This is done in TCP Vegas, which was first proposed in 1994 and I think is fairly common now. The problem with scaling this or any of the other common TCP implementations to high speed/high delay links is the reaction to detected congestion. "Normal" TCP aggressively scales back its send window (send rate) when it detects congestion, usually chopping it in half. The window/rate then grows linearly until something goes wrong again. This results in alot of lost throughput in high-speed networks especially if the amount of "real" congestion is low. The FAST group is working on a new TCP implementation that doesn't react so aggressively to congestion. This is great for those high-speed/low-congestion networks we all wish really existed but is not something you want to use on the always-backed-up Internet. Would probably make things worse.
TCP is extremely bursty - it pumps all the packets it can as fast as it can over the network as soon as the window opens. Then it waits for replies to all the packets. What typically happens is the burst from the NIC overloads the local router causing numerous dropped packets. This gives the imporession to the sending machine that the network is overloaded and results in a ~90% reduction in bandwidth utilization.
The change is to include a timer that allows the NIC to space the initial burst over the entire window. This prevents the overloading at the router and permits the NIC to reach near its theoretical maximum bandwidth.
In tests involving one router, the results were an order of magnitude increase in bandwith utilization. I'd be interested in seeing their test setup to see how they got such dramatic improvements. Normally TCP/IP is not that ineffecient - even with its extreme burstiness.
The number you have dialed is imaginary, please rotate your phone 90 degrees and try again.
The last sentence of both news articles suggests that this broadband-optimized TCP system could be used by corporations like Disney to provide video-on-demand. (If they're talking to Microsoft, on the other hand, the result will just be a modification to the TCP/IP stack in Windows(r), which doesn't care at all what kind of data it's transmitting)
That's just wrong, at least according to the ways media companies have traditionally desired their materials to be broadcast over the internet. They typically use streaming protocols, which not only gives the user one-click startup, but also makes it non-trivial to keep a local copy of the file (enhancing the corporation's feeling of control).
However, a well-designed streaming protocol won't use TCP at all. TCP hides many characteristics of the network from the application software, and to stream properly it needs to know as much as possible. One example of why TCP is bad for streaming: in streaming, you try to keep advancing time at a constant rate. Once 156 seconds of playing have elapsed, you want to be showing video from exactly 156 seconds into the source file. If at 155 seconds some packets were dropped, you should just skip over them and continue onward. TCP, however, will always try to retransmit any lost packets, even if that means they'll arrive too late to do any good. TCP has no knowledge that packets may expire after a fixed time, but a custom-built UDP protocol can be aware of that constraint.
(Here's a reference on preferring UDP in video streaming)
On the other hand, maybe a corporation will realize that properly controlled non-streaming playback can provide a better end-user experience (guaranteeing, for example, that once playing starts, network failures will never interrupt it). In that case, they might either try to push Microsoft to integrate this faster TCP/IP into Windows(r), or more interestingly, implement it themselves in customized player software.
It's possible to implement a protocol equivalent to TCP on top of UDP, with only a tiny constant amount of overhead. So a programmer for realplayer, quicktime, or mplayer might be able to add the techniques from this research to his own code, even without support in the operating system.
Actually if you read the New Scientist artictle you can see that that's a lie. What they actually did was bundle 10 FastTcp connetions (one must assume on fast lines) togeather and, fairly unsuprisingly, got speeds 6,000 times fast than a broadband connection... wow... 10 high speed lines are faster than broadband??
This would be more interesting had they actually tested it on a standard 512kbs connection and seen if there was a speed increase. IMO it most likely would not make a huge a difference anyway since alot of the slowdown on a consumer broadband connecting is the connection buffers at your ISP. For a better explanation read the Traffic Shaping HOWTO.
I believe this is HSTCP.
For more info, you can also take a look at:
Web100 and Net100.
It basically amounts to improving the AIMD algorithm and changing the way slow start works as well. Also, whoever said it before that this will not help your DSL connection is correct. It is meant to help high speed long RTT paths. And it does so -- quite well.