Slashdot Mirror
Fast TCP To Increase Speed Of File Transfers?
Wrighter writes "There's a new story at Yahoo about a new version of TCP called Fast TCP that might help increase the speed of file transfers. Sounds like it basically estimates the maximum efficient speed of your network, and then goes for it, dumping a lot of time-consuming error checking." There's also an article at the New Scientist with some additional information.
Faster pr0n!!!!
Let's see. Transmission without error-checking is called UDP, isn't it?
This would be badass when combined with BitTorrent!
THERE IS NO DATA. THERE IS O
SmartTCP. It sounds like equipment is constantly tweaking the connection for the optimum through put.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Why not just number all packets between two hosts and if the recipient doesn't recieve a packet it requests that particular packet to be resent? I see problems with man in the middle attacks but is there any other reason?
just wondering
I remember back in my BBS days what a big deal zmodem was when it started getting used all over the place. As I recall, it changed the block size that you would receive dynamically based on line quality.
So when you sent a block of 2k and got no errors, the frame size increased to 4k...8k... etc etc... Sounds like a similar approach.
Case
P.S. That was a long time ago in a FidoNet far far away, so my terms may be off.
"Caltech is already in talks with Microsoft and Disney about using it for video on demand," the magazine added. "Hey! Let's take a technology that's potentially revolutionary, and give it to Microsoft!" Yay for Caltech!
speed or accuracy..either one...
Isn't estimating the effective bandwidth of the link exactly what tcp window is all about?
I read the article and did not understand what do thay add that is better then the standerd tcp enhancements of selective ack and big window sizes.
clue anyone?
As of Postgres v6.2, time travel is no longer supported.
I'm sorry, but without further technical details, this sounds like the sort of technical mumbo-jumbo that snake-oil salesmen were peddling back in the dot-com era.
Who needs erro>*H~@}&)aA=cking anyway?
That's not a soda... it's a caffeine delivery device!
You people are giving me a headache. There is no mention about the removal of error checking. Actually, the article is lite on any technical details, but you can divine what they are doing well enough. Basically, it seems they keep tweaking mtu rwin, ect until they get the best connection they can.
This isn't revolutionary so much as evolutionary.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Looking at the information on their web page at caltech, the FAST network project is working with alternate TCP window sizing schemes.
Namely, instead of reducing window size in the case of packet loss, window size is changed based on round trip latency. The problem being that reducing the window size in response to loss works well on most networks, but has a serious problem when dealing with very high-bandwidth links.
In such a case, the conventional TCP windowing will shrink greatly in response to even one or two lost packets, which when you are sending a LOT of data, will occur.
The real work (and it seems to be somewhat covered in their web pages) is how to use latency for congestion detection/control, but I haven't read it in enough detail to quite understand this, NOR how this scheme will interact with conventional TCP streams.
Test your net with Netalyzr
Hmm, just think how much faster IIS can get infected with this one!
My sausage tree didn't grow, does that make me a bad mommy?
...will Fast TCP have the Evil Bit?
"When the researchers tested 10 Fast TCP systems together it boosted the speed to more than 6,000 times the capacity of the ordinary broadband links.
6,000 times? The tests done in labs are usually stripped-down and the results overstated just for statistical pleasure. In the real world, however, such figures are rarely achieved.
or you could also call it ReverseDoS..
or Self-Slashdotting! :-)
Measuring the round-trip time for packets and using this to information to predict the bandwidth delay product is nothing new. This is essentially one of the effects achieved with existing TCP congestion control algorithms such as TCP Tahoe, TCP Vegas and TCP Reno. The article is light on details and doesn't lead me to believe that they've done anything signicantly different from these three. Furthermore, if it *is* doing something different, how can it still obey the existing congestion control algorithms without thrashing? After all, we can all boost the speed of our TCP connections by simply turning off congestion control, as long as nobody else does it either. ;) [UDP's lack of congestion control is precisely why a few streaming video users can clog up an entire pipe for themselves, screwing everyone else who's using it.]
Does that mean TCP has 99.99% (humor me) overhead ?
But seriously, you can probably use large windows to send streams of packets such that a single ack is required for a bunch of them, but it's impossible to achieve 6000x more throughput just by "optimizing" the TCP protocol. Even over Internet (I'm not even talking LANs since there is obviously not that much room for improvement due to the low latency).
It's called congestion collapse and the condition is described by RFC 896 by John Nagle.
Just firing packets into the network willy-nilly is very bad; it's the "tragedy of the commons" all over again...
This protocol figures out ahead of time if it needs to slow down so its always getting acks back instead of waiting for timeouts. Also it avoids the binary backoff time that happens with timeouts.
So in response to many of the previous posts it loses none of the robustness of TCP. In the worst case its as slow as TCP and in the best case it should be equally as fast as TCP. In the average case, however, it shows a huge performance increase. Most of the time on the network is the average case so this is a good thing.
"You can now flame me, I am full of love,"
Looks like this this
SCTP is a reliable transport protocol operating on top of a connectionless packet network such as IP. It offers the following services to its users:
-- acknowledged error-free non-duplicated transfer of user data,
-- data fragmentation to conform to discovered path MTU size,
-- sequenced delivery of user messages within multiple streams,
with an option for order-of-arrival delivery of individual user
messages,
-- optional bundling of multiple user messages into a single SCTP
packet, and
-- network-level fault tolerance through supporting of multi-
homing at either or both ends of an association.
The design of SCTP includes appropriate congestion avoidance behavior
and resistance to flooding and masquerade attacks.
We substituted the coffee Slashdot normally drinks with "Sandoz Crystals", Lets see if they notice the difference
... to get a good name for this technology.
With Microsoft's, it would be ActiveTCP.
With Intel's, it would be HyperTCP.
And so on, and so forth.
This is part of a whole bunch of TCP and networking related work at CalTech.
I hate to do this to them, but the Caltech Networking Lab site has more info.
From what I see, the improvement here is to use packet delay instead of packet loss for congestion control. They claim this has a bunch of advantages for both speed and quality.
Here is a Google cached copy of their paper from March 2003.
If only I were using Fast TCP, this could have been first post!
As near as I can tell from the popular articles, and the web page referenced in the New Scientist article, "Fast TCP" is not a new protocol, but rather better congestion control for standard TCP. I'm not a network guru by any means, so please take the comments below with a grain of salt.
Currently, TCP implementations use a standard trick to play nice with small router queues. Using precise timing would be better. I hassled Mike Karels over it about 10-15 years ago, but the consensus at the time was that the hardware wasn't up to it. Now it is. Also, modern routers have gotten clever about queue management, which screws up the trick.
The new proposal is to take advantage of modern HW to measure latencies. Existing TCP could thus be used more efficiently, by allowing larger amounts of data to be outstanding on the network without trashing routers.
It is not widely understood that in 1988 the Internet DOSed itself because of a protocol design issue, and Van Jacobsen got everybody to fix it by a consensus change to the reference implementation of TCP. These articles appear to report (badly) ongoing research into that issue.
Actually the mention of removing error checking is in the /. article summary.
The problem is people are just reading the summary and assuming that it is actually correct and accurate.
Good points otherwise...
Yes indeed. 6000 times faster is possible, but 6000 times faster than what? 6000 times faster than broadband? Maybe if you're downloading a video with one very slow hop in between, and then you all implemented this it could be 6000 times faster. But there's no way it could be 6000 times faster always, or even on average. Go download aol instant messenger and look at the speed. I just did it and got 308 KB/sec, and I've seen as high as 440 from AOL's servers. You mean I could download that same file from them at 1,848,000 KB/sec or 2,640,000 KB/sec with this protocol? (Please just kindly correct my converson if I'm wrong, but that would be about 1.8 and 2.6 gigs a second would it not?) Forget about the protocol, that's rediculous for most hardware. Wide ultra2 scsi only transfers 80 megabytes per sec. How would I download. Can today's ram and motherboards even work with gigs a second?
Is this really as rediculous as it seems to me? I always get confused by the big B/little B byte/bit distinction.
Maybe they're measuring the round-trip delay, and then sending more data than can fit in the reciver's window, on the assumption that ACKs "should be" in flight. Maybe they also notice when an ACK is overdue, and send a duplicate packet early, rather than wait for the normal timeout or a duplicate ACK of earlier data. If they do that, then the duplicate would come 1 RTT after the original, and the reciever's window would be full of after-loss data (so it would catch up right away.) I suppose they could assume that only one packet would be lost, and send another window-full of data after that, before recieving an ACK. (If that assumption was wrong, then that data would be lost and bandwidth wasted.) ... but that's all just guessing.
I do hope that there is something to this (in spite of the fluff of these articles.) We're kindof stuck in terms of throughput with TCP right now.
The former inequality is reasonably easy to fix -- make bigger windows (and buffers). The latter is harder. 9.8 is a constant (I don't know where it comes from, but that value is often quoted, and seems to work out in my experience.) It would be great to fix MSS, but lots of hardware won't support more than 1500 bytes, and nobody benifits from this until everybody upgrades, so nobody upgrades. For fast links, RTT is mostly determined by the speed of light.without further technical details, this sounds like the sort of technical mumbo-jumbo that snake-oil salesmen were peddling back in the dot-com era.
The New Scientist makes it quite clear on how the Fast TCP is done, if you know anything about how TCP works (and how the window size halves in the event of packet losses)
shame on a relatively low-ID user making such trollish comments...
data link layer technology, like Ethernet, already has error checking built into it's frames, so why is there a need for another error checking at the higher transport layer?
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
The speed, er... rather the window size is changed according to a rigid design, and ideas like this in the past have failed because once everyone is doing them, they stability of the network decreases. Mind you this is distinctly different that "removing error checking" because it is basically taking two steps forward instead of one step at a time. If you leap ahead two steps, and fail you simply step back one (which is still one step forward). The packets are still resent on the TCP leaving the application to not worry about data quality. Looking at packet captures of ftp traffic show that ftp is an aggressive consumer of bandwidth, and that ramping up or down near the beginning or end of the TCP session is were the greatest amount of *inefficiency* is found. So the idea is to make tcp more aggressive near the ramp-up/down stages of the connection. The idea also being to remove some of the agonizingly redundant error checking in favor of self-throttling optimistic educated guessing. Fast TCP simply wants to do the dirty work ahead of time instead of gradually discovering the safe speed limit. Fast TCP will bump into glass bandwith ceiling at mach 10, instead of 10 Mph, quicly recovering by resending the big chucks at a fraction of the window size previously sent. So it could also be described as willig to find the threashold quickly in echange for knowing the boundrys instead of wasting precious time ramping up. Traditional TCP hates data loss to the point that is "drives slow in a parking lot" attempting to never have a colision, whne the protocal itself is well designed to recover from such an event already.
It isn't a lie if you belive it.
See caltech's press release on FAST for an article that actually makes sense.
Also, could someone please explain to me why boringly predictable stereotypical slashdot feedback is being modded up?
"Whoa! Faster pr0n!"
"Imagine a beowolf cluster of these!"
-Insert completely unrelated Microsoft bashing post here-
-Insert completely unrelated technobabble from some geek posting out of their ass (without reading the article first)-
News for nerds. Stuff that matters. Discussion that doesn't.
TFTP sucks in the presence of packet loss. Its mechaism to deal with lossy networks is, umm, "trivial" :) The server sends one packet at a time, and waits for an ack after each. If a data packet or ack packet is dropped the server or client retries after something like 5 seconds (configurable, but in seconds). TCP retries faster, and keeps multiple packets in the air at one time.
- AB
Totally ridiculous. That's the kind of crap you can expect from a totally clueless reporter. The sentence is ambiguous and in the end, it doesn't really say anything.
For some reason, he felt the need to mention that 10 systems were tested "together" without saying how.
It says that FastTCP can boost the speed past the "capacity" of ordinary broadband links. I don't know how you can get performance that exceeds the capacity (which I take to mean the theoretical limit). On the other hand, it isn't really clear if they even used "ordinary broadband links". For all we know, they could have been testing on a local GigE network (or on localhost!).
If he's trying to say that FastTCP gets 6,000 times the throughput of TCP, then this implies that TCP can be at most using 1/6,001 of the theoretical maximum. I'm guessing that they're advertising some corner case (or the reporter is retarded, which is probably just as likely).
FastTCP just seems like a better predictor of the delay/capacity properties of a connection. The New Scientist article says in one place that they were able to achieve a 4x increase in transmission speed between Geneva and California, which is impressive, but it's probably under strange conditions. The graph in their paper shows that FastTCP utilizes the link about twice as well as regular TCP when you get to 10Gbit links. The "reporter" must have done some serious mangling to get the number 6,000.
Second, it's intended for use for single big flows on gigabit networks with long latency. You have to be pumping a few hundred megabits per second on a single TCP connection over a link with 100ms latency before it really pays off. It won't do a thing for your DSL connection. It won't do a thing for your LAN. It won't do a thing for a site with a thousand TCP connections on a gigabit pipe.
Third, unlike some previously hokey attempts to modify TCP, this one has what looks, at first glance, like sound theory behind it. There's a stability criterion for window size adjustment. That's a major step forward.
(I first addressed these issues in RFC 896 and RFC 970, back in 1984-1985. Those are the RFCs that first addressed how a TCP should behave so as not to overload the network, and what to do if it misbehaves. So I know something about this.)
It sounds like they are working on a replacement for the Nagle algorithm. Nagle works well on clean connections, even if badly tuned slow-start gets annoying when you have gigabit connection and it still takes a minute to ramp up to full speed on an ISO download. Where "fast tcp" would really help is on a dirty connection. I had to connect to a supercomputer a few years ago over a 100Mbps link that corrupted or lost 30% of all packets and I had to use my own streaming on top of UDP to avoid getting hammered by shrinking windows (I still needed congestion control.) On this type of connection I'd expect their "fast tcp" might give a 10x speedup. On a normal non-noisy and relatively slow DSL connection, like I have at home, I'd be surprised by a 10% speedup.
In other words the story is all wrong, but what they are doing is actually worthwhile. You sometimes have noisy networks, especially when they are wireless or in an industrial environment. The big long haul telecoms lines are better off doing error correction on line, but in the last mile you never really know the noise characteristics so this should be handled better on the TCP level. I would probably do something like FEC with the number of recoverable errors per packet and per lost packet per logical block, tuned to the error characteristics of the network. Then call it TCP2 and release an RFC and some BSD licensed source code.. (I thought of doing this as part of building an ISP friendly P2P protocol but decided I didn't have the time..) Their solution has the advantage that it works just great with regular old TCP implementations.
They represent TCP totally wrong. Not only that, they describe the whole network infrastructure wrong.
:(
No wonder I have trouble explaining how the network works to my sister, or even to my mother who happens to have his masters in tech. (Albeit in mechanical engineering)
Let's see.
"The sending computer transmits a pack, waits for a signal from the recipient that acknowledges its safe arrival, and then sends the next packet"
No honey, thats why we have the buffers. So you could receive packets out of sequence and wait for the middle ones to arrive. This is why we have 32-bit seq and ack fields in the tcp header just after the src and dst ports. seq tells the packets order in the queue. Ack tells the seq ofthe next packet (from other peer) so we can use random increments to prevent spoofing of packets. Or make it harder atleast.
But that's out of the scope of this rant.
If no receipt comes back, the sender transmits the same packet at half the speed of the previous one, and repeats the process, getting slower each time, until it succeeds.
Umm, No. I'm not 100% sure but I think the network devices are dump thingies that talk to each other on predefined carriage frequencies. Thus, you can't really "slow down" the speed to increase possibility to get the packet through. And certainly this has nothing to do with TCP. Resending of failed packets is a Good Thing (TM). They are just sent again untill they reach their destination or the "I give up"-treshold has been reached.
"The difference (in Fast TCP) is in the software and hardware on the sending computer, which continually measures the time it takes for sent packets to arrive and how long acknowledgements take to come back"
This is the only difference? Wow! Shit. We are definetely going to get faster speeds by adding overhead with this calculation.
Now, I'm through with my rant.
I really really would like to see an actual white paper how this works. There has to be more to this. By the sound of just these articles, it seems to me that someone was paid to develop new, faster protocol that would magically be backward compliant with TCP. Finally the persons couldn't come up with anything smart but gobbled together something that might sound plausible.
Of course you can get "more than 6000 times the capacity of ordinary broadband links" by using your very own dedicated parallel LAN links. You just need fast enough computer to handle the TCP stack. You would also need some fricking fast BUS's on you computer to make any use of this bandwith. Remember, the hard drives, mem chips and other storages aren't exactly 'infinite' in speed either.
If the demo consisted only of two computers exchanging data, there would be no need to estimate the speeds as it would be very unlikely to get packet collisions because of disturbance from other network devices. Also, again, that has nothing to do with TCP-stack. Again, this useless speed calculation is just more overhead.
And now I'm rambling.
Shit, why can't i just stop.
I'm angry, that's why
I hope someone would answer me with insight of what am I overlooking. This looks so useless to me.
Bot Assisted Blogging
I don't know why they don't just get nVidia to design it. That way the sending machine will only send the packets it thinks you're actually going to be able to see instead of the entire datastream.
nTCP = instant speed increase.
Heh, I saw this article on Yahoo, and was immediately concerned. Stupid reporters cut out way too much information, and make the people on wee dialup systems think that they'll get the moon.
Anyway, I think this is primarily interesting for people on really fast connections (ranging in hundreds of megabits per second up to gigabits) with relatively large latencies (tens/hundreds of milliseconds as on a transcontinental link rather than nanoseconds/milliseconds like on a LAN), but I'm sure the research will have some effect on LANs and even the standard broadband connection. Impact on dialup and other not-quite-broadband connections would likely be miniscule.
One main issue with TCP is that it uses a "slow start" algorithm, which other people have mentioned. Real TCP stacks probably tweak the algorithm quite a bit, but from the description in Computer Networks (3rd edition, 1996) by Andrew Tannenbaum, TCP packets start off with a small "window"--how much data can be in transit at a time. The window grows exponentially as packets are transmitted and acknowledgements received until a pre-set threshold is reached, and then the window starts growing more slowly (Tannenbaum's example grows exponentially to 32kB at first, then by 1kB per transmitted packet).
If a packet is lost, the process starts over and the threshold is set to half the window size you had before the dropped packet (I imagine many systems reduce the window size by lesser amounts). Now, this particular algorithm can cause quite a bit of nastiness. It's possible the window size will never get very large. This isn't a really huge problem on low-latency links like in a LAN where you get acknowledgements really quickly, but a hyperfast transcontinental link could be reduced to mere kilobits per second even if the percentage of dropped packets is fairly low.
Additionally, this slow start algorithm will eventually force you to restart at a smaller window size. Given enough data, you'll eventually saturate the link and lose a packet, so until the window grows enough again, there will be considerable unused bandwidth. Good TCP stacks would attempt to guess the available link speed and stop growing the window at a certain point.
Smart system administrators can tweak kernel variables to make systems behave better (preventing the window from getting too small, having larger initial thresholds, for instance), but it looks like a lot of work on Fast TCP and related projects is related to making this a more automatic process and growing/reducing the transmit window size in a more intelligent manner.
Error checking hasn't been removed from TCP; we've removed it from Slashdot story summaries instead, speeding up the posting by nearly 6000x.
--
Linux-Universe
Freedom Is Universal
Linux-Universe
Congestion control based on roundtrip times is old news but is uncommon AFAIK. What really happens is direct feedback from routers along a transmission's path. This is done in TCP Vegas, which was first proposed in 1994 and I think is fairly common now. The problem with scaling this or any of the other common TCP implementations to high speed/high delay links is the reaction to detected congestion. "Normal" TCP aggressively scales back its send window (send rate) when it detects congestion, usually chopping it in half. The window/rate then grows linearly until something goes wrong again. This results in alot of lost throughput in high-speed networks especially if the amount of "real" congestion is low. The FAST group is working on a new TCP implementation that doesn't react so aggressively to congestion. This is great for those high-speed/low-congestion networks we all wish really existed but is not something you want to use on the always-backed-up Internet. Would probably make things worse.
TCP is extremely bursty - it pumps all the packets it can as fast as it can over the network as soon as the window opens. Then it waits for replies to all the packets. What typically happens is the burst from the NIC overloads the local router causing numerous dropped packets. This gives the imporession to the sending machine that the network is overloaded and results in a ~90% reduction in bandwidth utilization.
The change is to include a timer that allows the NIC to space the initial burst over the entire window. This prevents the overloading at the router and permits the NIC to reach near its theoretical maximum bandwidth.
In tests involving one router, the results were an order of magnitude increase in bandwith utilization. I'd be interested in seeing their test setup to see how they got such dramatic improvements. Normally TCP/IP is not that ineffecient - even with its extreme burstiness.
The number you have dialed is imaginary, please rotate your phone 90 degrees and try again.
Normally, when you're sending stuff from point A to point B, there is a lot of buffering on the way. At the point where you lose a packet, you've overflown a buffer somewhere. Halving your throughput at that point is probably a good idea - you may be backing off your data transmission a lot right now, but there should be plenty of data sitting in buffers out there that needs to be cleared anyway.
If you only back off a little bit, what happens is you just go overrun that same buffer again, and just send out a lot of data on your end that's just getting dumped halfway out.
Think of it like dumping bags of sugar into a big funnel. You start by dumping in a quarter bag per second, then a half bag per second, then 3/4ths bag per second, then a bag a second, then 1.25 bags per second, then 1.5 bags per second...
At which point you notice that the funnel is full and sugar is overflowing it. At this point, does it make more sense to go back to dumping 1.25 bags per second, or 3/4 bags per second? (Hint: It's 3/4ths.)
By the time you've lost a packet from overflowing a buffer, you definitely want to back off more than a little. The rate you're filling the buffers is likely at least an order of magnitude higher than the rate you're emptying it, so it only makes sense to back off by at least an order of magnitude.
paintball
I stand by my claim that the New Scientist article sounds like snake oil. It's a misleading article, pure and simple.
But, now that I've read some of the documents from the Caltech site, and I think I understand the claims, the research is fairly interesting, at least in the world of "ultrascale" networking. Of course, I'm just an unfrozen caveman engineer, and that world confuses and frightens me, so my understanding might be slightly off. Here goes anyways.
As I understand it, the authors are saying that current TCP congestion avoidance algorithms break down on very high speed, long-haul networks. They mention looking forward to 100Gbps and higher speeds for "ultrascale" supercomputing. They have papers analyzing TCP Vegas (which was designed in 1994) and show that for the networks they're looking at, Vegas "does not scale to this regime". Specifically, they examine throughput stability of Vegas, and show that with these ultrascale networks, performance can end up bouncing between the two states of "balls-to-the-wall" fast and molasses slow. (They do not actually use the phrase "ball-to-the-wall"; that's my addition.) Network performance doesn't reach equilibrium, and your average throughput is quite ugly. In this context, the "cars starting and stopping" analogy starts to make sense.
This is a fairly different "regime" than I'm used to; then again, I'm an unfrozen caveman engineer, and the fastest neworks I've dealth with are 10Gbps LANs, and not high speed WANs. It appears that network performance on such networks can be surprisingly bad, at least, it's surprising to me. (And heck, there's a world of conditions under which Vegas is a great TCP congestion control algorithm, and the network delivers fair bandwidth to everyone. They even link to plenty of papers which analyze, simulate, and measure conditions where current algorithms work just fine.)
They're really looking forward to where the network finally starts to reach pathological extremes, and this all breaks down, which is different than conditions I (and most readers of the New Scientist article) would have seen.
Their first solution, by the way, is a slightly modified algorithm called "stabilized Vegas". They prove that this algorithm avoids the oscilating behavior, and thus avoids the low throughput situation that results. Neat.
Let me summarize my summary: original article bad, sounds like a scam. Actual research interesting, but applies to network speeds and conditions that are forward-looking, and probably don't directly apply to Mr. Bob Homeuser for quite a while. Fire bad.
It comes with Solaris right now. You can also get it for Linux. Why?
It's useful when the ssh client has it built in because you get pretty much the same speed and the ability to download between clients.
By the way, I know about two zmodem-enabled ssh clients:
1) SecureCRT - nonfree/Windows only.
2) Zssh - open-source, cross-platform.
The actual applications which initiate the transfer are called "rz" and "sz."
Mod me down and I will become more powerful than you can possibly imagine!
As I see it (hard to say because the article wasn't exactly technical), here is how it's going to work:
That's wrong. You were possibly misled by a bad article. Your diagram of "Slow TCP" isn't describing TCP at all, but a more primitive protocol that can be called "Stop and Wait". Stop-and-Wait provides some of the same advantages of TCP (reliability and consistency), but obviously takes a painfully long time to transmit a file, so it's little used in real life.
The idea that you can send many data packets before getting an ACK is already part of normal TCP. It's called the "sending window", and it measures the number of packets that can be sent before waiting for an ACK. The sending window is basically an estimate of the bandwidth of the link, which TCP maintains as it runs. The estimate starts out low, and then is continually increased as the transmission goes on. As soon as ACKs fail to arrive, the assumption is that the bandwidth was exceeded, and the estimate is cut in half. It's a linear-increase, exponential falloff procedure- which is meant to prevent from impairing the TCP performance of other users (you increase your transmission speed slowly, but decrease it quickly if there's an overload). The lineary increase of the window size is called "slow start".
Apparently, this research uses a different technique to obtain the starting estimate of the bandwidth available. Instead of sequentially increasing the window size up to gigabit levels as the connection goes, they start out with a large estimate, because they already know the link is fast.
The last sentence of both news articles suggests that this broadband-optimized TCP system could be used by corporations like Disney to provide video-on-demand. (If they're talking to Microsoft, on the other hand, the result will just be a modification to the TCP/IP stack in Windows(r), which doesn't care at all what kind of data it's transmitting)
That's just wrong, at least according to the ways media companies have traditionally desired their materials to be broadcast over the internet. They typically use streaming protocols, which not only gives the user one-click startup, but also makes it non-trivial to keep a local copy of the file (enhancing the corporation's feeling of control).
However, a well-designed streaming protocol won't use TCP at all. TCP hides many characteristics of the network from the application software, and to stream properly it needs to know as much as possible. One example of why TCP is bad for streaming: in streaming, you try to keep advancing time at a constant rate. Once 156 seconds of playing have elapsed, you want to be showing video from exactly 156 seconds into the source file. If at 155 seconds some packets were dropped, you should just skip over them and continue onward. TCP, however, will always try to retransmit any lost packets, even if that means they'll arrive too late to do any good. TCP has no knowledge that packets may expire after a fixed time, but a custom-built UDP protocol can be aware of that constraint.
(Here's a reference on preferring UDP in video streaming)
On the other hand, maybe a corporation will realize that properly controlled non-streaming playback can provide a better end-user experience (guaranteeing, for example, that once playing starts, network failures will never interrupt it). In that case, they might either try to push Microsoft to integrate this faster TCP/IP into Windows(r), or more interestingly, implement it themselves in customized player software.
It's possible to implement a protocol equivalent to TCP on top of UDP, with only a tiny constant amount of overhead. So a programmer for realplayer, quicktime, or mplayer might be able to add the techniques from this research to his own code, even without support in the operating system.
Actually if you read the New Scientist artictle you can see that that's a lie. What they actually did was bundle 10 FastTcp connetions (one must assume on fast lines) togeather and, fairly unsuprisingly, got speeds 6,000 times fast than a broadband connection... wow... 10 high speed lines are faster than broadband??
This would be more interesting had they actually tested it on a standard 512kbs connection and seen if there was a speed increase. IMO it most likely would not make a huge a difference anyway since alot of the slowdown on a consumer broadband connecting is the connection buffers at your ISP. For a better explanation read the Traffic Shaping HOWTO.
The mention of "156 seconds" was an oversimplified example of the kind of problem. It does happen in real streaming applications, however, especially if playing from a live source where latency constraints don't permit lengthy buffers (most important in bidirectional VoIP apps)
.
Of course a streaming protocol uses a buffer. That's one reason why you don't want to run it over TCP. TCP provides its own buffer, which would be redundant with the one the higher-level protocol is also creating. Optimally, one fully-informed piece of software can manage all buffers.
Streaming music over TCP (even HTTP) is fine, because the bandwidth needs are so low that suboptimal solutions work. Streaming video is different- not only is the total bandwidth more than 10x higher, but you've got two separate streams of data whose buffer sizes need to be correlated.
Essentially, if the network is stressed enough so that framedrops (underflows of the application-level buffer) ever happen, then TCP will waste time retransmitting old data, while a hand-rolled UDP solution will recover quicker . Today's popular audio-only streaming doesn't usually push enough data to cause framedrop events
You can experimentally check that UDP is better for streaming video with applications like RealPlayer. Watch some videos with it. Then adjust your firewall to block it's UDP ports (6970+) and try again (or use the HTTP setting from inside realplayer).
TCP does not provide its own underflow buffer. I'm sure modern implementations provide an overflow buffer to make sure it doesn't overwhelm the physical transport layer, but it is not the kind of buffer an application needs to ensure the seamless delivery of streaming content. Does any implementation of TCP allow for the aggregation of an arbitrary amount of data before handing it off to the application? No. That would be counter productive.
Also, RTSP is a protocol-independent stream control mechanism. The RFC clearly states that all application data is transmitted out-of-band in another protocol. It is only a "transmission control" protocol (not to be confused with TCP, of course) and allows for the transport-independent control of data streaming to and from different sources on the net.
In other words, it's not used to make sure the content itself gets from one place to another reliably (or unreliably). In fact, the RFC also provides support to specify a lower-transport protocol for (out-of-band) streams, either UDP or TCP. This would presumably would allow for both reliable and unreliable transmission of data, depending on your specific application.
So in conclusion, reinventing TCP with UDP is a dumb idea and is not what RTSP aims to do. It probably plays a vital role in controlling the behavior of things such as VoIP phones. In such applications, when latency is negligible, buffers are not practical, and a reliable transport mechanism is assumed, then no, a buffer is not necessary or practical, but these assumptions do not apply to the internet-at-large in the least. If you want reliable data on the internet, you use TCP. Period.
The dangers of knowledge trigger emotional distress in human beings.
I have an idead! Fast TCP transfers with no error checking... And we'll call it UDP.
Karma: The shiznight, mostly because I am the Drizzle.
dumping a lot of time-consuming error checking
Sounds like a slashdot editor
I believe this is HSTCP.
For more info, you can also take a look at:
Web100 and Net100.
It basically amounts to improving the AIMD algorithm and changing the way slow start works as well. Also, whoever said it before that this will not help your DSL connection is correct. It is meant to help high speed long RTT paths. And it does so -- quite well.
I actually heard this on the ABC radio news whilst driving to work this morning.
Unfortunately they managed to summarise it in the most bogus form possible, and I quote (roughly)
"Existing Internet links could run six thousand times as fast!".
Even New Scientist, normally a beacon of good science journalism, is really dumbing it down to the noise level.
The analogy they gave is most ironic -
... as my car sat, motionless, on the congested freeway ...
Seems to me that the overhead required to estimate your speed would make small downloads slower. Since 99% of our downloads (web pages, images, etc.) are relatively small, wouldn't this be worse??
For large downloads, however, it seems there could be some definite advantages.
However, when it comes to transferring data files, I'd personally be happy sticking with the slower old version of TCP. Who would like to have a file 600 gigs in size that potentially has errors in it, and cannot be trusted? It really has problems when your calculating mountains of data to estimate future stock values, or something else requiring massive amounts of accurate data. An error here or there may not matter, but then again, it may, and you can't tell if there are errors in it or not.
9 23 4&mode=thread
Now, this may help with streaming content, such as perhaps streaming on demand high quality video. This may help in the future with an on demand video system much like Pay-Per-View, but over the internet. Though, lets not forget that Snail Mail is still beating out the Internet.
http://slashdot.org/article.pl?sid=02/09/23/171
A friend of mine posted this article to a private mailing list yesterday. I had the following to say.
Come on, don't buy into the media's interpretation of things. I am not saying the research is bogus, just the article makes things sound different then things are. If a physical wire operates at 1.5MHz serial, there is no way to transmit more than 1.5Mbit/s over that link. Obviously anyone who attempts to sell you software that does so is pulling your chain. That said, Fast TCP is about four times faster than Linux's TCP stack on a 1Gbit/s link (how many people do you know that have one of those). That is because most existing TCP stacks do not perform well at high speeds over long distances, because the demand hasn't been there yet.
Now with that said, different TCP's do make a big difference because of TCP's built in congestion control. The basic idea of congestion control is that a computer shouldn't send data faster than the routers along the path can handle. There are formal proofs that also show that TCP's congestion control guarentees that all TCP connections (using the same implementation and equal round trip times) are given equal priority. The basic idea is to pull back the transmit rate when a packet is dropped.
If all the Internet used UDP, which doesn't have congestion control, our routers would be more overwhelmed than they currently are and everything would slow down.
One can improve one's performance by pulling back less or by taking more than one's fair share.
The statement that all the Internet uses the TCP developed in the 1970's (called TCP Tahoe) is very much false. Most of the Internet runs TCP Reno (1990) now days which includes Jacobson's modification of TCP Tahoe (1988) and added fast recovery and fast retransmit. A number of improvements to that have been discussed in TCP/IP Illustrated by Stevens (1994) and in RFC 2581.
A newer version called TCP Vegas (1995) has been proposed which speeds up performance dramatically and provides a more consistent transmission rate. TCP Vegas hasn't really caught on yet. Fast TCP is a competitor to TCP Vegas.
If you are still reading at this point I will give a more thorough explanation. Whenever a recipient receives a packet it sends an ACK to the sender with the packet it is expecting next.
When TCP starts out it starts in a mode called "slow start." It starts off with a window size of 1, meaning it sends one packet and then waits for the ACK. In slow start mode it increases the window size by one each time an ACK is received until a packet is dropped. So next round it transmits 2, then 4, then 8, until it hits the threshold (Stevens[1994] suggests 65Kbytes). Once it hits the threshold it enters "collision avoidance" mode and increases the window size by one each round (each ACK by 1/window size).
If a sender transmits packets and does not receive any ACKs by the time the timeout for the first packet occurs it pulls back all the way to a windows size of one and drops the threshold in half in both TCP Tahoe and Reno. After going back to one, they start "slow start" again growing the window size exponentially. The difference lies when one packet is dropped but the next few packets are received in a timely manner. In this situation the receiver will send back what is called a triple-ACK all stating it is expecting the missing packet. When a triple-ACK is received TCP Tahoe behaves the same as a timeout (window size to one, threshold in half, slow start), while TCP Reno cuts both the window and the threshold in half, then enters collision avoidance mode.
TCP Vegas works totally differently. It measures round trip time and keeps track of the difference between expected and actual round trip time. If the difference is more than a certain amount it adjusts the window size in the appropriate direction. This method even detects router congestion before the routers start dropping packets in some cases. TCP Vegas also retransmits at a double-ACK
This message is encrypted with Quad ROT-13 to protect the author's copyright under the DMCA.
that all the other streams are going to back-off during collisions. TCP normally backs off when a packet is lost because it assumes that the router buffer is full. It seems like if *everyone* used FastTCP, and no one backed off, it would simply DOS routers all over the Internet. Basically, a router buffer fills up, but everyone keeps sending at an astronomical rate, leading to 100% packet loss. Eventually it would back down far enough that the buffer would open up again, but only briefly, and the process would repeat. TCP was designed to be fair, not a bully algorithm. This thing goes in and violates that fairness principle, by being a bully while everyone else plays nicely. TCP works for the benefit of everyone, by having a universal understanding that unless everyone plays nicely, everyone loses. Hopefully this won't catch on...
Sure it does. I'm thinking around 10000 seconds.
-- this is not a
This really devastates the credibility of CalTech
as an institution. It seems clear that some group
at CalTech pumped this to the media, to the point
where a categorically deceptive series of fluff
pieces entered the news stream.
Compare this to the "cold fusion" debacle in '89:
Pons and Fleischman reported valid, and eventually
reproducible results without hype, but the media
pumped it with speculation. Pons and Fleischman,
excellent, highly competent and productive stars in
their field, were essentially tainted by no fault
of their own, and run out of town on a rail.
It's galling.
-I like my women like I like my tea: green-
Been there, done that.
the challenge is to make this 'better tcp' to co-exist with other versions of tcp currently deployed on Internet, like Tahoe and NewReno.The key point is the notion of 'fairness'. The way TCP (at least the currently deployed versions) are designed is to cooperate (in a sense) as to provide the equal share of bandwidth to competing flows. That's one of the reasons why it is so difficult to introduce TCP-like protocol with radically new design. I believe one of the reasons TCP Vegas (which has a good potential) does not get deployed on a wide scale is that it is not entirely clear how the mix of Tahoe, NewReno and Vegas would perform in terms of fair share.
... I wouldn't call it a 'breakthrough' though.
however from what i've been reading on caltech site, it appears that one of the usages of this protocol would be to download very large file on dedicated pipe (like movie on demand). From movie server to the user through private connection. This makes sense. You can streamline lots of things. OPtimize protocol for a fat pipe. Whatever