Slashdot Mirror
Fixing the Unfairness of TCP Congestion Control
duncan99 writes "George Ou, Technical Director of ZDNet, has an analysis today of an engineering proposal to address congestion issues on the internet. It's an interesting read, with sections such as "The politicization of an engineering problem" and "Dismantling the dogma of flow rate fairness". Short and long term answers are suggested, along with some examples of what incentives it might take to get this to work. Whichever side of the neutrality debate you're on, this is worth consideration."
The author of this analysis seems to have missed the fact that each TCP session in a P2P application is communicating with a different network user and may not be experiencing the same congestion as other sessions. In most cases (those where the congestion is not on the first hop) It doesn't make sense to throttle all connections when one is effected by congestion.
Absolute power corrupts absolutely. indymedia
A New Way to Look at Networking is a Google Tech Talk. It's about an hour long, but there's a lot of very good and fascinating historical information, which sets the groundwork for this guy's proposal. Van Jacobson was around at the early days when TCP/IP were being invented. He's proposing a new protocol layered on top of TCP/IP that can turn the Internet into a true broadcast medium -- one which is even more proof against censorship than the current one!
There is a debate? I thought it was more like a few monied interests decided "there is a recognized correct way to handle this issue; I just make more profit and have more control if I ignore that." That's not the same thing as a debate.
For what it's worth, Net Neutrality IS a political fight, p2p is not the cause, but just the straw that broke the camels back. Fixing the fairness problem of tcp flow control will not make Net Neutrality go away. Nice fix though, too bad getting people to adopt it would be a nightmare. Where was this suggestion 15 years ago?
Weighted TCP is a great idea. That doesn't change the fact that net neutrality is a good thing, or that traffic shaping is a better fix for network congestion than forging RST packets.
The author of this article is clearly exploiting the novelty of a technological idea to promote his slightly related political agenda, and that's deplorable.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
How do they get off saying THAT line? By their own admission, the P2P apps simply TRANSFER MORE DATA, it's not an issue of congestion control if one guy uploads 500KB/Day and another uses 500MB in the same period. Hell you could up-prioritize all HTTP uploads and most P2P uploaders wouldn't care or notice. The issue with Comcast is that instead of prioritizing HTTP, they're dropping Bittorrent. There's a big difference in taking a small speed bump to non-critical protocols for the benefit of the network and having those protocols disabled entirely.
Between the data transfer amount, and THAT line, this reads like a puff piece. It's not as if the P2P applications were the first to come up with multiple connections either, I'm pretty sure download managers like "GetRight" did it.
How do you kill that which has no life?
because the Internet is a group of autonomous systems (hence the identifier "ASN") agreeing to exchange traffic for as long as it makes sense for them to do so. There is no central Internet "authority" (despite what Dept of Commerce, NetSol, Congress and others keep trying to assert) - your rules end at the edge of my network. Your choices are to exchange traffic with me, or not, but you don't get to tell me how to run things (modulo the usual civil and criminal codes regarding the four horsemen of the information apocalypse). Advocates of network neutrality legislation would clearly like to have some add'l regulatory framework in place to provide a stronger encouragement to "good behavior" (as set out in the RFCs and in the early history of internetworks and the hacking community) than the market provides in some cases. It remains to be seen whether the benefits provided by that framework would at all outweigh the inevitable loopholes, unintended consequences and general heavy-handed cluelessness that's been the hallmark of any federal technology legislation.
Those networks that show consistently boorish behavior to other networks eventually find themselves isolated or losing customers (e.g. Cogent, although somehow they still manage to retain some business - doubtless due to the fact that they're the cheapest transit you can scrape by with in most cases, although anybody who relies on them is inevitably sorry).
The Internet will be a democracy when every part of the network is funded, built and maintained by the general public. Until then, it's a loose confederation of independent networks who cooperate when it makes sense to do so. Fortunately, the exceedingly wise folks that wrote the protocols that made these networks possible did so in a manner that encourages interconnectivity (and most large networks tend to be operated by folks with similar clue - when they're not, see the previous paragraph).
Not everything can be (or even should be) a democracy. Now get off my lawn, you damn hippies.
illum oportet crescere me autem minui
This article is well fine and good, but it fails to recognize that there are two types of packet discrimination being kicked around. Protocol filtering/prioritization, and Source/Destination filtering/prioritization. There are certainly good and bad ways of doing the former, and some of the bad ways are really bad (for a "for instance", see Comcast). However, the basic concept, that network bandwidth is finite over a set period of time, and that finite resource must be utilized efficiently, is not one most geek types will disagree with you on. Smart treatment of packets is something few object to.
What brings a large objection is the Source/Destination filtering. I'm going to downgrade service on packets coming from Google video, because they haven't paid our "upgrade" tax, and coincidentally, we're invested in Youtube. This is an entirely different issue, and is not an engineering issue at all. It is entirely political. We know is technologically possible. People block sites today, for parental censorship reasons, among others. It would be little challenge, as an engineer, to set to a VERY low priority an arbitrary set of packets from a source address. This however violates what the internet is for, and in the end, if my ISP is doing this, am I really connected to the "Internet", or just a dispersed corporate net, similar to the old AOL.
This is, and will be, a political question, and if it goes the wrong way, will destroy what is fundamentally interesting about the net. The ability to, with one connection, talk to anyone else, anywhere in the world, no different then if they were in the next town over.
Where are we going, and why am I in this handbasket?
I need coffee before I'll really understand this, but here's a first attempt:
Ok, first of all, that isn't about TCP congestion avoidance, at least not directly. (Doesn't Skype use UDP, anyway?)
But the problem here, I think, is that George Ou is assuming that Comcast is deliberately targeting P2P, and moreover, that they have no choice but to deliberately target P2P. I'd assumed that they were simply targeting any application that uses too many TCP connections -- thus, BitTorrent can still work, and still be reasonably fast, by decreasing the number of connections. Make too many connections and Comcast starts dropping them, no matter what the protocol.
Well, where is our money going each month?
But more importantly, the trick here is that no ISP guarantees any peak bitrate, or average bitrate. Very few ISPs even tell you how much bandwidth you are allowed to use, but most reserve the right to terminate service for any reason, including "too much" bandwidth. Comcast tells you how much bandwidth you may use, in units of songs, videos, etc, rather than bits or bytes -- kind of insulting, isn't it?
I would be much happier if ISPs were required to disclose, straight up, how much total bandwidth they have (up and down), distributed among how many customers. Or, at least, full disclosure of how much bandwidth I may use as a customer. Otherwise, I'm going to continue to assume that I may use as much bandwidth as I want.
Yes, it is a tricky engineering problem. But it's also a political one, as any engineering solution would have to benefit everyone, and not single out individual users or protocols. Most solutions I've seen that accomplish this also create a central point of control, which makes them suspect -- who gets to choose what protocols and usage patterns are "fair"?
Alright. But as I understand it, this is a client-side implementation. How do you enforce it?
Nope. What I wonder is why a P2P user might want to do that, rather than install a different TCP implementation -- one which tags every single TCP connection as "weighted".
Oh, and who gets to tag a connection -- the source, or the destination? Remember that on average, some half of th
Don't thank God, thank a doctor!
The whole article is disingenuous. What he is describing are not "loopholes" being cynically exploited by those evil, and soon to be illegal, P2P applications. They are the intended behavior of the protocol stack. Are P2P applications gaming the system by opening multiple streams between each pair of endpoints? No. While we could have a legitimate debate on what is fair behavior, he poisons the whole issue by using it as a vehicle for his anti-P2P agenda.
Mea navis aericumbens anguillis abundat
There have been plenty of lessons, Japan most recently, that upping the avaible capacity simply ups the amount of bulk-data P2P, without helping the other flows nearly as much.
Test your net with Netalyzr
But it's ridiculous that when I'm spending 30 seconds downloading CNN.com during a high-demand period, some asshat is using twenty times my bandwidth downloading some file that could just as easily be sent at any time of day.
1. Could that possibly be to the processor demand on the CNN servers at peak times?
2. Does not certain companies like Blizzard force P2P patches onto their customers?
3. Is your 30 second video file just as important as a technician using torrents to download a Linux Distro to put on a server used for business they need up and running ASAP?
4. And lastly... Someone using a torrent shouldn't soak up an ISPs entire bandwidth... Unless someone at CNN is using the web server to host torrents but thats nothing you or your ISP can control.
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
This then lets the user put each packet into one of three buckets:
- Low delay.
- High throughput.
- Don't care.
A packet with the low delay flag set would go into a high priority queue, but only a limited fraction of the customer's allotted bandwidth could be used for these. Any more would either be dropped or have the low delay flag cleared. These would be suitable for VoIP use and would have low latency and (ideally) low jitter.Those with the high throughput flag set would have no guaranteed minimum latency. They would go into a low-priority, very wide queue. If you're doing a big download, you set this flag - you'll get the whole file faster, but you might get a lot of jitter and latency.
Perhaps the high reliability flag could be used to indicate which packets should not have the flags cleared if the quota was exceeded (and other packets without the high reliability flag set were available for demotion).
Of course, Microsoft's TCP/IP stack sets all of these flags by default, so most traffic would simply be placed into the default queue until they fixed it.
I am TheRaven on Soylent News
I'd think that a simple fix to Jacobson's algorithm could help a lot. Instead of resetting the transmission rate on just one connection on a dropped packet, reset all of them. This would have no effect on anyone using a single stream, and would eliminate problems with the source of the congestion is nearby. Variations on this theme would included resetting all connections for a single process or process group, which would throttle my P2P without affecting my browser. This alone would be more than enough incentive for me to adopt the patch: instead of having to schedule different bandwidth limits during the day, I could just let everything flow at full speed 24x7. And by putting the patch into the kernel, you'd have less to worry about individual applications and/or users deciding to adopt it.
Nothing for 6-digit uids?
Here is the glaring flaw in his proposal:
So he wants everyone, especially P2P users, to voluntarily update their TCP stack? Why in the world would a P2P user do that, when they know that (a) older stacks would be supported forever, and (b) a new stack would slow down their transfer rates? He does mention this problem:
There are two issues with this solution:
It's nice that he wants to find a solution to the P2P bandwidth problem, but this is not it.
In the UK bandwidth out of BTs ADSL network costs ~ £70/Mbit/Month wholesale. Consumer DSL costs ~ £20/month.
You've got three options,
#1 Have an uncapped uncontended link for the £20/month you pay - you'll get about 250kbps.
#2 Have a fast link with a low bandwidth cap - think 8Mbits with a 50GB cap and chargeable bandwidth after that at around ~ 50p-£1/GB
#3 Deal with an ISP who's selling bandwidth they don't have and expect them to try as hard as possible to make #1 look like #2 with no overage charges.
If you want a reliable fast internet connection you want to go with a company that advertises #2. If you can't afford #2, you can spend your time working against the techs at ISP #3, but expect them to go our of their way to make your life shit until you take your service elsewhere because you cost them money.
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. (Einstein)
If you're using Linux, which TCP Congestion algorithm are you using? Reno isn't very fair; if a single connection is congested beyond the first hop, you'll slow down the rest of your connections when the window slides to smaller units. Have you tried Bicubic, Veno, or any of the other 9 or 10 congestion algorithms?
You can change them on the fly by echoing the name into your procfs, IIRC. Also, if you have the stomache for it, and two connections to the internet, you can load balance and/or stripe them using Linux advanced Routing & Traffic Control (mostly the ip(1) command). Very cool stuff if you want to route around a slow node or two (check out the multiple path stuff) at your ISP(s).
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
WARNING ~! Core dump follows.
It occurred to me this morning that driving on public roadways and surfing the public networks were identical experiences for the vast majority of people. That experience being; "mine, mine, ALL MINE!....hahahaha!" AKA "screw you...it's all about me!"
Now, I have the joy of managing a global network with links to 150 countries AND a 30 mile one way commute. So, I get to see, in microcosm, how the average citizen behaves in both instances.
From a network perspective, charge by usage...period. Fairness only works in FAIRy tales.
We do very good traffic shaping and management across the world. QoS policies are very well designed and work. The end user locations do get charged an allocation for their network costs. So, you'd think the WAN would run nicely and fairly. After all, if the POS systems are impacted, we don't make money and that affects everyone, right?
Hardly. While we block obvious stuff like YouTube and Myspace, we have "smart" users who abuse the privilege. So, when we get a ticket about "poor network performance", we go back to a point before the problem report and look at the flows. 99 out of 100 times, it's one or more users hogging the pipe with their own agenda. Now, the branch manager gets a detailed report of what the employees were doing and how much it cost them. Of course, porn surfers get fired immediately. Abusers of the privilege just get to wonder what year they'll see a merit increase, if at all.
So, even with very robust network tuning and traffic shaping, the "me, me" crowd will still screw everybody else...and be proud that they did. Die a miserable death in prison you ignorant pieces of shit.
Likewise the flaming assholes I compete with on the concrete and asphalt network link between home and office every day. This morning, some idiot in a subcompact stuck herself about 2 feet from my rear bumper...at 70mph. If I apply ANY braking for ANY reason, this woman will collide with me. So, I tapped the brakes so she'd back off. She backed off with the upraised hand that seemed to be "yeah, I know I was in the wrong and being unsafe" She then performed 9 lane changes, all without signaling once, and managed to gain....wait for it.... a whole SEVEN SECONDS of time over 10 miles of driving.
I see it every day. People driving with little regard for anyone else and raising the costs for the rest of us. On the network, or on the highway, same deal. And they feel like they did something worthwhile. I've talked to many users at work and the VAST majority are not only unapologetic, but actually SMUG. Many times, I'll get the "I do this at home, so it must be okay at work". To which I say, "well you cannot beat your wife and molest your kids at the office, now can you?"
My tolerance of, and faith in, my fellow man to "do the right thing" are at zero.
A technical solution (to TCP Congestion Control, etc.) is teaching the pig to sing; horrible results. Charge the thieving, spamming bastards through the nose AND constrain their traffic. That'll get better results than any pollyanna crap about "fair".
I am my own gestalt.
Every year or so somebody else proposes to "fix TCP". It never happens. Why?
1) TCP works well.
2) TCP is in a lot of code and cannot easily be replaced
3) If you need something else, alternatives are there, e.g. UDP, RTSP and others.
Especially 3) is the killer. Applications that need it are already using other protocols. This article, like so many similar ones before it, is just hot air by somebody that did either not do their homework or want attention without deserving it.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I know that only a handful of these have been implemented for Linux or *BSD, even fewer for both. Instead of Summer of Code producing stuff nobody ever sees, how about one of the big players invest in students producing some of these meaty chunks of code?
Schemes for reducing packet loss by active queue management: REM, RED, GRED, WRED, SRED, Adaptive RED, RED-Worcester, Self-Configuring RED, Exponential RED, BLUE, SFB, GREEN, BLACK, PURPLE, WHITE
Schemes for adjusting packet queues: CBQ, Enhanced CBQ, HFSC, CSFQ, CSPFQ, PrFQ, Local Flow Separation,
Schemes for scheduling traffic: Gaussian, Least Attained Service, ABE, CSDPS
Schemes for shaping traffic flows: DSS, Constant bit Rate
Schemes for bandwidth allocation: RSVP, YESSIR, M-YESSIR
Schemes for active flow control: ECN, Mark Front ECN
Schemes for managing queues: Adaptive Virtual Queue, PRIO
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Personally, I think they should move to a supply and demand based system, where you are charged per packet or per megabyte, and per-unit prices rise during periods of peak demand.
There are a few power companies who announce 24 hours in advance how much they're going to charge per Kwh in any given hour, and their customers can time their usage to take advantage of slack space, since the prices are based on demand.
If we do the same thing with internet service *both in and out*, a real bandwidth hog is going to wind up paying a shitload of money for his service, especially if he tries to tie up the net during peak hours. However, a casual user won't get burned.
And, coincidentally, it would solve the nasty "RIAA's making me block bittorrent" by comcast, or at least make it much harder for them to hide behind such a statement.
One particular property shared by almost ALL multimedia is that it is friggin HUGE. A movie can easily run into multiple gigabytes.
So start charging per-unit fees, and you'll put a massive leash on filesharing of media files. Suddenly, all those shared movies are costing major beaucoup to get, and they start going away.
Since when is it forced? I keep it turned off and download patches from filefront.
As the one who devised much of this congestion control strategy (see my RFC 896 and RFC 970, years before Van Jacobson), I suppose should say something.
The way this was supposed to work is that TCP needs to be well-behaved because it is to the advantage of the endpoint to be well-behaved. What makes this work is enforcement of fair queuing at the first router entering the network. Fair queuing balances load by IP address, not TCP connection, and "weighted fair queueing" allows quality of service controls to be imposed at the entry router.
The problem now is that the DOCSIS approach to cable modems, at least in its earlier versions, doesn't impose fair queuing at entry to the network from the subscriber side. So congestion occurs further upstream, near the cable headend, in the "middle" of the network. By then, there are too many flows through the routers to do anything intelligent on a per-flow basis.
We still don't know how to handle congestion in the middle of an IP network. The best we have is "random early drop", but that's a hack. The whole Internet depends on stopping congestion near the entry point of the network. The cable guys didn't get this right in the upstream direction, and now they're hurting.
I'd argue for weighted fair queuing and QOS in the cable box. Try hard to push the congestion control out to the first router. DOCSIS 3 is a step in the right direction, if configured properly. But DOCSIS 3 is a huge collection of tuning parameters in search of a policy, and is likely to be grossly misconfigured.
The trick with quality of service is to offer either high-bandwidth or low latency service, but not both together. If you request low latency, your packets go into a per-IP queue with a high priority but a low queue length. Send too much and you lose packets. Send a little, and they get through fast. If you request high bandwidth, you get lower priority but a longer queue length, so you can fill up the pipe and wait for an ACK.
But I have no idea what to do about streaming video on demand, other than heavy buffering. Multicast works for broadcast (non-on-demand) video, but other than for sports fans who want to watch in real time, it doesn't help much. (I've previously suggested, sort of as a joke, that when a stream runs low on buffered content, the player should insert a pre-stored commercial while allowing the stream to catch up. Someone will probably try that.)
John Nagle
Back in 1994 to 1997 I was in many debates on just this subject.
We were buying T1 and T3 for use with video streaming and the ISP where getting upset that we were using 90% of the capacity they sold us. Apparently they specked out their cost based on office use doing web surfing. And based their models on older Telco traffic models where they needed 100 lines of outbound bandwidth for every 10000+ phone lines based on supporting 95% of the peak throughput.
But we concluded if you are selling us 1.5Mbps I dam well better be able to use 1.5Mbps, don't blame me when I use what was sold to me.
Well I see this as the same problem. If Comcast or Verizon sells me internet at at data rate, then I expect to be able to use all of it. There is nothing unfair about me using what I was sold. If they don't like it then they need to change their contractual agreements with me and change their hardware to match!
Same goes with the internal infrastructure, backbones and exchange point. If you can't support it don't sell it! Don't attack the P2P users, they are using what they PAID FOR and what was sold to them!!! If they are not getting it, they should file a class action suit.
No more then if you local cable company decided that 4 hr of TV was your limit and they would start to degrade your reception if you watched more, though this wasn't in the contract you signed up for.
On the other side, P2P should be given the means to hug the edges of the network. By this I mean communication between 2 cable modem or DSL users running off the same upstream routers (less hops) should be preferable and more efficient, not clogging up the more costly backbones. Currently P2P doesn't take any of that into consideration. Maybe ISP's could consider some technical solution to that rather then trying to deny customers the very access they are contractually bound to provide...
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso
- many P2P protocols use UDP (skype, anyone?)
- proposes a client/application side change in behaviour, while they're whining about a failure of the protocol
- enforcement proposal ignores how the interweb works, there's NO difference (at the IP level) between a user multi-streaming a TCP download of a single file, and a user opening multiple tcp connections to a webserver to simultaneously download *all* the crappy bits-n-shits that make up a web page (ie parallel non-pipelined http requests) rather than one-at-a-time
I could go on for hours.- (think my car needs a grease-and-oilchange, so I'll go walk the dog - proposed solution bears no relationship to the problem)
- yet the first would be argued as an "unfair use* while the second is perfectly normal and acceptable behaviour
- If 'the protocol' is broken, then 'the protocol' needs to change, recommending an app-level change only opens up further opportunity for abuse
- recommending an *external* enforcement will never work, that costs time and money and who is gonna pay me to implement it?
- P2P users are initiating "sessions" (assuming they're still using TCP) to different endpoints, so you don't have a beautiful and neat bundle of parallel-tubes as described in the metaphor
Of course, the entire article starts out from a baseless assumption (that users should get 'fair' access to the interweb).- after all, if the app developers were genuinely interested in playing nicely in the sandbox, they would already
- TCP congestion control "works" (ie as engineered) because it's inherent in the protocol implementation, does not require "enforcement" by the ISP
- ie most of your assumptions about "how this works" are wrong.
Anyone read their ISP Ts&Cs ? Ever?
IP is a *best effort* protocol.... we will punt your packet upstream and hope it gets there - have a nice day.
There is *no* guarantee of *anything*.
Now, as far as anything approaching a "solution" to the supposed "problem".
What about all the P2P developers marking their "data transmission" packets (whatever the protocol) with the lowest-of-the-low QoS markings.
--> "if you need to manage congestion, I am exceedingly eligible for shaping"
That would work nicely.
In fact, if YouTube (and friends) did the same, it would actually *encourage* ISPs to enable proper QoS processing throughout their entire networks.
If applications (and protocols) inherently played nicely in the sandbox, your ISP would bend-over-backwards to guarantee a near-perfect service. (mainly because it'd thusly be near-trivial to do)
And yes I realise this raises the spectre of "Net Neutrality" - but seriously folks how is that argument any different than "because of the terorists" or "think of the children"?
ISPs Applying QoS to traffic in order to guarantee the quality is not inherently bad. The *bad* ness comes about because they will (yes, I said WILL, not MIGHT or COULD) use said QoS practices to push their own services/enforce their own policies (we hate P2P/ignore client-QoS-markings, etc , etc, etc).
All those people who're frothing-at-the-mouth because QoS is BAD need a RABIES shot.
In an ideal world, we'd never need QoS. QoS is a congestion management mechanism. If you have no congestion, then you don't need to apply QoS techniques.
But until the day when we all have quantum-entangled communications processors with near-infinite effective bandwidth we're going to need QoS, somewhere.
Visit CryptoGnome in his home.
John,
Fairness is not the problem. Fairness is the wedge-issue that CATV-ISPs are trying to use to justify their behavior.
I personally like the rudimentary aspects of the weighted fair queuing proposal -- so let's image that we had it. Would Comcast still have a problem with too many upload bytes from too many homes competing for the upload path back to the CMTS? Yes.
The real problem is that CATV-ISPs are at their upper limits and FIOS is currently superiour. Most CATV nets are DOCSIS 1.1, neighborhoods of 400-500 homes sharing 9-10 Mbps back to the CMTS. Meanwhile, they have to compete with FIOS advertising 15/15, 15-20/5, 20/20, or etc. Mbps. Due to TCP and higher-layer protocol RETURN overhead, CATV ISPs can only offer download speeds if they can reserve 5% in the upload pipe -- so their download is limited by their upload. For example, downloading 8 Mbps from NNTP requires around 200 Kbps. Their upload was 256 Kbps. What happened next: to increase their download speed offering, they pushed out configuration files allowing uploads of 384 Kbps! Cost $0.00 -- no new equipment, no neighborhood splits -- just "let's pretend that we have the bandwidth." After all, customers don't care about upload speed, they just want to download.
Heh.
What he's describing is not just Freenet. There's also a little bit of Bittorrent in there as well, and some more ingredients. Freenet is about distribution to prevent censorship. What he's proposing is to decentralize to turn the *entire Internet* into a huge broadcast cache. This will also have the effect of making censorship difficult, but that's only a byproduct.