Slashdot Mirror
Bufferbloat — the Submarine That's Sinking the Net
gottabeme writes "Jim Gettys, one of the original X Window System developers and editor of the HTTP/1.1 spec, has posted a series of articles on his blog detailing his research on the relatively unknown problem of bufferbloat. Bufferbloat is affecting the entire Internet, slowly worsening as RAM prices drop and buffers enlarge, and is causing latency and jitter to spike, especially for home broadband users. Unchecked, this problem may continue to deteriorate the usability of interactive applications like VOIP and gaming, and being so widespread, will take years of engineering and education efforts to resolve. Being like 'frogs in heating water,' few people are even aware of the problem. Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"
http://en.wikipedia.org/wiki/X_Window_System
Latency is bad? Bigger buffers = more latency?
I'm so glad the term has been defined so that I know what the hell we're talking about here. Oh wait, no it hasn't.
Okay, then I'll RTFA. Oh wait, two screens worth of text later and it still hasn't.
I'd like to change the topic now to the submarine that's sinking the English language: jargonbloat.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
Isn't this something that people that setup QoS on their home router's know about? Isn't this what the program Wondershaper fixes when run on your home linux router. I think we have known about this for years.
He's Jim Gettys, not Getty.
Jim Getty, one of the original X Window System developers and editor of the HTTP/1.1 spec
I'd murder four people just to have TTY in my name. Five if I could capitalize them, and postfix with a number. I'd name my son Dev.
You'd get a business card with something like Dev GeTTY1, Armadillo Avenue 64, Seattle, Washington
8 of 13 people found this answer helpful. Did you?
If we all switched to ATM (Asynchronous transfer mode), would this issue be fixed, regardless of the size of RAM available at the endpoints? Yes, yes I realize that this would be utterly impractical; my question is theoretical.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
Just start RTFAing: "In my last post I outlined the general bufferbloat problem."
Follow the link:
"Each of these initial experiments were been designed to clearly demonstrate a now very common problem: excessive buffering in a network path. I call this bufferbloat
"Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"
> Yes, with IPv6 QoS.
I read TFA and I'm not seeing the problem. He can't duplicate this issue unless he maxes out his connection and then his latency goes to hell. No shit Sherlock, that's what happens when your pipe is full and the packets have to wait in the queue to be transmitted. Am I stupid or could he avoid this issue entirely by using QoS and/or rate-limiting his connection to some amount <100% of it's maximum throughout? I have QoS at the office that keeps our connection from pegging (it's limited to around 75% on the download and 90% on upload) and have never once encountered an issue with latency or jitter. At home I only throttle the upload (to 90% of maximum) and have successfully ran VPNs, bittorrent uploads and VoIP calls all at the same time without any headaches.
Really, what's the problem here?
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
What exactly are they buffering? Whole pages? Whole sites? Just packets? What?
And back in the late 90s, the buffering was supposed to speed things up and a few companies were IPO's or purchased by big companies (for billions of dollars) that did this.
I knew it wasn't my age thats making me worse at online games, its all the jittering and latency caused by bufferbloat!
It's what lets me watch youtube even over slow Dialup or Cellphone connections (buffer the video halfway, then press play).
Buffer also prevents one of the main flaws with broadcast TV. When there's interference the video skips a second or two. "Hi this is NBC News. And today in ...... met for its first session." What? Who? Where? That never happens with internet video (which just pauses for a second, buffers the video, and then resumes).
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/
Red Leader Standing By!
If we all switched to ATM, I'd find you in your sleep and murder you.
TBH though.. MPLS sorta tries to split the difference in the 'good' ways. Especially if you drink the Kool Aid (tm) and have the budget to spend on rolling it out.
----- The internet has given everyone the ability to have their voice heard equally as loud.. even if they shouldn't be
I thought the times of people being paid by the word were over. There's so much to read these days. Couldn't Getty state the situation in a few sentences before taking us on a journey through lightening, ICU and tracings?
I agree. I was having the same issues on a much smaller connection until I set up QOS. Now, I never have issues with any of my stuff.
"If you make people think they're thinking, they'll love you; But if you really make them think, they'll hate you." - DM
RAM is cheap.
High speed uplink is not cheap.
Peering agreements are manipulative, expensive, and sometimes extortionate.
So...
The poorly designed, poorly peered, under allocated back haul links can't handle the traffic that routers want to push through them -- but since RAM is cheap, operators just add RAM to the buffers so that when those back-haul lines slow down for a second the packets can get pushed through.
And we're blaming the buffer for the problem?
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
it's a long nightmarish tale of deception & death, followed by a 'brief', 'surprise ending', whereas the tail discards the dog as excess baggage. plus, it was a fairly good movie, in addition to being an ongoing real life horror flick/sit-com. evile never sleeps. are you still calling this 'weather'?
The internet will die? :(
From TFA: "Any time you have a large data transfer to or from a well provisioned server, you will have trouble."
Well there's your problem: you bought a home cable connection and actually expected to USE it. What are you, retarded?
I think he's making up a new word. It's oversubscription. There is SLA. And it's not caused by memory prices. I can similarly claim that it's caused by terrorism as people spend more time on the Internet saturating channels because they're scared to interact in real life.
Increase connection prices and decreases caps.
Have you no sense of humor?, he MUST be kidding...
PS: if he isn't, I'll second you
Easy, name him Devone = Dev1
There's much more to it than that - the connection gets maxed out too easily, or it maxes out way below where it should, the reason being that too much is buffered. Too much buffering = lots of latency = TCP/IP latency and bandwidth calculations go out the window and you can't get the transfer speed you ought to.
Or so I understand it.
Several issues:
1. People who aren't networking engineers don't know about QoS, or don't know/want to know how to configure it.
2. QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
3. How do you determine the maximum throughput? It's not necessarily the official line's speed. The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is. You're proposing a regression to having to tell the system by hand.
4. QoS is most effective on stuff you're sending, but in the current consumer-oriented internet most people download a lot more than they upload.
Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?
And here I was wondering when 3G networks would become usable for interactive apps.
if only
RTFA. I've been following his blog for the last few weeks as he's written about this. The problem is much more serious than most realize. In fact, I'd say most people completely misunderstand the issue.
What a load of bull. The possibility of a large buffer doesn't mean at all that everyone will overbuffer blindly even if it becomes a real problem.
Is this IT for retards?
The problem is that maxing your connection from one site is causing everything else you do on your connection to be delayed / dropped as well, because it ends up queued behind anything that got buffered mid-transit from the first site. With a smaller buffer the large transfer would start to drop packets and back off sooner, allowing packets from other sources to "hop the queue".
What Jim is saying is that TCP flows try to train themselves to the dynamically available bandwidth, such that there is a minimum of dropped packets, retransmits, etc.
But in order for TCP to do this, packets must be dropped _fast_.
When TCP was designed, the assumptions about the price of ram (and thus, the amount of onboard memory in all the devices in the virtual circuit) were different -- namely, buffers were going to be smaller, fill up faster, and send "i'm full" messages backwards much sooner.
What the experimentation has determined is that many network devices will buffer 1 megabyte or MORE of traffic before finally dropping something and telling the tcp originator to slow down. And yet with a 1 meg buffer and a rate of 1 megabyte per second.. it will take 1 second simply to drain the buffer.
The pervasive presence of large buffers all along the tcp vc, and the non-speified or tail-drop drop behavior of these large queues means that tcp's ability to rate limit is effectively nullified, and in situations where the link is highly utilized, many degenerate behaviors occur, such that the overall link has extremely high latency, and that bulk traffic will cause interesting traffic to be randomly dropped.
Personally, I used pf/altQ on openBSD to try and manage this somewhat.. but its a dicey business.
My opinions are my own, and do not necessarily represent those of my employer.
Really, what's the problem here?
You really don't see the problem? How can you be so naive. Maybe you're new to this. All signs show to the fact there is a problem.
Of course the problem is not obvious. The article itself says it'll completely surprise us. They know we won't believe it at first. But that's why we must believe it, or else it's Armageddon.
Would you risk an Armageddon, because of your inability to understand and see?
And that's, in short, why we must attack Iraq.
Wait, what were we talking about :P?
Yes, he could. What about all the non-technical users?
Rgasuya aata! : I have been coding Perl and cannot tell where my fingers are now!
As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?
The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small
It's not about *you* buffering - it's about the machine in the middle buffering. When that machine buffers instead of drops, your TCP connection will never become aware that it has to play nice and lower its transmission window.
Religion is what happens when nature strikes and groupthink goes wrong.
I wouldn't say you're stupid, but you're not understanding the problem right. ISPs configure high buffers on low bandwidth links so that the total throughput is higher, at the expense of latency, since, as you say, packets have to wait in queue until they go out when you max out the connection.
This is NOT the right way of doing things, buffers should be smaller, if you max out your connection, your packets will drop, this will cause either TCP (at the protocol level) or the application if there is no support at the protocol level (UDP for example) to back out and lower the transmission rate, WHILE keeping your latency at normal levels.
You'll still have some packet loss, sure, but the overall experience should be better if applications act nice.
And no, QOS is not the solution to this, QOS should work end to end on a connection and that's simply not possible for internet users. What you mean is shaping, or policing, at the interface level, but if you're doing that, you're just avoiding the filling up the buffers, so that you achieve what I described earlier.
Not quite the whole story. In peak hours when your 20 Mbit ADSL drops to 2 Mbit speeds it's simply because they oversold the line that much... The problem isn't buffering but trying to use 1000% of the bandwidth, that is never going to work smoothly. Buffers don't really change it because you have a choice for either large buffers + higher latencies or small buffers + more dropped packets, your throughput will suffer about the same...
In my experience most software will handle some higher latencies just fine, but too many dropped packets will fuck almost any protocol up (most are not so sturdy when you remove a few percentage of the packets). So the choice for larger buffers is a valid one as long as the connection is this saturated.
If you'd have used fewer lines in yer joke, yed have gotten some mod points for Funnie.
but in the current consumer-oriented internet most people download a lot more than they upload.
Because the current consumer infrastructure forces it onto you. I would happily seed my torrents all year long, except I only have 1/12th the uploading bandwidth as I have for downloading. Since I need some of it for other things, uploading becomes impractical.
It's easy to blame the consumer, but there's a certain model imposed on him from the start.
Seven puppies were harmed during the making of this post.
If you put a frog in a pot of water and slowly raise the temperature it will try to jump out before the water reaches a temperature that is fatal to the frog.
Best Slashdot Co
It's kinda like the Fed printing another 600 billion and refusing to raise interest rates, while at the same time saying everything is fine and the economy is improving.
Seven puppies were harmed during the making of this post.
Rate limiting is exactly what Getty suggests as a work around in a later blog post. However, most people would not be able to do that and bufferbloat can also occur elsewhere along the connection path.
The problem, moron, is that TCP is supposed to have congestion control to prevent that. The buffers are keeping congestion control from working as intended. You shouldn't *need* QoS unless you're *prioritizing* specific traffic.
Mod parent up. Half way down the comments, and this is the first post that actually explains why 'bufferbloat' is something I should care about.
I am TheRaven on Soylent News
| No shit Sherlock that's what happens when your
| pipe is full and the packets have to wait in
| the queue to be transmitted.
That is what happens with bufferbloat. Actually it is even worse, because the buffers will confuse TCP to think the pipe is thicker than it is and fill it even more.
The point is: The internet is not supposed to have latency only because there is some bottle-neck.
The latency is supposed to be around the same all the time. Things are supposed to send data at a rate that it can reach the other side. Both is lost if you have too much buffers. Mostly because TCP measures the width of your pipe by looking at lost packages. And if some smartass network router thinks it should not drop packets if there is not enough network for everyone, then the net grinds to halt periodically, killing latency, killing bandwith, causing sparks in dropped packages, causes timeouts, and so on and so forth.
...chime in please. It seems like the solution to this is potentially all user-side, and controllable? Adjust the buffers in your devices if you can, or perhaps find a way to reduce the TCP buffer in your modern operating system?
I won't discount the buffer problem because I just don't know. But the single biggest contributor to "latency" when I visit a webpage is connectivity to ad sourcers. I can click to go to a site and stare at a blank screen while my status bar flickers with: "Transferring data from ads.that.you.could.care.less.about.com."
"that's what happens when your pipe is full and the packets have to wait in the queue to be transmitted."
And his point is that said queue is so excessively long, it's screwing up TCP's congestion avoidance. Those queues mean delay. Serious delay.
So naturally, I instantly get modded down.
I've always wondered why RealVideo streams kept buffering... ad neauseam. Now we know why.
QoS does generally not work beyond the first hop. Your provider most likely will drop any QoS data. Some providers wil try to make their own QoS systems (e.g. to show a low ping). However if the lantency has a great variance due to all kind of buffers any algoritm will get the bandthwidt wrong.
QoS based on network types will get it wrong. For pure browsing /downloading it is relatively simple, but for VPN Encrypted skype udp traffic, game data it will never be optimal.
And as the blogger wrote, there is not a simple solution, because the end user has a "dad the internet is slow today" mentality. Couple that with a "reinstall your windows" helldesk and the solution becomes VERY HARD.
Another perspective: http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/
If we all switched to ATM (Asynchronous transfer mode), would this issue be fixed, regardless of the size of RAM available at the endpoints?
A good buffering algorithm SHOULD be implemented as a fifo queue, preferably a circular buffer. Nothing should stop transmitting in order to fill up the buffer, the front of the queue should be emptied as fast as possible and the back of the queue should be filled as fast as possible. Stopping to fill a buffer before transmitting is just asinine.
Buffers should work to even out transfer speeds and reduce jitter. What will happen is that a buffer just before the slowest link will start to fill as more data comes in than goes out. Once the buffer is full the packets will begin to be dropped and the TCP window will narrow, in the meantime the buffer should still continue to serve the slower link data. Ideally the transmission rate will balance out such that the buffer stays partially filled the entire time, in the real world there will always be some jitter but a properly-implemented buffer shouldn't make it much worse and should have the potential to greatly improve it.
Sapere aude!
Finally, someone understands why we went into Iraq and explained it so everyone else could understand. Nice job.
The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes.
That doesn't happen, not even close. Maybe if your machine only had a single TCP/IP session to the outside world or every connection was rerouted and merged into a single connection upstream somehow somewhere or if someone invented a new transport method that utilized a single TCP connection that only went from your machine to your ISP somewhere but that is not how it works now. Yes a system like that would suck but who the hell uses a buffing system like that?
The issue is that many IP stacks do not handle ECN (Explicit Congestion Notification) and only know when the link is saturated by packet loss. Huge buffers hide the problem. A solution is to use ECN, that's what it's for http://en.wikipedia.org/wiki/Explicit_Congestion_Notification
Can someone explain why broadband routers don't use the TOS bit to do some basic Qos.
Add this to a well behaved P2P (or Rsync in the case of TFA) client that sets this field, and you get rid of most issues.
This is an excellent explanation of what issues are happening here. I can clearly see that this is an issue, and the problem is something that over time will impact everybody.
The problem is really focused on trying to deal with differences in bandwidth between computers... always a problem but in this case trying to match up slow connections with fast connections is particularly difficult. Since memory is cheap, a 1 GB buffer certainly can be found in some devices now and perhaps much more. I don't see this example as being really too far off the mark in the near future.... which is the point being raised and why buffer bloat is such a big deal.
More to the point, some of the complaints that triggered the "quality of service" debate are rooted in this problem. As mentioned in the original article triggering this whole slashdot thread, setting up "quality of service" priorities only creates multiple buffer queues.... it doesn't solve the problem of the monster queue to begin with. That is why the author of the blog post suggests that the debate over network neutrality is not based upon the real problem that is facing network engineering and why it is a political solution in search of a problem.
It takes awhile to "grok" this problem, but once you do it becomes obvious why this is such a huge deal.
The problem is not with /his/ connection, it is with /some link somewhere/ in the internet which has maxed out. He has no idea where it is, nor probably does anyone else including that link's owner. Of course, you expect that to happen, and many internet protocols, particularly including TCP, have sophisticated and well tested mechanisms for throttling back when they detect congestion in their path and slowing their transmissions in a manner which actually works very well at sharing the available bandwidth, But there is a critical phrase in that last sentence: *when they detect congestion*. Large amounts of buffering in the system delays the point at which the sender realizes that packets have been dropped and throttles back. Then a large number of packets get dropped, so the sender scales back a long way to dead slow, and slowly speeds up until it detects packet loss. But, because of buffering, by the time packet loss is detected, it is already sending far faster than the remote choke-point can handle.
By analogy, think of driving a car whose brakes have a one second delay before they go on, and then go to full emergency stop. You would progress in a very stuttery and uncomfortable manner, and on average not very fast. That, he says, is what is increasingly happening to out internet connections. Huge buffers fool senders into thinking that there is lots of bandwidth, and then discard scads of data which has to be retransmitted.
He implies that QoS all the way would alleviate this problem - but that seems incompatible with current pressures for Net Neutrality. At the moment, the end consumer has no access to QoS.
Consciousness is an illusion caused by an excess of self consciousness.
People who aren't networking engineers don't know about QoS, or don't know/want to know how to configure it.
*shrug*, not my problem :)
QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
The issue is always going to be there. Pegged connection == FIFO queuing, absent some sort of QoS scheme.
How do you determine the maximum throughput? It's not necessarily the official line's speed.
If you aren't getting the line speed you paid for then you need to find another ISP.
The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is
And it does, even with QoS. All you do with QoS is force the buffering to happen on equipment that you control rather than equipment your ISP controls. In this manner you can ensure that time sensitive packets (interactive VPNs, VoIP, etc.) don't sit in the queue behind someone's Windows Update download.
QoS is most effective on stuff you're sending
It's not really all that difficult to shape downstream traffic. All you need is a router between your internet connection and LAN clients. I've done this for years at my office using the QoS functionality of the Linux kernel. We are located out in the middle of nowhere with T-1s as our only means of connectivity. Sharing a 3.0mbit/s connection with 60+ employees without QoS is virtually impossible if you need to run interactive protocols.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
That makes no sense. It doesn't matter how fat their pipe is because your computer needs to receive and ack those TCP packets. They can't just dump the file and close the connection.
No, The problem is diagnosed quite clearly, the TCP protocol is designed to deal with full buffers, ie no more transmit space left, and cope with that BUFFERBLOAT means that the transmit side KEEPS accepting packets, which may then time out or be dropped and THIS implies unstable latency and congestion spikes. Routers should allocate a small number of buffers for each connection and return an appropriate transmit window size.
Please READ the blog carefully, it all makes much sense even though the result is counter-intuitive.
To solve this problem, you first have to realize why these buffers exist: They are put there to avoid dropping packets. The buffer is only used when the network connection is so congested that without the buffer the arriving packets would have to be dropped because they can't be forwarded. With protocols which implement congestion control, the buffer size should not grow uncontrollably, because, for example with TCP, packets which are in the buffer do not get ACK'ed and TCP throttles the flow to match the available bandwidth. Recently there was a story about Google, Microsoft and others ignoring the TCP spec and sending more initial packets without waiting for ACKs. If you were wondering then why that's a bad idea, this is it. Of course the other option is to simply not use large buffer sizes on network equipment and avoid congestion by provisioning more network capacity.
It is also about "you" buffering, as even something so innocent as your home network router box can be at least a source of some of the problems. The problem is all across the whole network and the fact that buffering because memory is cheap isn't the solution to all networking problems.
If your home router (or the router connecting your LAN to the greater internet) has a huge buffer, it is possible for the outgoing packets to also start backing up where the latency of the network drops considerably. In the original article, the author goes into details about how a simple switch of routers at his home is something that triggered some huge problems with latency on the order of 10 seconds or more. When you are talking network connections, that is a huge latency issue and it completely kills connections with something like Skype that needs latency on the order of a fraction of a second to work effectively.
The funny thing is where he goes and explains how Bittorrent files exposed this problem early on, why it became an issue for cable modems in particular and not so much for DSL users. That analysis is brilliant, furthermore explaining why the "solution" by Comcast to filter out torrent uploads was a silly move in the first place and only kicked the problem down to a future resolution.
That is why the author of the blog post suggests that the debate over network neutrality is not based upon the real problem that is facing network engineering and why it is a political solution in search of a problem.
It takes awhile to "grok" this problem, but once you do it becomes obvious why this is such a huge deal.
At least the real solution is obvious. Invest in network infrastructure and eliminate those queues entirely.
Give me Classic Slashdot or give me death!
OK, not on the (intentionally ridiculous) scale used in the example, but people are doing something very similar to what you describe, even though they "can't do that". http://slashdot.org/article.pl?sid=10/11/26/1729218
# cat
Damn, my RAM is full of llamas.
That site won't have a 1GB SNDBUF, so that won't happen. ('man 7 socket', then search for SO_SNDBUF).
It's amazing how many people, Gettys apparently unfortunately included (note: of course I did not rtfa before I posted this), don't know how TCP/IP really works.
There is no 'bufferbloat because RAM is getting cheaper'. What he is seeing is what happens when you want to saturate your link. It's sort of an Heisenberg of communication, if you want low latency and low (or no) packet loss on a connection, then the bandwidth can't exceed the available bandwidth, and for any instance that it does, you get either a buffered or a dropped packet.
--- Hindsight is 20/20, but walking backwards is not the answer.
Slashdot is in a very murderous mood today. This guy would kill to have a name like Gettys, you will kill if people roll out ATM.
I'm a good cook. I'm a fantastic eater. - Steven Brust
| Stopping to fill a buffer before transmitting is just asinine.
Huh? Starting to fill a buffer before your output bandwith is saturated is something you can do to get throughput, but it kills latency. And latency is often
more important. (If I have a device with 1 MB buffer, should I get the first webpage only after I sent out enough requests that my requests are 1 MB in size?)
| What will happen is that a buffer just before the slowest link will start to fill as more data comes in than goes out.
Once you have a link that has more data than it can send, then there is no more sense in buffering. Just send out what you can, drop what you
cannot send. If you cannot send all you get then there is no sense in buffering anything, you only cause latency, but gain nothing.
The problem is that if you are not the slowest link but only some connections will somtimes shortly produce more data then you can sound out at
that moment, then a buffer is good to send it later. That means if you test in an artifical net where all pipes are bigger than needed, buffers will not harm but only have good effects. But once one connection is too slow for the data wanting to get through, the buffer causes problems. And the way TCP is made will made this little problem a very big problem.
| Once the buffer is full the packets will begin to be dropped and the TCP window will narrow,
But only packets getting in now will be dropped. Everything still in the queue will still be sent. The other side will receive it and send acknowledgements.
So it will be a very long time before TCP will even notice it should no longer increase traffic.
In the meantime you cannot serve the whole connection, you have absurd latency because you still let everything rot in the buffer.
So even dropping the whole contents of the buffer at this time is likely to produce better results in this case. (Unless your buffer is small enough
that TCP's algorithm is still able to calculate proper throughput, which it no longer if it only realized it's being too fast almost a second later).
The problem is that rate-limiting should happen automatically through TCP congestion avoidance protocol and it doesn't at least not for a sane value of latency.
the tcp connection which is maxing out the bandwidth should notice that it is doing that and throttle back down a bit until it it just shy of saturating the available bandwidth in order to keep latency to a minimum or at least to a sane value for the link. This would also allow for low latency for the // ping and therefore for low latency web browsing
2. QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
3. How do you determine the maximum throughput? It's not necessarily the official line's speed. The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is. You're proposing a regression to having to tell the system by hand.
4. QoS is most effective on stuff you're sending, but in the current consumer-oriented internet most people download a lot more than they upload.
While the Internet in-theory is beautiful, our modern implementation really is a series of layered hacks. And the solution to Bufferbloat is going to be another hack. You're crazy if you think that the solution to the Bufferbloat 'problem' is going to be some fundamental redesign of the TCP protocol (how would you force 10 people to use it?), or the total re-architecture of millions of consumer devices to remove buffering. You're also crazy if you think the ISPs and backbone providers are going to stand by while this thing kills the Internet.
So the question is: which hack will it be? The GP poster already identified one that works well enough --- using QoS to control flows. Your final objection about content providers stressing connections is the real one. But there's some probably a good hack to deal with it --- or more likely a series of hacks, some at the content providers themselves (e.g., Netflix), some in the backbone, and some at your ISP. It won't be elegant, but it will keep this problem from ever becoming anything more than a few cranky blog posts.
The issue is that, when packets are dropped and the TCP window narrows, it's pretty much a catastrophe from throughput perspective. In particular, the recipient won't know about a dropped packet until the latency delay, and the sender won't know about it until (minimum) twice that latency. And so the sender will keep on sending at its high rate, causing it and everyone else's packets to drop, and it'll take many round-trip-times before flow is re-established.
If there were smaller buffers in the internet routers, then receiver and sender would know much quicker about congestion, and they'd be able to get the correct transmission rates with only a tiny amount of disruption.
Why, what's wrong with ATM? (besides the cost of the switch) It has all the predictability and latency guarantees that people *wish* IP had.
Higher Logics: where programming meets science.
The point he's making is that in the days when TCP was developed, RAM was expensive, so we didn't have big queues. As a result, you didn't need to rate limit any connections to get low latency and high throughput.
Remember that no matter how big the queue is, if you saturate your link for long enough, you get a degree of packet loss. If, for example, the queue is 5 seconds long at maximum speed, and you saturate the link for 6 seconds, you lose some packets. TCP exploits this by using the packet drop as an indication that a link in the path between two hosts is saturated. When buffers are appropriately sized, and queue length appropriately managed by something active like RED, this is not a problem; latency stays low because the queue isn't that big compared to the link throughput, and packet drops genuinely indicate congestion on the path.
Bufferbloat creates the symptom you're working around by QoS and rate-limiting. Because the queue is immense, there's no packet drop until the latency is insane. Because there's no packet drop, the TCP stacks sending data your way don't believe that your link is congested, so don't slow down. Your rate-limiting and QoS fixes this by letting the packets come in via your Internet connection, then dropping them if the actual data rate exceeds a level below that which your line is capable of; Gettys is asking why you need to do that. Why can't your ISP shrink their queue, and drop packets when your line is just saturated, rather than building up an immense queue, which you promptly go and waste by throttling to less than the speed you've paid for?
I appear to have a blog. Odd.
Personally, I think the problem is in the assumption of fairness; as in first-come first-served.
If network infrastructure instead handled packets in LIFO order, than a large majority of packets will be delivered in a timely manner with a small percentage grossly delayed... or dropped when the LIFO buffers fill up which would eject the oldest packet from the buffer.
Protocols such as TCP would see network congestion effects more rapidly and if packets were dropped, TCP has ways of getting the lost, or grossly delayed, packet retransmitted.
No sig. Move along - nothing to see here.
let me summarize the problem that is being observed: On a given interface, if you have more buffer memory than is needed as packet buffer on the transmit side, it can induce latency. As an example, consider a 1Mb/s link. If you want to have a peak of .1s latency added by buffering at high load, then you want 1Mb*.1=12,500 bytes of buffer. If you have 1MB of buffer, then you have 8 seconds of buffer, therefore triggering the "buffer bloat" issue. Part of the problem is that buffer size would be set based on the top speed a piece of hardware could drive, i.e. if you want a 1Gb/s interface to be able to buffer .1s, then you use it at 100Mb/s, then it has 1s worth of buffer. In most home deployments where you have a router that may have a 1Gbps upstream, maybe 4 100Mb/s physical connections, and a 54Mbps wireless router, you probably have a shared buffer for all the interfaces. The result of this is that when using the 54Mb/s wireless, you can easily have the buffer over saturated, while the buffer size may be just right for the 100Mb/s interfaces.
What is the solution to this? Realistically, the alternative is to drop packets that have resided in the buffer longer than a configured amount of time, which causes it's own performance issues. Net result: TCP would slowdown for a period of time, but would speed up again resulting in a sawtooth behavior. This would result in periodic issues with other protocols as well, i.e. VOIP would have dropped packets every time TCP ramps up again, etc.
Solution: Don't download porn when you are trying to do VOIP calls.
Right. Because most people really are content providers at heart.
Latency is bad? Bigger buffers = more latency?
Buffers increasing latency is not exactly a new phenomena. Its been observed and taken into design considerations for quite some time. For example back-in-the-day serial chips essentially had a buffer of one byte. The CPU fed data one byte at a time as the buffer became available and latency was pretty low since data was immediately transmitted. As more capable serial chips became available larger buffers were introduced. A newer chip may have a larger buffer but it may also not transmit data as soon as it has a single byte. It was common to have two programmable thresholds to begin a data transmission, (1) when a certain amount of data has accumulated in the buffer or (2) when a certain amount of time has elapsed. So if a "packet" to transmit was small enough it may sit in the buffer until (2), hence more latency with larger buffers. Software that cared generally began to issue flush commands to cause anything in the buffer to be sent immediately.
Network cards and/or the operating system may try to similarly accumulate data before transmitting a packet.
oh, Iraq.... whew.... for a moment I was afraid you were referring to the CO2 emissions....
Sig ?
I discovered this series of blog posts about 2 months ago, when he accidentally published one of his blog posts prematurely. I started reading it and followed the links and saw that this was a like a sleuth tale-if I had started reading this with his very first blog on the topic I would have had no idea where he was going with this. Now as to why this contribution by Jim Gettys does the world a great service:
Hats of to Jim Gettys. Thanks for your service.
QOS does not always work that well.
If i have a router that manages QOS connected to cable modem.
The router can manage QOS in the routers buffer but it has no controll over the buffer in the cable modem.
It cant tell the router hey i have som importent packages here please put them early in the buffer.
I managed to get semi working by forcing my router to have the exact same buffer size as the cable modem so the modem will fill it's buffer every time
but it's not perfect since you cant send one buffer and wait until the cable modem buffer is empty to send the next.
Sure if you have QOS directly in the cable modem this is not a big issue, except that you cant controll the reciving end.
This is slowly but surely turning into a Orson Welles movie.
Mark my word: Rosebuds.
... too much deep packet inspection going on?
Have gnu, will travel.
Education. Talk to many network engineers these days and ask them about queueing theory. After the deafening silence, you may wish to rethink who is architecting your network.
"To those who are overly cautious, everything is impossible. "
If we don't have big enough buffers on our network devices, where will the network-based sentient self-forming AI store itself?
If the only way you can accept an assertion is by faith, then you are conceding that it can't be taken on its own merits
Except that never happens, since you donot request 1 GB. Instead you use a TCP link which uses packets, and which are only requested in chunks that your line can handle (using the window size which will scale according to available bandwidth on YOUR side).
In other words, your software will never request sizes that big in one go, and so you will not be saturating your line with it.
If however you do manage to saturate your line, then that is still a problem on your side, and can be solved on your side by simply limiting the rate just below maximum speed.
A: is bogus, there are many sites that can saturate my crappy line without blinking -- they do ALL rate limit though (well actually the receiving site is doing that by virtue of not requesting more data than the TCP window size allows).
B: is irrelevant -- the problem is on the side of the requester; see above, TCP links donot send data willy-nilly, they actually wait until your software acknowledges that earlier data was received before sending more, if they didn't then every site faster than your connection would be DDOS-ing you.
No, that was a pretty accurate description of network router buffers.
Most don't do anything fancy with their buffer queues because that would use too much router CPU. So it's first in first out. You might get RED (random early drop) if you're lucky.
To make multiple TCP sessions work as you think they work would require the ISP router sending packets to you to be using SFQ (stochastic fair queuing) or something similar. By hashing the packets by "flow" and placing them into separate queues the router can make the TCP sessions share fairly.
But does the ISP do this? I doubt it.
I wish I had mod points to you for such a clearly written post. Hats off to good writers everywhere. May I someday become one. :)
As the internet is used more and more, there are more and more layers, each with their own buffers, all getting bigger, so feedback on buffer capacity gets more and more stale and meaningless as it percolates back and forth according to protocol, leading to unanticipated side effects.
Infuriate left and right
As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?
This is an example of how NOT to implement a buffering system. Ideally you should have multiple buffers per connection and have TCP streams go to an available buffer. You then service the buffers according to some scheduling algorithm, reading from one buffer for a pre-determined amount and then switching to another. In this way you can have a large download buffered but if a small stream opens it isn't too burdened by the buffer of the large stream.
You just need a couple of buffers per connection in order to provide a balance between reducing latency and avoiding starvation.
This is a huge topic and a lot of these problems have already been worked out. Some compromises have been made and sometimes there are unintended consequences or bad assumptions but overall it does work pretty well.
Sapere aude!
Oh Lord, you'd better hope the fella I used to work with doesn't see this statement. He used to rail at length against ATM. Then again, it might be worth pointing out that his company with their toroidal mesh Terabit Switch Router (TSR) is no longer in business. Of course we'd usually substitute 's' for the 'h'. Engineering humour.
Actually with the HTTP spec you just request "the whole file", and the webserver sends it all to you. You don't request individual chunks of the file. Other protocols (e.g. bittorrent) work on file chunks and so can actually do this.
At the TCP level, the modern spec allows the sender to send more packets without acks on the previous packets, which massively increases throughput on high-latency connections. You can only scale the rate you receive the transfer at by dropping/delaying the packets, and that only works if the sender cares. Dropped packets just get resent, reducing your bandwidth even more, so routers normally delay (buffer) the packets, which (guess what) trashes your latency.
Actually, I think it is ONLY about you buffering. Other boxes buffering is completely irrelevant because YOUR tcp/ip stack should have adjusted the window size accordingly, no matter what is in between you and the destination.
I mean, if I have a super fast link, but some box in between has a super slow link with a 1 GB buffer, how does that affect me? It only affects me if I actually use a huge window size, but any properly written TCP/IP stack would never use such large window sizes as the receiver would have to acknowledge initial burst of data first...
The other way around? My link begin super slow? Same thing.
Now, if you simply make sure that the buffers on your router are always empty and therefore putting them out of play, then latency will be close to minimum. How to do that? Simple, don't request more data than your link can download, and donot send more data than your link can upload. There's software that can do that.
It is indeed an example on how not to implement buffering, and older buffered routers used to be even worse: they'd use one buffer per physical network socket, which may be serving multiple endpoints, allowing one person to trash everyone's latency.
For true fairness, you need one buffer per TCP connection.
Except the point is that the buffers screw up the TCP algorithms for detecting bandwidth and really make a mess of latency, regardless of what traffic is on the line and who's sharing it.
ATM: 53 byte frames should be big enough for everybody.
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
He explained why it works, not why you should care. The information you seek is in the summary, oddly enough.
In Canada references to the Bank of Canada in news stories have lately been abbreviated to BOC.
That's because unlike "Federal Reserve" and "Federal Express", "Bank of Canada" doesn't have a snappy, pronounceable contraction (Fed and FedEx respectively).
When I read "BOC to raise interest rates" I always wonder why the Blue Oyster Cult is doing that.
No, that'd be "BÖC to raise interest rates". BÖC was probably the first rock band to incorporate a gratuitous diaeresis in its name. The root problem here is that BOC's dis am bigger than yours.
As a service writer (as in SOAP, REST, Win32 system Service, unix daemon etc) I can say the trivial case - waiting for an entire file before processing is far more conceptually simple than writing a true streaming service. I see it all the time. Wait for the file to be done before processing it. Of course, for small files this makes sense. However when working on ever-more-common larger files this doesn't. Most calculations on input data can be done before the next packet arrives. The most trivial is if you're doing a file copy. A more complex example is if you are doing movie transcoding. Anyway, as it works out having a byte stream-oriented design allows you to process the data while you wait. This is seemingly for free. Consider a file-copy from internet to local storage. You can receive your TCP packet at line speed, then write it out on a remote volume far faster. If you do this, you won't have to wait X+Y time, where X is net transfer time (slow) and Y is local transfer time (fast). You will only need to wait X time. Yes, if there is an error you have to abort the local transfer and that takes slightly more intelligent error handling.
Case and point: I used a video transcoding service. I had to wait for 3 times X+Y+Z which are upload, transcode and download. Since my effective upload speed was a few dozen KBps, the transcoder CPU could have transcoded in real time, and sent me a byte stream back. meaning it would only take X time. Also note that if it is a multicore CPU, the transcoding can be done independent of the byte stream reading/writing.
In the case of errors - which are not common, it is ok to throw out those wasted CPU cycles, because the odds are they would have been idled anyway. On a server that handles many requests at the same time and isn't idle, errors (and cancellation) are rare enough that the time saved more than makes up for the few wasted transactions.
There are graphics libraries that support streaming pixel transforms like graphicsmagick ( http://www.graphicsmagick.org/ ) and I am sure VLC/ffmpeg supports a streaming conversion as well. using streams rather than whole files is the way to go.
Of course, this requires a but more error handling (and checksums, which can be problematic http://tools.ietf.org/id/draft-bryan-metalinkhttp-01.html#checksums ) because the checksum needs to be in the HTTP header, which means it can't be sent unless you've already ran through the file once... They way I addressed that is on the HTTP upload, you checksum as you go (again a streaming operation) and store that in a database or md5sum file.
I often wonder how much better the net would perform if amateur programmers didn't wait to get the whole file.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
SFQ implies adding an extra buffer.... and this extra buffering will cause the death of the internet.
The solution is that all hops keep a low buffer, and use the same time (not size) the buffer. That will solve the bufferbloat, but it will not solve other problems.
The description of the parent comment is correct, but it is an exaggeration, and exaggeration of buffers (to minimize packet loss) is exactly the cause of the bufferbloat.
The problem is that the whole point of TCP is to 1. maximize your throughput while 2. avoid congestion. Without excessive buffering, TCP actually can provide both goals. Add to much buffer, you break TCP congestion avoidance and thus your connection. Rate limiting is not a solution, but a mere work-around. TCP should (and in earlier times, could) provide congestion avoidance on its own.
It does not matter how big your ISP's buffer is. Unless they actually make data WAIT in an EMPTY buffer. I don't think they would be that stupid though, in which case the solution always is....
Donot REQUEST more data than your link can download -- this can be done with throttling, and not acknowledging previously received data until you are ready for more.
Donot SEND more data than your link can upload -- simply limit the rate of outgoing data to be slightly below your Routers maximum send speed.
The result: the ISP's buffer will be running close to empty at all times. Your Router's buffer will also be running close to empty at all times (since it can always send/receive the data slightly faster than the rate limit that you set your software to).
Nice side benefit: Effectively the queue will be on your computer's side. Inserting something at the head of the queue would then skip whatever is already queued up (for sending). Prioritize your games/pings/interactive stuff and you won't even notice that big download in the background.
For uploads, only if you allow it by not paying attention to ACK's you get from the receiver side and simply keep sending data.
For downloads, only if you are so stupid to request more data than your link can handle.
TCP will handle both of these as part of the spec. The only way you can still fuck it up is by having many connections going at once. In that case, use some proper rate limiting software.
There is no 'bufferbloat because RAM is getting cheaper'. What he is seeing is what happens when you want to saturate your link. ... ...you get either a buffered or a dropped packet.
Yes, and if a link is saturated, there should be packet drops, which TCP senses, then automatically throttles back to reduce the required bandwidth and avoid saturation. But what is happening, is that these huge buffers are holding packets that would otherwise be dropped, and so TCP doesn't get the feedback it needs to detect saturation. So it continues transmitting at full speed, believing it has uncongested pipes, which in turn continues to fill the buffers, and so on.
Because of the buffers, most of these packets are eventually getting through, but maybe in seconds instead of tens or low hundreds of milliseconds. Thus you're getting huge latency.
Jitter is caused by the buffers eventually filling or TCP timing out (registering packet loss), dropping the rate for a little bit, the buffers draining, then TCP upping the rate again as the buffers refill, hiding the saturation, until they're full again. Rinse and repeat.
It's related to the "bloat" of buffering (due to the increasing affordability of RAM and the "more of a good thing must be better than a little of a good thing - QED" mindset) because, if the size of the buffer is kept below a certain point related to the pipe bandwidth and number of traffic streams, it tends to act just as a temporary "buffer" against spikes in the traffic (the intention of buffering), and can't cause the scenario above, having insufficient capacity to overload the bandwidth just from buffer contents alone. Above this threshold, the latency issues and back-and-forth thrashing noted above occurs. The bigger the buffers, the worse the effect.
And it's not just a "well, keep your traffic below x mbit if you're on ADSL2" issue, because it happens anywhere a high capacity pipe interfaces with a low capacity or otherwise congested (of any capacity) pipe. This might be your ISP's backbone which is getting hit by several thousand people downloading the latest WOW patch simultaneously, causing your 300kbps Skype call to go to hell through latency and jitter. If the ISP's equipment had smaller buffers, the servers would be throttling back as packet loss occurred. You'd probably still be losing packets, but they'd be detected and re-transmitted pretty quickly and you possibly wouldn't notice the latency or have jitter.
What he is seeing is what happens when you want to saturate your link.
So, no, what you get with appropriate buffers is your TCP connection moderating itself to the appropriate link capacity and availability, and latency remaining approximately the same (relative to what you're seeing in bufferbloat, but worse than an uncongested link, obviously).
With bufferbloat, your bandwidth appears to remain about the same, but your latency balloons massively and you get jitter effects as above.
Is my inability to understand and see the problem, or is it how educated stupid I am? I keep losing track.
This is been enlightening. I've suffered very similar problems at home, but instead of figuring out the problem I replaced the hardware... After reading TFA all fits perfectly, I had occasional "chokes" - sites would take ages to load, ping's wouldn't return from my Wi-Fi router, DNS queries took ages. All while downloading a big file or something. But what's significant the throughput would stay high. It was strange as hell - high thoughput (an ongoing large transfer [but not large enough to saturate connection]) but other things choke.
The parent in exaggerating, but wmem_max can be easy tuned up a lot maybe even to 1 GB.
TCP (or any protocol) cannot simply handle a large jitter due to extrermely large buffers.
If the network price is higher than ram prices, the mistake is very easy made to make the buffer bigger to reduce packet loss on a particular link.
After reading the explanation the real problem seems to be in the US 10Mbps is fast whereas in Korea 1000Mbps is fast. Korea will not have trouble with bufferbloat but the US will. I do not see the buffer as being the problem here.
How did you get that immense queue in the first place? Sounds to me the TCP Window size is set WAY too large, which would be a problem on the initiating side (ie, your own computer). A TCP download won't send more data than you ask it to give, and will wait for ACK's of the previous data before sending more.
If you are silly enough to tell it, YES, send me that 1 GB right now! And then it takes your line 3 minutes to handle it all, then who's exactly at fault?
Instead, you tell it: send me that 1 GB but do it in chunks of 32 kB, and don't send more until I acknowledge the previous chunk was received correctly -- which is a gross simplification of what TCP does.
I don't see how buffers come into play... not to mention that RAM sizes have scaled by several orders of magnitude since the beginning of the internet and now, and only now we start noticing it? Sure, it may not have been a problem before with tiny buffers, but it always could happen if you (or your software) is stupid enough to ask for more than it can handle. Rate limiting is the correct solution, even going as far as dropping your own outgoing packets if you are trying to oversaturate your link.
I'd murder four people just to have TTY in my name.
No need; just change your first name to Betty. This change may require SRS and HRT to be believable. (Only some faiths consider SRS to be murder, usually the same ones that ban condom use.)
How much bandwidth can I have, though? Take the link between my desktop and a Slashdot server; is the correct answer "1GBit/s, no more" (speed of my network card)? Is is "20MBit/s, no more" (speed of my current Internet connection)? Is it "0.5MBit/s, no more" (my fair share of this office's Internet connection)? In practice, you need the answer to change rapidly, depending on network conditions - maybe I can have the full 20MBit/s if no-one else is using the Internet, maybe I should slow down briefly while someone else handles their e-mail.
TCP doesn't slam the network; it starts off slowly (TCP slow start currently sends just two packets initially), and gradually ramps up as it finds that packets aren't dropped. When packet drop happens, it realises that it's pushing too hard, and drops back. If there's been no packet drop for a while, it goes back to trying to ramp up. RFC 5681 talks about the gory details. It's possible (bar idiots with firewalls that block it) to use ECN (explicit congestion notification) instead of packet drop to indicate congestion, but the presence of people who think that ECN-enabled packets should be dropped (regardless of whether congestion has happened) means that you can't implement ECN on the wider Internet.
This works well in practice, given sane buffers; it dynamically shares the link bandwidth, without overflowing it. Bufferbloat destroys this, because TCP no longer gets the feedback it expects until the latency is immense. As a result, instead of sending typically 20MBit/s (assuming I'm the only user of the connection), and occasionally trying 20.01MBit/s, my TCP stack tries 20.01MBit/s, finds it works (thanks to the queue), speeds up to 20.10MBit/s, and still no failure, until it's trying to send at (say) 25MBit/s over a 20MBit/s bottleneck. Then packet loss kicks in, and brings it back down to 20MBit/s, but now the link latency is 5 seconds, not 5 milliseconds.
I appear to have a blog. Odd.
echo 1 > /proc/sys/net/ipv4/tcp_low_latency
http://serverfault.com/questions/215674/latency-in-tcp-ip-over-ethernet-networks
Many people are championing Jim Gettys now, but he is being a quintessential nerd and pontificating like he just stepped out of the basement, discovered the sun, and now needs to communicate this fantastic news to humanity.
Those of us in the high performance computing and networking community have been observing and publishing about this problem for about 15 years, but nobody cares to listen. And I suspect we were being quintessential nerds when WE first started doing this too! Some Internet priests were dismissive of us when we first walked down the hallway and told them about our troubles... Look up "gridftp" as an example that has evolved for a long time, starting with expanding TCP window buffers and ending with many parallel TCP streams, each with large buffers. In the last decade, bittorrent came along to do much the same thing but in a more scatter/gather pattern instead of a point-to-point transfer. The many streams are more effective (and more damaging to the general network) because they increase the number of TCP state machines and change the statistical properties of congestion avoidance and congestion. You are able to get your application much closer to saturating the path bottleneck(s) because random packet drops may shut down one TCP stream window, but the other soldier on and compensate while it ramps back up, and vice versa.
The problem is that there are very real applications where people want/need to saturate "their" path to make large transfers. The high performance scientific crowd wants to be able to saturate their expensive, multi-gigabit links between regions and continents to transfer terabytes to petabytes of scientific data. The home user wants to saturate their expensive broadband to transfer movies and such. Expensive is a relative term, but the mindset is much the same. The Internet priests shrug it off as selfish users who should grow up and live with the meager performance of one TCP stream per user.
It is a fundamental emergent behavior of larger networks, not due simply to some inappropriate buffer setting but to the collision between the best-effort/no-QoS philosophy which requires over-provisioning, the economic incentives of over-subscription/reselling of network capacity, and the population of self-interested parties. It's yet another Inconvenient Truth, of which most parties are ignorant and some willfully so; yet all these parties need to be aware and thoughtful to avoid the emergent pathological behavior. In other words, it will not go away until the Internet is dramatically reshaped to inject some kind of QoS, whether through path negotiation, tiered service, or some hybrid.
People who talk about setting bandwidth limits in their transfers are really just trying to emulate a QoS manager on their own path, because ISPs do not give us the control we want. But in general, you need to shape traffic around the bottleneck and the entire point of the Internet philosophy is that you don't know the path a priori. In a perfect world, you'd be able to push outward (much like route publishing) your own QoS policies which should be applied to packets destined to you, so other routers could help enforce the QoS policy you are trying to achieve whenever they become congested with your traffic.
However, this does nothing for the case of routers congested with traffic from multiple parties; someone has to synthesize a policy that differentiates between your traffic and others' and nobody ever agrees on what a fair policy is for this case. Does it depend on how much capacity you bought? Bought from who? You don't purchase capacity from every router operator who bears your Internet traffic. Does it depend on how many IP addresses you have, i.e. each destination gets an equal share? That will suck for everyone behind NAT and be great for everyone with clusters for parallel transfer. Does it depend on each destination/port combination? Now we're back to the current problem with TCP...
If you aren't getting the line speed you paid for then you need to find another ISP.
In many places, that's the same as saying "If you aren't getting the line speed you paid for then you need to find another place to live that has a better ISP." What solution do you have to work around last-mile monopolies?
I read TFA and I'm not seeing the problem. He can't duplicate this issue unless he maxes out his connection
After reading both blog posts, I get that what the author is saying that the problem may not be in just your uplink to the ISP. It can be inside the ISP, or somewhere in the transit path. In other words, someone else can cause you problems by being too much of a hog.
Also, has anyone twigged that there is a marketing aspect to the problem? ISPs are being measured on file download speed, so they do everything they can to maximize download speed measurements. Even if what they do harms the network. So it isn't just people being stupid, it's also people being selfish.
Buffer bloat is only part of the problem. How many Web server operators want their pages to download as quickly as possible, so they turn off TCP slow-start so the stuff goes out SMASH instead of dribbing out as TCP was designed to do? "Oh, shit, I'm losing packets during high load, so I have to increase the number of outgoing packet buffers in my outgoing router so it can absorb the SMASHes and send the packets out at line speed." And the Web hosting company is wondering why their Skype phones work so badly, especially as they cut over to Skype over land lines for technical support. :)
Unfortunately, the setting for TCP slow-start tends to be server-wide, so not only do the small pages and graphics get the SMASH treatment, so do those monster files that are downloaded. So much for congestion control, even if there wasn't buffer-bloat to make it worse.
Sounds similar to a router that uses FIFO versus Fair Queuing. Most by default use FIFO which has this type of problem. However many higher end ones also support FQ or Weighted FQ (like QoS) which wouldn't have this issue.
The 1GB of data going across the ADSL link still has to be broken up into packets usually around 1500 bytes so your other connections (dns lookup, etc) would still get a turn and not have to wait until the entire 1GB of data was finished. It depends on how the scheduler is setup in the router. Even with FIFO based scheduling other packets will usually get sent forward.
The issue is that, when packets are dropped and the TCP window narrows, it's pretty much a catastrophe from throughput perspective. In particular, the recipient won't know about a dropped packet until the latency delay, and the sender won't know about it until (minimum) twice that latency.
I'm a computer science major, not a network engineer.
It's been a bit since I've closely studied TCP/IP but I believe what is proper is for the router to participate in the congestion control through notifications and randomly dropping packets early based on a computed probability that the buffer will eventually begin doing tail drops, among other techniques. A good router will try to prevent the buffer from ever getting too full or too empty in the first place through these methods. It shouldn't ever get to the point where there's a dropped packet that belongs at the end of a large buffer.
Of course I'm sure there are plenty of poorly-implemented flow control mechanisms out there that are wreaking all sorts of havoc. A huge buffer will certainly exacerbate a poor network congestion algorithm. Buffers are generally intended to reduce jitter and they can definitely introduce latency in a bad scenario such as this.
Sapere aude!
That site won't have a 1GB SNDBUF, so that won't happen. ('man 7 socket', then search for SO_SNDBUF).
It's amazing how many people, Gettys apparently unfortunately included (note: of course I did not rtfa before I posted this), don't know how TCP/IP really works.
There is no 'bufferbloat because RAM is getting cheaper'. What he is seeing is what happens when you want to saturate your link. It's sort of an Heisenberg of communication, if you want low latency and low (or no) packet loss on a connection, then the bandwidth can't exceed the available bandwidth, and for any instance that it does, you get either a buffered or a dropped packet.
Interesting how you mouth off after confessing to not having read the article. Are you an idiot or a troll?
But that's exactly the point, you can do that, and would be the same as the ISP configuring low (normal) sized buffers.
And yes, ISPs are stupid, or not, depending on the way you see it. The side effect of reducing buffer sizes is that you reduce throughput, so you will have to configure more bandwidth to the customer, so that the customer "sees" it's 3M, 7M, or whatever that he paid for. But believe me, ISPs do this, first hand experience.
...where N is inversely proportional to the rate at which your huge buffer is filling. This will provide some differential feedback and help stabilize your loop. This is just a hand-waving idea I pulled out of the air, of course, but it indicates what you can do if you learn control theory and apply it to the problem (you want queueing theory as well but I assume you already know that as a network engineer).
This would be easier if everyone would enable ECN.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
It must have been something you assimilated. . . .
Just did some reading on the TCP window size, I apparently had it confused with the congestion window size (which is estimated by the sender).
Again though, it only works if the sender sticks to it, and if you advertise a correct window size (which would be very difficult to do).
Looking past your sarcasm I find it hard to understand why you believe the internet is created by aliens, since obviously people are not "content providers".
Seven puppies were harmed during the making of this post.
As I understand it, the problem is:
- Under max load, your connection slows to much less than is possible.
- Restarting the transfer doesn't solve the problem.
- Testing with anything less than that max load doesn't reveal a problem. Indeed, testing just latency or bandwidth doesn't show a problem.
- In some cases, the connection 'lopes', I think, starting out at full speed and then getting throttled for NO GOOD REASON except that the buffers have defeated normal TCP bandwidth management.
- The cause can be anywhere along the system.
- Buffers sound good, but in practice are essentially defeating TCP throughput management features.
Why is this bad?
- As we hurdle headlong into a cloud-based environment, we will be sending more and larger transfers. This 'bufferbloat' will get worse, and we risk the hardware makers responding by including BIGGER buffers. Making the problem worse and worse.
- Other protocols such as VOIP, various video, etc. will be significantly impacted, and it will get worse no better unless the hardware makers understand this and both test and modify their designs as needed.
- With the large number of routers out there, this is a big $$$ investment overall. Adopting IPv6 may help by flushing out old routers, if the makers again understand and both test and modify their designs.
- And this seems to affect virtually every type of device, from home routers and smartphones/3-4G networks to 'big-iron' routers and switches. Collectively, this has become a serious problem.
So, in a nutshell, when you start up a download of your new favorite ISO, and the connection slows to a trickle, this might be caused just by 'bufferbloat'. When your video goes to hell halfway through the show, this might be caused just by 'bufferbloat'. When your Skype goes to hell regularly, this might be caused just by 'bufferbloat'.
I'm wondering how much this impacts my pet peeve, waiting for a page to load and seeing it is just the damned ads holding it up. And ads are getting bigger and heavier every day.
Gettys was dead-on when he mentioned that in the 'old days', a T-1 was pretty snappy. I had a T-1 at work for a long time, and it was great in the mid 90s. Even my 128k ISDN line was plenty enough for serious surfing, which back then wasn't quite as deanding as now, but even Flash sites were just fine. Now I have a 20MB cable connection and it just seems slow, even when I adjust for expectations.
Obviously, I blame this 'bufferbloat' for my inability to prevail in COD.
deleting the extra space after periods so i can stay relevant, yeah.
TCP contains some of the most incredible heuristic algorithms I've ever seen. Each algorithm, like Slow Start, RTT Estimation, SACK, etc. are relatively simple but together they work incredibly well at keeping data flowing across heterogeneous networks. They work so well that I've seen TCP overcome broken ethernet drivers and make them appear to work. Unfortunately, as someone who use to look at TCP traces for a living, I can tell you it can be really hard to work backwards from packet traces to figure out what is going on in the TCP/IP stack because there can be so much going on at the same time. This means that Wireshark in the hands of a weekend-hacker can easily lead to erroneous conclusions. If you follow this link and go to section 14.5 Random Early Detection (RED) you can see that the issue is already known and there are already solutions to mitigate the problem.
Relax and take a deep breath. Now you can move on to something more important......... like where you're going to spend your eternity
"Meaningless!, Meaningless!" says the Teacher. "Utterly meaningless!"
For TCP connections, dropped packets are actually a good thing, and they're part of how the protocol is supposed to work.
The way TCP is supposed to work is that the sender sends out a bunch of packets and waits to see if any are dropped. Then it sends a few more, and a few more. It keeps doing this until it detects packet loss and slows down and resends the dropped packets. The end result of this slowing down is supposed to be that eventually the line is full, but packets aren't sent any faster than the line can handle them.
The problem with large buffers in the middle is that they mess up the analysis of the line capacity. TCP starts sending packets much faster than the capacity of the line. When those packets get stuck in the buffer, it thinks they were dropped and resends them. The first buffered packets then make it to the receiver, and TCP thinks it can speed up again. Problem is, the new packets get stuck behind (or even get dropped due to) the duplicate packets which are now needlessly clogging up the buffer.
There may be ways to alter the TCP congestion analysis algorithms to better handle buffered connections, but the buffers also need to be sensible. If the packet has been in the buffer long enough that the TCP connection is going to think that it was lost, then it needs to be removed from the buffer.
NOTE: This is similar to the congestion/bandwidth problems that come up with high-bandwidth TCP-over-TCP connections like SSH port forwarding.
Fair queuing in the routers is the real solution to this.
You did set that 1GB packet size through all the network layers, didn't you? ;)
The TCP window has only grown that large because there's been no congestion signalled; thanks to slow start, my TCP window started out at just 2 packets, but grew and grew until TCP experienced congestion. Bufferbloat has meant that TCP has not experienced any congestion until the latency has reached insane values (as congestion is signalled in the Internet by dropping packets).
This is the root cause of the problem; we have lots of workarounds for it, but at heart, the issue is that the only working mechanism for indicating congestion on the Internet is packet drops (packets marked as ECN-capable are blocked by too many idiots with firewalls to be useful). TCP by design will increase the current in-use window size until such point as it experiences congestion, then it will scale back to fit within the link; because we've removed congestion notification in the name of zero packet loss (yet we still have packet loss on these "zero packet loss" links - go figure), we hit pain.
If you artificially constrain TCP so that it cannot fill a link (i.e. make the maximum window tiny compared to BDP - noting that on today's Internet, I experience BDPs from as little as 1 kilobyte, to as high as 10 megabytes), then, yes, you won't hit bufferbloat - you won't hit saturation, either. If you rate limit outside the application layer (and how exactly is Slashdot's web server meant to know what the bottleneck rate between Slashdot and my PC is, exactly?), you have to signal congestion back to TCP somehow. On the Internet, that's done by dropping packets instead of queueing them; but, thanks to bufferbloat, my link doesn't drop packets until the latency is very high. As a result, TCP doesn't scale back its send rate until it's too late; I can fix that locally, by rate limiting at my router to some fraction of my link speed, but then I have to drop the packets that exceed that fraction. Why shouldn't the ISP drop them instead, and thus let me use more of the link speed I'm paying for?
I appear to have a blog. Odd.
It's exactly what happens. A packet is a packet is a packet and there's one device sending stuff to your cable modem. If there's 200 packets waiting in the buffer then a new one gets put at the end and is only sent forward after those 200 are processed.
Obviously buffers aren't big enough to hold an hour's worth of packets (and servers don't send 1GB worth of packets before getting any acks) that's just to make the situation more obvious in an example.
May I introduce you to the HTTP/1.1 Range header:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
"HTTP retrieval requests using conditional or unconditional GET methods MAY request one or more sub-ranges of the entity, instead of the entire entity, using the Range request header, which applies to the entity returned as the result of the request..."
The router can manage QOS in the routers buffer but it has no controll over the buffer in the cable modem.
The solution to this is to rate-limit your downloads to 75%-80% of your total download speed at your router. If there is going to be any buffering you want to force it to happen at your equipment, rather than at the ISP's equipment.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
He could, however that doesn't deal with network congestion after that point. If you've got an ISP that is over sold, i.e., all of the ones in the US, then this is a real problem and I don't think that QoS is going to solve that. At least not the stuff that your end user gets to fiddle around with.
You understand! And, as you pointed out, output buffers are never sized in this manner precisely due to the latency. Hardware engineers typically know a little bit about queuing theory. Which make the article a non-issue.
No the example does not make sense at all. You guys are getting your layers of the OSI model confused. Your machine requests PACKETS of data, not files. I request some packets of data from point B. Point B sends them. I acknowledge them and request more packets. That 1 GB file or packets representing that file are NEVER going to be sitting on any router in limbo because no one requested now more than a few packets at once, your TCP/IP stack does not keep requesting more and more future packets without ever getting some of them back, it doesnt even have a method to keep asking for the "next" packet until it knows what was in the previous ones (to some extent) because it doesnt know what to ask for and the router sure as hell doesn know what you doing, it is just forwarding packets. The example with the 1 GB file is describing what a proxy server does, we are talking about TCP/IP communications at the packet level.
Right, then you solved the problem by ensuring the queue in question never gets filled. You added your own queues (more than one, prioritized), and you probably made those queues shorter. That works, but it's a shame that you had to throw away 25% of your download capacity and 10% of your upload capacity when it wouldn't be necessary if the equipment you were using had properly configured queues of its own. It's also unusual that you were able to do so: most people (including almost all home users) are not in a position to set up QoS on their download side. Imagine I called Comcast and asked them to set custom QoS settings on data they are sending to me. How do you think that conversation would go? And even for the upload side, most consumers don't have the equipment or knowledge to set up their own queueing.
There's no a priori reason to assign a particular percentage of bandwidth to upstream or downstream. The best approach you can come up with is to optimize for expected usage, namely, mostly downstream. There's nothing sinister about it, and as needs change (VoIP?) one would expect things to change.
Now, ideal would be if your DSL connection could determine that you need extra upstream bandwidth and temporarily reassign some of your downstream frequency bins to give it to you.
He's been researching it for months, and posting evidence as he finds it. This was a weird place in the saga to link to, but that wasn't Gettys' fault.
There's nothing inherently wrong with big in-transit buffers for TCP streams. The real question is not which packets get dropped; it's which packets get sent next. That's what "fair queuing" and some other quality of service algorithms are about. Unfortunately, most routers are basically FIFO devices with some packet drop algorithm. If the router is FIFO, dumb, and has big buffers, there's trouble.
Back in the 1980s, when I was working on this, I was applying fair queuing at choke points. My basic thinking was that the network should not drop packets for congestion unless a sender is badly behaved and isn't obeying the congestion avoidance rules. This is well-behaved, and will work well when bandwidth is a scarce resource. But for years, the Internet had more bandwidth than was needed, and so people stopped worrying about congestion. Now that everybody is trying to stream high-definition video, it's a big problem again.
The problem used to be that the CPU overhead for fair queuing was too high. Today we can afford enough transistors in ASICs and FPGAs to do queuing right, even in fast routers. That's already happened. The big players have already put the necessary hardware into their newer routers. Cisco supports weighted fair queuing in their current DOCSIS cable routers. So does Motorola. But it has to be set up and configured. Motorola has a very clear management level presentation on the need for fair queuing on their DOCSIS cable routers. That short piece of PowerPoint is a must read for anybody involved in managing a cable Internet system. Read the slides staring with "If RED is not good enough, what is?". A key point for managers: "There are no parameters to set". There are other parts of DOCSIS routers that have way too many tuning knobs. That's not true of fair queuing.
So, if your cable system is showing this problem, they probably have older routers, or misconfigured routers, or routers from some clueless vendor, or need a software upgrade. Cisco only supported this fully in DOCSIS routers starting in 2008. Earlier cable routers tended to be rather dumb. If you're in the industry, pass around that Motorola PowerPoint.
This has nothing to do with "buffer bloat". It's a queuing problem.
end-user are demanding more download speed, a bigger road connecting them to the internet-world.
backbone carrier, put in bigger roads, but NOT more lanes.
sure you got some OC-super-max connecting S.F. to N.Y but how to you fairly put all
the big end-user roads into that one pipe? surely there's a limit (time) to switch or let them on
the OC-super-max? not that the OC-super-max connection is congested.
-
the solution is to not have ONE OC-super-max from S.F. to N.Y. so to speak a serial connection,
but put in MORE OC-super minis, like a parallel cable. the switching problem is less a problem?
(parallel connection, think old skool printer-cable)
(switching: think many many cable strands go into ONE!)
Hallelujah.
Seriously, maybe your /. ID is too high ;-) You've grown up always using network connections which have this issue, and, since you're technical enough, you've learned workarounds like the one you (and Jim) suggest of rate limiting your connection. All your throttling does is allow your local network to start dropping packets so TCP will work and do its own throttling.
The problem is that all this is way over non slashdot-poster's heads. People should be able plug in their routers and it have it work without knowing their line's maximum throughput and configuring throttling. There's no justification for you and I having to configure our routers and set up QoS, and for everyone else to live with bad latency.
And it wasn't always like this. Back in the 80s, the creators of the internet spent plenty of effort building protocols that did scale. In Jim's day, despite slower network speeds, it was perfectly possible to download over FTP (not HTTP, that didn't exist) while interacting on telnet (not SSH that didn't exist either). Of course latency on a 100% network will be affected somewhat, but not unusably so. Transmission Control Protocol exists for a reason, let that do the throttling.
http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-mastermind-bufferbloat/
I never understood the appeal of replacing every occurrence of a single quote with a bitmap of a turd.
Especially, for technical blogs, where people might post shell commands, which break if their quotes get changed into something else.
So, why exactly are people still continuing to use wordpress? And if this is configurable, why doesn't anybody switch this option off?
That being said, does anybody have a link to a non-wordpress source of this story?
Oh I wasn't implying that there was some sort of plot (apart from the ISP greedily oversubscribing lines). However the model HAS changed - people now can easily produce and stream their own video and audio with little or no additional equipment. ISPs, however, are failing to keep up, insisting on a top down, one to many type model where you are the many. Unless of course you want to pay $1000/month (or whatever) for a T1 line.
Seven puppies were harmed during the making of this post.
"I read TFA and I'm not seeing the problem. He can't duplicate this issue unless he maxes out his connection and then his latency goes to hell. No shit Sherlock, that's what happens when your pipe is full and the packets have to wait in the queue to be transmitted."
It doesn't have to be as bad as it is - the queue doesn't have to be so long. If you have a shorter queue, it will just back up in each sending application (they send slower) and everyone gets a fair turn at the queue. With big buffers the queue is already full and any packet that wants to go has an extra long wait - latency is too high and useless for real-time applications. Equal access - fairness - but high latency.
If you keep the queue small and still give equal access, you get the same throughput for all applications but lower latency, which is very nice for real time applications like conferencing.
blog.sam.liddicott.com
How are you supposed to traffic shape your "best effort" cable modem? The maximum bandwidth is rarely what will be available, and the busy bandwidth will be far below average. A set of scripts to automatically determine the current bandwidth and adjust your traffic shaping percentage appropriately? Oh wait, that's just the TCP congestion avoidance algorithm!
Wasn't kidding. Don't kill me, bro!
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
I tried QoS once on my home Linksys router and it was awful, it simply wouldn't work with high traffic/connections. (using original firmware)
Jitter is caused by the buffers eventually filling or TCP timing out (registering packet loss), dropping the rate for a little bit, the buffers draining, then TCP upping the rate again as the buffers refill, hiding the saturation, until they're full again. Rinse and repeat.
Brings to mind a situation where I work. ping times between two sites went like (5,10,20,40,80,160,320,5,10,20.....) all milliseconds. I emailed that to our network guys but they just suggested putting an extra compression device on the link, which I persuaded them not to do. Their solution to latency was always to increase end to end bandwidth.
http://michaelsmith.id.au
QoS is not the solution to latency caused by bufferbloat--it's like putting a square peg into a round hole: it will either not fit or still leak.
If buffers were sized appropriately, latency for interactive apps wouldn't be ruined by a single HTTP download using all available bandwidth. QoS can only be controlled at each end separately, which means end users giving up some of their bandwidth for the sake of creating their own bottleneck to control the buffers. Users shouldn't have to artificially limit their bandwidth--properly-implemented networks can handle simultaneous connections without multi-second latency. TCP was designed and improved to handle the problem itself, but buffers have destroyed the very mechanism TCP uses to handle the problem.
QoS might be unnecessary for VOIP-type apps if it wasn't for bufferbloat. QoS should provide a small boost to interactive performance, at most, because latency shouldn't be that bad in the first place.
Think of it this way: we could either create additional, complicated mechanisms to manage packet queues and explicitly notify nodes of congestion...or we could just use smaller buffers and let TCP do what it used to do already, what it was engineered to do. We could add more layers of software to compensate for hardware problems...or we could remove the unnecessary hardware buffers and let the existing software work the way it used to. Which is simpler, and cheaper?
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Damn Nagle. ;) Gamers have been hating buffers since modems. Dropped packets are not evil. Apps which require low latency but humble bandwidth should be able to interrupt the queue, but I can't think of a way that ISPs could really distinguish these apps reliably and it falls back to the problem of ISP choosing favorites. With latency and max bandwidth to an arbitrary host as unknown and varying, as well as the volume of traffic exchanged with the host, the empirical ramp-up/back off seems to be best for a single connection to get good bandwidth. Contention between high bandwidth transfer with less important latency needs and low bandwith transfer with important latency needs is not working well. Makes me think about profiling or tailoring TCP parameters for different connection types, people jumped through all sorts of hoops to try to tailor their stuff to low latency when dial-up was king.
The problem here isn't that the buffers are big, it's that you don't have the bandwidth needed for your traffic load, period.
What would be enough bandwidth for a given load? If a user is downloading an ISO of a Linux distro, that can't be downloaded in mere seconds, period. That's going to saturate the connection at one end or the other, period. And thus, it's probably going to end up filling buffers at one end or the other, leading to cascading latency and failure of interactive apps and really frustrated people. This is the 21st century, after all. Even when I was using a 56k modem, I sometimes had less latency when downloading a file and browsing the web at the same time. Things should be getting better, not worse.
You seem to be suggesting that if we could just have enough bandwidth, buffers wouldn't matter. But we're not talking so much about backbones, we're talking about end-to-end paths with bottlenecks. We can't just add more bandwidth to everyone's connection--we have to fix the buffers.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
or the total re-architecture of millions of consumer devices to remove buffering
Total re-architecture? A tiny software patch could shrink the buffers on any device, regardless of how much RAM a device has. The problem is getting vendors to make the patches and users to install them--it's that simple, and that difficult.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
You're right. But every new or updated device that has smaller, appropriate buffers will help to solve the problem. Over time, as devices are updated or replaced, the problem could be eradicated, almost like a vaccine gradually wiping out a virus.
I think it's a bit silly to add yet more layers of complex software to cover over a problem that could simply be fixed by using less in the first place. It's like over-engineering a solution to an over-engineered solution!
It's like building larger and more complex recycling plants to compensate for huge, overflowing landfills while ignoring the obvious solution of making less in the first place (smaller, more efficient, reusable packaging, etc). Then there wouldn't be as much of a need for recycling. "REDUCE, REUSE, recycle."
Do the simple, obvious fix--that's actually undoing what shouldn't have been done in the first place--before creating a complex workaround for the workaround!
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
There are no layers involved. It's a high level description of something that ignores lots of stuff and "oh shock horror" even gets some details wrong.
Did they ever tell you in school that electrons orbit the nucleus in shells? Oh shit! They lied! Clearly they made no sense at all!
The example is describing what the packets do when you request the file. And yes its exaggerating things and yes it's simplifying things. But that's how you describe things to people who don't know the technical details when those technical details don't really matter to understanding the overall issue.
Obviously you don't get 1GB of packets in the queue then again the queue isn't 1GB in size in the first place anyway - oh look I explicitely mentioned that too but you're clearly to stupid to read.
Getting stuck up on trivial technical details is what your bringing, lots of other people manage to understand how analogies and high level approximations work.
Good post, my friend. "All progress depends on the 'fool.'" "Fools" like you are the leaders we need. (I mean "fool" in a positive way; I'm with you.)
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Netalyzr does exactly this, among other things. It will tell you how big your perceived buffers are along your network path, upstream and downstream.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
The issue is always going to be there. Pegged connection == FIFO queuing, absent some sort of QoS scheme.
You don't understand. The point is not that FIFOs shouldn't exist, the point is that the FIFOs are too big. If you think of the buffer sizes in milliseconds rather than bytes (buffer size / connection speed), you'll see that big buffers == big latency. Appropriately-sized--smaller--buffers will also fill up, but being much shorter (in terms of how long they take to empty), latency won't skyrocket, because a new packet won't have to wait seconds to make it from the back to the front of the queue.
And it does, even with QoS. All you do with QoS is force the buffering to happen on equipment that you control rather than equipment your ISP controls. In this manner you can ensure that time sensitive packets (interactive VPNs, VoIP, etc.) don't sit in the queue behind someone's Windows Update download.
As was mentioned, you can only truly effect QoS on your upstream data. You can affect ACKs going upstream, but if the node at the other end wants to saturate your downstream connection, you can't stop it from doing so. And since you can't effect QoS on downstream data, your VOIP/etc. may have good upstream latency, but downstream latency and RTT will suffer, because those incoming VOIP packets will wait their turn in the queue just like the Windows Update download's packets.
I think your first line said it all. :)
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Tomato's QoS works well. It was well worth the $40 for the ASUS router so I could play games while my neighbor was downloading TV with BitTorrent. After configuring it well, I tested it extensively and got no increase in game latency even when the connection was saturated with torrents.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Almost. I think the solution is to continue to invest in infrastructure where and when needed, while returning buffers to sizes that are appropriate (there have to be some buffers).
Really, if it were possible to convince vendors and users, all we need is software patches to make buffers smaller. So what if the RAM goes unused?
Hey, I know: for devices with the CPU to burn, use the extra RAM as HTTP cache for really popular sites. Then there'd be less traffic on the backbones.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
It's amazing how many people will accuse an innocent man of something they are guilty of themselves, and incriminate themselves in the process.
Go read and educate yourself. "Bufferbloat because RAM is getting cheaper" is exactly the problem. Too many packets are buffered, and not enough are dropped.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Haha, compression? Adding another stage, another buffer, another process, to try to reduce latency? That's like turning on another light because it's too bright in the room.
And these guys are "the network guys"? Sad. Just shows what this problem is up against.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
The use of 1 GB was merely as an illustration.
You have just demonstrated the uphill battle that is fixing this problem. You do not understand.
It's absurd for the end-user to rate limit his connection at his end--TCP is engineered to take care of that on its own. Unnecessarily large buffers defeat the very mechanism TCP uses to control congestion and data rates.
You're right about one thing: TCP links do not send data willy-nilly. But when a buffer is, e.g. 3500ms in length (buffer size in bytes / connection speed), any change in rate resulting from the receiver acknowledging packets won't happen for at least 3500ms. If packets aren't dropped for 3500ms, then the data rate won't reduce as a result of the packet loss for at least 3500ms. Then it will drop by 50%, and gradually increase again until the buffers are full. Repeat ad nauseam (e.g. jitter).
The best, simplest, and cheapest way to fix this problem is to patch software in routers, etc. to reduce buffers to sane sizes and let TCP do what it's already engineered to do. QoS and rate-limiting by every user is absurd, illogical, and wasteful--it's throwing useful bandwidth away because of a problem that shouldn't exist in the first place.
Please, at least study the issue before you try to debunk it.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
You're a fool. You don't understand the difference between an HTTP request and the resulting TCP session that fulfills the request. The foolishness comes from refusing to admit your mistake and refusing to learn.
The HTTP request for a file or a range of a file is made once. The resulting TCP session works by the server sending packets and the receiver ACKnowledging packets--not requesting packets.
You shouldn't attempt to comment authoritatively on an issue until you actually understand how the systems work.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I disagree that it works pretty well overall. At my apartment in one place, I can use my AT&T DSL connection to download large files or use BitTorrent or stream Netflix and still browse the web with acceptable latency. Visiting my family in another place, also on an AT&T DSL connection, the bufferbloat is so bad (over 3000ms according to Netalyzr) that downloading a single large file via HTTP makes for 5-10 second latency in browsing other web sites. I'm not kidding. And it hasn't always been this way. Years ago, at this same location, with the same AT&T DSL service, it didn't behave so poorly. And ten years ago, living in another state but also with the same speed AT&T DSL service, such latency was never a problem, even with two people constantly using the connection. My best guess is that at some point the AT&T equipment here was changed and the buffers got much bigger.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I can see why you're posting as an AC, because you don't understand the difference between an HTTP request and the TCP connection that fulfills it. There is no requesting of packets; the request is made via HTTP, and the receiver then ACKnowledges TCP packets from the server, which may send more quickly than it receives ACKs so as to increase throughput--this then fills buffers and causes cascading latency.
You are compounding the problem by spreading misinformation. Please stop and go educate yourself.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Perhaps, but that would require more router CPU. Simply using sensible buffer sizes would allow the existing TCP flow control mechanisms to do what they were engineered to do without additional software complexity. I think that's the "real" solution: to undo that which should never have been done.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Mr. AC, you will have to explain how it's a non-issue, since Mr. Gettys has shown how it is indeed an issue. I can testify to the problem myself, as Netalyzr shows 3000ms+ buffers on my current DSL connection, and as downloading a single large file via HTTP is causing these buffers to fill and resulting in multi-second latency for simple web browsing and jittery pings while downloading.
I think you give too much credit to these "hardware engineers" of yours. If "they" knew so much about queueing, "they" wouldn't have made buffers which are so large that they defeat the built-in TCP congestion-control mechanisms. "They" tend to think that more is always better.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
As others mentioned, dropped packets are intended--they're part of how the flow-control mechanisms work.
Dropped packets may be quite annoying for VOIP, gaming, perhaps even streaming media (though multi-second application-level buffers should compensate for streaming media), but for most protocols--like HTTP web browsing, BitTorrent, etc), dropped packets aren't really a big deal.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
A few people do get it.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Theoretically, hardware wouldn't have to be replaced--all that's needed is to patch the software to use smaller buffers. So some RAM goes unused; big whoop.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
You don't really understand QoS, do you?
Read up on QoS capabilities of equipment with multiple queues and multiple thresholds per queue. Read up on micro-flow policing. Read up on allocation of queue memory to different queues. QoS is likely much more complex than you realize. That's all I'm trying to say - that if ISP's properly configured QoS then there would not be a problem with "buffer-bloat," at least not on the Internet backbone. Cheap, non-configurable consumer grade equipment may contribute to buffer-bloat, but the core Internet equipment if properly configured should be able to handle multiple flows and congestion properly.
The problem for ISP's is two-fold. One that as I said QoS can be extremely complex given the mix of equipment, agreements between providers or exchanges to accept traffic as marked or to remark traffic, common usage of marking into appropriate classes, etc.
The other is that proper QoS can be mis-construed to be some kind of non-net neutrality and favor some traffic over others. While that is true, the favoring of traffic should and has to be done via traffic class, as in what kind of traffic it is, rather than source/destination address. An example of "good" traffic classification would be to single out VOIP traffic based on ports, packet contents, or other means as opposed to "bulk" traffic such as FTP transfers. An example of "bad" traffic classification would be identifying traffic coming from Google and giving it a higher DSCP value than traffic from other vendors just because Google paid the provider / exchange some extortion money. The first example is necessary for proper QoS in congested networks, and is an example of "good" non-net neutrality. The second is an example of using QoS for "bad" non-net neutrality.
It is the fundamental misunderstanding by people that don't understand the technology, don't know how specific hardware and software on Internet infrastructure devices operate, and couldn't configure QoS properly if they were tasked to do so that obfuscates the conversation and makes it almost meaningless to respond.
Another piece of the puzzle, perhaps.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Hear, hear! A voice of reason!
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
It's not that simple. Ideally, if, for example, a 1 Mbit DSL connection were saturated with one HTTP download, and the user then started loading a web page, the packets of the web page's request should have no more latency than the download's packets. However, when buffers are several thousand milliseconds in length, it doesn't work that way; the download's packets are constant and fill the buffers, and the web page's packets, being small and bursty, have to wait to get through the queue. If the user started a second download, both downloads would end up having equal shares of bandwidth. However, when one connection is constant and saturates the buffers, and another connection is interactive and bursty, the bursty one will suffer latency.
The correct solution is to size buffers so they are short in length of time (size in bytes being relative to bandwidth). This way, even if the buffers are saturated, our hypothetical web page's packets won't have to wait long to get through the buffers, and latency won't be much worse than on an idle connection. TCP is made to rate limit connections itself--it's just that these pesky, oversized buffers defeat TCP's rate-limiting mechanisms. Having to rate limit your software or your own upstream bandwidth is an ugly hack that wastes bandwidth. Just shrink the buffers with software patches.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
That's not really a solution, merely a hackish workaround--an extra layer of software complexity to work around a hardware problem that shouldn't exist. If the buffers were sized appropriately, rate-limiting wouldn't be necessary to avoid latency. Rate-limiting and QoS should be used to prioritize bandwidth, not latency.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Solution: fix the software in the routers, modems, etc. to use appropriate buffer sizes, and use the Internet as it was intended, to do whatever you want whenever you want, sharing bandwidth and latency equally.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
While I generally agree with you regarding UDP for streaming, it's not quite that simple. For one thing, application-level buffering combined with application-level retransmitting should handle dropped UDP packets (though reimplementing what TCP does might not perform so well in practice). However, it doesn't work out to a simple dropped frame here and there. Media transport stream packets aren't going to have video frames byte-aligned with network packets. Besides, variable-rate encoding and partial-frame encoding throws all that out the window--compressed video isn't a stream of bitmaps. When video data is dropped or corrupted, you see ugly artifacts that sometimes aren't resolved for a few seconds until the next keyframe. Dropped audio data might result in ugly, loud sound artifacts.
For VOIP or Skype-type stuff, maybe those artifacts would be tolerable. But not for all streaming video.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Uhh, sorry - but didnt you get the memo? IRAN! We must attack IRAN!!!
I'm not sure why I'm bothering to respond to an AC, but anyway...
I'm sure you're right about QoS and buffers on backbones working well on backbones, but I'm talking about the problem end-to-end. I think bufferbloat is more an edge phenomenon, what with last-mile cable/DSL/fiber connections interfacing with poor-quality home networking gear--and wireless networks have their own special bufferbloat problems.
Most people on here have been touting QoS as a solution--mostly claiming that rate-limiting connections on LAN routers and throwing away a percentage of your rated bandwidth is the way to solve the latency problems. Your backbone/enterprise-grade QoS is a whole 'nother matter, but even that can't fix the problems at the edges.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Fix the software to use appropriate queueing algorithms. Large buffers can improve performance but they are not compatible with simple tail-dropping algorithms.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Thank you for the kind words.
It helps for the several months of lonely work setting the stage for this for it to have succeeded this way.
Option A: Revise how the TCP protocol works, requiring re-programming TCP clients (where the code, competent programmers, and management-allocated time for those programmers are available). Same thing for network gear. Expect LOTS of landfillbloat. Who's gonna fix all those OS/2-driven ATMs, Windows 3.1-driven mass-transit ticket-vending machines, and NT4-driven lab devices, etc.? (Yes, I've seen all of the above with my own eyes.)
Option B: Revise how TCP clients and network gear uses the hardware buffers they have (no need to physically remove RAM), requiring requiring re-programming TCP clients (where the code, competent programmers, and management-allocated time for those programmers are available). Same thing for network gear. Expect SOMEWHAT-LESS landfillbloat than Option A above. Who's gonna fix all those OS/2-driven ATMs, Windows 3.1-driven mass-transit ticket-vending machines, and NT4-driven lab devices, etc.?
Option C, D, E...: ??
Option Z: Ignore the problem until the Internet collapses. I'm sure there'll be many PHB votes for that one ("I don't see the need to invest resources solving a possible problem which won't occur until the far future...")
I'd prefer Option B to Option A, because it's more-likely-reliable. Protocol changes affect EVERYBODY, and if the protocol people screw up, we're all screwed. Look at all the time that many smart people have invested in creating computer security protocols and algorithms, and look at how many times they've made significant errors: RC4, WPA, WEP...
Very well explained, thank you.
Ideally I suppose you're right, depending on the CPU in the router.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Yes, thank you. I should have said that sooner. :) I hope we can look back on your work (and the contributions of others you mentioned) as helping to avoid serious problems in the future.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I read the article, and author admits himself: "(I’m not a full fledged TCP expert)."
I'm not saying there isn't a problem with latency, bandwidth and saturation. This 'bufferbloat' is just something he made up and then he attributes network behaviour findings to this. That doesn't mean that 'bufferbloat' is anything that exists and causes anything. When I say that, I don't deny the symptoms, I'm only saying that the symptoms are not caused by what is claimed.
--- Hindsight is 20/20, but walking backwards is not the answer.
I'm not denying the symptoms, all I'm saying is that the buffer causing the jitter is the window size, and it's not the network operator that chooses or configures that. The total 'buffers' between you and the receiver will never be fuller than your window size. The TCP window size is the maximum amount of data sent and not yet acknowledged by the receiver. When the window size is reached, you stop transmitting until you receive acks.
You can modulate the window size on an existing tcp/ip connection at the (endpoint) application level to control latency. I've done it and it works.
--- Hindsight is 20/20, but walking backwards is not the answer.
please use and honour the TOS/DSCP/Traffic Class flags in IP packets. These seem to be there to avoid "bufferbloat"
Make high-latency, high-throughput, and high-reliability packets the users cost some fee and give the OS an indicator on which program requires how much money.
I don't think he was blaming the consumer, merely pointing out the reality of things.
It's not really all that difficult to shape downstream traffic. All you need is a router between your internet connection and LAN clients. I've done this for years at my office using the QoS functionality of the Linux kernel. We are located out in the middle of nowhere with T-1s as our only means of connectivity. Sharing a 3.0mbit/s connection with 60+ employees without QoS is virtually impossible if you need to run interactive protocols.
And just how, exactly, do you plan to have this router on your local network control the rate of incoming packets over your T1? Once they get to your router, to be "rate controlled" they've already been transmitted through the limited size network pipe. At that point, what reason is there to hold up delivery to your local user? If you want to queue and prioritise, it has to be before the slow link (and that in itself gives you the problem not-linked-to-but-described in the summary).
I'm quite sure you can control the rate you send, but you receive data when you receive data - you don't control the packet order, size or content, the sender does.
If the buffers were sized appropriately, rate-limiting wouldn't be necessary to avoid latency
Yes, actually it would. Interactive protocols are going to suck period if they wind up sitting in a buffer. A smaller buffer size might make them suck less but it's still not going to deliver acceptable performance. FIFO queuing can not handle interactive protocols even with smallish buffer sizes.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
By taking the high road and not pointing fingers he is able address an issue in such a way that a lot of the people who did contribute to this problem can recognize what they have done and own it, without being labelled, accused or feeling attacked.
But we aren't all bozos on this bus, and pretending "everybody contributed to it" is not necessarily the best way to fix this particular problem, or to reduce the likelihood of this kind of engineering failure in the future. Understanding how this happened is important.
My elevator version of what happened: the bellhead model of a communication service is a reliable circuit-switched connection. "Reliable" sounds good, and circuits are a familiar model. But the Internet is based on a model of best-effort delivery of packets. Every product group experienced in Internet infrastructure knew horror stories about confusing TCP. New entrants did not know this, or had system design teams tilted towards bellhead decision-making.
Cisco has all the cool toys for queue management in their routers. Are they bozos? People who have even skimmed the Linux traffic shaping HOWTO are sensitized to the issues. They're not bozos.
I have a copy of the first edition of Comer in front of me (the 1988 one that talked about the inevitable transition from TCP to OSI TP4.) The advice to implementors of gateways tells you to read RFC 1009 very carefully, which has a bunch of congestion cites, including John Nagle's (he's downthread) RFC 970 explaining why infinite buffers are a disaster. These are foundational documents of the Internet, and sure, they're from 1987 and routing to a T1 by processing over 9,000 packets a second is no longer something you would need a supermini for (you probably get faster computers free with your breakfast cereal.) But scanning forward through the RFCs you'll see lots and lots of very pointed advice to the effect of "please do not confuse TCP or you'll be sorry."
So some of the people building the network hardware with these problems weren't alive when this was being figured out. They didn't do their homework; fine. The people running the companies designing and building the hardware don't have that excuse, and it was their job to either get a clue or hire one. Their customers are going to be the ones paying to fix this.
So if you're buying Internet infrastructure, you might want to look for companies (and more particularly, product groups) hanging out on nanog and participating in IETF, since although that's not proof their products are not fighting the Internet, maybe it correlates.
My current guess is that organizational decision-making was tilted towards bellhead thinking for a variety of reasons (stereotype: they dress better and do nicer PowerPoint architecture.) Skimming through documentation of bearers such as 1xRTT makes it pretty clear that the design center was "reliable pipe first, then put packets on it." Which makes perfect sense if your company has history in non-Internet telecoms--your senior people are the ones who shipped products that did reliable circuit-switched pipes. But that's just wrong if you're doing IP, and for reasons known in the Internet communications world for decades.
I've been trying to figure out whether I wanted to link some version of this to the blog posts. I figure it's safely out of sight here and won't interfere with the public diplomacy.
I don't understand why you think he made up "bufferbloat" (the problem, that is, not the term). He clearly explained his methods and his work with other parties who are indeed experts. I've used Netalyzr on my own connection and it reports very large buffers on my DSL connection, and I've observed the exact same symptoms that Mr. Gettys has observed. Other people commenting on this article (even some who work in ISPs) have concurred.
If you want to convince me that bufferbloat is not a real problem, and not the cause of the observed symptoms, you'll need to provide some evidence of alternative explanations, because Mr. Gettys has done an excellent job of proving his case. He's also not the only person who's written about the problem, only the most recent.
If you read the series of articles and understand enough about how TCP works, it's clear that it is indeed a real problem.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I disagree. If a connection's buffers were 50-100ms in size (perhaps even less than 50ms would be enough sometimes), the latency could never exceed 50-100ms, which is plenty for most interactive protocols. Maybe you're still thinking of buffers in terms of bytes, but it's much more helpful in this case to consider them in terms of the time it takes to empty them.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I disagree. If a connection's buffers were 50-100ms in size (perhaps even less than 50ms would be enough sometimes), the latency could never exceed 50-100ms, which is plenty for most interactive protocols.
Unless they get dropped. Buffers are still FIFO.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Isn't that where retransmitting comes in?
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Which imposes an even bigger delay and makes interactive protocols a PITA (telnet/ssh) at best and nearly unusable (VOIP/streaming video) at worst.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
How about sane defaults on the web sever? Have it set by file.
I know tobacco is bad for you, so I smoke weed with crack.