Bufferbloat — the Submarine That's Sinking the Net

gottabeme writes "Jim Gettys, one of the original X Window System developers and editor of the HTTP/1.1 spec, has posted a series of articles on his blog detailing his research on the relatively unknown problem of bufferbloat. Bufferbloat is affecting the entire Internet, slowly worsening as RAM prices drop and buffers enlarge, and is causing latency and jitter to spike, especially for home broadband users. Unchecked, this problem may continue to deteriorate the usability of interactive applications like VOIP and gaming, and being so widespread, will take years of engineering and education efforts to resolve. Being like 'frogs in heating water,' few people are even aware of the problem. Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"

Comments

cringley explains

[szo]

http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/

Re:pegged connection == latency, who'd of thunk it

[TheThiefMaster]

As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?

The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small

Re:I think buffers are a good thing

[Coriolis]

He's not arguing against application-level caching. He's saying that too much caching at the IP layer is confusing TCP's algorithm for deciding how fast the link between two points is. This in turn causes massive variability in how fast the data can be downloaded; or in your terms, how fast the video can be buffered (and, in fact, how much buffer the video player needs).

Re:QoS

[Shakrai]

Given that most traffic on a domestic connection is incoming, that doesn't help much.

It's not that hard to shape downstream traffic. Take a Linux router with two ethernet cards. eth0 is the LAN and eth1 is the internet. You shape eth0 with a maximum throughput of 75%-80% of your line speed. All of the downstream traffic has to go out on that interface so that's your opportunity to shape it. I do this at work and successfully share a 3.0mbit/s connection with 60+ employees. We use latency sensitive services like VoIP and RDP alongside streaming video and other large downloads without any major hassles. It stinks to lose some of your bandwidth because of this (you have to shape it to a number less than 100% of your line speed, otherwise buffering occurs at your ISP and your QoS scheme is defeated) but I'll take responsiveness over throughout any day of the week.

Re:First link in the first article

[GooberToo]

Demands for definition are a bit pompous...

A bit?

Even more pompous is making a post about it when everyone can clearly see, "bufferbloat" is shorter than constantly saying something tedious like, "excessive packet buffering in the entirety of a network path."

Perhaps this will help the uninitiated. The article describes a wide problem of excessive packet buffering in the entirety of a network path, which has been dubbed, "bufferbloat."

And this is a known problem, and fairly intuitive

[ebrandsberg]

let me summarize the problem that is being observed: On a given interface, if you have more buffer memory than is needed as packet buffer on the transmit side, it can induce latency. As an example, consider a 1Mb/s link. If you want to have a peak of .1s latency added by buffering at high load, then you want 1Mb*.1=12,500 bytes of buffer. If you have 1MB of buffer, then you have 8 seconds of buffer, therefore triggering the "buffer bloat" issue. Part of the problem is that buffer size would be set based on the top speed a piece of hardware could drive, i.e. if you want a 1Gb/s interface to be able to buffer .1s, then you use it at 100Mb/s, then it has 1s worth of buffer. In most home deployments where you have a router that may have a 1Gbps upstream, maybe 4 100Mb/s physical connections, and a 54Mbps wireless router, you probably have a shared buffer for all the interfaces. The result of this is that when using the 54Mb/s wireless, you can easily have the buffer over saturated, while the buffer size may be just right for the 100Mb/s interfaces.

What is the solution to this? Realistically, the alternative is to drop packets that have resided in the buffer longer than a configured amount of time, which causes it's own performance issues. Net result: TCP would slowdown for a period of time, but would speed up again resulting in a sawtooth behavior. This would result in periodic issues with other protocols as well, i.e. VOIP would have dropped packets every time TCP ramps up again, etc.

Solution: Don't download porn when you are trying to do VOIP calls.

Jim Gettys did the world a great service with this

[iwbcman]

I discovered this series of blog posts about 2 months ago, when he accidentally published one of his blog posts prematurely. I started reading it and followed the links and saw that this was a like a sleuth tale-if I had started reading this with his very first blog on the topic I would have had no idea where he was going with this. Now as to why this contribution by Jim Gettys does the world a great service:

• Gettys is not pointing fingers at someone. The problem he is describing is truly vast, and involves lots of different people in different industries(router manufactures, ISP's, kernel driver authors, carrier grade network manufactures, etc.) with, presumably, a myriad of different intentions. The problem has been building over a long time-this didn't start yesterday, and won't be solved in a short time span, without a concerted effort on the part of everyone involved in all of these divergent industries, who often have quite divergent interests.
• This approach that Gettys takes allows him to describe a problem which confronts everyone. By taking the high road and not pointing fingers he is able address an issue in such a way that a lot of the people who did contribute to this problem can recognize what they have done and own it, without being labelled, accused or feeling attacked. This should be a lesson to anyone who really wants to redress an issue that effects everyone.
• Gettys develops this theme over many, many blog posts. It makes for some of the best internet reading I have experienced in years. Things only gradually become clearer-not merely what the problem is, but also all of the issues involved in it. I can read away in the internet for months at a time and not learn as much as I did by reading this series of posts.
• Gettys knows what he is talking about. He developed this theme by talking with lots of experts -engineers at the ISP, people who played a pivotal role in the creation of the www and network specialists. He himself is not a network specialist, but he went out and met with people to discuss his findings and took clues and information from these exchanges to inform him and his quest to find out what was going on.
• The series is short on answers. It may prove frustrating to many that he offers so little in the way of solutions to this problem. But this this due to the fact that the problem cannot be resolved by you, the end user. To solve this problem means rearchitecting countless millions of devices and altering hundreds of thousands of lines of code in multiple OS's.
• Failure to redress this problem means that every effort to decrease latency by upping available bandwidth or upgrading network infrastructure will fail to deliver. If packets are not dropped fast, due to excessive buffering, the negotiation process fails, which invariably means congestion, which means latency-only something that addresses this issue has any chance of actually effecting change. Saying that this problem is just an issue already solved by QOS show that you don't understand the problem.
• One of the first thoughts I had reading this was: if the techs on wallstreet read this article they will inevitably exploit this issue to win precious milliseconds on the stock exchange-ring a bell?
• Any ISP could exploit this issue to offer a relative market advantage. Sadly when resolving an issue is in everybody's interest, market players will exploit the issue for their own relative gain. Getting everyone to actually tackle this is going to a gargantuan task.

Hats of to Jim Gettys. Thanks for your service.

Re:Definition, please

[hairyfeet]

Now someone can point out if this humble repair guy is wrong, but from what I read of TFA it sounds to me like TCP is the problem and not the buffers. What TCP is doing is slamming the network until it drops packets, and then using that to determine speed, correct? Doesn't sound like an efficient way to allocate resources when even grandma has a fat cable or DSL pipe. It would be like everyone trying to shove their way to the front of a line and only backing off when getting punched in the face.

So again feel free to correct if I'm missing something, but wouldn't the better solution be to come up with a new way to allocate space? Perhaps a packet every so often that says "Hi, how much bandwidth may I have please?" which the server or node would reply "You can have X" and then everyone wouldn't be trying to slam and your route would be based on which gives you the largest X.

So maybe I'm missing something, but it seems to be a choice of that or rip every buffer out of everything from the home modem on up. Considering even the el crappo motherboards are gigabit now, and even the CCC (Cheapo Chinese Crap) home routers have decent sized buffers on them this would seem like a more doable solution. Because while the "slam and back off" method probably worked really well in the past I just don't see having millions of people trying to slam their way to the head of the line as the most efficient way to utilize a limited resource, and no matter how big a pipe you have it is still just that, limited.