Bufferbloat — the Submarine That's Sinking the Net

gottabeme writes "Jim Gettys, one of the original X Window System developers and editor of the HTTP/1.1 spec, has posted a series of articles on his blog detailing his research on the relatively unknown problem of bufferbloat. Bufferbloat is affecting the entire Internet, slowly worsening as RAM prices drop and buffers enlarge, and is causing latency and jitter to spike, especially for home broadband users. Unchecked, this problem may continue to deteriorate the usability of interactive applications like VOIP and gaming, and being so widespread, will take years of engineering and education efforts to resolve. Being like 'frogs in heating water,' few people are even aware of the problem. Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"

Comments

cringley explains

[szo]

http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/

Re:pegged connection == latency, who'd of thunk it

[TheThiefMaster]

As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?

The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small

Re:I think buffers are a good thing

[Coriolis]

He's not arguing against application-level caching. He's saying that too much caching at the IP layer is confusing TCP's algorithm for deciding how fast the link between two points is. This in turn causes massive variability in how fast the data can be downloaded; or in your terms, how fast the video can be buffered (and, in fact, how much buffer the video player needs).

Re:QoS

[Shakrai]

Given that most traffic on a domestic connection is incoming, that doesn't help much.

It's not that hard to shape downstream traffic. Take a Linux router with two ethernet cards. eth0 is the LAN and eth1 is the internet. You shape eth0 with a maximum throughput of 75%-80% of your line speed. All of the downstream traffic has to go out on that interface so that's your opportunity to shape it. I do this at work and successfully share a 3.0mbit/s connection with 60+ employees. We use latency sensitive services like VoIP and RDP alongside streaming video and other large downloads without any major hassles. It stinks to lose some of your bandwidth because of this (you have to shape it to a number less than 100% of your line speed, otherwise buffering occurs at your ISP and your QoS scheme is defeated) but I'll take responsiveness over throughout any day of the week.

Jim Gettys did the world a great service with this

[iwbcman]

I discovered this series of blog posts about 2 months ago, when he accidentally published one of his blog posts prematurely. I started reading it and followed the links and saw that this was a like a sleuth tale-if I had started reading this with his very first blog on the topic I would have had no idea where he was going with this. Now as to why this contribution by Jim Gettys does the world a great service:

• Gettys is not pointing fingers at someone. The problem he is describing is truly vast, and involves lots of different people in different industries(router manufactures, ISP's, kernel driver authors, carrier grade network manufactures, etc.) with, presumably, a myriad of different intentions. The problem has been building over a long time-this didn't start yesterday, and won't be solved in a short time span, without a concerted effort on the part of everyone involved in all of these divergent industries, who often have quite divergent interests.
• This approach that Gettys takes allows him to describe a problem which confronts everyone. By taking the high road and not pointing fingers he is able address an issue in such a way that a lot of the people who did contribute to this problem can recognize what they have done and own it, without being labelled, accused or feeling attacked. This should be a lesson to anyone who really wants to redress an issue that effects everyone.
• Gettys develops this theme over many, many blog posts. It makes for some of the best internet reading I have experienced in years. Things only gradually become clearer-not merely what the problem is, but also all of the issues involved in it. I can read away in the internet for months at a time and not learn as much as I did by reading this series of posts.
• Gettys knows what he is talking about. He developed this theme by talking with lots of experts -engineers at the ISP, people who played a pivotal role in the creation of the www and network specialists. He himself is not a network specialist, but he went out and met with people to discuss his findings and took clues and information from these exchanges to inform him and his quest to find out what was going on.
• The series is short on answers. It may prove frustrating to many that he offers so little in the way of solutions to this problem. But this this due to the fact that the problem cannot be resolved by you, the end user. To solve this problem means rearchitecting countless millions of devices and altering hundreds of thousands of lines of code in multiple OS's.
• Failure to redress this problem means that every effort to decrease latency by upping available bandwidth or upgrading network infrastructure will fail to deliver. If packets are not dropped fast, due to excessive buffering, the negotiation process fails, which invariably means congestion, which means latency-only something that addresses this issue has any chance of actually effecting change. Saying that this problem is just an issue already solved by QOS show that you don't understand the problem.
• One of the first thoughts I had reading this was: if the techs on wallstreet read this article they will inevitably exploit this issue to win precious milliseconds on the stock exchange-ring a bell?
• Any ISP could exploit this issue to offer a relative market advantage. Sadly when resolving an issue is in everybody's interest, market players will exploit the issue for their own relative gain. Getting everyone to actually tackle this is going to a gargantuan task.

Hats of to Jim Gettys. Thanks for your service.