Slashdot Mirror
Got (Buffer) Bloat?
mtaht writes, "After a very intense month of development, the Bufferbloat project has announced the debloat-testing git kernel tree, featuring (via suggestion of Van Jacobson) a wireless network latency smashing algorithm, (called eBDP), the SFB and CHOKe packet schedulers, and a slew of driver level fixes that reduce network latency across the Linux kernel by over 2 orders of magnitude. Got Bloat?"
It's about the downside of memory becoming cheap causing latency problems with congestion control mechinisms that rely on the endpoints being able to inform the sender when it's sending too fast.
Jim Getty's research blog entry explains the problem in detail.
My understanding may not be correct but:
Bufferbloat (I first came across the term bufferbloat in this blog post by Jim Gettys) is the nickname that has been given to the high latency that can occur in modern network connections due to large buffers in the network. An example could be the way that a network game on one computer starts to stutter if another computer starts to use a protocol like bittorrent to transfer files on the same network connection.
The large buffers seem to have arisen from a desire to maximise download throughput regardless of network condition. This can give rise to the situation where small urgent packets are delayed because big packets (which perhaps should not have been sent) are queued up in front of them. The system sending the big packets is not told to stop sending them so quickly because its packets are being delivered...
The linked article sounds like people have modified the Linux kernel source to allow people who know how to compile their own kernels to test ideas people have had for reducing the bufferbloat effect on their hardware and to report back their results.
Does this help explain things a bit?
The difference is that you can write an smtp server by reading in strings line by line and treating them as commands, then watch the logs and kludge it until it seems to interoperate well enough. With the OSI way of doing things you have to wear a blue tie for a start then you have to print out all the interface definition documents and spread them out on your desk and write the software to the interface.
man.. I want your desk if you can spread out all the iso interface definition documents on it and be able to read them
Most traffic throttling algorithms are based on the idea that the router will say "hey, slow down" if a client overloads it -- but when the router has lots of RAM, there is a tendency for it to just keep accepting and accepting, with the client happily pushing data at full speed, while the router is queuing up the data and only moving it upstream very slowly. Because the queues end up being huge, traffic going through that router gets lagged.
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
It is about reducing latency. So it helps prevent problems with VoIP failing when there is a lot of other data flowing over the same connection(s).
Some buffers are really large and it turns out some introduce latency of several seconds (!).
New things are always on the horizon
A lot of our problems today would not be here if.
OSI stack instead of TCP/IP.
DCE & DFS instead of passwd/whatever + the bastard abomination which is NFS.
Meh. People are lazy and cheap. Free with the network effect always wins. The Lowest Common Denominator. It's going to take another 15 years before we are near where we were 15 years ago. But this time it will be in Java!
Deleted
Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.
I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.
Live today, because you never know what tomorrow brings
IMO it's fine for buffers to be very big.
What routers should do is keep track of how long packets have been in the router (in milliseconds or even microseconds) and use that with QoS stuff (and maybe some heuristics) to figure out which packets to send first, or to drop.
For example, "bulk/throughput" packets might be kept around for hundreds of milliseconds, but while latency sensitive packets get priority they are dropped if they cannot be sent within tens of milliseconds (then the sender will faster realize that it should slow down).
Oh come on, you only have to follow four links to get to the definition. What are you, lazy?
Seriously, this is a true failure of web design. You click from the summary, then you go to the wiki, then you go to the faq, and the faq doesn't even tell you, it references a blog post.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
oblig. car analogy, by Eric Raymond no less:
https://lists.bufferbloat.net/pipermail/bloat/2011-February/000050.html
== Packets on the Highway ==
To fix bufferbloat, you first have to understand it. Start by
imagining cars traveling down an imaginary road. They're trying to get
from one end to the other as fast as possible, so they travel nearly
bumper to bumper at the road's highest safe speed.
[snipped]
That's a much more complex solution than "don't buffer so much damn stuff for no good reason."
Brian Fundakowski Feldman
I've seen it time and time again, people just generally don't care about latency, or even deny it exists in many cases (buffer bloat is certainly one cause of latency).
Everything from changing channel on your TV remote, to a mobile phone number entry, to the frame delays you get from LCD monitors, to the soundcard delay, to the GUI widgets you click on;......... it's all over the place, and it can wreck the experience, or reduce it somewhat according to how big the delay is. Just because latency is harder to measure, that doesn't mean it isn't very important, especially when it builds up with lots of other 'tiny' delays to make one big delay.
Why OpalCalc is the best Windows calc
The core work where we saw latency under load drop by 2 orders of magnitude was in the wireless driver stack on Linux. Examples were the iwl driver (130ms to ~2), and the ath9k driver ( > 200ms to ~d) (and these numbers were for GOOD connections, at high wifi rates. You can get 3 orders of magnitude improvement if you are on a slow wifi connection.) There's a new rate sensitive algorithm for wireless (eBDP) that we are trying in this kernel tree. It's not fully baked yet. 802.11n wireless package aggregation is HARD. That said there's bloat in all the other wired drivers too. We are doing far too much uncontrolled buffering in the kernel - specifically the dma tx ring on many devices - for slower networks. As one example, A gigE interface, connnected to a 3Mbit cable modem - does bad, subtle, things to the stack.
You further penalize the endpoint by cutting it down to 56k modem speeds for a month if you see it making too many short-lived (<5s) connections to multiple IPs and declaring them isoch. By definition, any such connections are an abuse of the standard.
But isn't that exactly how channel surfing would work on IP-based television?
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
Let the heathen drop theirs
When their RAM is spent.
TCP shall make them pay for
Each packet that can't be sent.
Every packet is wanted.
To this we are sworn.
From real-time data from CERN
To the filthiest of porn.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
.
Prisencolinensinainciusol. Ol Rait!
re:"Interesting problem for dd-wrt"
Agreed.
We are throwing efforts at both the mainline kernel and openwrt.
Openwrt is foundational for dd-wrt and several other (commercial) distributions of Linux on the router. I have a large set of debloated routers already, I'm just awaiting further work on the eBDP algorithm to make better....
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700
re: "using pings"
httpping is a much saner approach than ping, in many cases. Get it from:
http://www.vanheusden.com/httping/
re: RED & AQM
SFB and CHOKEe are in the debloat-testing kernel, as is eBDP.
RED 93 isn't going to work. nRED may. Experimentation and scripts highly desired. See the bloat and bloat-devel mailing lists for discussions.
Also:
http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle
Also:
I've seen some VERY interesting behavior with tcp vegas over bloated connections.
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_TCP_cubic_vs_TCP_vegas
The link in the /. story to debloat-testing should go here: git://git.infradead.org/debloat-testing.git.
git:gitinfradeadorgdebloat-testinggit is not a valid URL.