Slashdot Mirror
Got (Buffer) Bloat?
mtaht writes, "After a very intense month of development, the Bufferbloat project has announced the debloat-testing git kernel tree, featuring (via suggestion of Van Jacobson) a wireless network latency smashing algorithm, (called eBDP), the SFB and CHOKe packet schedulers, and a slew of driver level fixes that reduce network latency across the Linux kernel by over 2 orders of magnitude. Got Bloat?"
It is good to write in summary what program nobody hear before actually do
My understanding may not be correct but:
Bufferbloat (I first came across the term bufferbloat in this blog post by Jim Gettys) is the nickname that has been given to the high latency that can occur in modern network connections due to large buffers in the network. An example could be the way that a network game on one computer starts to stutter if another computer starts to use a protocol like bittorrent to transfer files on the same network connection.
The large buffers seem to have arisen from a desire to maximise download throughput regardless of network condition. This can give rise to the situation where small urgent packets are delayed because big packets (which perhaps should not have been sent) are queued up in front of them. The system sending the big packets is not told to stop sending them so quickly because its packets are being delivered...
The linked article sounds like people have modified the Linux kernel source to allow people who know how to compile their own kernels to test ideas people have had for reducing the bufferbloat effect on their hardware and to report back their results.
Does this help explain things a bit?
Instead of using TCP/IP (bastardized version of ISO), people should start using real OSI implementations such as the ISO protocol, with 20 byte addresses and QoS level settings for each of the 7 OSI layers.
Once upon a time it was an issue of cost of h/w logic, IP was the cheaper alternative, today the difference is nil and the benefits of ISO are orders of magnitude better than IP.
bufferbloat, IP address exhaustion, etc are just a few of the reason why we should drop IP altogether.
Does that mean the network becomes a hundred times faster? Oh well, at least I can dream.
A lot of our problems today would not be here if.
OSI stack instead of TCP/IP.
DCE & DFS instead of passwd/whatever + the bastard abomination which is NFS.
Meh. People are lazy and cheap. Free with the network effect always wins. The Lowest Common Denominator. It's going to take another 15 years before we are near where we were 15 years ago. But this time it will be in Java!
Deleted
I've seen it time and time again, people just generally don't care about latency, or even deny it exists in many cases (buffer bloat is certainly one cause of latency).
Everything from changing channel on your TV remote, to a mobile phone number entry, to the frame delays you get from LCD monitors, to the soundcard delay, to the GUI widgets you click on;......... it's all over the place, and it can wreck the experience, or reduce it somewhat according to how big the delay is. Just because latency is harder to measure, that doesn't mean it isn't very important, especially when it builds up with lots of other 'tiny' delays to make one big delay.
Why OpalCalc is the best Windows calc
Move to Nicaragua. Surf a lot. Hack from the beach. http://www.nicaraguasurfreport.com/reportlist.php?id_secc=25 It worked for me.
The first problem is that a ton of transit systems on the Internet (like indeed a ton of systems everywhere) are effectively running the default behaviors in this respect, with no special tuning. That means FIFO with whatever queue size is available.
The second is that even if all the ISP operators decided to fix this, "QoS stuff" has the potential to run afoul of Network Neutrality. The current thinking is that they shouldn't be discriminating between "bulk/throughput" packets and others. Some would suggest discrimination between traffic types is okay, so long as you don't discriminate between traffic sources (ie prioritize all VoIP, but don't let Comcast give Comcast VoIP preference over Vonage VoIP) but implementation would be tricky - all too easy to implement a "vendor neutral" policy that coincidentally doesn't seem to identify Vonage's traffic quite right.
The simplest and most neutral solution to all this is simply to decrease the buffer size in those big default FIFO queues. Even the bulk/throughput packets won't really suffer from that - TCP is specifically designed to have packets dropped in a timely manner, rather than held in a queue for a long time. One of the problem behaviors that Jim identifies is that if your real RTT to say, a server in California, is 40 msec, but there's 4 seconds worth of delay in the buffers. TCP will send a window full of data (let's say 64K) then wait for a reply for 40, 80, maybe 120 msec. Not getting it, he sends the whole window again. And again. And again. Finally an ACK squeezes through, and the process begins again. Instead of shrinking his transmission size, he does the opposite - he sends big multiples of every packet, making the congestion worse.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
Let the heathen drop theirs
When their RAM is spent.
TCP shall make them pay for
Each packet that can't be sent.
Every packet is wanted.
To this we are sworn.
From real-time data from CERN
To the filthiest of porn.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
.
Prisencolinensinainciusol. Ol Rait!
Sorry Dave. A Coral cache version of the animation.
Most delays are due to users connecting to their ADSL modem via Ethernet and not traffic managing properly.
On a congested link this can cause large delays as Ethernet normally has a 1000 packet buffer in the Linux kernel and the ADSL modem has a similar buffer. You only need a couple of heavy connections which want to go faster than the ADSL will support and those buffers start to fill up real fast. You can easily end up with latencies measured in seconds if you have a lot of connections running (say bittorrent).
There are several solutions to this but the best in my experience is to change the queuing discipline to SFQ and rate limit using HTB. This has been in the kernel for years and works extremely well. You need to limit the traffic upstream and downstream to slightly less (5% less) than the ADSL link speed. This ensures that the modem never queues traffic. Uplink you can use all sorts of fancy queuing but downlink all you can really do is policing of traffic unless you install the IMQ patch to the kernel.
I've a script which I got from somewhere a while ago, don't remember where though. I've put it at http://ams1.x31.com/~andy/ppp0-ratelimit.sh if anyone wants to look at it. It expects to work on ppp0 but can be adapted as required.
I've played a lot more recently with Linux kernel disciplines and it has produced surprising performance on congested links. One link is running mail, remote access and Internet access over a 1mbit symmetric link for about 60 users. in the morning it hits 95% link capacity at the start of work and stays there until everyone goes home but ssh sessions are fully interactive without noticeable lag all this time. Yes web browsing is a little slow but it is the same for everyone and one user can't flood the link and upset everyone else.
Linux QOS is the future, pity about the documentation
Have you done the research to see just who you're disagreeing with about this?
And why they engineered TCP the way they did?
I won't pretend that I've walked through the experiments to try to verify their conclusions. I'm not even sure I know enough to interpret to interpret But...the people shouting the warnings aren't your average Chicken Littles.
After reading this guys buffer bloat rant I largly agree with him with some exceptions:
1. What does multiple TCP sessions have to do with circumvention of congestion avoidance? TCP congestion avoidance needs to work with lots and lots of TCP sessions at once not just one or two. HTTP 1.1 sessions need NOT be short lived. I don't see why a large number of TCP sessions can't all be subject to congestion avoidance...responding individually to the conditions they see? How does this work to effectivly bypass congestion avoidance? I've seen this talking point in a few places but noone has ever explained WHY this is so. I can see an argument based on a static suboptimal initial congestion window but HTTP 1.1 supports pipelining...
2. Two connections per server is not sufficient for browsers. TCP is a stream protocol with head of line blocking... High latency links will never use the available bandwidth properly unless they either use lots of sessions or start with massive windows which is not good for congestion. There is also a problem of ordering dependancies of resource requests within the web content. Without lots of concurrent fetches the user gets to wait longer for page loads. The presentation sounded to me like someone either not understanding necessary details of TCP and higher layer considerations or trying to have their cake and eat it too.
Lastly we don't need to replace HTTP - we need to replace TCP... HTTP over SCTP would be a much more significant improvement than any reasonable change to HTTP. No matter what you do to HTTP you still have to live with the underlying transports limitations!
The link in the /. story to debloat-testing should go here: git://git.infradead.org/debloat-testing.git.
git:gitinfradeadorgdebloat-testinggit is not a valid URL.
Thanks. I was hoping someone would post something like this. My experience with ADSL modems has been a very slow responsiveness (ie. choppy ssh, slow browsing) whenever the [smaller, say 1mbps] upstream bandwidth is exceeded. This slowness can happen even with just one fast upload capable of saturating the link. From my understanding, the bulk data (ie. big/many ftp/bittorent packets) fill the FIFO buffers on the ADSL modems. Now when an outgoing http request is made, it gets stuck at the back of the FIFO and even though the [larger, say 3mbps] downstream bandwidth is nowhere near capacity, since the http request [click] has to wait (say a second) before it goes out, the response [web page or image] obviously isn't going to start on it's way. That's at the http level.
Now say there is a download already happening... stuff , say a "big" 2MB print-ready version of the bufferbloat logo, is zipping down your nearly empty download pipe... well, it comes in "little" chunks - say 1500bytes at a time - but for each chunk that is sent by the bufferbloat webserver, it expects to receive a "tiny" ACK packet back before it sends the next chunk (or decides to retransmit). Well, your browser gets the "little" chunk and your quad-core glowing blue machine immediately fires that ACK back at lightning speed... SMACK! there it lands at the back of the ADSL modem's FIFO buffer and patiently waits there for a second until it get's it's turn to zip up the pipe and say "right on! all's cool, send me the next chunk of bufferbloat logo!"
Of course in the meantime, the bufferbloat webserver has probably taken this delay to be an indication that the client is probably in orbit around the moon and will certainly not try to send the next chunk in a big rush.
So yes, one fix would be to limit your output (or that of all the machines behind the ADSL modem) to slightly less than the upstream bandwidth since you probably cannot have much influence on the FIFO buffer -- my impression was that this buffer might actually be on the far side of the ADSL modem.
But, when it comes to doing this (or to control the similar thing that happens when being hammered [slashdotted] by excessive incoming traffic, or to police incoming traffic) and your link to the world is not an ADSL modem, but a wireless link -- you have a similar (and perhaps less solvable) problem:
The wireless link is say nice fairly stable 30mbps link but when under [a shared] load it get's a little shaky and drops speed to 20mbps -- do you set your policing/SFQ whatever settings for 30 or 20 (or 10mbps)?
I always (as an AC slashdotter) suspected that these buffers were introduced as a "freedom of speech" impediment to punish "producers/publishers" of information by, by default, destroying their interactivity, voip, surfing, whatever, should they decide to broadcast/publish/share. The buffers were certainly not implemented in the opposite direction because, well, that would interfere with "consuming" bandwidth.
Interesting problems to solve, but I think by bringing the realities of bufferbloat to the "consumer/producers" aka public we can move towards a situation where ISP charges trend towards the transit charges that really more closely reflect the true (non-capital) cost of providing last-mile internet -- really a call to cut out the middleman/ISP and put the bottlenecks back where they belong - (intercontinental/backbone links). :D
[Changing channels] should generally look like a single, continuous stream from the head end at your cable company. [...] Now if by "changing channels" you mean rapidly switching between separate streams from content sources on the Internet, then yes, but on the other hand, most streaming videos are almost invariably not being transmitted live
In other words, streams MUST NOT be live unless the viewer subscribes to a cable television service and the streams have been approved by the operator of such service.
In general, isoch connections should be used for live streams from special events, video chat, VoIP, and a single isoch connection to the head end for anything that resembles IPTV.
This raises the question: should each IPTV user have access to multiple head ends? News and sporting events appear to qualify as "special events" for this purpose, but someone might want to watch news and sports offered by a provider without a specific carriage agreement with the head end owned by your particular ISP. For example, a small or remote provider might offer a live stream. Otherwise, it's cable TV carriage disputes all over again.