Slashdot Mirror
Got (Buffer) Bloat?
mtaht writes, "After a very intense month of development, the Bufferbloat project has announced the debloat-testing git kernel tree, featuring (via suggestion of Van Jacobson) a wireless network latency smashing algorithm, (called eBDP), the SFB and CHOKe packet schedulers, and a slew of driver level fixes that reduce network latency across the Linux kernel by over 2 orders of magnitude. Got Bloat?"
It is good to write in summary what program nobody hear before actually do
My understanding may not be correct but:
Bufferbloat (I first came across the term bufferbloat in this blog post by Jim Gettys) is the nickname that has been given to the high latency that can occur in modern network connections due to large buffers in the network. An example could be the way that a network game on one computer starts to stutter if another computer starts to use a protocol like bittorrent to transfer files on the same network connection.
The large buffers seem to have arisen from a desire to maximise download throughput regardless of network condition. This can give rise to the situation where small urgent packets are delayed because big packets (which perhaps should not have been sent) are queued up in front of them. The system sending the big packets is not told to stop sending them so quickly because its packets are being delivered...
The linked article sounds like people have modified the Linux kernel source to allow people who know how to compile their own kernels to test ideas people have had for reducing the bufferbloat effect on their hardware and to report back their results.
Does this help explain things a bit?
Instead of using TCP/IP (bastardized version of ISO), people should start using real OSI implementations such as the ISO protocol, with 20 byte addresses and QoS level settings for each of the 7 OSI layers.
Once upon a time it was an issue of cost of h/w logic, IP was the cheaper alternative, today the difference is nil and the benefits of ISO are orders of magnitude better than IP.
bufferbloat, IP address exhaustion, etc are just a few of the reason why we should drop IP altogether.
It is about reducing latency. So it helps prevent problems with VoIP failing when there is a lot of other data flowing over the same connection(s).
Some buffers are really large and it turns out some introduce latency of several seconds (!).
New things are always on the horizon
A lot of our problems today would not be here if.
OSI stack instead of TCP/IP.
DCE & DFS instead of passwd/whatever + the bastard abomination which is NFS.
Meh. People are lazy and cheap. Free with the network effect always wins. The Lowest Common Denominator. It's going to take another 15 years before we are near where we were 15 years ago. But this time it will be in Java!
Deleted
Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.
I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.
Live today, because you never know what tomorrow brings
I've seen it time and time again, people just generally don't care about latency, or even deny it exists in many cases (buffer bloat is certainly one cause of latency).
Everything from changing channel on your TV remote, to a mobile phone number entry, to the frame delays you get from LCD monitors, to the soundcard delay, to the GUI widgets you click on;......... it's all over the place, and it can wreck the experience, or reduce it somewhat according to how big the delay is. Just because latency is harder to measure, that doesn't mean it isn't very important, especially when it builds up with lots of other 'tiny' delays to make one big delay.
Why OpalCalc is the best Windows calc
The core work where we saw latency under load drop by 2 orders of magnitude was in the wireless driver stack on Linux. Examples were the iwl driver (130ms to ~2), and the ath9k driver ( > 200ms to ~d) (and these numbers were for GOOD connections, at high wifi rates. You can get 3 orders of magnitude improvement if you are on a slow wifi connection.) There's a new rate sensitive algorithm for wireless (eBDP) that we are trying in this kernel tree. It's not fully baked yet. 802.11n wireless package aggregation is HARD. That said there's bloat in all the other wired drivers too. We are doing far too much uncontrolled buffering in the kernel - specifically the dma tx ring on many devices - for slower networks. As one example, A gigE interface, connnected to a 3Mbit cable modem - does bad, subtle, things to the stack.
not just voip. Think: dns, dhcp, nd, ntp & arp.
Move to Nicaragua. Surf a lot. Hack from the beach. http://www.nicaraguasurfreport.com/reportlist.php?id_secc=25 It worked for me.
The first problem is that a ton of transit systems on the Internet (like indeed a ton of systems everywhere) are effectively running the default behaviors in this respect, with no special tuning. That means FIFO with whatever queue size is available.
The second is that even if all the ISP operators decided to fix this, "QoS stuff" has the potential to run afoul of Network Neutrality. The current thinking is that they shouldn't be discriminating between "bulk/throughput" packets and others. Some would suggest discrimination between traffic types is okay, so long as you don't discriminate between traffic sources (ie prioritize all VoIP, but don't let Comcast give Comcast VoIP preference over Vonage VoIP) but implementation would be tricky - all too easy to implement a "vendor neutral" policy that coincidentally doesn't seem to identify Vonage's traffic quite right.
The simplest and most neutral solution to all this is simply to decrease the buffer size in those big default FIFO queues. Even the bulk/throughput packets won't really suffer from that - TCP is specifically designed to have packets dropped in a timely manner, rather than held in a queue for a long time. One of the problem behaviors that Jim identifies is that if your real RTT to say, a server in California, is 40 msec, but there's 4 seconds worth of delay in the buffers. TCP will send a window full of data (let's say 64K) then wait for a reply for 40, 80, maybe 120 msec. Not getting it, he sends the whole window again. And again. And again. Finally an ACK squeezes through, and the process begins again. Instead of shrinking his transmission size, he does the opposite - he sends big multiples of every packet, making the congestion worse.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
Let the heathen drop theirs
When their RAM is spent.
TCP shall make them pay for
Each packet that can't be sent.
Every packet is wanted.
To this we are sworn.
From real-time data from CERN
To the filthiest of porn.
Every packet is sacred.
Every packet is great.
If a packet is wasted,
TCP gets quite irate.
.
Prisencolinensinainciusol. Ol Rait!
Sorry Dave. A Coral cache version of the animation.
Most delays are due to users connecting to their ADSL modem via Ethernet and not traffic managing properly.
On a congested link this can cause large delays as Ethernet normally has a 1000 packet buffer in the Linux kernel and the ADSL modem has a similar buffer. You only need a couple of heavy connections which want to go faster than the ADSL will support and those buffers start to fill up real fast. You can easily end up with latencies measured in seconds if you have a lot of connections running (say bittorrent).
There are several solutions to this but the best in my experience is to change the queuing discipline to SFQ and rate limit using HTB. This has been in the kernel for years and works extremely well. You need to limit the traffic upstream and downstream to slightly less (5% less) than the ADSL link speed. This ensures that the modem never queues traffic. Uplink you can use all sorts of fancy queuing but downlink all you can really do is policing of traffic unless you install the IMQ patch to the kernel.
I've a script which I got from somewhere a while ago, don't remember where though. I've put it at http://ams1.x31.com/~andy/ppp0-ratelimit.sh if anyone wants to look at it. It expects to work on ppp0 but can be adapted as required.
I've played a lot more recently with Linux kernel disciplines and it has produced surprising performance on congested links. One link is running mail, remote access and Internet access over a 1mbit symmetric link for about 60 users. in the morning it hits 95% link capacity at the start of work and stays there until everyone goes home but ssh sessions are fully interactive without noticeable lag all this time. Yes web browsing is a little slow but it is the same for everyone and one user can't flood the link and upset everyone else.
Linux QOS is the future, pity about the documentation
Have you done the research to see just who you're disagreeing with about this?
And why they engineered TCP the way they did?
I won't pretend that I've walked through the experiments to try to verify their conclusions. I'm not even sure I know enough to interpret to interpret But...the people shouting the warnings aren't your average Chicken Littles.
And that's exactly what this is about. That "weirdness [that] can happen under load..."
After reading this guys buffer bloat rant I largly agree with him with some exceptions:
1. What does multiple TCP sessions have to do with circumvention of congestion avoidance? TCP congestion avoidance needs to work with lots and lots of TCP sessions at once not just one or two. HTTP 1.1 sessions need NOT be short lived. I don't see why a large number of TCP sessions can't all be subject to congestion avoidance...responding individually to the conditions they see? How does this work to effectivly bypass congestion avoidance? I've seen this talking point in a few places but noone has ever explained WHY this is so. I can see an argument based on a static suboptimal initial congestion window but HTTP 1.1 supports pipelining...
2. Two connections per server is not sufficient for browsers. TCP is a stream protocol with head of line blocking... High latency links will never use the available bandwidth properly unless they either use lots of sessions or start with massive windows which is not good for congestion. There is also a problem of ordering dependancies of resource requests within the web content. Without lots of concurrent fetches the user gets to wait longer for page loads. The presentation sounded to me like someone either not understanding necessary details of TCP and higher layer considerations or trying to have their cake and eat it too.
Lastly we don't need to replace HTTP - we need to replace TCP... HTTP over SCTP would be a much more significant improvement than any reasonable change to HTTP. No matter what you do to HTTP you still have to live with the underlying transports limitations!
All those technical terms are confusing to slashdotters. /end sarcasm But, seriously - let's just reduce all of that to two words: online gaming. I'm most certanly not a serious gamer - and the wife even less so. But, we both complain about LAG!! I seriously doubt that our ISP uses any form of optimization for VOIP or anything else. Whatever their QoS, it's probably the most basic and most outdated jury rigged trash in use at the time the servers were manufactured, and still running at default settings. All of which discourages me from experimenting with this stuff. I mean - big deal if I optimize MY stuff - I still have to rely on the ISP's ancient junk.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
The link in the /. story to debloat-testing should go here: git://git.infradead.org/debloat-testing.git.
git:gitinfradeadorgdebloat-testinggit is not a valid URL.
Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.
I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.
Doesn't this also mean that under high bittorrent pressure, pings will not time out and and ssh will remain responsive.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
[Changing channels] should generally look like a single, continuous stream from the head end at your cable company. [...] Now if by "changing channels" you mean rapidly switching between separate streams from content sources on the Internet, then yes, but on the other hand, most streaming videos are almost invariably not being transmitted live
In other words, streams MUST NOT be live unless the viewer subscribes to a cable television service and the streams have been approved by the operator of such service.
In general, isoch connections should be used for live streams from special events, video chat, VoIP, and a single isoch connection to the head end for anything that resembles IPTV.
This raises the question: should each IPTV user have access to multiple head ends? News and sporting events appear to qualify as "special events" for this purpose, but someone might want to watch news and sports offered by a provider without a specific carriage agreement with the head end owned by your particular ISP. For example, a small or remote provider might offer a live stream. Otherwise, it's cable TV carriage disputes all over again.
DHCP? Are 2 second leases the new black or something?