Slashdot Mirror
Controlling Bufferbloat With Queue Delay
CowboyRobot writes "We all can see that the Internet is getting slower. According to researchers, the cause is persistently full buffers, and the problem is only made worse by the increasing availability of cheap memory, which is then immediately filled with buffered data. The metaphor is grocery store checkout lines: a cramped system where one individual task can block many other tasks waiting in line. But you can avoid the worst problems by having someone actively managing the checkout queues, and this is the solution for bufferbloat as well: AQM (Active Queue Management). However, AQM (and the metaphor) break down in the modern age when Queues are long and implementation is not quite so straightforward. Kathleen Nichols at Pollere and Van Jacobson at Parc have a new solution that they call CoDel (Controlled Delay), which has several features that distinguish it from other AQM systems. 'A modern AQM is just one piece of the solution to bufferbloat. Concatenated queues are common in packet communications with the bottleneck queue often invisible to users and many network engineers. A full solution has to include raising awareness so that the relevant vendors are both empowered and given incentive to market devices with buffer management.'"
The Internet is not getting slower. It is becoming laggier. Comeon people, learn the difference.
Wonder what the public key field is for?
Today, there is no incentive for an ISP to consider spending money on this. For their private customers, they sell QoS, which guarantees their customers a better queuing method. Extremely profitable. For consumers, it makes sense to simply continue investing in infrastructure. Adding capacity from the street to the CO not only eliminates the issue, but also allows the ISP to provide better, more profitable services. In short, we will likely see better queuing methods integrated with future routers. The may be one of them, but only time will tell, and nobody will discard all of their equipment today to get it. The issue is just too minor while capacity remains cheap and QoS profitable.
Yep, same cause. They attempted to minimize packet loss by increasing the buffers in their network. The user experience was horrible.
http://blogs.broughturner.com/2009/10/is-att-wireless-data-congestion-selfinflicted.html
Is the internet getting slower? (laggier)
because the simplest pages are HUGE BLOATED MONSTROSITIES!
Between flash and ads. And every single page loading crap from all around the world as their 'ad partners', hit counters, click counters, +1 this, like this, digg this, and all the other stupid social media crap that has invaded the web. All this shit that serves no purpose other than to some marketers. And EVERY SINGLE PAGE has to have a 'comment' section and other totally useless shit tacked on as well.
Just this little page here on slashdot. With less than a dozen replies. Tops 80k so far. And that's with everything being blocked that can be.
slower? laggier? no... the signal to noise raito is sucking major ass.
Maybe Slashdot should get one of these AQM things........
"First they came for the slanderers and i said nothing."
WAT? No mention of Fry's?
Single-queue multi-cashier. Best. Checkout. System. Evar.
We all can see that the Internet is getting slower.
Can we? It looks like it has been getting faster to me....
According to researchers, the cause is persistently full buffers,
What researchers? What buffers? Server buffers? Router buffers? Local browser buffers? Your statements are so vague as to be useless.
and the problem is only made worse by the increasing availability of cheap memory, which is then immediately filled with buffered data.
Buffering is a way of speeding servers up immensely. Memory is orders of magnitude faster than disk, and piling RAM on and creating huge caches can only help performance. I call bullshit on your entire claim. This summary is so awful, I don't even want to read whatever article you linked to.
HA! I just wasted some of your bandwidth with a frivolous sig!
We all can see that the Internet is getting slower.
Can we? I'd suggest that most people are unaware of any such trend, perhaps because it has happened too gradually and too unevenly. Indeed:
A full solution has to include raising awareness so that the relevant vendors are both empowered and given incentive to market devices with buffer management.
Exactly. Consumers don't know or care about low latency, so the market doesn't deliver it (that plus lack of competition among ISPs in general, but that's another kettle of fish).
We need a simple, clear way for ISPs to measure latency. It needs to boil down to a single number that ISPs can report alongside bandwidth and that non-techies can easily understand. It doesn't need to be completely accurate, and can't be: ISPs will exaggerate just like they do with bandwidth, just like auto manufacturers do with fuel efficiency, etc. What matters is that ISPs can't outright make up numbers, so that a so-called "40 ms" connection will reliably have lower average latency than a "50 ms" connection. That should be enough for the market to start putting competitive pressure on ISPs.
What kind of measure could be used for this purpose? Perhaps some kind of standardized latency test suite, like what the Acid tests were to web standards compliance? Certainly there would be significant additional difficulties, but could it be done?
The boundary they are transiting is one between a fast network and a slower network, similar to what you see at a head-end at a LATA or broadband distribution point and leaf nodes like peoples houses, or one the other end, on the pipe into a NOC with multi gigabit interconnects much bigger than the pipes into or out of the NOC.
The obvious answer is the same as it was in 1997 when it was first implemented on the Whistle InterJet: lie about the available window size on the slow end of the link so as to keep the fast end of the link from becoming congested by having all its buffers filled up with competing traffic.
In this way, even if you have tasks which would otherwise eat all of your bandwidth (at the time, it was mostly FTP and SMTP traffic), you can still set aside enough buffer space on the fast side of the router on the other end of the slow link to let ssh or HTTP traffic streams make it through. Beats the heck out of things like AltQ, which do absolutely nothing to prevent a system with a fast link that has data to send you crap-flooding the upstream router so that it has no free buffers to receive any other traffic, and which it can't possibly hope to shove down the smaller pipe at the rate it's coming in the large one.
Ideally this would be cooperatively managed, as was suggested at one point by Jeff Mogul (which is likely barred due to the lack of a trust relationship between the upstream and downstream routers, if nothing else). Think of it like half your router lives at each end of the link wire, instead of both sides living on one end.
It's the job of the device on the border who happens to know there's a pipe size differential to control what it asks for from the upstream side int terms of the outstanding buffer space it's possible for inbound packets to consume (and to likewise lie about the upstream windows to the downstream higher speed LAN on the other end of the slow link).
I'm pretty sure Julian Elischer tried to push the patches for lying about window size out to FreeBSD as oart of Whistle contributing Netgraph to FreeBSD.
While people are looking at that, they might also want to reconsider the TCP Rate Halving work at CMU, and the LRP implementation from Peter Druschel's group out of Rice University.
-- Terry
My InterTubes are BETTER because I HAVE ZERO LOSS!!!
Oddly enough such a business model turned out to be unsustainable due to
(1) it's finanically expensive (between one thing an another)
(2) doing this the less expensive way (ie by slathering on bigger buffers) introduces excessive latency (for some customer designated value of "excessive")
For the life of me I don't understand how ANYBODY can be allowed to run a company without at least vaguely understanding the concept TANSTAAFL.
And, finally: No that Hot Blonde Supermodel with MASSIVE Bazingas does NOT find you attractive, not in the slightest, no matter how much she may drink/snort/inject or pop.
If you want to fix throughput issues without spending lots of money or sacrificing latency then you're going to need a beter algorithm (yes folks, hard research and careful thinking).
Visit CryptoGnome in his home.
The first time I saw this was back about the mid '60s, when a bank got the idea to equalize wait time this way. It was called "Zip Line".
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
It's not clear from the paper whether packet dropping is per-flow, in some fair sense, or per link. There's a brief mention of fairness, but it isn't explored. It sounds like the new approach has no built-in fair queuing.
Without fair queuing, whoever sends the most gets the most data through. Windows (especially) starts up TCP connections by sending as many packets as it can at connection opening. There used to be a convention in TCP called "slow start", where new connections started up sending only two packets, increasing if the round trip time turned out to be good. That was too pessimistic. But Windows now starts out by blasting out 25 or so packets at once. This hogs the available bandwidth through everything with FIFO queues.
If the routers at choke points (where bandwidth out is much less than bandwidth in, like the entry to a DSL line) do fair queuing by flow, the problem gets dealt with there, as the excessive sending fights with itself, trailing packets on the biggest flows are sent last, and everything works out OK.
"Bufferbloat" is only a problem when a small flow gets stuck behind a big one. A flow getting stuck behind the preceding packets of the same flow is fine; you want those packets delivered. Packet reordering is better than packet dropping, although more computationally expensive. Most CIsco routers offer it on slower links. Currently, this means links below 2Mb/s, which is very slow by modern standards. That's why we still have kludgy solutions like RED. This new thing is a better kludge, though.
This seems like such an unstable system that it's practically a security issue. Could someone, in theory, purposely send bad traffic to as many internet relays (or whatever) as possible, causing them to stall and shutting down huge chunks of the internet?
I was wondering what a butterfloat was, how delicious it might be, and whether it was anything like butterbeer.
I was excited until re-reading.
We all can see that the Internet is getting slower *Citation needed* Have you tried turning your modem off and on again?
http://www.awfullybigmoustache.com
My first thought after reading the story was 'Hope whoever patents those ideas doesn't charge too much for them."
I want a list of atrocities done in your name - Recoil
If your ISP doesn't respect your TOS values then you're only ever going to get best effort service.
Changing the technology of the queuing in the network won't help because your traffic is all going into the 'whatever is left' queue at the bottom of the priority stack (or next to the bottom of the priority stack if your ISP has implemented worse than best effort queuing to control p2p and worm traffic) below the provider's own traffic (ie voice, video services they sell) and any business traffic where the business customer has paid for guarantees (for whatever that's actually worth, depending on the provider.
blindly antisocialist = antisocial
> But you can avoid the worst problems by having someone actively managing the checkout
> queues, and this is the solution for bufferbloat as well: AQM (Active Queue Management).
Can someone please implement this system at Heathrow to reduce the queues there?
Correction: There is no get-rich-quick scheme with a high probability of success. There are a few (like the lottery) which may get you rich quick, but with only a small probability.
Heh, I know exactly what you mean. The same thought definitely crossed my mind. Fortunately, if you read carefully, you'll see that they seem to be releasing their code as open source.
"The open source project CeroWrt is using OpenWrt to explore solutions to bufferbloat. A CoDel implementation is in the works, after which real-world data can be studied. We plan to make our ns-2 simulation code available, as well as some further results."
Not a guarantee, but it sounds promising.
1. Have brilliant idea
2. Patent it
3. Create open source implementation
4. Wait for widespread adoption
5. ????
6. Profit!
Smells like packet shaping by stealth to me. New hardware providing a base infrastructure to kill net neutrality.
This paper is total BS: the only real remedy would be to REDUCE latency.
But of course, our telcos and ISPs want to disable people using internet for low-latency application, like telephone. So that they can charge for it more. And thus, they are financing research that produces results like this paper.
Assuming you are a fellow USian, what competition?
Should have called it "AQtive Queue Magnagement".
It is *any* transition from fast to slow, including your computer to your wireless link or vice versa from your home router to your computer.
Bufferbloat is an equal opportunity destroyer of time.
And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material.
let's get rid of all the semaphores and put in roundabouts..
I disagree. It's fine for buffers to be very big, practically "infinite" even. Buffer size does not have to be linked with latency at all. The reason it is currently is because most routers are doing it wrong ;).
If you really want to address latency what routers should do is keep track of how long packets have been in the router (in clock ticks, milliseconds or even microseconds) and use that with QoS stuff (and maybe some heuristics) to figure out which packets to send first, or to drop.
For example, "bulk/throughput" packets (think email) might be kept around for 100+ milliseconds. In contrast, while latency sensitive packets get priority, they might be dropped if they cannot be sent within tens of milliseconds ( so both sides can earlier detect the comms channel can't cope and possibly deal with it sooner rather than wait till the latency gets intolerable).
With my proposed approach a latency insensitive burst of big packets through a high bandwidth channel does not necessarily have to cause packet loss at all - whether from latency sensitive or latency insensitive channels. There would be enough buffer space to hold all the big packets while letting the low latency stuff through. Whereas with the smaller buffer size approach you are more likely to have unnecessary packet loss - which reduces throughput.
Even if the router doesn't do any QoS stuff it's way more useful to people who care about latency to be able to configure a router so that the max latency it will ever cause is 10 milliseconds. Especially if the router has multiple network connections with different bandwidths (e.g. 2Mbps, 100Mbps).
See: http://www.grc.com/securitynow.htm - Episode #345
K Man
Nope. I never saw that it was going slower. The original poster immediately fails in the first line of text. Honestly it seems as fast or faster than ever lately.
So basically i didn't read the rest of the article because
YOU ARE STUPID
See Jim Getty's blog for more (and better) info.
the problem is you can't just 'fix' TCP. even if you make it work with both IPv4 TCP and IPv6 TCP, you're still missing the other protocols out there (SCTP, UDP, etc, etc), and more importantly, can NEVER fix protocols that are being encrypted (ie TCP inside VPN packets). the protocols that you aren't 'fixing' will just continue barging their way through.
the only workable solution is to drop packets and assume the protocols themselves will adjust (which they will, if they are working correctly, and if they're not working correctly.. they'll be penalized with a lot of dropped packets).
dropping packets is unpleasant, because it is wasting bandwidth, and causing TCP stalls.. but ECN barely works because half the routers out there drop or mangle it, so it's the only solution.
It's a part of the solution but not a complete solution. Boundary problems are different from center-of-network problems. Playing with TCP window sizes only works well at the edges and only in the outgoing direction (the 'inflight' sysctls that we've had forever), but does not completely solve the problem because you always need to add one or two additional packets above and beyond the calculated sweet spot to absorb changes in latency from other links and give the algorithm time to respond whenever reality changes.
This solution in both incoming and outgoing directions suffers from a connection multiplication problem. That is, it works fine if you have only a few simultaneous TCP connections running but it breaks down when you have dozens or hundreds due to the need to have 1-2 extra packets of slop in the reported window. It degrades gracefully in the outgoing direction but blows up very quickly when you are trying to control bandwidth in the incoming direction by changing the window size you report available in the outgoing ACKs (anyone running torrents can tell you this but the problem occurs for any busy network).
However, bandwidth limiting in this fashion DOES reduce packet backlogs and queues significantly. Not enough (it's simply impossible to make the algorithm stable at the sweet spot so you always need 1-2 additional packets), but significantly. Hence it is part of the solution.
Step 2: On speed boundary changes you have to run fair queuing, period. Not only that but you have to do it on both sides of the boundary (i.e. in both directions each at the choke point). In my case I could never get truly reliable operation by only running fair queuing in the outgoing direction. I had to actually run the fair queue on *both* ends of the link, meaning I had to colocate a server to serve as the terminus for a VPN and run all the traffic over the VPN so I could control both ends. The fair queue also has to reserve bandwidth for pure TCP ACKs to prevent restricting the bandwidth in the opposite direction due to ACK starvation.
Fair queuing takes care of all remaining packet buffering issues at the edges of the network.
Center-of-network issues can't use the above solutions simply because there are too many connections flowing through the center of the network to track and not enough packet buffer space to sufficiently buffers all those connections for fairq operation (N x B is just too big). It's impossible to calculate where the choke points are based on connection tracking at the center-of-network. Easy at the edges, impossible in the center.
So the center-of-network has to provide additional congestion control through some sort of AQM, and if early-warning requests go unheeded it must start dropping packets. Personally speaking I hate the idea of having to drop packets, it leads to all sorts of problems everywhere, but the simple fact of the matter is that there is not enough per-connection buffer space at the center of the network to run fairq, nor is it an appropriate place for that. Without tracking you cannot mess with tcp window sizes in the center-of-network and even if you could track you can't bundle the connections based on where the choke point is at the moment, there is simply not enough information.
So we are talking at least three and probably closer to half a dozen different mechanisms being needed to reduce network latencies and still provide good and fair performance.
-Matt
I would understand a "redundant" or perhaps a "troll".
But 'offtopic?'
Can I suggest the moderators to effectivelly READ Slashdot a little often?
http://tech.slashdot.org/story/11/12/03/0218257/bufferbloat-dark-buffers-in-the-internet
Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
From TFA-
"...the longer delays are not excessive and permit link utilizations to be maintained at high levels."
So we're adding another layer of management and still have "not excessive" delays. Awesome, sign me up.
Vote monkeys into Congress. They are cheaper and more trustworthy.
(The linked article, by Jacobson and a collaborator, cites two sources: one is Jacobson's original article, and the other is by Jacobson and the same collaborator.)
I'm not saying he's wrong; I'm saying this isn't very scientific.