Slashdot Mirror

← Back to Stories (view on slashdot.org)

Got (Buffer) Bloat?

Posted by Roblimo on from the good-buffering-is-better-than-licking-a-cake-batter-bowl dept.

mtaht writes, "After a very intense month of development, the Bufferbloat project has announced the debloat-testing git kernel tree, featuring (via suggestion of Van Jacobson) a wireless network latency smashing algorithm, (called eBDP), the SFB and CHOKe packet schedulers, and a slew of driver level fixes that reduce network latency across the Linux kernel by over 2 orders of magnitude. Got Bloat?"

31 of 121 comments (clear)

  1. Re:what it is by firstnevyn · · Score: 5, Informative

    It's about the downside of memory becoming cheap causing latency problems with congestion control mechinisms that rely on the endpoints being able to inform the sender when it's sending too fast.

    Jim Getty's research blog entry explains the problem in detail.

  2. What bufferbloat is by Sits · · Score: 5, Informative

    My understanding may not be correct but:

    Bufferbloat (I first came across the term bufferbloat in this blog post by Jim Gettys) is the nickname that has been given to the high latency that can occur in modern network connections due to large buffers in the network. An example could be the way that a network game on one computer starts to stutter if another computer starts to use a protocol like bittorrent to transfer files on the same network connection.

    The large buffers seem to have arisen from a desire to maximise download throughput regardless of network condition. This can give rise to the situation where small urgent packets are delayed because big packets (which perhaps should not have been sent) are queued up in front of them. The system sending the big packets is not told to stop sending them so quickly because its packets are being delivered...

    The linked article sounds like people have modified the Linux kernel source to allow people who know how to compile their own kernels to test ideas people have had for reducing the bufferbloat effect on their hardware and to report back their results.

    Does this help explain things a bit?

    1. Re:What bufferbloat is by MichaelSmith · · Score: 4, Interesting

      high latency that can occur in modern network connections due to large buffers in the network

      Nobody ever explained this to me but I was using ping to measure latency on a network where I was actually most interested in ssh. Ping times went something like 10ms, 50ms, 90ms, 130ms... up to about 500ms, then started again at 10ms, 50ms and so on. Maybe some of my pings shared a buffer with a large, periodic data transfer and when that transfer filled a buffer somewhere my latency dropped.

      I am pretty sure the people actually operating the WAN in question had no idea what was going on either.

      --
      http://michaelsmith.id.au
    2. Re:What bufferbloat is by Sits · · Score: 5, Informative

      I should have also linked to a definition of bufferbloat by Jim Gettys. For the curious here's a page of links to bufferbloat resources and a 5 minute animation that shows the impact of large buffers on network communication (.avi).

    3. Re:What bufferbloat is by Lennie · · Score: 4, Informative

      The code changes to the Linux kernel also reduce the size and ill effects of buffers inside the kernel and drivers.

      --
      New things are always on the horizon
    4. Re:What bufferbloat is by Predius · · Score: 3, Informative

      I think what you were seeing was more due to ATM overhead than the DSLAM trying to be cute with throttling. Because ADSL encapsulates everything in ATM even small IP / Ethernet frames get broken up into lots of ATM cells which can add upwards of 20% overhead. So an ADSL line trained at 8Mb/s will never provide 8Mb/s of usable throughput to the end user. Some ISPs actually advertise targeted throughput instead of train rate and set the train rate a certain percentage above the target throughput to compensate. Others just advertise train rates and have disclaimers in the fine print.

      I've had my hands inside most Gen 1, 1.5 and 2nd Gen DSLAMs and never seen any with automatic throttling like you described.

      (Gen 1 being units that just function at the ATM layer requiring an external system to bridge to Ethernet or IP. Gen 1.5's being upgraded Gen 1s with crude bridging, and Gen 2 being units that were designed to terminate connections directly from the ground up.)

  3. Re:what it is by xming · · Score: 2

    this explains it: https://lists.bufferbloat.net/pipermail/bloat/2011-February/000050.html

  4. Re:Solution: Use a proper protocol (aka ISO) by MichaelSmith · · Score: 2

    The difference is that you can write an smtp server by reading in strings line by line and treating them as commands, then watch the logs and kludge it until it seems to interoperate well enough. With the OSI way of doing things you have to wear a blue tie for a start then you have to print out all the interface definition documents and spread them out on your desk and write the software to the interface.

    You correctly point out that IP is cheaper, but that means all the people who work with it will be cheaper too and the product which is slightly cheaper will always win.

    --
    http://michaelsmith.id.au
  5. Re:Solution: Use a proper protocol (aka ISO) by firstnevyn · · Score: 3, Funny

    The difference is that you can write an smtp server by reading in strings line by line and treating them as commands, then watch the logs and kludge it until it seems to interoperate well enough. With the OSI way of doing things you have to wear a blue tie for a start then you have to print out all the interface definition documents and spread them out on your desk and write the software to the interface.

    man.. I want your desk if you can spread out all the iso interface definition documents on it and be able to read them

  6. Re:what it is by shish · · Score: 4, Informative

    Most traffic throttling algorithms are based on the idea that the router will say "hey, slow down" if a client overloads it -- but when the router has lots of RAM, there is a tendency for it to just keep accepting and accepting, with the client happily pushing data at full speed, while the router is queuing up the data and only moving it upstream very slowly. Because the queues end up being huge, traffic going through that router gets lagged.

    --
    I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
  7. Re:Two orders of magnitude! ? by Lennie · · Score: 3, Interesting

    It is about reducing latency. So it helps prevent problems with VoIP failing when there is a lot of other data flowing over the same connection(s).

    Some buffers are really large and it turns out some introduce latency of several seconds (!).

    --
    New things are always on the horizon
  8. That is a battle which was lost 20 years ago by Colin+Smith · · Score: 3, Interesting

    A lot of our problems today would not be here if.

    OSI stack instead of TCP/IP.
    DCE & DFS instead of passwd/whatever + the bastard abomination which is NFS.

    Meh. People are lazy and cheap. Free with the network effect always wins. The Lowest Common Denominator. It's going to take another 15 years before we are near where we were 15 years ago. But this time it will be in Java!
     

    --
    Deleted
    1. Re:That is a battle which was lost 20 years ago by thanasakis · · Score: 2

      OSI stack instead of TCP/IP

      Can you please elaborate?

  9. Re:Two orders of magnitude! ? by Kjella · · Score: 4, Interesting

    Most network throughput is at least 80-90% efficient already, so it won't get much faster. It will make it more responsive though, which is good if you're browsing the web, playing an online game or something else interactive.

    I assume this is under load though, because on ping there's not much to be saved. On local sites I have 8-12 ms ping, on slashdot I have 140-150 ms. Since the theoretical round trip in a straight line at light speed is some 110 ms, there's not even room for a 50 ms drop. A lot of weirdness can happen under load though if stuff gets buffered up various places.

    --
    Live today, because you never know what tomorrow brings
  10. Re:what it is by TheLink · · Score: 4, Insightful

    IMO it's fine for buffers to be very big.

    What routers should do is keep track of how long packets have been in the router (in milliseconds or even microseconds) and use that with QoS stuff (and maybe some heuristics) to figure out which packets to send first, or to drop.

    For example, "bulk/throughput" packets might be kept around for hundreds of milliseconds, but while latency sensitive packets get priority they are dropped if they cannot be sent within tens of milliseconds (then the sender will faster realize that it should slow down).

    --
  11. Re:what it is by drinkypoo · · Score: 3, Informative

    Oh come on, you only have to follow four links to get to the definition. What are you, lazy?

    Seriously, this is a true failure of web design. You click from the summary, then you go to the wiki, then you go to the faq, and the faq doesn't even tell you, it references a blog post.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  12. Re:what it is by Anonymous Coward · · Score: 3, Informative

    oblig. car analogy, by Eric Raymond no less:
        https://lists.bufferbloat.net/pipermail/bloat/2011-February/000050.html

    == Packets on the Highway ==

    To fix bufferbloat, you first have to understand it. Start by
    imagining cars traveling down an imaginary road. They're trying to get
    from one end to the other as fast as possible, so they travel nearly
    bumper to bumper at the road's highest safe speed.
    [snipped]

  13. Re:what it is by Brian+Feldman · · Score: 3, Insightful

    That's a much more complex solution than "don't buffer so much damn stuff for no good reason."

    --
    Brian Fundakowski Feldman
  14. Re:what it is by mtaht · · Score: 2

    The web site has only been up for a month, and I agree with you very much that it is hard to get to the core ideas of bufferbloat from the get-go, we are incorporating information from dozens of very large blog posts, and hundreds of comments, and have been very busy (among other things) getting hardware running and kernel patches done. bufferbloat.net IS a wiki, however, and registration is open to all. If you can help improve the quality, PLEASE join and do so.

  15. Latency again by Twinbee · · Score: 3, Insightful

    I've seen it time and time again, people just generally don't care about latency, or even deny it exists in many cases (buffer bloat is certainly one cause of latency).

    Everything from changing channel on your TV remote, to a mobile phone number entry, to the frame delays you get from LCD monitors, to the soundcard delay, to the GUI widgets you click on;......... it's all over the place, and it can wreck the experience, or reduce it somewhat according to how big the delay is. Just because latency is harder to measure, that doesn't mean it isn't very important, especially when it builds up with lots of other 'tiny' delays to make one big delay.

    --
    Why OpalCalc is the best Windows calc
    1. Re:Latency again by mtaht · · Score: 3, Insightful

      "The bulk of Internet traffic in the US these days is streaming video. For that you need big bandwidth and big buffers, not low latency. " Emphatically not true. ESPECIALLY for streaming video, you need a functioning feedback mechanism (tcp acks or ECN or some other mechanism) to slow down the video periodically, so that it *doesn't* overflow what buffers you have, and catastrophically drop all the packets in the queue, resulting in stuttering video.

    2. Re:Latency again by rcpitt · · Score: 3, Informative
      I deal with streaming video daily - from a producer, distributor and support point of view.

      "Why does the web site load so slowly?" is the classic question - caused in many cases by the "eagleholic" having 4 live eagle nest video streams running in one window while trying to post observations and screencaps to the web site in another.

      Believe me - there is ample reason to deal with the problem as most of today's home networks are used for more than just one thing at a time. Mom is watching video, sis is uploading pictures of her party to Facebook, son is playing online games and dad is trying to listen to streaming audio - and NOTHING is working correctly despite the fact that this is a trivial load for even a T1 (1.45Mbps) let alone today's high-speed cable (30Mbps down and 5Mbps up). We used to run 30+ modems and web sites and email and all manner of stuff over bonded 56K ISDN lines for pity sake - and we got better latency than the links today.

      What's the problem? The latency for the "twitch" game packets has gone from 10ms to 4000ms or more - and the isochronos audio stream is jerky because it's bandwidth starved and the upload takes forever because the ACKs from FB can't get through the incoming video dump from YouTube (with its fast start window pushed from default 3 to 11 or 12) and by the time the video is half over, the link to YouTube has dropped because it took 30 seconds or more for the buffer to drain after the first push and the link had timed out.

      That's the problem - you need low latency for some things at the same time you need high throughput for others - and it is possible and can be done - and IS done if things are tuned correctly. But correctly tuning the use of buffers is an art today, not a science - and the ever-changing (by 3-4 orders of magnitude) needs of today's end-point routers has pushed the limits of what AQM (automated queue management) algorithms are currently available, even if they're turned on (which in most cases they're not it seems)

      --
      Been there, done that, paid for the T-shirt
      and didn't get it
  16. Re:Two orders of magnitude! ? by mtaht · · Score: 5, Informative

    The core work where we saw latency under load drop by 2 orders of magnitude was in the wireless driver stack on Linux. Examples were the iwl driver (130ms to ~2), and the ath9k driver ( > 200ms to ~d) (and these numbers were for GOOD connections, at high wifi rates. You can get 3 orders of magnitude improvement if you are on a slow wifi connection.) There's a new rate sensitive algorithm for wireless (eBDP) that we are trying in this kernel tree. It's not fully baked yet. 802.11n wireless package aggregation is HARD. That said there's bloat in all the other wired drivers too. We are doing far too much uncontrolled buffering in the kernel - specifically the dma tx ring on many devices - for slower networks. As one example, A gigE interface, connnected to a 3Mbit cable modem - does bad, subtle, things to the stack.

  17. Re:Solution: Use a proper protocol (aka ISO) by tepples · · Score: 3, Interesting

    You further penalize the endpoint by cutting it down to 56k modem speeds for a month if you see it making too many short-lived (<5s) connections to multiple IPs and declaring them isoch. By definition, any such connections are an abuse of the standard.

    But isn't that exactly how channel surfing would work on IP-based television?

  18. Keeping big buffers but managing them better by Geoff-with-a-G · · Score: 2
    That is indeed part of the solution Jim Gettys suggests - Active Queue Management or Random Early Detect.

    The first problem is that a ton of transit systems on the Internet (like indeed a ton of systems everywhere) are effectively running the default behaviors in this respect, with no special tuning. That means FIFO with whatever queue size is available.

    The second is that even if all the ISP operators decided to fix this, "QoS stuff" has the potential to run afoul of Network Neutrality. The current thinking is that they shouldn't be discriminating between "bulk/throughput" packets and others. Some would suggest discrimination between traffic types is okay, so long as you don't discriminate between traffic sources (ie prioritize all VoIP, but don't let Comcast give Comcast VoIP preference over Vonage VoIP) but implementation would be tricky - all too easy to implement a "vendor neutral" policy that coincidentally doesn't seem to identify Vonage's traffic quite right.

    The simplest and most neutral solution to all this is simply to decrease the buffer size in those big default FIFO queues. Even the bulk/throughput packets won't really suffer from that - TCP is specifically designed to have packets dropped in a timely manner, rather than held in a queue for a long time. One of the problem behaviors that Jim identifies is that if your real RTT to say, a server in California, is 40 msec, but there's 4 seconds worth of delay in the buffers. TCP will send a window full of data (let's say 64K) then wait for a reply for 40, 80, maybe 120 msec. Not getting it, he sends the whole window again. And again. And again. Finally an ACK squeezes through, and the process begins again. Instead of shrinking his transmission size, he does the opposite - he sends big multiples of every packet, making the congestion worse.

  19. Every packet is sacred by RevWaldo · · Score: 4, Funny

    Every packet is sacred.
    Every packet is great.
    If a packet is wasted,
    TCP gets quite irate.

    Let the heathen drop theirs
    When their RAM is spent.
    TCP shall make them pay for
    Each packet that can't be sent.

    Every packet is wanted.
    To this we are sworn.
    From real-time data from CERN
    To the filthiest of porn.

    Every packet is sacred.
    Every packet is great.
    If a packet is wasted,
    TCP gets quite irate.

    .

    --
    Prisencolinensinainciusol. Ol Rait!
  20. Buffer bloat is (not) an illusion... by WaffleMonster · · Score: 2
    For home users with a linux router set an HTB queue /w maximum egress rate to modem a little less than than your sustained upstream rate. At least this worked for me... never had problems with saturated upstream causing huge lags after doing this.

    After reading this guys buffer bloat rant I largly agree with him with some exceptions:

    1. What does multiple TCP sessions have to do with circumvention of congestion avoidance? TCP congestion avoidance needs to work with lots and lots of TCP sessions at once not just one or two. HTTP 1.1 sessions need NOT be short lived. I don't see why a large number of TCP sessions can't all be subject to congestion avoidance...responding individually to the conditions they see? How does this work to effectivly bypass congestion avoidance? I've seen this talking point in a few places but noone has ever explained WHY this is so. I can see an argument based on a static suboptimal initial congestion window but HTTP 1.1 supports pipelining...

    2. Two connections per server is not sufficient for browsers. TCP is a stream protocol with head of line blocking... High latency links will never use the available bandwidth properly unless they either use lots of sessions or start with massive windows which is not good for congestion. There is also a problem of ordering dependancies of resource requests within the web content. Without lots of concurrent fetches the user gets to wait longer for page loads. The presentation sounded to me like someone either not understanding necessary details of TCP and higher layer considerations or trying to have their cake and eat it too.

    Lastly we don't need to replace HTTP - we need to replace TCP... HTTP over SCTP would be a much more significant improvement than any reasonable change to HTTP. No matter what you do to HTTP you still have to live with the underlying transports limitations!

    1. Re:Buffer bloat is (not) an illusion... by jg · · Score: 2

      Re: 1. I've always thought that the congestion window to the same end-point should be shared: but that's not the way TCP implementations work, and wishing they worked that way won't make the problem go away. And, as I've shown, bufferbloat is not a TCP phenomena in any case.

      Re: 2. HTTP is a lousy protocol in and of itself, and having to do it on top of TCP makes it yet harder. It is the fact that HTTP is so ugly that makes so much else difficult. And I disagree with your claim that high latency links won't use the bandwidth; in fact, lots of sessions is just making things harder. You can read our HTTP/1.1 Sigcomm performance paper.

      I'll be writing more on this topic this week.

      And we need to replace HTTP, and something other than TCP would be highly desirable. Personally, I'm much more fond of CCNx than any IP based transport.

  21. Re:what it is by ischorr · · Score: 2

    And you know, in my experience that's where one of the real problems - and one of the most commonly undiagnosed problems - exists. In nearly 99% of networks I've looked at where buffer overflows were occurring and drops were happening, network admins were not only unaware of the severity of packet drops and didn't understand the impact this was having on their *critical* workloads, they had no idea how to even look for it.

  22. Re:what it is by mtaht · · Score: 3, Informative

    re:"Interesting problem for dd-wrt"

    Agreed.
    We are throwing efforts at both the mainline kernel and openwrt.
    Openwrt is foundational for dd-wrt and several other (commercial) distributions of Linux on the router. I have a large set of debloated routers already, I'm just awaiting further work on the eBDP algorithm to make better....

    http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700

    re: "using pings"
    httpping is a much saner approach than ping, in many cases. Get it from:

    http://www.vanheusden.com/httping/

    re: RED & AQM

    SFB and CHOKEe are in the debloat-testing kernel, as is eBDP.
    RED 93 isn't going to work. nRED may. Experimentation and scripts highly desired. See the bloat and bloat-devel mailing lists for discussions.

    Also:

    http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle

    Also:

    I've seen some VERY interesting behavior with tcp vegas over bloated connections.

    http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_TCP_cubic_vs_TCP_vegas

  23. Corrected Git URL by JamieF · · Score: 3, Informative

    The link in the /. story to debloat-testing should go here: git://git.infradead.org/debloat-testing.git.

    git:gitinfradeadorgdebloat-testinggit is not a valid URL.