Slashdot Mirror
Is Your Internet Connection Free From Bufferbloat? (blogspot.com)
Bufferbloat is that "undesirable latency that comes from a router or other network equipment buffering too much data," according to the site for
an ongoing project trying to address it. Now long-time Slashdot reader mtaht writes:Inside the lede-project, two core new bufferbloat-fighting techniques are poised to enter the linux mainline kernel and thousands of routers -- the first being a fq-codel'd and airtime fair scheduler for wifi, and the second, the new "cake" qdisc, which outperforms fq_codel across the board for shaping inbound and outbound connections.
His submission ends with a question for Slashdot readers. "It's been nearly six years since the start of the bufferbloat project. Have you or has your ISP fixed your bufferbloat yet?"
His submission ends with a question for Slashdot readers. "It's been nearly six years since the start of the bufferbloat project. Have you or has your ISP fixed your bufferbloat yet?"
Say, I wasn't born yesterday. I know very well that if I just disconnect the cables and put the router in the microwave for 45 seconds at 50% power it'll do the same thing.
You are welcome on my lawn.
...is what slows my connection speed down. Fuck, I could have a gigabit connection and would spend 80% of my time waiting for the next version of ad.doubleclick.net, etc. Really? Bufferbloat? I wish!
DSL is unfortuantely the best internet connection in the small town I live in. The upload rate of these connections is really slow, and for large uploads, can saturate the connection. What this translates to in the real world is constant complaints from people about how their internet connection has just died for no good reason. What's happening in 99% of these cases is that some iPad in their house is backing up to iCloud, and bufferbloat from this upload is temporarily wiping out download speeds.
What I did was install the OpenWRT firmware on my TP-Link router, and install the SQM (Smart Queue Management) QoS application on it. This shapes uploads so that bufferbloat is greatly reduced. I tested all of this on DSLReport's Bufferbloat page, and it works great.
If you are referring to the cake article, the baseline latency of the path is ~11ms. It grows to about 250ms under pressure from a tcp transfer with a "normal" cable modem, and to only 16ms or so with cake. See the bar graph... wifi could get much much worse. but we fixed it in the upcoming linux 3.10 release. Not that anybody seems to understand....
As for the wifi article, yes, we have seen 10+ seconds of excess latency in the wifi stack. 1-2 seconds is typical with normal traffic at lower rates, as most protocols time out in that range.
It is entirely probable we've been inside our own filter bubble so long (6 years) we cannot properly communicate with first time readers! some folk explaining the problem... the ietf video shows the benefit from fixing it. https://www.bufferbloat.net/pr... showing the extent: http://www.dslreports.com/spee... you have this entirely backwards: "Buffering can reduce latency, especially under heavy load, by better bandwidth utilization, and allowing faster retransmission of dropped packets. If it is slowing things down, then you should fix the buffering rather than eliminating it." You want enough buffering to absorb bursts, but any more just adds latency. Van Jacobson and kathie nichols calls this distinction good queue and bad queue: https://tools.ietf.org/html/dr... Less buffering (and fair queuing) allows for faster retransmission in particular.
With dislreports and other aggregation tests, the bloat for download and upload may not be symmetric. So the resulting score might not be as good as it looks.
Paying for a commercial connection? Test for this kind of performance daily and scream as soon as it drops. Otherwise why bother to pay so much?
In the United States and other jurisdictions a home 'customer' user is not expected to run a "server" on their paid for Internet connection. Downloads may be finely tuned to low bloat. But upload may have significant bufferbloat, caps and gradual dropout. For financial reasons, of course.
This upload problem may get to be much worse in the future. More and more services push data from "client" devices in the home or office. Camera phone videos, twitch streams, shared google docs and your home automation spyware upend the upload/download assumptions of last-hop telcos. P2P is impacted now. The highly asymmetric buffering of uploads is detectable using protocols like bittorrent that don't have client-server separation.
"You cannot have a General Will unless you have shared experiences. You cannot be fair to people you don't know."
I am not huge on basic web tests, preferring the finer grained results we get from flent. (https://flent.org).
And I totally agree that the trendline is to ever more devices doing ever more stuff randomly when you least desire it. We need to have edge routers AND ISPs ready for this change in traffic patterns.
The article you cited was quite good, although it missed completely the outputs of the ietf aqm working group, of which both I and fred baker are members.
https://tools.ietf.org/wg/aqm/
Buffers are not a problem for latency, the growing internet is. Back in the early 2000s from a particular place in Europe to the west coast in US we averaged over 220ms RTT because it was going up to a satellite, landing in Newark and then traveling over 4 hops to the west coast. Around 2003 when we switched to fiber we got down to about 110ms, with the fiber going via two landing stations on the north shore of Africa, then via France via the Atlantic Ocean to Maine (or dalaware) then over land to the west coast. That was something like 12 to 14 hops.
As the years passed newer and faster fibers were put in place, but also more routers were added to branch the backbone more. Now the same geographic location in Europe to the west coast of US is again at 220ms RTT, because the hop count is around 36. Almost 3 times more routers today than 14 years ago. This is where the latency problem comes from - packet switching in the multitude of hops and MPLS tunnels that you don't even see, not from some imaginative buffers.
Now it is after I got my fiber connection it is all gone. My old *DSL connected at 50/10 mbit(errorfree) but I couldn't get anywhere near that(30mbit at most) and latency were way too high. Only place it caused me some problems was when I worked from home and the Citrix connection as I don't play online games.
Badly managed buffers are a massive problem for latency. Just look at this graph from the article. You see the four ping time measurements on the right? You see how one of them is 100-250ms and the rest are more like 20ms? That's exactly the same link in all cases, but the first measurement has a giant pile of latency introduced purely by poor buffer management.
I'm not going to dismiss the problem you described, because I agree it's a problem. But it makes no sense to worry about 100ms on cross-Atlantic links and yet completely dismiss 200ms right on the first hop.
Thank you for such excellent contributions to Linux kernel :-)
You're right, of course. The trouble is, the latency increases aren't reasonable for common consumer networks under load.
Two speedtests I just did on my lightly-loaded hardwired home network (30Mbsp cable from Time Warner):
With QoS
Without QoS
Throughput is less (rather surprisingly less -- I may want to check some things) with my QoS rules that group connections into individually-throttled categories, but bufferbloat is sane-ish (a brief peak at 250ms was observed, but otherwise under 100ms).
Without QoS, bufferbloat starts at around 1000ms (x10 increase!) and goes up from there.
I'm currently using Shibby's version of Tomato-USB on an overkill dual-core Asus router to accomplish this, though I have used other consumer-ish hardware with reasonable success (including the venerable WRT54G/L/GS) using similar software.
The trick, as I see it, is primarily to ensure that the cable modem (and whatever is directly upstream of it at the head-end) never see enough throughput for their buffers to begin filling by keeping all nearby bottlenecks under my own control.
The other benefit of QoS is that on heavily bandwidth-constrained networks, some tasks can be given higher priorities than other tasks, which is easy when we control the neck of the bottle.
I dated a girl for a bit who had the cheapest Internet she could get: 2Mbps down. Her kids hated it, and web browsing with tablets and phones and laptops was terrible for all of them if anyone was streaming a video (badly) or downloading (slowly). Loud banter over who was "hogging the Internet" and ruining gaming was common, and not unreasonable. It got worse when people would visit. It was really bad.
Best case: They were taking turns using the Internet. In 2014.
After observing this and suggesting she get faster Internet ("no, it's not important to me," she said) I gave her a router with Tomato, did some obvious QoS priorities that were tweaked for that particular situation, and voila: The games worked fine. Web browsing was always quite responsive. Youtube worked (worked meh, but worked), and downloads and BT didn't trash any of the above. Anyone could do whatever they wanted, and the inevitable slowdowns were graceful while responsiveness remained good. The gamer of the house didn't get upset anymore seemingly-randomly.
But that's just one success story. I've been doing tricks like this for over a decade on a myriad of non-enterprise networks, using cheap hardware and thoughtful software.
(Now it's time for someone to pop up and tell me that I've done it all wrong, and that my results are impossible. This always happens on /. when I write about using Tomato and QoS to solve real, practical problems. I'm ready.)
Kid-proof tablet..
Few here can understand anymore. Admittedly it is at the edge of my skill and knowledge level, but I understand enough to respect it. I think most of the real engineers have gone from here.....
Silence is a state of mime.
You've done it all wrong! Those results are impossible!
Silence is a state of mime.
I built my own router because I don't want any of these mass-produced, consumer piece of shit routers with more holes in them than swiss cheese.
Many enterprise APs are pretty good, btw - and while I have not tested the current crop of stuff from eero, and google and so on, I'm pretty sure they've been paying attention to the work. (portions of the make-wifi-fast project were funded both by google and comcast research) So I hope you've been making your stuff great in the first place, and not having to deal with paying off all the technical debt we've been paying off here: https://docs.google.com/docume... But please go test for the things we are testing for and fixing!
It doesn't matter how big or small the buffer is, what matters is why it's filling up to start with.
If you're buffering because of a transient traffic spike or network load, then the buffer helps. If it's constantly filling up and evicting then there's a deeper problem that won't be solved either by using, eliminating, or changing the buffering strategy.
The router needs to manage the bottleneck link, since that's the place all the data gets queued up (in those buffers that are the topic of interest). The router is the only device that has visibility into the amount of data that's in transit "to the internet". Your browser doesn't know that your spouse's/kid's iPhone just decided to upload all new images to the cloud. Nor can your gaming system. Browsers are designed to send the data as fast as possible. Gaming systems are designed to send immediately after you click. It's the responsibility of good networking equipment to regulate all the flows of data so that *everyone* gets good performance. (And while you *could* spend your life optimizing qos rules, the beauty of fq_codel/cake is that they take one parameter (link speed) and they automate all the rest.)
Well... I disagree that the "modern internet does not suffer from this problem." I have seen it at my house, and at measurements at many other places. (If you're only considering FTTH as "modern", I have still seen bufferbloat there...)
The ISP does have an opportunity to control buffering in two places: at both ends of the bottleneck (which is likely to be your cable/DSL/FTTH/etc.) link between your house and their facility.
a) Their "head end" gear might control queues for traffic going *to* you
b) Their Customer Premise Equipment (CPE) also would have the ability to control outgoing queues
If the ISP did both, then no one would have need to coin the term "bufferbloat". But the fact of the matter is that the vast majority of ISPs do *neither*.
Consequently, in late 2016, I believe it's prudent to provide my own solution and use one of the Smart Queue Management solutions (fq_codel, cake) that's available in LEDE/OpenWrt/DD-WRT so that I can get on with useful work.
Buffering can reduce latency, especially under heavy load, by better bandwidth utilization
You have no idea what you're talking about. Buffering is one of the main causes of latency. Ever see a 1,000ms ping? That's not because the speed of light is too slow, that's because there is a backlog of packets in the buffer. With the speed of light through fiber, no one should ever see a ping above 300ms to anywhere in the world. The highest ping I see from Midwest USA to Australia, India, or China is about 220ms.
Buffers are not inherently bad, but "bufferbloat" is because buffers are too large. Too large of buffers actually reduce throughput because TCP takes longer to respond to changes in congestion. Even worse is when bufferbloat starts to get up into the 3second range, yes seconds not milliseconds, TCP treats it as a lost packet and resends the data. I regularly see bloated Linux ISO seeders with 2k-4kms pings resending nearly 50% of their packets, most of which were not actually lost but only highly delayed.
Good anti-bufferbloat AQMs like fq_Codel and Cake increase effective bandwidth, while isolating light traffic from heavy traffic and keeping latency almost idle-link low. Want 10ms pings while paying games and downloading/uploading torrents, I have that already.
The article talks about shaping download, which isn't possible at the endpoint. The traffic is already there and you have to deal with it. Dropping it will create retransmits for TCP and make the problem worse.
Wrong. Dropped packets signal congestion. If you don't signal congestion, the congestion will only get worse. You eventually have to drop a packet. The sooner you drop the packet after congestion has started, the less the congestion will be. The flip side is if you signal too early, you lose effective bandwidth. I shape my download and it has caused my average to go up because it stabilizes the flows.
With normal fifo buffers, once the buffer is full, you get a burst of lost packets. This is much worst than dropping a single packet earlier.
It's pretty much not that at all. It's closer to:
* The provider is selling 100/100 Mbit/s to 20 people with a 1 Gbit/s uplink.
* You hook a WiFi router up to the 100/100 connection.
* While trying to VoIP/Skype on one WiFi device, somebody else starts watching Netflix on another.
* The latency on your WiFi (and thus your VoIP call) jumps up to 50-100ms due to bad buffer management on the WiFi.
* A third device starts trying to sync photos to a backup service, introducing another 100-250ms* of latency by tying up your upstream and generating another badly managed queue on your router.
That's a ton of unnecessary latency being generated right in your own house, by your own gear, and none of it will be helped by the ISP putting in more upstream bandwidth.
...on the topic of which, it would be insanely unnecessary to have 2 Gbit/s of bandwidth for twenty 100 Mbit/s users. You don't need enough bandwidth for every user to max out their connection simultaneously, because that never happens; you only need enough to cover whatever your actual peak traffic is without dropping any packets. When averaging over thousands of customers, this actually works out to needing something around 100 Kbit/s(!) per customer today.
Of course 20 is much less than "thousands" and the traffic profile of 20 customers will be much more peaky than the one of 1000 customers, but I suspect even then that 1000 Mbit/s would be enough to cover twenty 100 Mbit/s connections without dropping any packets. It certainly wouldn't be anywhere near having a "real uplink of 50 Mbit/s".
(*: Probably it wouldn't be this bad with a symmetric 100/100 connection; the graph I linked is for a 140/12 connection, but those are probably more common than symmetric connections anyway.)