Slashdot Mirror
Bufferbloat — the Submarine That's Sinking the Net
gottabeme writes "Jim Gettys, one of the original X Window System developers and editor of the HTTP/1.1 spec, has posted a series of articles on his blog detailing his research on the relatively unknown problem of bufferbloat. Bufferbloat is affecting the entire Internet, slowly worsening as RAM prices drop and buffers enlarge, and is causing latency and jitter to spike, especially for home broadband users. Unchecked, this problem may continue to deteriorate the usability of interactive applications like VOIP and gaming, and being so widespread, will take years of engineering and education efforts to resolve. Being like 'frogs in heating water,' few people are even aware of the problem. Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"
http://en.wikipedia.org/wiki/X_Window_System
Latency is bad? Bigger buffers = more latency?
I'm so glad the term has been defined so that I know what the hell we're talking about here. Oh wait, no it hasn't.
Okay, then I'll RTFA. Oh wait, two screens worth of text later and it still hasn't.
I'd like to change the topic now to the submarine that's sinking the English language: jargonbloat.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
He's Jim Gettys, not Getty.
Jim Getty, one of the original X Window System developers and editor of the HTTP/1.1 spec
I'd murder four people just to have TTY in my name. Five if I could capitalize them, and postfix with a number. I'd name my son Dev.
You'd get a business card with something like Dev GeTTY1, Armadillo Avenue 64, Seattle, Washington
8 of 13 people found this answer helpful. Did you?
Just start RTFAing: "In my last post I outlined the general bufferbloat problem."
Follow the link:
"Each of these initial experiments were been designed to clearly demonstrate a now very common problem: excessive buffering in a network path. I call this bufferbloat
I read TFA and I'm not seeing the problem. He can't duplicate this issue unless he maxes out his connection and then his latency goes to hell. No shit Sherlock, that's what happens when your pipe is full and the packets have to wait in the queue to be transmitted. Am I stupid or could he avoid this issue entirely by using QoS and/or rate-limiting his connection to some amount <100% of it's maximum throughout? I have QoS at the office that keeps our connection from pegging (it's limited to around 75% on the download and 90% on upload) and have never once encountered an issue with latency or jitter. At home I only throttle the upload (to 90% of maximum) and have successfully ran VPNs, bittorrent uploads and VoIP calls all at the same time without any headaches.
Really, what's the problem here?
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/
Red Leader Standing By!
RAM is cheap.
High speed uplink is not cheap.
Peering agreements are manipulative, expensive, and sometimes extortionate.
So...
The poorly designed, poorly peered, under allocated back haul links can't handle the traffic that routers want to push through them -- but since RAM is cheap, operators just add RAM to the buffers so that when those back-haul lines slow down for a second the packets can get pushed through.
And we're blaming the buffer for the problem?
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
Easy, name him Devone = Dev1
Several issues:
1. People who aren't networking engineers don't know about QoS, or don't know/want to know how to configure it.
2. QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
3. How do you determine the maximum throughput? It's not necessarily the official line's speed. The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is. You're proposing a regression to having to tell the system by hand.
4. QoS is most effective on stuff you're sending, but in the current consumer-oriented internet most people download a lot more than they upload.
The problem is that maxing your connection from one site is causing everything else you do on your connection to be delayed / dropped as well, because it ends up queued behind anything that got buffered mid-transit from the first site. With a smaller buffer the large transfer would start to drop packets and back off sooner, allowing packets from other sources to "hop the queue".
What Jim is saying is that TCP flows try to train themselves to the dynamically available bandwidth, such that there is a minimum of dropped packets, retransmits, etc.
But in order for TCP to do this, packets must be dropped _fast_.
When TCP was designed, the assumptions about the price of ram (and thus, the amount of onboard memory in all the devices in the virtual circuit) were different -- namely, buffers were going to be smaller, fill up faster, and send "i'm full" messages backwards much sooner.
What the experimentation has determined is that many network devices will buffer 1 megabyte or MORE of traffic before finally dropping something and telling the tcp originator to slow down. And yet with a 1 meg buffer and a rate of 1 megabyte per second.. it will take 1 second simply to drain the buffer.
The pervasive presence of large buffers all along the tcp vc, and the non-speified or tail-drop drop behavior of these large queues means that tcp's ability to rate limit is effectively nullified, and in situations where the link is highly utilized, many degenerate behaviors occur, such that the overall link has extremely high latency, and that bulk traffic will cause interesting traffic to be randomly dropped.
Personally, I used pf/altQ on openBSD to try and manage this somewhat.. but its a dicey business.
My opinions are my own, and do not necessarily represent those of my employer.
Really, what's the problem here?
You really don't see the problem? How can you be so naive. Maybe you're new to this. All signs show to the fact there is a problem.
Of course the problem is not obvious. The article itself says it'll completely surprise us. They know we won't believe it at first. But that's why we must believe it, or else it's Armageddon.
Would you risk an Armageddon, because of your inability to understand and see?
And that's, in short, why we must attack Iraq.
Wait, what were we talking about :P?
As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?
The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small
He's not arguing against application-level caching. He's saying that too much caching at the IP layer is confusing TCP's algorithm for deciding how fast the link between two points is. This in turn causes massive variability in how fast the data can be downloaded; or in your terms, how fast the video can be buffered (and, in fact, how much buffer the video player needs).
Rgasuya aata! : I have been coding Perl and cannot tell where my fingers are now!
After reading TFSeries, the problem is excessive buffering (as in 1-10 or more seconds worth of data) screwing up TCP/IP's automatic bandwidth detection. QoS helps a little bit by getting the important packets (especially ACKs) through, but high-bandwidth TCP connections are still going nuts when they hit a slower link with excessive buffering.
And one of the major offenders is Linux commonly defaulting to a txqueuelen of 1000.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
Buffering of packets, "network path" can't be refering to anything else.
but in the current consumer-oriented internet most people download a lot more than they upload.
Because the current consumer infrastructure forces it onto you. I would happily seed my torrents all year long, except I only have 1/12th the uploading bandwidth as I have for downloading. Since I need some of it for other things, uploading becomes impractical.
It's easy to blame the consumer, but there's a certain model imposed on him from the start.
Seven puppies were harmed during the making of this post.
If you put a frog in a pot of water and slowly raise the temperature it will try to jump out before the water reaches a temperature that is fatal to the frog.
Best Slashdot Co
Packets.
Mod parent up. Half way down the comments, and this is the first post that actually explains why 'bufferbloat' is something I should care about.
I am TheRaven on Soylent News
You haven't read the article (or the many others around on LWN.net on the same topic). Basically, large buffers in networking gear, from DSL routers on your home network through to ISP's, mean that interactivity is *shite*. You might download Gb's but in terms of interactive applications it's useless and we're facing ever-increasing latency and problems through wanting to cope too much with errors and delays (e.g. huge buffers to keep resending instead of just letting packets drop and having TCP sort it out by retransmission). TCP windows never shrink because errors and buffered and retried so much from intermediate devices that any sort of window scaling is worthless because it doesn't *see* any packet-loss.
Same devices, smaller buffers, everything works fine and "faster" / "more responsive" all around. It actually would *save* money on new devices because you don't need some huge artificial buffer, you can just drop the occasional packet. But the problem is so deeply embedded into run-of-the-mill hardware that it's almost impossible to escape at the moment and thus EVERYONE from large businesses to home users are running on a completely sub-optimal setup because of it. Almost every networking device made in the last few years has buffers so large that they cause problems with interactivity, bandwidth control, QoS, etc. It's NOT just that a "faster connection" solves the problem - we are getting a percentage of optimal service that's steadily decreasing as buffers increase even though we're improving all the time. That's the point. And it *is* caused by memory prices because memory is so cheap that a huge thoughtless buffer costs no more than a tiny, thought-out buffer.
So naturally, I instantly get modded down.
QoS does generally not work beyond the first hop. Your provider most likely will drop any QoS data. Some providers wil try to make their own QoS systems (e.g. to show a low ping). However if the lantency has a great variance due to all kind of buffers any algoritm will get the bandthwidt wrong.
QoS based on network types will get it wrong. For pure browsing /downloading it is relatively simple, but for VPN Encrypted skype udp traffic, game data it will never be optimal.
And as the blogger wrote, there is not a simple solution, because the end user has a "dad the internet is slow today" mentality. Couple that with a "reinstall your windows" helldesk and the solution becomes VERY HARD.
Given that most traffic on a domestic connection is incoming, that doesn't help much.
It's not that hard to shape downstream traffic. Take a Linux router with two ethernet cards. eth0 is the LAN and eth1 is the internet. You shape eth0 with a maximum throughput of 75%-80% of your line speed. All of the downstream traffic has to go out on that interface so that's your opportunity to shape it. I do this at work and successfully share a 3.0mbit/s connection with 60+ employees. We use latency sensitive services like VoIP and RDP alongside streaming video and other large downloads without any major hassles. It stinks to lose some of your bandwidth because of this (you have to shape it to a number less than 100% of your line speed, otherwise buffering occurs at your ISP and your QoS scheme is defeated) but I'll take responsiveness over throughout any day of the week.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
The issue is that many IP stacks do not handle ECN (Explicit Congestion Notification) and only know when the link is saturated by packet loss. Huge buffers hide the problem. A solution is to use ECN, that's what it's for http://en.wikipedia.org/wiki/Explicit_Congestion_Notification
This is an excellent explanation of what issues are happening here. I can clearly see that this is an issue, and the problem is something that over time will impact everybody.
The problem is really focused on trying to deal with differences in bandwidth between computers... always a problem but in this case trying to match up slow connections with fast connections is particularly difficult. Since memory is cheap, a 1 GB buffer certainly can be found in some devices now and perhaps much more. I don't see this example as being really too far off the mark in the near future.... which is the point being raised and why buffer bloat is such a big deal.
More to the point, some of the complaints that triggered the "quality of service" debate are rooted in this problem. As mentioned in the original article triggering this whole slashdot thread, setting up "quality of service" priorities only creates multiple buffer queues.... it doesn't solve the problem of the monster queue to begin with. That is why the author of the blog post suggests that the debate over network neutrality is not based upon the real problem that is facing network engineering and why it is a political solution in search of a problem.
It takes awhile to "grok" this problem, but once you do it becomes obvious why this is such a huge deal.
People who aren't networking engineers don't know about QoS, or don't know/want to know how to configure it.
*shrug*, not my problem :)
QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
The issue is always going to be there. Pegged connection == FIFO queuing, absent some sort of QoS scheme.
How do you determine the maximum throughput? It's not necessarily the official line's speed.
If you aren't getting the line speed you paid for then you need to find another ISP.
The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is
And it does, even with QoS. All you do with QoS is force the buffering to happen on equipment that you control rather than equipment your ISP controls. In this manner you can ensure that time sensitive packets (interactive VPNs, VoIP, etc.) don't sit in the queue behind someone's Windows Update download.
QoS is most effective on stuff you're sending
It's not really all that difficult to shape downstream traffic. All you need is a router between your internet connection and LAN clients. I've done this for years at my office using the QoS functionality of the Linux kernel. We are located out in the middle of nowhere with T-1s as our only means of connectivity. Sharing a 3.0mbit/s connection with 60+ employees without QoS is virtually impossible if you need to run interactive protocols.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
It's not, really. There are many buffers involved. Some of them will be in the network infrastructure - routers, firewalls - that users have no control over. A lot more are in devices they control, but that don't allow configuration of such low-level parameters. Cable modems, home routers, access points.
OK, not on the (intentionally ridiculous) scale used in the example, but people are doing something very similar to what you describe, even though they "can't do that". http://slashdot.org/article.pl?sid=10/11/26/1729218
# cat
Damn, my RAM is full of llamas.
2. QoS used that way is a hack to work around an issue that doesn't have to be there in the first place
3. How do you determine the maximum throughput? It's not necessarily the official line's speed. The nice thing about TCP is that it's supposed to figure out on its own how much bandwidth there is. You're proposing a regression to having to tell the system by hand.
4. QoS is most effective on stuff you're sending, but in the current consumer-oriented internet most people download a lot more than they upload.
While the Internet in-theory is beautiful, our modern implementation really is a series of layered hacks. And the solution to Bufferbloat is going to be another hack. You're crazy if you think that the solution to the Bufferbloat 'problem' is going to be some fundamental redesign of the TCP protocol (how would you force 10 people to use it?), or the total re-architecture of millions of consumer devices to remove buffering. You're also crazy if you think the ISPs and backbone providers are going to stand by while this thing kills the Internet.
So the question is: which hack will it be? The GP poster already identified one that works well enough --- using QoS to control flows. Your final objection about content providers stressing connections is the real one. But there's some probably a good hack to deal with it --- or more likely a series of hacks, some at the content providers themselves (e.g., Netflix), some in the backbone, and some at your ISP. It won't be elegant, but it will keep this problem from ever becoming anything more than a few cranky blog posts.
A big part of the problem is customers complain mightily when network devices congest (drop packets). Congestion is easily monitored. Additional latency is much harder to measure and most customers are less likely to notice it.
Less support calls == better margins
For some devices, it's clear that the engineers that built it don't understand networking very well and that's where the problem crops up. For others, it reduces pain on the field support organization so, in some ways, customers are doing it to themselves.
let me summarize the problem that is being observed: On a given interface, if you have more buffer memory than is needed as packet buffer on the transmit side, it can induce latency. As an example, consider a 1Mb/s link. If you want to have a peak of .1s latency added by buffering at high load, then you want 1Mb*.1=12,500 bytes of buffer. If you have 1MB of buffer, then you have 8 seconds of buffer, therefore triggering the "buffer bloat" issue. Part of the problem is that buffer size would be set based on the top speed a piece of hardware could drive, i.e. if you want a 1Gb/s interface to be able to buffer .1s, then you use it at 100Mb/s, then it has 1s worth of buffer. In most home deployments where you have a router that may have a 1Gbps upstream, maybe 4 100Mb/s physical connections, and a 54Mbps wireless router, you probably have a shared buffer for all the interfaces. The result of this is that when using the 54Mb/s wireless, you can easily have the buffer over saturated, while the buffer size may be just right for the 100Mb/s interfaces.
What is the solution to this? Realistically, the alternative is to drop packets that have resided in the buffer longer than a configured amount of time, which causes it's own performance issues. Net result: TCP would slowdown for a period of time, but would speed up again resulting in a sawtooth behavior. This would result in periodic issues with other protocols as well, i.e. VOIP would have dropped packets every time TCP ramps up again, etc.
Solution: Don't download porn when you are trying to do VOIP calls.
Latency is bad? Bigger buffers = more latency?
Buffers increasing latency is not exactly a new phenomena. Its been observed and taken into design considerations for quite some time. For example back-in-the-day serial chips essentially had a buffer of one byte. The CPU fed data one byte at a time as the buffer became available and latency was pretty low since data was immediately transmitted. As more capable serial chips became available larger buffers were introduced. A newer chip may have a larger buffer but it may also not transmit data as soon as it has a single byte. It was common to have two programmable thresholds to begin a data transmission, (1) when a certain amount of data has accumulated in the buffer or (2) when a certain amount of time has elapsed. So if a "packet" to transmit was small enough it may sit in the buffer until (2), hence more latency with larger buffers. Software that cared generally began to issue flush commands to cause anything in the buffer to be sent immediately.
Network cards and/or the operating system may try to similarly accumulate data before transmitting a packet.
I discovered this series of blog posts about 2 months ago, when he accidentally published one of his blog posts prematurely. I started reading it and followed the links and saw that this was a like a sleuth tale-if I had started reading this with his very first blog on the topic I would have had no idea where he was going with this. Now as to why this contribution by Jim Gettys does the world a great service:
Hats of to Jim Gettys. Thanks for your service.
Within a router it would be the actual IP data packets that are being buffered. A standard router has a number of network interfaces (token ring, ethernet, wireless, ISDN, whatever....) . Each network interface is piece of hardware that is memory mapped to allow the CPU to send and receive packets. Each hardware device also has a small online memory buffer to store the most recently received or transmitted addressed data packets (every protocol layer down to the MAC source and destination address, IP address, sequence number as well as the data). Depending on system and packet size, that could be anything between 1 and 16.
The usual implementation was to have each hardware device generate an interrupt whenever some data had been received and to transfer the data from internal memory to a common pool in system RAM. The latter was divided up into pre-allocated blocks with a few large blocks (>1000 bytes) and many smaller blocks (512 bytes). Some one might have done a statistical analysis onto the theoretical distribution of the size of packet data being sent through the network. Most of the time this worked out, but there were problems that happened some times. If all the smaller blocks were in use, then the larger blocks were used instead. For efficiency, these wouldn't be transferred through the system, until all the entire block has been filled up with data, so if you have a stream of 128 byte packets, it would take eight of them before the larger block was filled. For some systems, packet sizes were enhanced to 4K or even 8K. A constant high-speed stream of small packets was most likely to do this.
Also, many of the hardware devices would simply overwrite the contents of one unprocessed data packet with the contents of the latest arrival if it wasn't collected fast enough. So that could really mess up sequence numbers.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
In Canada references to the Bank of Canada in news stories have lately been abbreviated to BOC.
That's because unlike "Federal Reserve" and "Federal Express", "Bank of Canada" doesn't have a snappy, pronounceable contraction (Fed and FedEx respectively).
When I read "BOC to raise interest rates" I always wonder why the Blue Oyster Cult is doing that.
No, that'd be "BÖC to raise interest rates". BÖC was probably the first rock band to incorporate a gratuitous diaeresis in its name. The root problem here is that BOC's dis am bigger than yours.
There is no 'bufferbloat because RAM is getting cheaper'. What he is seeing is what happens when you want to saturate your link. ... ...you get either a buffered or a dropped packet.
Yes, and if a link is saturated, there should be packet drops, which TCP senses, then automatically throttles back to reduce the required bandwidth and avoid saturation. But what is happening, is that these huge buffers are holding packets that would otherwise be dropped, and so TCP doesn't get the feedback it needs to detect saturation. So it continues transmitting at full speed, believing it has uncongested pipes, which in turn continues to fill the buffers, and so on.
Because of the buffers, most of these packets are eventually getting through, but maybe in seconds instead of tens or low hundreds of milliseconds. Thus you're getting huge latency.
Jitter is caused by the buffers eventually filling or TCP timing out (registering packet loss), dropping the rate for a little bit, the buffers draining, then TCP upping the rate again as the buffers refill, hiding the saturation, until they're full again. Rinse and repeat.
It's related to the "bloat" of buffering (due to the increasing affordability of RAM and the "more of a good thing must be better than a little of a good thing - QED" mindset) because, if the size of the buffer is kept below a certain point related to the pipe bandwidth and number of traffic streams, it tends to act just as a temporary "buffer" against spikes in the traffic (the intention of buffering), and can't cause the scenario above, having insufficient capacity to overload the bandwidth just from buffer contents alone. Above this threshold, the latency issues and back-and-forth thrashing noted above occurs. The bigger the buffers, the worse the effect.
And it's not just a "well, keep your traffic below x mbit if you're on ADSL2" issue, because it happens anywhere a high capacity pipe interfaces with a low capacity or otherwise congested (of any capacity) pipe. This might be your ISP's backbone which is getting hit by several thousand people downloading the latest WOW patch simultaneously, causing your 300kbps Skype call to go to hell through latency and jitter. If the ISP's equipment had smaller buffers, the servers would be throttling back as packet loss occurred. You'd probably still be losing packets, but they'd be detected and re-transmitted pretty quickly and you possibly wouldn't notice the latency or have jitter.
What he is seeing is what happens when you want to saturate your link.
So, no, what you get with appropriate buffers is your TCP connection moderating itself to the appropriate link capacity and availability, and latency remaining approximately the same (relative to what you're seeing in bufferbloat, but worse than an uncongested link, obviously).
With bufferbloat, your bandwidth appears to remain about the same, but your latency balloons massively and you get jitter effects as above.
This is been enlightening. I've suffered very similar problems at home, but instead of figuring out the problem I replaced the hardware... After reading TFA all fits perfectly, I had occasional "chokes" - sites would take ages to load, ping's wouldn't return from my Wi-Fi router, DNS queries took ages. All while downloading a big file or something. But what's significant the throughput would stay high. It was strange as hell - high thoughput (an ongoing large transfer [but not large enough to saturate connection]) but other things choke.
Larger buffers do not really decrease congestion as far as TCP is concerned: With a large buffer TCP will simply send more/faster, untill the buffer overflows. The congestion will simply manifest a tiny bit later, but much, much severe.
How much bandwidth can I have, though? Take the link between my desktop and a Slashdot server; is the correct answer "1GBit/s, no more" (speed of my network card)? Is is "20MBit/s, no more" (speed of my current Internet connection)? Is it "0.5MBit/s, no more" (my fair share of this office's Internet connection)? In practice, you need the answer to change rapidly, depending on network conditions - maybe I can have the full 20MBit/s if no-one else is using the Internet, maybe I should slow down briefly while someone else handles their e-mail.
TCP doesn't slam the network; it starts off slowly (TCP slow start currently sends just two packets initially), and gradually ramps up as it finds that packets aren't dropped. When packet drop happens, it realises that it's pushing too hard, and drops back. If there's been no packet drop for a while, it goes back to trying to ramp up. RFC 5681 talks about the gory details. It's possible (bar idiots with firewalls that block it) to use ECN (explicit congestion notification) instead of packet drop to indicate congestion, but the presence of people who think that ECN-enabled packets should be dropped (regardless of whether congestion has happened) means that you can't implement ECN on the wider Internet.
This works well in practice, given sane buffers; it dynamically shares the link bandwidth, without overflowing it. Bufferbloat destroys this, because TCP no longer gets the feedback it expects until the latency is immense. As a result, instead of sending typically 20MBit/s (assuming I'm the only user of the connection), and occasionally trying 20.01MBit/s, my TCP stack tries 20.01MBit/s, finds it works (thanks to the queue), speeds up to 20.10MBit/s, and still no failure, until it's trying to send at (say) 25MBit/s over a 20MBit/s bottleneck. Then packet loss kicks in, and brings it back down to 20MBit/s, but now the link latency is 5 seconds, not 5 milliseconds.
I appear to have a blog. Odd.
I don't think buffering is the real problem. Buffering can help with throughput especially TCP throughput.
:).
For example if there is a brief burst of packets higher than a router's outbound connection bandwidth supports, the router has two options:
a) buffering or queuing up the packets (assuming enough buffer space), till the output queue empties.
b) dropping the packets.
UDP latency doesn't go up if UDP packets are dropped - since there are no retransmissions.
But if TCP packets are dropped too often just because of bursts (from other connections), the speed drops significantly and the latency goes up a lot too - because the relevant parties have to wait for acks and retransmissions and the TCP window size also decreases. If the TCP packets are buffered, the TCP transmitter won't slow it's transmission rate drastically (it will still slow a bit when waiting for acks), but there would be a brief increase in latency for the period the packets are being buffered.
Trouble is if it's not a brief burst, it means you're just delaying the inevitable packet drops, meanwhile the sustained delay becomes more noticeable.
What the author doesn't seem to realize is the upstream network providers should buffer (within reason - even up to a few seconds worth) - because they usually don't know which of your packets are more important to you, and thus which packets to drop first.
The solution is for the network providers to have enough internal bandwidth so that THEIR buffers rarely start to fill up, and there is minimal packet loss, AND for the user to do traffic shaping and policying at their connection (and the servers too).
Say you have a 10Mbps connection, and the ISP actually does give you a full 10Mbps. If you download from a fast site, the packets queue up when they reach your 10Mbps connection before dropping and finally reaching and equilibrium at 10Mbps. But meanwhile your interactive ssh or game connections suffer from increased latency since they share the queue - as mentioned before the ISP doesn't know which packets/connections are more important to you (it can guess of course and favour small packets, ACKs, DNS; or you can give QOS hints, but most ISPs don't bother).
To prevent this, you get your modem/firewall to "shape" your traffic at say 9.5Mbps (leave enough headroom for the initial TCP connection bursts) and choose which packets you want to drop when it goes past 9.5Mbps (for instance drop http packets first instead of your interactive or VOIP stuff, or the other way round if http traffic is more important to you for some reason). This way you control what happens AND the buffers never fill up at the ISP end since the traffic doesn't even hit 10Mbps (if their buffers don't fill up, there is no increase in latency). This also applies to your outbound connections - if your upload bandwidth is 5Mbps, shape it at 4.5Mbps or so.
The narrowest part of the straw controls the flow. If the ISP is the narrowest part of the straw - the buffering and shaping is up to them. If your system is the narrowest part of the straw - it controls the flow.
If there's something wrong and somehow there's a 1Mbps bottleneck in the ISP for ALL your traffic, that bottleneck will control the flow - since it will drop packets before it reaches 9.5Mbps point where you start dropping packets - in which case the ISP is cheating you and giving you a 1Mbps connection (note though that if the 1Mbps bottleneck is only for some of the traffic your other traffic will be fine ).
Note: I'm assuming the case of sustained TCP connections, and lower UDP traffic speeds. This all doesn't work if some site is blasting UDP packets at 100Mbps to you[1].
[1] The UDP blasting could actually be a way of bypassing certain ISP throttling methods - sending lots of packets, and coping with packet loss by using forward error correction and some feedback. Fortunately not many are doing this yet
As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?
The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small
Well, I might have a way, but it only works on a semi spherical planet in a vacuum.
TCP contains some of the most incredible heuristic algorithms I've ever seen. Each algorithm, like Slow Start, RTT Estimation, SACK, etc. are relatively simple but together they work incredibly well at keeping data flowing across heterogeneous networks. They work so well that I've seen TCP overcome broken ethernet drivers and make them appear to work. Unfortunately, as someone who use to look at TCP traces for a living, I can tell you it can be really hard to work backwards from packet traces to figure out what is going on in the TCP/IP stack because there can be so much going on at the same time. This means that Wireshark in the hands of a weekend-hacker can easily lead to erroneous conclusions. If you follow this link and go to section 14.5 Random Early Detection (RED) you can see that the issue is already known and there are already solutions to mitigate the problem.
Relax and take a deep breath. Now you can move on to something more important......... like where you're going to spend your eternity
"Meaningless!, Meaningless!" says the Teacher. "Utterly meaningless!"
No the problem is that the self regulation built into TCP is delayed by buffers. The size of the pipe is only relevant in how long it takes to fill. (And like a garage, it will always fill) If you read the articles (Have a few hours free) he shows how to find the issue on a gigabit link. That is even fast in Korea.
The solution is for the network providers to have enough internal bandwidth so that THEIR buffers rarely start to fill up, and there is minimal packet loss, AND for the user to do traffic shaping and policying at their connection (and the servers too).
There is no such thing a enough bandwidth. It will always fill. You need to allow the built in mechanisms to recognize when it is full. And while I agree that traffic shaping is nice (and easy with firewalls like m0n0wall) it is not in most home routers. Besides, expecting most users to do this properly when they can not even patch their systems is folly at best.
Another possible fix is to shorten the TTL on packets, where the packets are discarded if the route has too much delay in it.
Umm, that's not what the TTL does. The TTL gets decremented by 1 for every router that touches the packet. If it hits zero without reaching it's final destination it's dropped. It has nothing to do with latency. Traceroute works by sending out packets with increasing TTLs (i.e: the first packet has a TTL of 1, the second has a TTL of 2, and so on) and looking at the returning ICMP time exceeded packets.
Or use UDP for streaming applications rather than TCP, like it was intended, and let the end points buffer the difference.
This. UDP is tailor made for streaming applications. Nobody is going to notice if their video feed drops a frame here or there. They are going to notice if it freezes because a TCP connection is waiting for a packet retransmit.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
You just perfectly described the procedure for shapeing upstream traffic.
You are shaping the "upstream" from your LAN interface. As I described it:
Linux router eth1 == internet
Linux router eth0 == lan
Path of packet from the internet: the cloud -> your ISP -> eth1 -> (nat/routing occurs here), eth0 -> your LAN PC
If you shape it before it goes out on your LAN the TCP stack of your clients will respond accordingly and the overall bandwidth consumption is appropriately limited.
I don't know if wondershaper can do this, as I've always configured my QoS by hand. More flexible that way. What I've described is very easy to set up with iptables, tc and the relevant Linux kernel modules.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
You forgot the tiny little fact that unless one pulls his connection to the limit with a lot of tcp connections, there isn't any problem.
Turn off your bittorrent client while you're playing starcraft online, and the problem disappears.
The post fails to explain what happens in the case of insufficient buffers - and dropped packets : it can take up to 2 minutes for tcp to recover from a single dropped packet (granted - on slow links or long distance connections). Would you really feel that interactive response has improved if things work fast 95% of the time, and then your web browser* - for no apparent reason at all - takes 2 minutes to load pages** ?
(yes, I'm an ISP's network engineer. Big buffers or small buffers ? Trust me, you want big)
* the fun thing about webbrowsers is that they open lots of tcp connections, then barely send any packets at all (ie. connection generally closes after 4-5 packets tops - sometimes after 2 packets). If you lose the first packet in a connection, which is quite likely when browsing, the SRTT algorithm has no choice but to wait 2 minutes before retry - guaranteeing the user will have to interfere (ie. "F5"). This results in the massive deterioration of web browsing experience with trivially small packet loss. Unless you've never wondered why internet is near-unusable with 0.1% packet loss on your link, and nothing at all gets through at 1% packet loss. You'd think 99% correctly transmitted packets would translate in 99% of bandwidth available, no ? (in case you have this problem : a simple trick to put everything through a single tcp connection. ssh -D 1025 server_at_work; set up firefox to use socks5 proxy at port localhost:1025. You'll have 40-50% of your link bandwidth available on recent windows or linux)
** Just try, go to a big company that's upgraded to cheap gigabit switches, with tiny buffers. Ask them if, perchance, they've been experiencing sudden "timeouts" all of a sudden. Ask again if they like this.
The way to fix this - not that I'm expecting political interference from large groups of idiots - all large groups are large groups of idiots, because most people are idiots in most subjects - to go in a sensible direction all of a sudden, so "let's get the mob to 'fix' this" doesn't work regardless of good intentions - is not to go with small buffers but to have intelligent queuing algorithms in all devices. Of course, bittorrent will always cause this behavior, because one of two things will happen when bittorrent opens it's 5000 connections
1) either routers slow down bittorrent traffic in favor of http, much better performance, but results in underwear kids who haven't seen sunlight in a year shouting "NET NEUTRALITY !"
2) or they "treat all traffic the same" - and with tcp the one with the most connections "wins" the most bandwidth - meaning if you open 500 connections, your web browser is only going to get 1/500th of the link bandwidth - resulting in abysmal performance
This is what network engineers mean when they're saying bittorrent is destroying network performance. As to what lawyers and politicians mean, yes that's something else, and frankly, I don't care.
There are no layers involved. It's a high level description of something that ignores lots of stuff and "oh shock horror" even gets some details wrong.
Did they ever tell you in school that electrons orbit the nucleus in shells? Oh shit! They lied! Clearly they made no sense at all!
The example is describing what the packets do when you request the file. And yes its exaggerating things and yes it's simplifying things. But that's how you describe things to people who don't know the technical details when those technical details don't really matter to understanding the overall issue.
Obviously you don't get 1GB of packets in the queue then again the queue isn't 1GB in size in the first place anyway - oh look I explicitely mentioned that too but you're clearly to stupid to read.
Getting stuck up on trivial technical details is what your bringing, lots of other people manage to understand how analogies and high level approximations work.
I can see why you're posting as an AC, because you don't understand the difference between an HTTP request and the TCP connection that fulfills it. There is no requesting of packets; the request is made via HTTP, and the receiver then ACKnowledges TCP packets from the server, which may send more quickly than it receives ACKs so as to increase throughput--this then fills buffers and causes cascading latency.
You are compounding the problem by spreading misinformation. Please stop and go educate yourself.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."