Agreed. We are throwing efforts at both the mainline kernel and openwrt. Openwrt is foundational for dd-wrt and several other (commercial) distributions of Linux on the router. I have a large set of debloated routers already, I'm just awaiting further work on the eBDP algorithm to make better....
SFB and CHOKEe are in the debloat-testing kernel, as is eBDP. RED 93 isn't going to work. nRED may. Experimentation and scripts highly desired. See the bloat and bloat-devel mailing lists for discussions.
Well... yes, I'm one of the first 144 on bufferbloat.net
But jg defined Bufferbloat so well, that the packet traces I'd seen on my wisp6 design in South America, suddenly made complete and total sense, when I first saw his back in November. I'd had no idea what I was dealing with was actually a worldwide probem.
but thank you for the thanks. I blush.
you are conflating - to some extent - two things that a lot of people get mixed up.
1) on the TCP/ip sending and receiving side of the hosts, you already have very big, dynamic buffers in the stack for managing BDP. In this case, without very smart queue management, the TX_QUEUE and the DMA TX ring are completely "extra", and mess up the BDP calculation.
There are no "extra buffers" in the TCP equations for the host side.
2) on switches and routers, large receive buffers are ok, for BURSTS, with queue management, flow control, and other AQM features. TX, not so much, and in the general case, small buffers are simply better as you need to signal end-to-end that there is indeed congestion, before catastrophy happens.
There are other ways to lower the impact of flooding an individual host from a big fat server, for example multiple forms of weighted fair queueing.
oh, that was a wonderful parody of an already wonderful parody.
Thanks for that.
I'd probably modify it a little bit for accuracy, if you'd let me paste a copy over to our humor page?
http://www.bufferbloat.net/projects/bloat/wiki/Humor
The bufferbloat problem is so big - in hundreds of millions of devices today and millions more in the pipeline, that if we didn't laugh sometimes, we'd explode.
"The bulk of Internet traffic in the US these days is streaming video. For that you need big bandwidth and big buffers, not low latency. "
Emphatically not true.
ESPECIALLY for streaming video, you need a functioning feedback mechanism (tcp acks or ECN or some other mechanism) to slow down the video periodically, so that it *doesn't* overflow what buffers you have, and catastrophically drop all the packets in the queue, resulting in stuttering video.
Re:Two orders of magnitude! ?
on
Got (Buffer) Bloat?
·
· Score: 5, Informative
The core work where we saw latency under load drop by 2 orders of magnitude was in the wireless driver stack on Linux. Examples were the iwl driver (130ms to ~2), and the ath9k driver ( > 200ms to ~d)
(and these numbers were for GOOD connections, at high wifi rates. You can get 3 orders of magnitude improvement if you are on a slow wifi connection.)
There's a new rate sensitive algorithm for wireless (eBDP) that we are trying in this kernel tree. It's not fully baked yet. 802.11n wireless package aggregation is HARD.
That said there's bloat in all the other wired drivers too. We are doing far too much uncontrolled buffering in the kernel - specifically the dma tx ring on many devices - for slower networks.
As one example, A gigE interface, connnected to a 3Mbit cable modem - does bad, subtle, things to the stack.
latency issues were driving me insane upon my return to the USA.
http://nex-6.taht.net/posts/Beating_the_speed_of_light_on_the_web/
the huge bandwidths advertised here, and the actual "speed" reminded me of a scene in the Marching Morons, where someone steps into their hot looking sportscar, smoke and sounds come out like he is doing 100 mph, the speedo says the same, and then he looks outside the car to see the countryside slooooowly going by.
Many Americans have confused "Bandwidth", with "Speed".
Bandwidth = **capacity**, not speed.
Latency - the time it takes to click on something and get a result - is far more important to your perception of speed.
Which would you rather drive down the information superhighway? The Titanic? or a Tesla?
The web site has only been up for a month, and I agree with you very much that it is hard to get to the core ideas of bufferbloat from the get-go, we are incorporating information from dozens of very large blog posts, and hundreds of comments, and have been very busy (among other things) getting hardware running and kernel patches done.
bufferbloat.net IS a wiki, however, and registration is open to all. If you can help improve the quality, PLEASE join and do so.
I think Charles Stross has prior art. From Accelerando, published in 2005:
"In IP geek circles, Manfred is legendary; he's the guy who patented the business practice of moving your e-business somewhere with a slack intellectual property regime in order to evade licensing encumbrances. He's the guy who patented using genetic algorithms to patent everything they can permutate from an initial description of a problem domain - not just a better mousetrap, but the set of all possible better mousetraps. Roughly a third of his inventions are legal, a third are illegal, and the remainder are legal but will become illegal as soon as the legislatosaurus wakes up, smells the coffee, and panics. There are patent attorneys in Reno who swear that Manfred Macx is a pseudo, a net alias fronting for a bunch of crazed anonymous hackers armed with the Genetic Algorithm That Ate Calcutta: a kind of Serdar Argic of intellectual property, or maybe another Bourbaki math borg. There are lawyers in San Diego and Redmond who swear blind that Macx is an economic saboteur bent on wrecking the underpinning of capitalism, and there are communists in Prague who think he's the bastard spawn of Bill Gates by way of the Pope."
I am aware of the problems I may have introduced by speaking openly about the case, if it comes to a jury trial.
However, I was engaged as a fact witness, not as an expert witness, and a deposition was also taken from the co-author of the howto.
In that light, I felt it ok to ask the questions of my "tribe", publicly, that plague me, in the hope that I (and others) might learn from them.
I am grateful to all the commenters here today that have taken time out to discuss the issues to the best of their knowledge and abilities.
I have learnt more about patent law, prior art and about more prior art in the wireless world, in half a day, via slashdot, than in a month and a half of part-time research.
There have been a bunch of links and other useful material posted here today that are GREAT.
I will (ultimately) summarize on my blog, but quite possibly not until after the case is closed.
Thanks again, all!
I note that I would have been more interested in the analysis had your paper compared to apples to apples - e.g - the linux core vs the wrk core.
Linux has a notably lower standard of excellence for device driver code.
You can build a spaceship from the things you find
on
Make Your Own Sputnik
·
· Score: 3, Funny
The filk song "You can build a spaceship from the things you find at home" comes to mind.
Now next on my agenda was to find a rocket drive Strong enough to launch the ship and still keep me alive I found the right propellant when I scouted out the bars Six kegs of Old Peculier that will shoot me to the shtars! *hic*
(chorus) Lockheed, Bell and Boeing, MDC and Grumman too
Pratt and Whitney, BAE, they'll keep it all from you
They make big bucks off NASA so they never want it known
That you can build a spaceship from the things you find at home!
I note that even more popular than Ardour 2 on linux is Ardour 2 on Mac OSX. It works pretty good on a mini -
Aside from getting X installed, which seems to be a painful process for some users (answers for this are on the forums) Ardour is much more sophisticated than garageband. For me, the killer app in ardour is the anywhere to anywhere routing model.
RME-Audio Multiface - up to 14 channels of sweet sounding 96khz/24 bit converters - 8 line inputs + ADAT + SPDIF
Prosonus Digimax FS - 8 nice pre's with an ADAT out.
Dual processor opteron (3 years old) - with 3GB of ram. Given the huge samples I use (bardstown bosendorfer being one), I have linuxsampler compiled for 128 voices, and configured to use up 1.6GB of ram all by itself.
4 drives in a striped terabyte.
System works way better than my motu ever did under the evil os - works like a champ at latency levels down to 1.5ms. I generally run at 5.2ms however, as I tend to run linuxsampler+rosegarden+ardour+hydrogen a lot. One day soon I hope to get a dual core with 8GB of ram.
The RME-audio design might be 5+ years old, but it's still superior to "normal" firewire, IMHO. The fact that I have both PCI and PCMCIA cards for it means I can take the gear on the road easily...
Rest of the machine: a bunch of edirol midi converters (they just work), a roland XV88, and PodXT (fully supported by rosegarden) - the M-audio keyboard....
Dual heads provided by a 19 dollar matrox M450 card. I tried the latest nvidia card in this machine, could never get it to work...
Last important note:
[m@mingus ~]$ uptime 09:23:22 up 12 days, 6 min, 11 users, load average: 1.39, 1.31, 1.33
I used to own a motu 24i and run Sonar - and tascam gigastudio on a different machine. Sonar 1 and 2 was unreliable as hell, I could never record at low latency, and sonar over the course of 3 versions kept crashing - permanently - so it would not restart without a complete reinstall, and I was always misplacing the license key.
I had this happen in the middle of a critical, paid gig, and I lost not only a lot of money, but a lot of respect from the customer. I was incredibly angry, as you might imagine, and resolved to never again be dependent on code I couldn't fix.
100 bucks a year for sonar upgrades wasn't worth it as my bugs weren't getting fixed.
So... After begging the motu guys *for years* for specs for their board so I could write a driver for linux, and/or begging them for a driver, and getting the same "hell, no" response over and over again...
1) I researched companies that had a good history of linux support, and chose the RME-audio multiface.
2) Publically denounced motu's squareheadedness as loudly and bitterly as possible. I sold my motu 24i's to a dedicated mac-head.
3) Threw out my windows PC and Sonar and upgraded to a dual opteron 64 bit linux box...
... And, today, admittedly after some rough spots - I couldn't be happier.
Ardour2 ROCKS! It works great 64 bit Linuxsampler does a great job with gigastudio files
And I just added a digimax FS (via ADAT) to the rme-audio multiface, giving me 12 tracks of 96khz audio or 16 tracks of 44.1 - and it sounds great.
I sold the used Motu 24is for something like 400 dollars each. I haven't upgraded my sonar in a few years - so I've saved at least 300-400 bucks in upgrade fees, just on sonar. Gigastudio has come out with a few new versions (but is worth buying just for the sample libraries). There's a new windows version out - doesn't work terribly well for 64 bit, and costs some serious money.
So, all in all, throwing windows out of the studio entirely has resulted in:
1) Vastly improved reliability, with an os (linux-rt)truly targeted at multimedia
2) A huge cost savings in software, letting me buy much better hardware
3) I can run all my applications on a single dual-core machine with very low latency
4) A sense of satisfaction of "sticking it to the man"
5) The ability to participate in the process at any level you might choose. In my case, I've been speeding up plugins lately...
A windows based platform costs a lot more than linux platform. Windows + Sonar + Gigastudio is nearly a thousand dollar investment just in software. Linux + ardour + rosegarden + linuxsampler are subscriber supported.
Slashdot Mirror
Please feel free to write some code. Writing a new qdisc for Linux and BSD is not very hard.
Writing a good qdisc, insanely so.
That said, I tend to feel that time-stamping more packets and doing more guessing may make sense, as does concepts in TCP vegas.
But: Before starting, read these:
http://pollere.net/Pdfdocs/bcit_6.2001.pdf Kathleen Nichols - who proved to Van Jacobson that RED was wrong -
And Van Jacobson: http://pollere.net/Pdfdocs/QrantJul06.pdf
Which would you rather have? CPU time or personal time?
re:"Interesting problem for dd-wrt"
Agreed.
We are throwing efforts at both the mainline kernel and openwrt.
Openwrt is foundational for dd-wrt and several other (commercial) distributions of Linux on the router. I have a large set of debloated routers already, I'm just awaiting further work on the eBDP algorithm to make better....
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700
re: "using pings"
httpping is a much saner approach than ping, in many cases. Get it from:
http://www.vanheusden.com/httping/
re: RED & AQM
SFB and CHOKEe are in the debloat-testing kernel, as is eBDP.
RED 93 isn't going to work. nRED may. Experimentation and scripts highly desired. See the bloat and bloat-devel mailing lists for discussions.
Also:
http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle
Also:
I've seen some VERY interesting behavior with tcp vegas over bloated connections.
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_TCP_cubic_vs_TCP_vegas
Well... yes, I'm one of the first 144 on bufferbloat.net But jg defined Bufferbloat so well, that the packet traces I'd seen on my wisp6 design in South America, suddenly made complete and total sense, when I first saw his back in November. I'd had no idea what I was dealing with was actually a worldwide probem. but thank you for the thanks. I blush.
I think you are referring to an early draft of an article by Eric Raymond, which was wildly incorrect on this point, which was thoroughly discussed on the bloat mailing list. See the incredibly long thread starting at: https://lists.bufferbloat.net/pipermail/bloat/2011-February/000050.html There is a newer, fully corrected piece coming out soon. In the meantime, consider the words of Van Jacobson, Vint Cerf, and others. Carefully. https://lists.bufferbloat.net/pipermail/bloat/2011-February/000108.html http://www.bufferbloat.net/projects/bloat/wiki/Quotes
you are conflating - to some extent - two things that a lot of people get mixed up. 1) on the TCP/ip sending and receiving side of the hosts, you already have very big, dynamic buffers in the stack for managing BDP. In this case, without very smart queue management, the TX_QUEUE and the DMA TX ring are completely "extra", and mess up the BDP calculation. There are no "extra buffers" in the TCP equations for the host side. 2) on switches and routers, large receive buffers are ok, for BURSTS, with queue management, flow control, and other AQM features. TX, not so much, and in the general case, small buffers are simply better as you need to signal end-to-end that there is indeed congestion, before catastrophy happens. There are other ways to lower the impact of flooding an individual host from a big fat server, for example multiple forms of weighted fair queueing.
oh, that was a wonderful parody of an already wonderful parody. Thanks for that. I'd probably modify it a little bit for accuracy, if you'd let me paste a copy over to our humor page? http://www.bufferbloat.net/projects/bloat/wiki/Humor The bufferbloat problem is so big - in hundreds of millions of devices today and millions more in the pipeline, that if we didn't laugh sometimes, we'd explode.
"The bulk of Internet traffic in the US these days is streaming video. For that you need big bandwidth and big buffers, not low latency. " Emphatically not true. ESPECIALLY for streaming video, you need a functioning feedback mechanism (tcp acks or ECN or some other mechanism) to slow down the video periodically, so that it *doesn't* overflow what buffers you have, and catastrophically drop all the packets in the queue, resulting in stuttering video.
oh, great... point to a HUGE animation on the same website that is being slashdotted... thx...
Move to Nicaragua. Surf a lot. Hack from the beach. http://www.nicaraguasurfreport.com/reportlist.php?id_secc=25 It worked for me.
not just voip. Think: dns, dhcp, nd, ntp & arp.
The core work where we saw latency under load drop by 2 orders of magnitude was in the wireless driver stack on Linux. Examples were the iwl driver (130ms to ~2), and the ath9k driver ( > 200ms to ~d) (and these numbers were for GOOD connections, at high wifi rates. You can get 3 orders of magnitude improvement if you are on a slow wifi connection.) There's a new rate sensitive algorithm for wireless (eBDP) that we are trying in this kernel tree. It's not fully baked yet. 802.11n wireless package aggregation is HARD. That said there's bloat in all the other wired drivers too. We are doing far too much uncontrolled buffering in the kernel - specifically the dma tx ring on many devices - for slower networks. As one example, A gigE interface, connnected to a 3Mbit cable modem - does bad, subtle, things to the stack.
latency issues were driving me insane upon my return to the USA. http://nex-6.taht.net/posts/Beating_the_speed_of_light_on_the_web/ the huge bandwidths advertised here, and the actual "speed" reminded me of a scene in the Marching Morons, where someone steps into their hot looking sportscar, smoke and sounds come out like he is doing 100 mph, the speedo says the same, and then he looks outside the car to see the countryside slooooowly going by. Many Americans have confused "Bandwidth", with "Speed". Bandwidth = **capacity**, not speed. Latency - the time it takes to click on something and get a result - is far more important to your perception of speed. Which would you rather drive down the information superhighway? The Titanic? or a Tesla?
The web site has only been up for a month, and I agree with you very much that it is hard to get to the core ideas of bufferbloat from the get-go, we are incorporating information from dozens of very large blog posts, and hundreds of comments, and have been very busy (among other things) getting hardware running and kernel patches done. bufferbloat.net IS a wiki, however, and registration is open to all. If you can help improve the quality, PLEASE join and do so.
I run it as root, for weeks at a time. Who needs vi?
I think Charles Stross has prior art. From Accelerando, published in 2005: "In IP geek circles, Manfred is legendary; he's the guy who patented the business practice of moving your e-business somewhere with a slack intellectual property regime in order to evade licensing encumbrances. He's the guy who patented using genetic algorithms to patent everything they can permutate from an initial description of a problem domain - not just a better mousetrap, but the set of all possible better mousetraps. Roughly a third of his inventions are legal, a third are illegal, and the remainder are legal but will become illegal as soon as the legislatosaurus wakes up, smells the coffee, and panics. There are patent attorneys in Reno who swear that Manfred Macx is a pseudo, a net alias fronting for a bunch of crazed anonymous hackers armed with the Genetic Algorithm That Ate Calcutta: a kind of Serdar Argic of intellectual property, or maybe another Bourbaki math borg. There are lawyers in San Diego and Redmond who swear blind that Macx is an economic saboteur bent on wrecking the underpinning of capitalism, and there are communists in Prague who think he's the bastard spawn of Bill Gates by way of the Pope."
Thank you! this is excellent, so far.
I am aware of the problems I may have introduced by speaking openly about the case, if it comes to a jury trial. However, I was engaged as a fact witness, not as an expert witness, and a deposition was also taken from the co-author of the howto. In that light, I felt it ok to ask the questions of my "tribe", publicly, that plague me, in the hope that I (and others) might learn from them. I am grateful to all the commenters here today that have taken time out to discuss the issues to the best of their knowledge and abilities. I have learnt more about patent law, prior art and about more prior art in the wireless world, in half a day, via slashdot, than in a month and a half of part-time research. There have been a bunch of links and other useful material posted here today that are GREAT. I will (ultimately) summarize on my blog, but quite possibly not until after the case is closed. Thanks again, all!
The Pot in Nicaragua is terrible. Maybe open source can help?!
I note that I would have been more interested in the analysis had your paper compared to apples to apples - e.g - the linux core vs the wrk core. Linux has a notably lower standard of excellence for device driver code.
The filk song "You can build a spaceship from the things you find at home" comes to mind.
http://www.khaosworks.org/filk/spaceship.html
Now next on my agenda was to find a rocket drive
Strong enough to launch the ship and still keep me alive
I found the right propellant when I scouted out the bars
Six kegs of Old Peculier that will shoot me to the shtars! *hic*
(chorus) Lockheed, Bell and Boeing, MDC and Grumman too
Pratt and Whitney, BAE, they'll keep it all from you
They make big bucks off NASA so they never want it known
That you can build a spaceship from the things you find at home!
A few other cool things: Ardour under the 3d window manager, beryl. Jamin - mastering software. Bristol - softsynth.
I note that even more popular than Ardour 2 on linux is Ardour 2 on Mac OSX. It works pretty good on a mini - Aside from getting X installed, which seems to be a painful process for some users (answers for this are on the forums)
Ardour is much more sophisticated than garageband. For me, the killer app in ardour is the anywhere to anywhere routing model.
RME-Audio Multiface - up to 14 channels of sweet sounding 96khz/24 bit converters - 8 line inputs + ADAT + SPDIF
Prosonus Digimax FS - 8 nice pre's with an ADAT out.
Dual processor opteron (3 years old) - with 3GB of ram. Given the huge samples I use (bardstown bosendorfer being one), I have linuxsampler compiled for 128 voices, and configured to use up 1.6GB of ram all by itself.
4 drives in a striped terabyte.
System works way better than my motu ever did under the evil os - works like a champ at latency levels down to 1.5ms. I generally run at 5.2ms however, as I tend to run linuxsampler+rosegarden+ardour+hydrogen a lot. One day soon I hope to get a dual core with 8GB of ram.
The RME-audio design might be 5+ years old, but it's still superior to "normal" firewire, IMHO. The fact that I have both PCI and PCMCIA cards for it means I can take the gear on the road easily...
Rest of the machine: a bunch of edirol midi converters (they just work), a roland XV88, and PodXT (fully supported by rosegarden) - the M-audio keyboard.... Dual heads provided by a 19 dollar matrox M450 card. I tried the latest nvidia card in this machine, could never get it to work...
Last important note:
[m@mingus ~]$ uptime 09:23:22 up 12 days, 6 min, 11 users, load average: 1.39, 1.31, 1.33
I had this happen in the middle of a critical, paid gig, and I lost not only a lot of money, but a lot of respect from the customer. I was incredibly angry, as you might imagine, and resolved to never again be dependent on code I couldn't fix.
100 bucks a year for sonar upgrades wasn't worth it as my bugs weren't getting fixed.
So... After begging the motu guys *for years* for specs for their board so I could write a driver for linux, and/or begging them for a driver, and getting the same "hell, no" response over and over again...
1) I researched companies that had a good history of linux support, and chose the RME-audio multiface.
2) Publically denounced motu's squareheadedness as loudly and bitterly as possible. I sold my motu 24i's to a dedicated mac-head.
3) Threw out my windows PC and Sonar and upgraded to a dual opteron 64 bit linux box...
I sold the used Motu 24is for something like 400 dollars each. I haven't upgraded my sonar in a few years - so I've saved at least 300-400 bucks in upgrade fees, just on sonar. Gigastudio has come out with a few new versions (but is worth buying just for the sample libraries). There's a new windows version out - doesn't work terribly well for 64 bit, and costs some serious money.
So, all in all, throwing windows out of the studio entirely has resulted in:
1) Vastly improved reliability, with an os (linux-rt)truly targeted at multimedia
2) A huge cost savings in software, letting me buy much better hardware
3) I can run all my applications on a single dual-core machine with very low latency
4) A sense of satisfaction of "sticking it to the man"
5) The ability to participate in the process at any level you might choose. In my case, I've been speeding up plugins lately...
A windows based platform costs a lot more than linux platform. Windows + Sonar + Gigastudio is nearly a thousand dollar investment just in software. Linux + ardour + rosegarden + linuxsampler are subscriber supported.