As for the wifi article, yes, we have seen 10+ seconds of excess latency in the wifi stack. 1-2 seconds is typical with normal traffic at lower rates, as most protocols time out in that range.
If you are referring to the cake article, the baseline latency of the path is ~11ms. It grows to about 250ms under pressure from a tcp transfer with a "normal" cable modem, and to only 16ms or so with cake. See the bar graph...
wifi could get much much worse. but we fixed it in the upcoming linux 3.10 release. Not that anybody seems to understand....
Oh, I strongly recommend ublock, too! I go around installing that on friends and family's computers every christmas.:) But this christmas, I reflashed a ton of routers, too.
Judging from the first 25 replies, the slashdot readership is suffering from an overdose of eggnog.
Here's a link (which has links to results from every ISP), which shows latency under load often measured in seconds.
http://www.dslreports.com/spee...
The problem with this survey is that there are now plenty of folk that get sub-30ms latencies on their internet - which is what those using bufferbloat fixes get, and the question was if you or your isp was driving improved hardware to get those results. Problem seems to be 99% of the results are worse than that, still, 4+ years after the code to fix first arrived in Linux.
The five most commonly used routers in the bufferbloat project for reflashing with lede are:
low end: netgear wndr3800
less low end: tp-link archer c7v2
midrange: ubntlite, edgerouter
higher end: apu2 + ath9k and ath10 wifi cards
The turris omnia is becoming worth looking at, also.
"Ubiquity, like great power, requires of us great responsibility. It changes our duties, and it changes the kind of people we have to be to meet those duties. It is no longer enough for hackers to think like explorers and artists and revolutionaries; now we have to be civil engineers as well, and identify with the people who keep the sewers unclogged and the electrical grid humming and the roads mended. Creativity was never enough by itself, it always had to be backed up with craftsmanship and care â" but now, our standards of craftsmanship and care must rise to new levels because the consequences of failure are so much more grave." -
Eric Raymond, in: http://esr.ibiblio.org/?p=4196
see also: icei.org
I am inclined to put this result in the "win" column, provisionally.
June 2 came and went, tp-link's router firmware returned to field upgradable, and other manufacturers did nothing to make flashing other firmwares any harder than it already was. Hopefully, our arguments buttressed the legal case ongoing at the time against tplink (I knew there was one, but not against whom, or over what, I hope to get more details).
This does not mean the war is won, however. Certainly binary blob firmware that completely controls the radio remains a problem - but progress is being made with the very thin firmware in the 802.11ac mt76 chipset, I am not aware of 5ghz ath9k chips requiring blobs, and other binary only firmwares are improving to support APIs that fq_codel on wifi needs. http://blog.cerowrt.org/post/f...
(Recently a few new *major* chipsets had wifi drivers submitted to the linux kernel, but I haven't looked at what, exactly the firmware controls. The state of most wifi drivers and firmware is thoroughly depressing - and a very smart and fast co-processor is seemingly needed to run at very high rates)
Five things I learned from this exercise:
1) If a legalistic solution can be vague, it will be. It then can be spun many ways for many audiences. Read Ed Bernays. Still, sometimes what is said publicly, continues to matter, and the FCC has said some very nice things.
2) The FCC was not the enemy, but a harried organization attempting to fulfill its mandates. As minimally outlined, their problem was the FAA complaining about wifi interference with weather radars. The first solution was overbroad. They have a much better understanding of the roles of open source, third party firmware now - after the keruffle - of the usefulness of user control, better security, and more frequent updates.
The FCC has WAY bigger problems than linux wifi. The number of wireless capable devices requiring certification and testing is skyrocketing, among other things.
3) If you really want attention in D.C., it is a good idea to make a good argument, with a lot of well known people, file it somewhere inside the agency's process, and then issue (buy) a press release, and make the biggest stink you can. As it turned out many of the recommendations we made above cannot be implemented inside the FCC's mandates, but the FTCs.
4) Chipmakers can now no longer hide behind an argument that the FCC will not let them open up their firmware.
5) The best "proof of the pudding" I can think of would be to push through a new product with much more or entirely open wifi firmware through the FCC processes, using the CRDA library to enforce the rules. Lining up a vendor willing to try that has so far not happened, although I expected a few mt76 chipsets to enter the US by now, I have not been actively watching their RSS feed for progress.
All in all, honestly, I do think we moved the dial a few notches in the right direction, and I'm going to sleep pretty well tonight.
Dear Bruce:
In your slashdot posting today you mischaracterized our efforts as attempting to "open source" all routers. (as have multiple other reporters and people)
I lost sleep for years trying to create a third not "open source" or "closed source" *option* for making society's safety critical source code *public* vs what is currently buried in inauditable binary blobs - and in this letter, tried to shift the core fcc licensing requirements to mandating that the source code at the lowest layers of the network stack be "public, maintained, and regularly updated".
What license is slapped on this "public" code I totally do not care about - it could mandate you have to sell off your first born child, or slit your throat after reading, for all I care.
I care only that the sources be public, buildable, maintained and updated. http://www.bufferbloat.net/pro...
Open source and closed source alike have been doing a terrible job of maintenance, and in the embedded market - aside from higher end devices like android and mainline OSes like redhat/ubuntu - are not being updated. That is the *real problem* here that we are trying to solve.
thx in advance for any efforts you might make to correct your messaging, particularly when talking about our efforts! I have been busting my b**ls to make these points with every reporter I've talked to.
Aside from that... I think extremely highly of your characterization of the problem's stakeholders, the quality of your letter is even better than ours overall, and your proposed solution quite possibly one that could succeed (although I would shoot for a new licensing regime that made the git committer more responsible, perhaps - it is very worthy of discussion!)
I am totally willing to discuss restrictions on "how public" things become - and how fast they become so! particularly as I am well aware dismal code quality in many mission and public safety critical pieces of software that is out there. Mandating that all that be made public all at once would induce a terrifying amount of risk to society as a whole, and a staged approach towards making the core blobby bits public would be best.
...which is why I have tried to initially limit the call to merely opening up the binary blobs going into wifi, particularly as getting the current 802.11ac trends towards doing so have failed so dismally and wifi far less safety critical than many other things.
I would dearly like, also, to fix the dsl drivers and firmware worldwide, at least in part, because I strongly suspect quite a lot of it, in light of snowden's revelations, is compromised already, and they just need 50 lines of code or so, and a firmware update, to eliminate the bufferbloat in them - and verify, it really is doing what the authors say in the tin, to the FCC.
Sincerely,
Dave Taht
lead author, the cerowrt project's letter to the fcc http://fqcodel.bufferbloat.net...
I incidentally came up with a way to make remote compromise MUCH harder recently, but I don't know how to implement it in tcp.
by default, emit replies to ssh/telnet/web requests with a TTL of 1, thus limiting all admin access to the local link.
The article recommends updating the firmware to the latest provided by the vendor - which is quite often, no help. First, check to see if that latest firmware is corrected...
But preferably - install better 3rd party firmware - like openwrt - designed by people that care about your security, reliability, and uptime.
Over at bufferbloat.net we have developed several pretty accurate bandwidth and latency measurement tests, that work at speeds up to 40GigE.
We wrap the popular with the linux-netdev's "netperf" tool with something that can aggregate and plot the results, called "netperf-wrapper". The most popular test in the suite is called "rrul" which stands for "Realtime Response Under Load", but there are many others in the suite.
It has been used to extensively tune several fair queuing and aqm algorithms, notably "fq_codel" which is in cerowrt, openwrt, and many other 3rd party firmwares. Its been used to debug network hardware, wifi, cablemodems, and most recently during the 40GigE batch-bql patchset now entering the linux kernel.
Some examples of use to tune a smarter queue management system against modern day cable modems:
http://burntchrome.blogspot.co...http://snapon.lab.bufferbloat....
There are also netperf-wrapper results for 40GigE, DSL, and wifi spread around the Internet. The intermediate format netperf-wrapper uses to store its results are in json and parsable by anything, and I keep hoping someone will get around to writing a web interface for the datafiles...
Nothing else I've ever seen even comes close to netperf-wrapper for finding good, accurate, long term numbers and multiple forms of anomoly. Pretty much all the web based tests get increasingly inaccurate above 20Mbits. Single threaded TCP tests are bad also as they generally result in someone defeating TCP congestion avoidance in pursuit of the best benchmark numbers. [2]
Far more important to the debloaters is not the bandwidth attained but the latency induced while getting it. [1]
We maintain several public servers for netperf-wrapper, all connected via a gigE connection to the internet. Thus far we haven't overloaded them (nor advertised them widely), but if you want to give netperf-wrapper a try, and can't set up your own netperf server on the other side, feel free to ping us on the bloat mailing list for some addresses on various continents.
[1] A brief rant: Bandwidth != speed. Bandwidth is capacity/interval. Real perceived speed is obtained via low
latency.
[2] I really hate that all the web network measurement tests don't simultaneously measure ping while running their upload and downloads. IF ONLY those tests would do that, people would start to realize that there is a huge tradeoff between good latency and high bandwidth, and that they are doing their networks in, by optimizing for bandwidth only.
Networks engineered for speedtest's current test, *suck* for voip and gaming. I'd like to petition them to at least report ping times under load to the 98th percentile.
It's not just simple backdoors like the dlink one that are a problem.
There is a systemic complete and total regard for basic tenets of security in nearly the entire home router/cpe market.
Start with crypto - no hwrng and a known "less than ideal" version of/dev/random to feed your "secure" wpa and ssh sessions.
Worse:
There is no privilege separation in most routers, which was ok when they were single function devices - BUT: not ok, when vulnerability via services like samba can be used to root most of the top 10 current home routers:
or have it participate in botnets, or inflict further attacks on unsuspecting devices both inside and outside your firewall, or sniff your traffic - there is no security when your front door is left wide open.
What nearly every home router and cpe manufacturer is shipping is **rotware**, running 4-7 year old kernels with known CVEs, and 10 year old versions of critical services like dnsmasq. You'd think that new 802.11ac devices available for this christmas might have some modern software on it, but just to pick out a recent example - the "new" netgear nighthawk router runs Linux 2.6.36.4 and dnsmasq 2.15, according to their R7000 gpl code drop -
Brand new hardware - 4+ and 10 year old software respectively.
It's unfair of me to pick on Netgear, every router I've looked at this christmas season has some major issues.
Right now, the only current hope for decent security in home routers is in open, modern, and maintained firmware. And I wish the manufacturers (and ISPs, AND users, and governments) understood that, and there was (in particular) a sustainable model for continuous updates and upgrades as effective as android's in this market. I don't care if it came from taxation, isp fees, or built into the price of the device - would you willingly leave your networks' front door open if you understood the consequences?
Rotten routers with closed source code, and no maintenance, are a huge security risk, and they are holding back the ipv6 transition, (and nearly all current models have bufferbloat, besides)
How can the dysfunctional edge of the Internet be fixed?
I lost two friends and my father this year. I dedicated this release of cerowrt ( http://cero2.bufferbloat.net/cerowrt/credits.html ) to them.
Most of the machines we have are named after someone that has passed, for example our main build box is named after http://en.wikipedia.org/wiki/John_Huchra
It helped a lot to channel them all as we struggled to get the releases out.
And, surprisingly, making ice cream, with liquid nitrogen as the coolant, has got to be a healing ritual, around here.
What Jim and the bufferbloat.net's group of volunteers have accomplished in a year - on nearly no money - boggles my mind.
Today's commentary on slashdot is a hundred times more clueful than it was last year - and a few days back Byte Queue Limits went into linux's net-next tree, which fixes much of the bloat problems that exist at the ethernet driver layer.
What has been discussed as 'Time in Queue' limits in the higher level schedulers is still awaiting a clean way to avoid layer violations. I've been too distracted by the BQL merge to pursue that next phase of fixes.
What we could have done this year with *some money* - nowhere near the amounts you describe above! - could have been amazing, and as for the next year, well, who knows? It is going to take many man-years worth of effort to make the internet responsive again.
And even with that said, to have harnessed the powers of hundreds first, now thousands, of talented minds, to help solve the bufferbloat problem - has been a far more effective - and wonderful! thing than all the money in the world.
Nearly every Linux machine ships with named (bind9) available and often, even turned on, in a caching-only configuration.
To use it by default you just disable/etc/dhcp/dhclient's domain-name-servers request and point your resolv.conf to localhost.
By doing this you get NXdomain back, too... and your local cache of dns entries is likely to be more performant than an ISPs 10s of ms away for cached entries.
You can also run dnssec, if you so choose. Latest versions of bind can do dnssec, you can enable it with one line in the conf file.
Ever since multiple services started messing with DNS a decade ago... returning broken queries, pointing to ad sites, not doing ipv6, not returning mx records, etc... I've run my own dns server.
Now that dns is being mis-used for censorship, perhaps more will rebell.
As servers go, in memory it's rather small...
The original gatech study showed not only bufferbloat, but enormous variation of base latencies in the first mile for different brands of cable modem as well as for different kinds of DSL and wireless technologies.
Slashdot Mirror
As for the wifi article, yes, we have seen 10+ seconds of excess latency in the wifi stack. 1-2 seconds is typical with normal traffic at lower rates, as most protocols time out in that range.
If you are referring to the cake article, the baseline latency of the path is ~11ms. It grows to about 250ms under pressure from a tcp transfer with a "normal" cable modem, and to only 16ms or so with cake. See the bar graph... wifi could get much much worse. but we fixed it in the upcoming linux 3.10 release. Not that anybody seems to understand....
do you ever use skype, play games, surf the web, while someone or something else is more heavily using you connection?
which sqm mode are you using?
Oh, I strongly recommend ublock, too! I go around installing that on friends and family's computers every christmas. :) But this christmas, I reflashed a ton of routers, too.
Judging from the first 25 replies, the slashdot readership is suffering from an overdose of eggnog. Here's a link (which has links to results from every ISP), which shows latency under load often measured in seconds. http://www.dslreports.com/spee... The problem with this survey is that there are now plenty of folk that get sub-30ms latencies on their internet - which is what those using bufferbloat fixes get, and the question was if you or your isp was driving improved hardware to get those results. Problem seems to be 99% of the results are worse than that, still, 4+ years after the code to fix first arrived in Linux.
The five most commonly used routers in the bufferbloat project for reflashing with lede are: low end: netgear wndr3800 less low end: tp-link archer c7v2 midrange: ubntlite, edgerouter higher end: apu2 + ath9k and ath10 wifi cards The turris omnia is becoming worth looking at, also.
"Ubiquity, like great power, requires of us great responsibility. It changes our duties, and it changes the kind of people we have to be to meet those duties. It is no longer enough for hackers to think like explorers and artists and revolutionaries; now we have to be civil engineers as well, and identify with the people who keep the sewers unclogged and the electrical grid humming and the roads mended. Creativity was never enough by itself, it always had to be backed up with craftsmanship and care â" but now, our standards of craftsmanship and care must rise to new levels because the consequences of failure are so much more grave." - Eric Raymond, in: http://esr.ibiblio.org/?p=4196 see also: icei.org
Instead of a new AP for every villiage, how about one reflashed with some modern software (openwrt/lede)?
just came out.
http://blog.cerowrt.org/post/n... It's weird how sometimes I'm tapped into the global meme generator. My life is better without all the blinkenlights.
As someone who helped put together one of the biggest filings with the FCC on this matter, with 260+ other people...
http://fqcodel.bufferbloat.net......
(in addition to 1300? 1700? filings from other orgs)
And later met in person with many of the top people there:
https://www.fcc.gov/ecfs/filin...
I am inclined to put this result in the "win" column, provisionally.
June 2 came and went, tp-link's router firmware returned to field upgradable, and other manufacturers did nothing to make flashing other firmwares any harder than it already was. Hopefully, our arguments buttressed the legal case ongoing at the time against tplink (I knew there was one, but not against whom, or over what, I hope to get more details).
This does not mean the war is won, however. Certainly binary blob firmware that completely controls the radio remains a problem - but progress is being made with the very thin firmware in the 802.11ac mt76 chipset, I am not aware of 5ghz ath9k chips requiring blobs, and other binary only firmwares are improving to support APIs that fq_codel on wifi needs.
http://blog.cerowrt.org/post/f...
(Recently a few new *major* chipsets had wifi drivers submitted to the linux kernel, but I haven't looked at what, exactly the firmware controls. The state of most wifi drivers and firmware is thoroughly depressing - and a very smart and fast co-processor is seemingly needed to run at very high rates)
Five things I learned from this exercise:
1) If a legalistic solution can be vague, it will be. It then can be spun many ways for many audiences. Read Ed Bernays.
Still, sometimes what is said publicly, continues to matter, and the FCC has said some very nice things.
2) The FCC was not the enemy, but a harried organization attempting to fulfill its mandates. As minimally outlined, their problem was the FAA complaining about wifi interference with weather radars. The first solution was overbroad. They have a much better understanding of the roles of open source, third party firmware now - after the keruffle - of the usefulness of user control, better security, and more frequent updates.
The FCC has WAY bigger problems than linux wifi. The number of wireless capable devices requiring certification and testing is skyrocketing, among other things.
https://twitter.com/FCC is a good source for the FCC's other concerns.
3) If you really want attention in D.C., it is a good idea to make a good argument, with a lot of well known people, file it somewhere inside the agency's process, and then issue (buy) a press release, and make the biggest stink you can.
As it turned out many of the recommendations we made above cannot be implemented inside the FCC's mandates, but the FTCs.
4) Chipmakers can now no longer hide behind an argument that the FCC will not let them open up their firmware.
5) The best "proof of the pudding" I can think of would be to push through a new product with much more or entirely open wifi firmware through the FCC processes, using the CRDA library to enforce the rules. Lining up a vendor willing to try that has so far not happened, although I expected a few mt76 chipsets to enter the US by now, I have not been actively watching their RSS feed for progress.
All in all, honestly, I do think we moved the dial a few notches in the right direction, and I'm going to sleep pretty well tonight.
I thought Nadia, here, did a bang up job about what is wrong in the open source world, here. https://medium.com/@nayafia/ho...
Dear Bruce:
...which is why I have tried to initially limit the call to merely opening up the binary blobs going into wifi, particularly as getting the current 802.11ac trends towards doing so have failed so dismally and wifi far less safety critical than many other things.
In your slashdot posting today you mischaracterized our efforts as attempting to "open source" all routers. (as have multiple other reporters and people)
I lost sleep for years trying to create a third not "open source" or "closed source" *option* for making society's safety critical source code *public* vs what is currently buried in inauditable binary blobs - and in this letter, tried to shift the core fcc licensing requirements to mandating that the source code at the lowest layers of the network stack be "public, maintained, and regularly updated".
What license is slapped on this "public" code I totally do not care about - it could mandate you have to sell off your first born child, or slit your throat after reading, for all I care.
I care only that the sources be public, buildable, maintained and updated.
http://www.bufferbloat.net/pro...
Open source and closed source alike have been doing a terrible job of maintenance, and in the embedded market - aside from higher end devices like android and mainline OSes like redhat/ubuntu - are not being updated. That is the *real problem* here that we are trying to solve.
thx in advance for any efforts you might make to correct your messaging, particularly when talking about our efforts! I have been busting my b**ls to make these points with every reporter I've talked to.
Aside from that... I think extremely highly of your characterization of the problem's stakeholders, the quality of your letter is even better than ours overall, and your proposed solution quite possibly one that could succeed (although I would shoot for a new licensing regime that made the git committer more responsible, perhaps - it is very worthy of discussion!)
I am totally willing to discuss restrictions on "how public" things become - and how fast they become so! particularly as I am well aware dismal code quality in many mission and public safety critical pieces of software that is out there. Mandating that all that be made public all at once would induce a terrifying amount of risk to society as a whole, and a staged approach towards making the core blobby bits public would be best.
I would dearly like, also, to fix the dsl drivers and firmware worldwide, at least in part, because I strongly suspect quite a lot of it, in light of snowden's revelations, is compromised already, and they just need 50 lines of code or so, and a firmware update, to eliminate the bufferbloat in them - and verify, it really is doing what the authors say in the tin, to the FCC.
Sincerely,
Dave Taht
lead author, the cerowrt project's letter to the fcc
http://fqcodel.bufferbloat.net...
I incidentally came up with a way to make remote compromise MUCH harder recently, but I don't know how to implement it in tcp. by default, emit replies to ssh/telnet/web requests with a TTL of 1, thus limiting all admin access to the local link.
The article recommends updating the firmware to the latest provided by the vendor - which is quite often, no help. First, check to see if that latest firmware is corrected... But preferably - install better 3rd party firmware - like openwrt - designed by people that care about your security, reliability, and uptime.
I just gave up and filed a bug: https://github.com/facebook/au...
Over at bufferbloat.net we have developed several pretty accurate bandwidth and latency measurement tests, that work at speeds up to 40GigE. We wrap the popular with the linux-netdev's "netperf" tool with something that can aggregate and plot the results, called "netperf-wrapper". The most popular test in the suite is called "rrul" which stands for "Realtime Response Under Load", but there are many others in the suite. It has been used to extensively tune several fair queuing and aqm algorithms, notably "fq_codel" which is in cerowrt, openwrt, and many other 3rd party firmwares. Its been used to debug network hardware, wifi, cablemodems, and most recently during the 40GigE batch-bql patchset now entering the linux kernel. Some examples of use to tune a smarter queue management system against modern day cable modems: http://burntchrome.blogspot.co... http://snapon.lab.bufferbloat.... There are also netperf-wrapper results for 40GigE, DSL, and wifi spread around the Internet. The intermediate format netperf-wrapper uses to store its results are in json and parsable by anything, and I keep hoping someone will get around to writing a web interface for the datafiles... Nothing else I've ever seen even comes close to netperf-wrapper for finding good, accurate, long term numbers and multiple forms of anomoly. Pretty much all the web based tests get increasingly inaccurate above 20Mbits. Single threaded TCP tests are bad also as they generally result in someone defeating TCP congestion avoidance in pursuit of the best benchmark numbers. [2] Far more important to the debloaters is not the bandwidth attained but the latency induced while getting it. [1] We maintain several public servers for netperf-wrapper, all connected via a gigE connection to the internet. Thus far we haven't overloaded them (nor advertised them widely), but if you want to give netperf-wrapper a try, and can't set up your own netperf server on the other side, feel free to ping us on the bloat mailing list for some addresses on various continents. [1] A brief rant: Bandwidth != speed. Bandwidth is capacity/interval. Real perceived speed is obtained via low latency. [2] I really hate that all the web network measurement tests don't simultaneously measure ping while running their upload and downloads. IF ONLY those tests would do that, people would start to realize that there is a huge tradeoff between good latency and high bandwidth, and that they are doing their networks in, by optimizing for bandwidth only. Networks engineered for speedtest's current test, *suck* for voip and gaming. I'd like to petition them to at least report ping times under load to the 98th percentile.
There is a systemic complete and total regard for basic tenets of security in nearly the entire home router/cpe market.
Start with crypto - no hwrng and a known "less than ideal" version of /dev/random to feed your "secure" wpa and ssh sessions.
Worse:
There is no privilege separation in most routers, which was ok when they were single function devices - BUT: not ok, when vulnerability via services like samba can be used to root most of the top 10 current home routers:
http://securityevaluators.com/content/case-studies/routers/soho_service_hacks.jsp
Once an attacker p0wns your home gateway they can change your dns to malicious sites, as dnschanger did:
http://www.dcwg.org/
or have it participate in botnets, or inflict further attacks on unsuspecting devices both inside and outside your firewall, or sniff your traffic - there is no security when your front door is left wide open.
What nearly every home router and cpe manufacturer is shipping is **rotware**, running 4-7 year old kernels with known CVEs, and 10 year old versions of critical services like dnsmasq. You'd think that new 802.11ac devices available for this christmas might have some modern software on it, but just to pick out a recent example - the "new" netgear nighthawk router runs Linux 2.6.36.4 and dnsmasq 2.15, according to their R7000 gpl code drop -
http://kb.netgear.com/app/answers/detail/a_id/2649
Brand new hardware - 4+ and 10 year old software respectively.
It's unfair of me to pick on Netgear, every router I've looked at this christmas season has some major issues.
Right now, the only current hope for decent security in home routers is in open, modern, and maintained firmware. And I wish the manufacturers (and ISPs, AND users, and governments) understood that, and there was (in particular) a sustainable model for continuous updates and upgrades as effective as android's in this market. I don't care if it came from taxation, isp fees, or built into the price of the device - would you willingly leave your networks' front door open if you understood the consequences?
Rotten routers with closed source code, and no maintenance, are a huge security risk, and they are holding back the ipv6 transition, (and nearly all current models have bufferbloat, besides)
How can the dysfunctional edge of the Internet be fixed?
I lost two friends and my father this year. I dedicated this release of cerowrt ( http://cero2.bufferbloat.net/cerowrt/credits.html ) to them. Most of the machines we have are named after someone that has passed, for example our main build box is named after http://en.wikipedia.org/wiki/John_Huchra It helped a lot to channel them all as we struggled to get the releases out. And, surprisingly, making ice cream, with liquid nitrogen as the coolant, has got to be a healing ritual, around here.
What Jim and the bufferbloat.net's group of volunteers have accomplished in a year - on nearly no money - boggles my mind.
Today's commentary on slashdot is a hundred times more clueful than it was last year - and a few days back Byte Queue Limits went into linux's net-next tree, which fixes much of the bloat problems that exist at the ethernet driver layer.
What has been discussed as 'Time in Queue' limits in the higher level schedulers is still awaiting a clean way to avoid layer violations. I've been too distracted by the BQL merge to pursue that next phase of fixes.
What we could have done this year with *some money* - nowhere near the amounts you describe above! - could have been amazing, and as for the next year, well, who knows? It is going to take many man-years worth of effort to make the internet responsive again.
And even with that said, to have harnessed the powers of hundreds first, now thousands, of talented minds, to help solve the bufferbloat problem - has been a far more effective - and wonderful! thing than all the money in the world.
Nearly every Linux machine ships with named (bind9) available and often, even turned on, in a caching-only configuration. To use it by default you just disable /etc/dhcp/dhclient's domain-name-servers request and point your resolv.conf to localhost.
By doing this you get NXdomain back, too... and your local cache of dns entries is likely to be more performant than an ISPs 10s of ms away for cached entries.
You can also run dnssec, if you so choose. Latest versions of bind can do dnssec, you can enable it with one line in the conf file.
Ever since multiple services started messing with DNS a decade ago... returning broken queries, pointing to ad sites, not doing ipv6, not returning mx records, etc... I've run my own dns server.
Now that dns is being mis-used for censorship, perhaps more will rebell.
As servers go, in memory it's rather small...
on google maps http://arstechnica.com/tech-policy/news/2011/03/google-maps-300tb-of-real-world-internet-speeds.ars
The original gatech study showed not only bufferbloat, but enormous variation of base latencies in the first mile for different brands of cable modem as well as for different kinds of DSL and wireless technologies.
Slides: http://www.caida.org/workshops/isma/1102/slides/aims1102_ssundaresan.pdf
Some commentary: http://gettys.wordpress.com/2011/02/17/caida-workshop/
I look forward to the followup!
Sort of in answer to both of your questions the bufferbloat.net servers are configured as follows:
http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle
trying at every point to make sure http 1.1 actually got used.
We survived today's slashdotting. Handily.
That said, your points are well made. SPDY is part of the chromium browser and looks to have some potential.
In my case, I like the idea of smarter - and eventually sctp-enabled - proxies, especially on wireless hops. See thread at:
https://lists.bufferbloat.net/pipermail/bloat/2011-February/000068.html