Slashdot Mirror
FCC Relying On Faulty ISP Performance Data
alphadogg writes "The FCC recently used speed test results from comScore as an absolute indicator of specific ISPs' performance. Consulting firm NetForecast analyzed comScore's testing methodology and data to assess whether it accurately reflects broadband ISP performance, and to assess the appropriateness of using the data to reach general conclusions about the actual performance ISPs deliver to their subscribers. NetForecast uncovered problems on both counts. They found that the effective service speeds comScore reports are low by a large margin (PDF) because its data calculations under-report performance and place many subscribers in a higher performance tier than they purchased."
Waiting for an ACK before transmitting the next packet doesn't seem like a way of measuring bandwidth. Sounds like a measure of bandwidth + latency.
I just am so surprised. Its run by a bunch of government employees, and they are rarely faulty.
I wonder which ISP owns comScore. Who got the worst rating?
comScore got the data more or less right. The OP's main complaint seems to be that the speed is under-reported because packet loss causes the TCP session they used to slow down. Guess what? Packet loss causes the TCP session to slow down. Customers on ISPs with noticeable loss rates experience slower performance than the line's rated speed. Hello!
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
At least they don't have data saying that our speeds are faster than they really are. This way the problem of 3rd world net speeds can be addressed.
And really the argument 'these scores are low because X slowed them down' is really not sound. If X really exists then the connection is slowed in real life. I would bet this is the results of what people expierence over the net. And should be plenty fine to help determine what needs to be done to get back in the same ballpark as the rest of the industrialized world.
My addiction: Arguing with idiots. AKA Slashdot!
Both sides need to learn more about statistics.
The report fails to mention that across a large enough population, the results will be more-or-less correct within a certain percentage point because as he mentions, some people will test with a lot of bandwidth available at a certain point but others will test with their available bandwidth constricted. Overall, out of a large enough population the outliers are washed away.
comScore needs to realize that correlation != causation. It's not because your bandwidth correlates with other users' high-bandwidth plans, that it is caused by you actually buying the plan. But even then, even in the report the statistics show that it evens out pretty good with only a small percentage error.
Off course this brief report reeks more like paid research. Off course comScore measures the users' experience connecting to large-bandwidth centers like Akamai which has a lot of large sites on it and it doesn't accurately measures what the provider offers in the last mile. I don't care that I actually get my 10Mbps connecting to my neighborhood (unless a bunch of my neighbors actually host the Linux-ISO torrent I want) I care about getting on average getting maybe 50% of what I pay for which I usually don't get (I get closer to 1-10% depending on what I'm doing). comScore accurately reflects the poor status of broadband in this metropolitan area - dual-ISDN speeds (early 90's) on the best high-tier packages money can buy in this area. The only alternative is DSL which is horribly outdated.
Custom electronics and digital signage for your business: www.evcircuits.com
I guess I need a new ISP. According to the article there is roughly 17% of the population who at any given time is getting MORE bandwidth from their ISP than what they're paying for? Is that right? Did I misread the article? I'm sure comScore would have at least put me in a much lower tier than what I pay for. Something here doesn't seem right.
In those times of traffic shapping and deep packet inspection, test results are always awesome. I can get more than 20Mbit/s on tests with my 6Mbit/s rate comcast line. I can see 2MB sustained download speed over http. I can also see crippled 256kbit/s SSH transfer rate, dut to traffic shapping, and out the roof latency (say 400ms) on anything that is not going to port 80. Tests always look good, but for all practical purposes, this connexion is crap.
As if dslreports.com isn't useful?
Sheesh
deleting the extra space after periods so i can stay relevant, yeah.
So here is the outline of their claims, with responses.
Data gathering errors
Only one TCP connection is used
Basically valid, it's a pretty rare net activity nowadays that actually maxes out the connection by itself, no idea if the promises the ISPs make contractually include any wording about per-connection performance.
Client-server delay is variable
Tough, this is a reality of how the network operates, if an ISP promises speed X, they need to invest in the infrastructure necessary to deliver speed X.
Participants’ computers may be resource constrained
Outside of listing minimum requirements for client computers, this is also a reality of how the customer will perceive network performance, and this is the important measure.
Test traffic may conflict with home traffic
semi valid-ish point, but I'm skeptical that it has a noticeable impact.
Decimal math is incorrect
This one seems like utter crap, they seem to be assuming that the testing company is saying MB and meaning MiB in one case, but that they say MB and really mean MB in another case. It's far more likely that they are saying MB and they mean MiB in both cases, in which case this point is moot.
Protocol overhead is unaccounted for
Another semi-valid point, but they claim the testers have the responsibility to make the ISPs numbers look better, why isn't it instead the ISPs responsibility to make their numbers more meaningful? IIRC, speeds are often advertised on the basis of file downloads, which means the protocol overhead should NOT be accounted for.
Data interpretation errors
Purchased speed tiers are incorrectly identified
This is probably the most significant claim, if true. However it's also the most wishy-washy of all the claims, going so far as to specifically state that it's the opinion of the company that it is even happening, rather than a factual claim:
"NetForecast estimates that it is highly likely that comScore incorrectly places many panelists' PCs into higher tiers than the subscribers purchased."
Overall, the report looks like a tiny bit of valid criticism of the testing methodology wrapped in a whole lot of weaseling about what the ISP should be expected to provide, and always siding with the ISP. The end result for me is that the validity of the entire report is fatally undermined by the obvious grasping at straws being done, and the impression that I get that if there were any errors in the opposite direction, they will not be reported.
I wish I could get the horrible low speeds these poor people are getting. I'm in Australia, feel free to commence your laughter now.
"The Secure features a fingerprint scanner and a thermal sensor 'so that the finger alone, detached from the body, will still not give access to the memory stick's contents."
I'm sure that if someone went through the trouble of removing the finger to access the Secure Pro, then they'd go through the trouble of warming the dead finger up so that they could have access. seems kinda gimmicky to me...
like a man without arms, you can't hang......