Slashdot Mirror
FCC Relying On Faulty ISP Performance Data
alphadogg writes "The FCC recently used speed test results from comScore as an absolute indicator of specific ISPs' performance. Consulting firm NetForecast analyzed comScore's testing methodology and data to assess whether it accurately reflects broadband ISP performance, and to assess the appropriateness of using the data to reach general conclusions about the actual performance ISPs deliver to their subscribers. NetForecast uncovered problems on both counts. They found that the effective service speeds comScore reports are low by a large margin (PDF) because its data calculations under-report performance and place many subscribers in a higher performance tier than they purchased."
I just am so surprised. Its run by a bunch of government employees, and they are rarely faulty.
comScore got the data more or less right. The OP's main complaint seems to be that the speed is under-reported because packet loss causes the TCP session they used to slow down. Guess what? Packet loss causes the TCP session to slow down. Customers on ISPs with noticeable loss rates experience slower performance than the line's rated speed. Hello!
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Both sides need to learn more about statistics.
The report fails to mention that across a large enough population, the results will be more-or-less correct within a certain percentage point because as he mentions, some people will test with a lot of bandwidth available at a certain point but others will test with their available bandwidth constricted. Overall, out of a large enough population the outliers are washed away.
comScore needs to realize that correlation != causation. It's not because your bandwidth correlates with other users' high-bandwidth plans, that it is caused by you actually buying the plan. But even then, even in the report the statistics show that it evens out pretty good with only a small percentage error.
Off course this brief report reeks more like paid research. Off course comScore measures the users' experience connecting to large-bandwidth centers like Akamai which has a lot of large sites on it and it doesn't accurately measures what the provider offers in the last mile. I don't care that I actually get my 10Mbps connecting to my neighborhood (unless a bunch of my neighbors actually host the Linux-ISO torrent I want) I care about getting on average getting maybe 50% of what I pay for which I usually don't get (I get closer to 1-10% depending on what I'm doing). comScore accurately reflects the poor status of broadband in this metropolitan area - dual-ISDN speeds (early 90's) on the best high-tier packages money can buy in this area. The only alternative is DSL which is horribly outdated.
Custom electronics and digital signage for your business: www.evcircuits.com
TCP/IP doesn't wait for the ACK. It keeps sending until the Window is full, or the ACK is received. If the Window fills it will wait until the ACK is received (or timeout and retry, etc).
If the test is trying to automatically place the users in specific Tiers then there could be a problem, however the rest of the issues are mostly a red herring. I use Speedtest.net and can readily attest to it's general accuracy, and I seriously doubt any other services are all that different.
by the way, I'm not in the U.S., I actually get what I pay for.
I guess I need a new ISP. According to the article there is roughly 17% of the population who at any given time is getting MORE bandwidth from their ISP than what they're paying for? Is that right? Did I misread the article? I'm sure comScore would have at least put me in a much lower tier than what I pay for. Something here doesn't seem right.
I pay for 15 Mbit down and .5 Mbit up. .5 Mbit up.
I get 15-20 Mbit down and
I even tested it with the FCC test since I posted the first message. It rated my connection slightly faster than speedtest.net did. (Not significantly, and I'm sure it would vary).
Until recently I paid for 20 Mbit / 1.0 Mbit; but I wanted to save some money.
The general illusion in the U.S. is many markets are the numbers for a zip-code are good, or even fantastic, but only a tiny fraction of the zip-code may actually get any service at all. Other zip-codes where they actually have significant penetration often have poor, or even dismal results compared to what the consumer is promised. Due to the prior reporting requirements the FCC had the ISPs were using this difference to fudge their numbers and service levels. The truth took a back seat.
by the way, I'm not in the U.S., I actually get what I pay for.
You might have worded that a little bit better. Canada and Australia have worse broadband networks than the US does. Most US users on DSL get what they pay for. Cable networks may or may not deliver the promised performance at all hours, but that's simply the nature of the beast. In my area Time Warner provides 10MBit/s service on a DOCSIS 1.1 network. That means that just four customers are enough to max out a node that serves dozens to hundreds of customers.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
So here is the outline of their claims, with responses.
Data gathering errors
Only one TCP connection is used
Basically valid, it's a pretty rare net activity nowadays that actually maxes out the connection by itself, no idea if the promises the ISPs make contractually include any wording about per-connection performance.
Client-server delay is variable
Tough, this is a reality of how the network operates, if an ISP promises speed X, they need to invest in the infrastructure necessary to deliver speed X.
Participants’ computers may be resource constrained
Outside of listing minimum requirements for client computers, this is also a reality of how the customer will perceive network performance, and this is the important measure.
Test traffic may conflict with home traffic
semi valid-ish point, but I'm skeptical that it has a noticeable impact.
Decimal math is incorrect
This one seems like utter crap, they seem to be assuming that the testing company is saying MB and meaning MiB in one case, but that they say MB and really mean MB in another case. It's far more likely that they are saying MB and they mean MiB in both cases, in which case this point is moot.
Protocol overhead is unaccounted for
Another semi-valid point, but they claim the testers have the responsibility to make the ISPs numbers look better, why isn't it instead the ISPs responsibility to make their numbers more meaningful? IIRC, speeds are often advertised on the basis of file downloads, which means the protocol overhead should NOT be accounted for.
Data interpretation errors
Purchased speed tiers are incorrectly identified
This is probably the most significant claim, if true. However it's also the most wishy-washy of all the claims, going so far as to specifically state that it's the opinion of the company that it is even happening, rather than a factual claim:
"NetForecast estimates that it is highly likely that comScore incorrectly places many panelists' PCs into higher tiers than the subscribers purchased."
Overall, the report looks like a tiny bit of valid criticism of the testing methodology wrapped in a whole lot of weaseling about what the ISP should be expected to provide, and always siding with the ISP. The end result for me is that the validity of the entire report is fatally undermined by the obvious grasping at straws being done, and the impression that I get that if there were any errors in the opposite direction, they will not be reported.
Course my Zipcode has 10MB charter available in it. Of course, thats the homes right around the golf course. And the only ones that have cable. The rest of us get either Satellite, 3G, or Wisp coverage that is spotty, and drops alot. But according to the FCC, I have high speed available, since those houses on the golf course have it, and were all in the same zip code.
And I live a few miles from a very large town. So if the majority of the EU is at 7.8Mb/s and they have access to that, I think its awesome. I have 600k\s. I pay $50/month for that, and feel lucky.
What are we going to do tonight Brain?