A Broadband Survey That Asks the Right Questions

Lauren Weinstein writes "I've just deployed the first ever Broadband Survey under the auspices of GCTIP, which asks questions that the FCC neglected to ask about service types, promised vs. actual broadband speeds, user satisfaction (or lack thereof) with their ISPs and local ISP competition, etc. I'm already finding the detailed comments many persons are leaving on the survey form to be extremely illuminating and with sufficient participation I'm hoping my reports from this data will be useful to the Internet community broadly."

Comments

Uhmmmmm

[MyLongNickName]

So, you post a survey on Slashdot. Now, I am not a statistician, and I have never played one on TV, but I think I have heard a thing or two about selection bias. Is your organization run by two college kids and an IIS server?

Re:Uhmmmmm

[socsoc]

I would never admit to running IIS on slashdot. I'd rather run a fake survey with a skewed population.

Re:Uhmmmmm

[Dorsch]

Well... If I wanted to create a proper survey, I wouldn't use the free version of www.123contactform.com (I just assume that the creator used the free version, since it's limited to 10 questions per form and the survey has 10 questions...). Get yourself a proper server (IIS should do it too) and install a proper system like LimeSurvey (http://www.limesurvey.org/) - you'll find that more useful than some ugly online service. Have fun evaluating all those textboxes!

Re:Uhmmmmm

[darkpixel2k]

Have fun evaluating all those textboxes!

No kidding. Who the hell in their right mind has a free-form text box to enter *both* your download and upload speed in bits?

My answer: People stopped using 9600 baud modems a long time ago, so I'm not sure how many bits I get--or even kilobits. Sure, I could do the math, but your survey is retarded. I get 15 MEGABITS (or should I sound like a retard and use the prefix MEBI?!?) down burstable to 30, and 2 megabits up, burstable to 0.0005 gigabits up. Have fun with the conversion, fuckers.

Re:Uhmmmmm

[hldn]

i didnt check out the article, but that's a youtube link.

http://www.youtube.com/watch?v=4fCLFKlYW3c&fmt=18

Getting relevant responses? Gosh!

[BadAnalogyGuy]

This is a self-selected survey

You're getting relevant responses from people who are already actively interested in discussing the topic? Will wonders never cease?

Re:Getting relevant responses? Gosh!

[noidentity]

Because we, as tech-savvy geeks, are most likely to be aware of the problems with the internet service we're using, and the ones most interested in them being fixed. Not that I can complain, as I get better speed than advertised on my entry-level cable connection (which I reported on the survey form).

A Broadband Survey That Asks the Right Questions

[John Hasler]

I.e., ones that are loaded so as to produce the results that the author wants to see.

In the source of the page:

[trickotomy]

""

really? really?!

Re:In the source of the page:

[zippthorne]

That is one incredibly precise versioning scheme...

FCC speed test vs. this post

[NonUniqueNickname]

The FCC-did-it-wrong tone of the post made me expect a speed test. There isn't one. It's just a questionnaire.
But I must concede this survey gets the upper hand against the FCC speed test in two aspects:
It's even later to the party than the FCC test was.
It covers an even smaller portion of the population than the FCC test did.

The Australian Broadband Survey... compare?

[mitchells00]

Shouldn't one run a survey much like the Australian Broadband survey? I mean, really, your survey is limited and open ended. With the ABs, it's interesting comparing the results from year to year... http://whirlpool.net.au/survey/2009/ http://whirlpool.net.au/survey/2008/ http://whirlpool.net.au/survey/2007/ http://whirlpool.net.au/survey/2006/ http://whirlpool.net.au/survey/2005/ http://whirlpool.net.au/survey/2004/ http://whirlpool.net.au/survey/2003/ This is how a survey should be done! We actually have serious issues with our ISP's here, so this is done to perhaps give them a bit of a kick up the arse.

Fast Enough

[Gothmolly]

768k DSL is fast enough for most people - posting on Facebook, checking CNN, sending webmail. The people who need 10MBit are the warez hounds and ISO downloaders.

Re:Fast Enough

[mmcxii]

Or people watching NetFlix/Hulu or a multi-user home or someone who's video conferencing....

Your assumptions may have been fine 5 or 6 years ago but today they're nonsense and we're only going to see more of the same.

Re:Fast Enough

[Bigjeff5]

Warez Hound! Burn Him!

Re:Fast Enough

[adamofgreyskull]

768k DSL is fast enough for most people - posting on Facebook, checking CNN, sending webmail. The people who need 10MBit are the warez hounds and ISO downloaders.

This was true, and possibly still is true for some values of "most people", but there are quite a few uses for broadband which are legal, increasingly mainstream, and which greatly benefit from increased bandwidth. E.g. Legitimately buying/downloading games (Steam,2dBoy,Telltale Games), watching streaming video (BBC iPlayer, Netflix), working/video-conferencing from home. Arguably, if all your strawman is doing is "posting on Facebook, checking CNN, sending webmail" there's no real reason to have any more than 56k or maybe 128k ISDN, except that would make watching their friends'/CNNs' embedded video or opening that attachment of the panda sneezing an incredibly painful endeavour.

10Mbit is certainly not a necessity to most people, but 768k is simply pathetic by today's standards. For one person, it's bad enough, but as soon as you have more than a couple of people (family home/student house) with the aforementioned increasingly typical use cases (streaming media/large downloads) who want to do different things at the same time, 768k soon becomes saturated.

Re:Fast Enough

[Nikker]

I used to think about the same, I would consider my self above average for downloading ISO's multiple times just to clock my fastest time and figured the party line was correct, most people only use maybe a GB / month, and in many cases they do. Now I start getting into Youtube videos at minimum 480p and 720p also comedy central videos and after about 3 hours of video I'm rounding 1.5GB! If I drop my cable TV subscription and continue to watch Internet video of similar quality to replace it for the same amount of time 5 days a week your looking at 1.5 x 30 = 45GB, still well below many bandwidth caps. Now if you have more than one person in the house with the same tastes even if they are watching the same videos on different displays it multiplies from there. Some people are into Netflix, Hulu, iTunes movies, etc and multiply that by the people in the house hold that are doing the same thing your coming around 60+GB / person/month. In some places 64GB is a common cap and just basic stuff all legit and your cap is blown.

ISP's seem to build for todays standards for the future, which is not right. How can they assume news, social media and email is the only thing many are going to use? TV's already come with internet connectivity as well as pretty much every other electronic device. OnLive and other similar ventures will be eating that cap just to play a few games. The usage is not leveling is it about to explode. So now they fail to meet serious future demands and seem "shocked" when they get saturated? Then they complain that supply and demand will kick in because of the saturation and your connection is now charged at a premium?

Saying X is fast enough for Y is fine since you are commenting on present day usage and you in many cases are correct, the trick is that ISP's are banking on your dumb ass to build the network of tomorrow.

Re:A Broadband Survey That Asks the Right Question

[JesseMcDonald]

Which of the questions did you consider loaded?

• Broadband Internet Service Type (DSL, Cable, U-verse, FiOS, Satellite, Cellular Data, Non-Cellular Wireless (WISP), T1, T3, Other)
• Type of Service if you specified 'Other' above (free-form)
• Name of ISP (e.g. AT&T, Comcast, Verizon, etc.) (free-form)
• Maximum Download and Upload Speeds (in bits per second) for your service level as specified by your ISP (including any promised initial speed boosts - e.g. 'Powerboost'), also if your ISP has imposed a traffic or bandwidth cap on your service, please briefly describe it if possible (free-form)
• City, State, Country (Zip or City code would also be appreciated) (free-form)
• Contact Info (E-mail address preferred, plus your name and/or organization name would be appreciated - This info will only be used for statistical purposes or to contact you if we have questions - [optional] (free-form)
• If you've tried the FCC Broadband Tests [link], please enter the results - Download and Upload Speeds, Latency, Jitter, and Test Type (M-Lab or Ookla) - Please also include the day of the week and approximate time of day (including Time Zone) that any tests were conducted - [optional] (free-form)
• Overall - considering performance, cost, and any other factors - how would you rate your ISP? (Perfect, Excellent, Very Good, Good, Adequate, Neutral, Disappointing, Bad, Very Bad, Abysmal, Dial-up might be better, No Opinion)
• In your opinion, is there sufficient ISP competition available to you at your location? (Yes, No, Maybe, I don't know)
• Questions, Comments, etc. - [optional] (free-form)

Right Questions, Wrong Format

[TubeSteak]

That page desperately needs text boxes to input all the answers about bandwidth/latency/jitter.
I filled out the survey, but sweet tap dancing Jesus do I pity the person(s) who have to turn the results into useable data.

Re:Right Questions, Wrong Format

[criznach]

No kidding - it just plain will not be useable data. Gotta learn somehow :)

Rigged Bandwidth Testing

[bengoerz]

I challenge whether you can even trust bandwidth tests. The OOKLA-powered bandwidth test on Broadband.gov shows 80Mbit down on my 10Mbit connection. I never see similar numbers from any other source. So, perhaps my ISP (Time Warner) is pulling one over?

I teach survey design... This is terrible.

[kklein]

This, in all honesty, is the worst survey I have ever seen, and I work with language teachers.

Not only are you setting yourself up for selection bias (as many others have pointed out), you've got all these free-answer text boxes all over it. Have you given any thought whatsoever to what you're going to do with the "data" that you get from this instrument?

Things like network speed should be in set categories. Satisfaction should be on a Likert scale, and should be broken down into aspects of interest (satisfaction with upload, download, etc.). The ISPs should be on a drop-down menu, not free answer (you'll need to include an "Other"). ZIP and City should be in separate fields (how are you going to parse those?--yes, it can be done, BUT WHY???).

Your question about maximum upload and download speed and limit and favorite color... Son, you make me want to stab out my eyes with a fork. What are you asking with that question? Whatever it is, it should be several questions with constrained responses.

One of the cardinal rules of survey design is that it should be quick and easy for people to fill out. Do the hard work for them, and let them just tick boxes. If you don't, they won't take it and all you'll get is data skewed toward people who--like you--actually care enough to type up a bunch of thoughts. I care about broadband, but even I am not interested in blathering away into a text box.

Pray tell, what "statistical purposes" would my email address be used for? Last I knew, principal components analysis only took numeric data... Same for cluster analysis. "This will only be used for magical statistics that use email addresses as variables... Or if we want to drop a line and say hi." Please.

You are setting yourself up for a world of hurt. You will need to go through with Nvivo or something to categorize all the garbage you get from this, and even if you present results, all you're really going to be presenting is "here is some stuff that people said." I have no time for listening to results of surveys like that. It's softheaded gibberish.

You are lucky you're not a student in my research practicum. There's no way I'd sign off on this as a research instrument.

Re:I teach survey design... This is terrible.

[Anonymous Coward]

So, you are the guy responsible for all these horrible cookie cutter type surveys.

You see them over and over again. Multiple choice and the answer isn't one of the choices. Talk about invalidating the results. Nothing borks the results more than forcing someone to give inaccurate answers to complete the survey.

I used to work for a survey company and hear the frustration all night long. Frustration compounded by the fact that I wasn't allowed to give any explanation or even apologize lest I corrupt the data.

For example, I gave a government sponsored survey on the effectiveness of seat belt advertising campaigns. The entire survey consisted of questions and answers that didn't fit if someone wasn't in favor of government enforced seat belt laws in the first place... 20 mins of them.

Re:I teach survey design... This is terrible.

[story645]

Do you know what they are driving at?

They're trying to figure out if you're lying, trying to make yourself look good, trying to make yourself look bad, or otherwise screwing with the survey answers. They actually need to be about 10 pages for reliability purposes and the like. The field is called psychometrics, and I've gotta agree with the OP on his rant. I just looked at the data someone collected for her masters thesis, and it's all open ended survey stuff, so she's crying at the thought of getting it into a usable form in SPSS and a good chunk of the data will either have to chucked or reworked to get anything useful out of it.

Re:I teach survey design... This is terrible.

[kklein]

A well-designed survey would have been born as an open-answer one first, administered, and the resultant data categorized into constrained responses. Then it would have been given again and checked for reliability. There would probably be some manner of factor analysis done at this point to identify patterns in the responses (make sure that items that should be similar are similar, etc.). Then you give it again and make sure that the factors or paths look the same. Then you'd give it for real. Each time, though, you'd need a unique sample.

Virtually no one does this, though, for obvious reasons.

So what you were working with there was a poorly-developed survey.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:A Broadband Survey That Asks the Right Question

[quickOnTheUptake]

In your opinion, is there sufficient ISP competition available to you at your location? (Yes, No, Maybe, I don't know)

This should have asked something objective, like: how many companies offer residential broadband service at your address (not counting satellite).