Slashdot Mirror
How 136 People Became 7 Million Illegal File-Sharers
Barence writes "The British government's official figures on the level of illegal file sharing in the UK come from questionable research commissioned by the music industry. The Radio 4 show named More or Less examined the government's claim that 7m people in Britain are engaged in illegal file sharing. The 7m figure actually came from a report written about music industry losses for Forrester subsidiary Jupiter Research. The report was privately commissioned by none other than the UK's music trade body, the BPI. The 7m figure had been rounded up from an actual figure of 6.7m, gleaned from a 2008 survey of 1,176 net-connected households, 11.6% of which admitted to having used file-sharing software — in other words, only 136 people. That 11.6% was adjusted upwards to 16.3% 'to reflect the assumption that fewer people admit to file sharing than actually do it.' The 6.7m figure was then calculated based on an estimated number of internet users that disagreed with the government's own estimate. The wholly unsubstantiated 7m figure was then released as an official statistic."
Some of the estimation steps might be sketchy, but the basic practice of estimating a population proportion from a sample of that population is not particularly questionable. That's how almost all studies of populations work, because taking censuses of all people in a country is rarely feasible. We have century-old statistical theory on how to put bounds on the sampling error, too, assuming the sample was indeed random.
You could have a whole slew of these stories if you really objected to that basic methodology, e.g. nearly every estimate of N million people suffering from a disease or disorder is based on a sample.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
1. the same size is small.. probably too small to make the claims they did. 2. they altered the numbers on an estimate of how many people fileshare on the assumption that the number was under-reported. 3. conflict of interest... it's like the tobacco industry sponsoring studies claiming that smoking doesn't have anything to do with lung cancer... there is significant reason to believe that the study carries significant bias in favor of their conclusion and must at the least be repeated by other sources.
N. real statistics researchers know that this study has numerable crippling flaws and should not be held as gospel by anyone. Even a first year stats student can see it. The reason this story is important is that it may influence governmental policy and it's flawed... That's dangerous.
Sigs are too short to say anything truly profound so read the above post instead.
It doesn't really make sense to claim "sample size is small" for an 1,100-person sample. If the sampling was done in a random, unbiased manner, that size sample gives a margin of error of +/- 3%. If there are flaws in the sampling method, that's another thing, but the sample size alone doesn't seem problematic, unless you need accuracy better than +/- 3%.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
A margin of error of +/- 3% is the Maximum margin of error for a random sample of 1100 drawn from a large enough population at the 95% significance level (actually its really +/-2.95%), i.e this is the margin of error when the observed % is 50% , The margin of error is less when the observed % approaches 0 or 100%.
In the case of an observed % of 11.6 the margin of error is +/-1.9% so it is 95% likely that the population figure is between 9.8% and 13.5%
No, it's not.
http://www.raosoft.com/samplesize.html
About 60 million people in the UK, sample size of 1,176, confidence interval of 96% gives a margin of error of 2.99%. So, it's 96% likely that they got within 2.99% of the right answer (to the question of how many people admit to it).
I hate seeing this "that's too small a sample size" objection to every single study, from people who clearly don't know enough about how sample sizes work.
"A great democracy must be progressive or it will soon cease to be a great democracy." --Theodore Roosevelt
This is almost as cliche in arguments of statistics as the car analogy is on slashdot, and it's the sign of a scoundrel. If you actually had a first year stat student's understanding of stats you'd know where the weaknesses actually are, and where all the rest of the smoke blown in this discussion goes laghably wrong.
So let's apply some first year stats to the issue.
First, the sample size. Whether it is numerically large enough to be useful is a matter not only of it's size but also the number of positive results. IOW, a sample size of 1176 is too small if you found 3 of what you're looking for, but if you found 136 (11.6% of 1176), you have plenty of samples. The question is then only whether you had a representative sample.
My next concern would be precision. Using data with three or four significant digits (136, 1176) to make conclusions to seven significant digits (11.56463%) is silly, but that doesn't seem to have happened here. The only number in all of this that is fishy is the 16.3% number. To get three significant digits they'd have to know the number of lying households to that precision. If they had another study that determined this number they might very well have a number to that precision, but I'm assuming they just guessed.
That's still not a problem. If you guess, you run your confidence interval through your formulae (here it's a simple product) to put a range on your results. If it's a from-your-ass guess you might put a 100% failure estimate on your low end (i.e. there might be no lying households at all) to arrive at a conservative range. Here, it looks like they used an estimate of 40%. They should have (and might have; I didn't RTFA) run the un-adjusted 11.6% through the formulae to get a conservative low-end range.
Anyway, the number they finally used was 7%. One significant digit. That doesn't imply the same precision as, say, 6.7% would. In fact, if their figure for the number of lying households really was accurate to one digit (i.e. 35-45%) then rounding their final result to one digit was the correct procedure. If it was just a guess they should have run the absolute low estimate (probably, zero lying households) through to get a range.
So, with actual first year stat knowledge it's possible to actually state what might be wrong with the study, and not resort to "any first year stat student" hand-waving. It's clear that the most-cited criticism (the sample size) is the result of ignorance and group think, not actual knowledge of statistics.