Slashdot Mirror
Busting the MythBusters' Yawn Experiment
markmcb writes "Most everyone knows and loves the MythBusters, two guys who attempt to set the story straight on things people just take for granted. Well, maybe everyone except Brandon Hansen, who has offered them a taste of their own medicine as he busts the MythBusters' improper use of statistics in their experiment to determine whether yawning is contagious. While the article maintains that the contagion of yawns is still a possibility, Hansen is clearly giving the MythBusters no credit for proving such a claim, 'not with a correlation coefficient of .045835.'"
TFA's conclusion is correct but their methods are wrong. For these kind of data, correlations aren't the appropriate test; they should have used a chi-square distribution test. Using TFA's assumptions -- total sample size of 50, 4 yawners out of 16 not seeded, 10 yawners out of 34 seeded -- the chi-square value is .10, which pretty strongly misses the critical value of 3.84 for significance.
Not that it matters anyway, but it's pretty funny to read an article debunking statistics that employs inappropriate statistics itself...
Not only was MythBusters embarassingly statistics-free, but the "busting" was done using a wholly inappropriate statistical technique. Hansen used a correlation-based test, which assumes that the data follows a Normal distribution (which a bunch of 1s and 0s do not).
There is a very well-known test, the chi-square test, that deals with exactly this case. (Given the small sample sizes, the Fisher exact test may give better results.) Someone should point Hansen to the Wikipedia page on the topic.
For example, if there are 16 non-primed people, with 4 yawning and 12 not (for 25%), and there are 34 primed people, with 10 yawning and 24 not (for 29%), the chi square test gives a p value of 0.74.
The values Hansen supposes are significant 4,12 and 12,24 are not: p = 0.29.
You have to go all the way to 4,12 and 17,19 (i.e. 47% on a sample of 36) to get significance.
MythBusters was wrong to conclude that their results were significant, but Hansen was equally wrong to conclude that he had shown that Mythbusters was wrong.
Would that be Rough Science? In particular, it sounds like the second series. I've seen a couple of the series over the past few years, and I believe it did a pretty good job of being a science show – the interest comes from watching people who actually know what they're doing, designing and building ingenious solutions (admittedly with very convenient tools and materials available) to problems that aren't inherently interesting (like making toothpaste or measuring the speed of a glacier), rather than relying on 'interesting' problems that are large/dangerous/explosive and lacking focus on the solution process.
They have stated both on the show and in other interviews that a lot more testing goes on than just what we see on the show. For the "showcase" experiment on each show (the one that opens and closes the program), the producers have taken to placing video of most or all of the tests on their Discovery website: http://www.discovery.com/mythbusters
The number of significant figures in an answer depends on how the function propagates errors. It's INCORRECT in general to think that if the inputs are given with two significant digits (say), then the output is only good for two significant digits.
The CORRECT way is to perform error analysis on the function being computed. If the function is linear, then the error magnitude is essentially multiplied by a constant. If that constant is close to 1 (and only then) will the output accuracy be close to the input accuracy.
In general, a function being computed is nonlinear, and the resulting number of significant digits can be either more or less than for the input. Examples are chaotic systems (high accuracy in input -> low accuracy in output) or stable attractive systems (low accuracy in input -> high accuracy in output).