Slashdot Mirror
Busting the MythBusters' Yawn Experiment
markmcb writes "Most everyone knows and loves the MythBusters, two guys who attempt to set the story straight on things people just take for granted. Well, maybe everyone except Brandon Hansen, who has offered them a taste of their own medicine as he busts the MythBusters' improper use of statistics in their experiment to determine whether yawning is contagious. While the article maintains that the contagion of yawns is still a possibility, Hansen is clearly giving the MythBusters no credit for proving such a claim, 'not with a correlation coefficient of .045835.'"
The example I like to use, though apparently they revisited this one (I "can't" afford cable unfortunately), is they were trying to figure out whether the aerodynamic drag of running your car with your windows down was greater than the engine drag of running the A/C.
But to test this, they used SUVs (if you are concerned about fuel efficiency, are you driving one?) going at about 40 mph (air drag I think increases by the square of the speed at those speeds, so highway speeds could significantly change the results), and, most stupidly, running the A/C cold enough that Jamie was commenting that he was glad that he was wearing a fairly heavy jacket and (IIRC) a scarf!
Yeah, real useful result that test was.
TFA's conclusion is correct but their methods are wrong. For these kind of data, correlations aren't the appropriate test; they should have used a chi-square distribution test. Using TFA's assumptions -- total sample size of 50, 4 yawners out of 16 not seeded, 10 yawners out of 34 seeded -- the chi-square value is .10, which pretty strongly misses the critical value of 3.84 for significance.
Not that it matters anyway, but it's pretty funny to read an article debunking statistics that employs inappropriate statistics itself...
Not only was MythBusters embarassingly statistics-free, but the "busting" was done using a wholly inappropriate statistical technique. Hansen used a correlation-based test, which assumes that the data follows a Normal distribution (which a bunch of 1s and 0s do not).
There is a very well-known test, the chi-square test, that deals with exactly this case. (Given the small sample sizes, the Fisher exact test may give better results.) Someone should point Hansen to the Wikipedia page on the topic.
For example, if there are 16 non-primed people, with 4 yawning and 12 not (for 25%), and there are 34 primed people, with 10 yawning and 24 not (for 29%), the chi square test gives a p value of 0.74.
The values Hansen supposes are significant 4,12 and 12,24 are not: p = 0.29.
You have to go all the way to 4,12 and 17,19 (i.e. 47% on a sample of 36) to get significance.
MythBusters was wrong to conclude that their results were significant, but Hansen was equally wrong to conclude that he had shown that Mythbusters was wrong.
MythBusters numbers may mean that someone is 20% more likely to yawn if seeded. Now, what's important is to evaluate the margin of error for this statement given the sample size.
What the article is definitely wrong about is that the sample size does not change anything. The sample size basically reduces the probability of error. The higher the sample size, the more likely that the statement "someone is 20% more likely to yawn if seeded" is true. However, at their sample size, it is not unlikely that the error marging is comparable with that 20% difference, which would invalidate the experiment.
The detailed calculations for sufficient sample size are left as an excercise for the reader.
Would that be Rough Science? In particular, it sounds like the second series. I've seen a couple of the series over the past few years, and I believe it did a pretty good job of being a science show – the interest comes from watching people who actually know what they're doing, designing and building ingenious solutions (admittedly with very convenient tools and materials available) to problems that aren't inherently interesting (like making toothpaste or measuring the speed of a glacier), rather than relying on 'interesting' problems that are large/dangerous/explosive and lacking focus on the solution process.
They have stated both on the show and in other interviews that a lot more testing goes on than just what we see on the show. For the "showcase" experiment on each show (the one that opens and closes the program), the producers have taken to placing video of most or all of the tests on their Discovery website: http://www.discovery.com/mythbusters
Well, a chi-squared test would have worked too, but so would Phi correlation (a correlation between two dichotomous / binary variables), which can be computed exactly the same as ... Pearson correlation, which TFA used. In fact, if you take the chi-square value you worked out to a few more decimal places: 0.10504 (from R), divide by 50 (=N, the sample size), and then take the square root, you get 0.046, which is the phi (and hence Pearson) correlation coefficient for the TFA's data. I can't tell if OmniNerd knew this or if s/he got lucky, but there's nothing wrong with the test employed.
So, TFA's conclusion was correct, and so - whether intentionally or not - was the method. It's just not nearly as common as chi-squared test for a 2x2 table.
The number of significant figures in an answer depends on how the function propagates errors. It's INCORRECT in general to think that if the inputs are given with two significant digits (say), then the output is only good for two significant digits.
The CORRECT way is to perform error analysis on the function being computed. If the function is linear, then the error magnitude is essentially multiplied by a constant. If that constant is close to 1 (and only then) will the output accuracy be close to the input accuracy.
In general, a function being computed is nonlinear, and the resulting number of significant digits can be either more or less than for the input. Examples are chaotic systems (high accuracy in input -> low accuracy in output) or stable attractive systems (low accuracy in input -> high accuracy in output).