Slashdot Mirror
Busting the MythBusters' Yawn Experiment
markmcb writes "Most everyone knows and loves the MythBusters, two guys who attempt to set the story straight on things people just take for granted. Well, maybe everyone except Brandon Hansen, who has offered them a taste of their own medicine as he busts the MythBusters' improper use of statistics in their experiment to determine whether yawning is contagious. While the article maintains that the contagion of yawns is still a possibility, Hansen is clearly giving the MythBusters no credit for proving such a claim, 'not with a correlation coefficient of .045835.'"
If they find out, they may very well make an announcement on the show. Wouldn't be the first time.
it always seems to me that their conclusions are specious. I can't think of any specific episodes right now but they over simplify the data, build elaborate setups that are prone to error, and use inadequate controls.
not to mention that they always try to prove stupid crap like "a rolling stone gathers no moss". I'm waiting for them to try "the grass is always greener on the other side", or "it takes one to know one".
I dunno, the fact that he's willing to state the correlation coefficient so precisely makes me leary of his own statistical expertise.
Please, for the love of God, no more car analogies.
Facinatingly detailed observations like this must go down a treat at the parties you attend Brandon Hansen.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
In almost every episode they do something that invalidates their own findings.
Sometimes they don't things more than once (even when required), other times they don't adequately recreate the conditions of the "myth."
The show is entertaining as hell, and sometimes they do conclusively prove things.
Latewire
I'm often surprised at how many people take the MythBusters seriously. Their show is entertaining, but it's important to realize that neither Jamie nor Adam really have a scientific or engineering background. To think that they could "bust" a "myth" with any degree of certainty is laughable. But every so often I hear somebody use MythBusters as a reference, even intelligent people with at least some scientific background, like medical doctors and geologists.
I'm all for watching their show for its entertainment value. But that's about it. I'd feel like a vagine if I were to use one of their "experiments" to back up my claims.
Science and entertainment do not play well together, it is mostly true because science requires real thought and watching TV basicly does not. If you attempt to put real science on TV today you will watch the other 6.9 million TV stations each gain 1 more viewer while you get a dust bowl rolling through. Maybe it's time we started to realize what the mass public want are crappy reality shows, cooking and some bullshit made to look information but that is infact 75% CGI or "docudrama".
The above is why I wouldn't trust Mythbusters as far as I could throw them. The entire show screams entertainment rather than Science. Unfortunaely I can't find the name of a program that aired in the UK about 6 months ago. It took a team of 4 people to a deserted island and each week they had a task to complete each, they were only allowed to use what was on the island and what was given to them each week (as well as a tool set because, well no tools = screwed). They had to do things like make fireworks, record a song and various other "minor" things which required them to render down various things to achieve the chemicals they needed to complete each task. What they did and what it resulted in was very clearly labeled, having real science explained behind it.
Saddly as I recall it basicly got replaced with some crappy school based soap opera where the kids say "innit" and the teachers fuck anything with two legs (including the kids as the current trailer at least implies). So after this long rant, I guess we just give up on science and go back to the discoery channel, maybe we can catch the 3 minutes of it that isn't Nazis or some form of sport!
I like muppets.
TFA's conclusion is correct but their methods are wrong. For these kind of data, correlations aren't the appropriate test; they should have used a chi-square distribution test. Using TFA's assumptions -- total sample size of 50, 4 yawners out of 16 not seeded, 10 yawners out of 34 seeded -- the chi-square value is .10, which pretty strongly misses the critical value of 3.84 for significance.
Not that it matters anyway, but it's pretty funny to read an article debunking statistics that employs inappropriate statistics itself...
You do not report five significant figures derived from data with only two.
If Slashdot were chemistry it would look like this:Cadaverine
Not only was MythBusters embarassingly statistics-free, but the "busting" was done using a wholly inappropriate statistical technique. Hansen used a correlation-based test, which assumes that the data follows a Normal distribution (which a bunch of 1s and 0s do not).
There is a very well-known test, the chi-square test, that deals with exactly this case. (Given the small sample sizes, the Fisher exact test may give better results.) Someone should point Hansen to the Wikipedia page on the topic.
For example, if there are 16 non-primed people, with 4 yawning and 12 not (for 25%), and there are 34 primed people, with 10 yawning and 24 not (for 29%), the chi square test gives a p value of 0.74.
The values Hansen supposes are significant 4,12 and 12,24 are not: p = 0.29.
You have to go all the way to 4,12 and 17,19 (i.e. 47% on a sample of 36) to get significance.
MythBusters was wrong to conclude that their results were significant, but Hansen was equally wrong to conclude that he had shown that Mythbusters was wrong.
MythBusters numbers may mean that someone is 20% more likely to yawn if seeded. Now, what's important is to evaluate the margin of error for this statement given the sample size.
What the article is definitely wrong about is that the sample size does not change anything. The sample size basically reduces the probability of error. The higher the sample size, the more likely that the statement "someone is 20% more likely to yawn if seeded" is true. However, at their sample size, it is not unlikely that the error marging is comparable with that 20% difference, which would invalidate the experiment.
The detailed calculations for sufficient sample size are left as an excercise for the reader.
I like his references, too..
reference 5 is an episode that won't air for 2 days (maybe he's from the future!)
references 7 and 8 are forum posts (ref. 8 has just 2 replies)
two references are news stories..
these do not suggest a thorough exploration of the matter, but he cites them as if they are authoritative sources
Dr. Seuss has already proved the contagion of yawns.
At least, it can be. A quick search at Pubmed brings up eight studies that examine the phenomenon of 'contagious yawning,' including in macaques and chimps. So even if the mythbusters experimental setup was pretty crappy, and their sample was too small to have enough power to find an effect, at least their conclusion agreed with the literature.
-Ted
-=-=- Quantum physics - the dreams stuff are made of.
Well, a chi-squared test would have worked too, but so would Phi correlation (a correlation between two dichotomous / binary variables), which can be computed exactly the same as ... Pearson correlation, which TFA used. In fact, if you take the chi-square value you worked out to a few more decimal places: 0.10504 (from R), divide by 50 (=N, the sample size), and then take the square root, you get 0.046, which is the phi (and hence Pearson) correlation coefficient for the TFA's data. I can't tell if OmniNerd knew this or if s/he got lucky, but there's nothing wrong with the test employed.
So, TFA's conclusion was correct, and so - whether intentionally or not - was the method. It's just not nearly as common as chi-squared test for a 2x2 table.
Engineering is the art of compromise.
Look, to spare everyone the continued arguing over which statistical test to use at what probability level and the lack of proper control groups, let me say that MythBusters has never claimed to be a science show like Mr. Wizard. The guys are special effects designers for crying out loud! They are good at what they do, and while their scientific methodology and statistics may be a bit wonky at times, there are some experiments I've seen in peer-reviewed journals that aren't much better. Science education in the United States gets worse all the time, and if these guys can inject some life and curiosity into the current generation to get them interested in science, I applaud the effort.
GetOuttaMySpace - The Anti-Social Network
*yawn*
*yawn*
No, it's just a coincidence.
Forget thrust, drag, lift and weight. Airplanes fly because of money.
I've been reading through the comments and I'm fairly alarmed by how many people think Mythbusters isn't worthwhile based solely on scientific merit.
Look, the show never said it was teaching people about science. Adam and Jaime themselves have said many times they're more entertainment than science. They're special effects people by trade, not scientists. They build things and blow shit up. It's what they enjoy doing. You can even see it on Jaime's face when they're doing myths that don't involve blowing things up (e.g. Adam building a wind tunnel for the penny drop myth).
When the show first started, there wasn't even mention of science. They looked at urban legends such as rocket car and getting airborne in a lawnchair. The show was about the stories themselves, not the methods. Only in about season 2 or so did they start including things like "controls" and "variables" (probably by Discovery's request), but they never lost sight of the fact that they're a TV show, and television (by and large) is meant to entertain.
But that leads to an interesting question: even if they DID follow proper scientific method, how do you even apply that to some of the myths they examine? For example, they did a myth where a hillbilly chased a raccoon into a sewer pipe, decided to throw gas down it, attempted to fill the thing with fire to kill the raccoon and was purportedly "shot out". How on earth do you test that scientifically? Nowhere in the myth does it says how big the pipe was, how much gasoline was used, etc. Nowhere does it mention if he was stuck (which is important, as they found the man could only be shot out of he was wrapped in a sabot). All they have is a fun story to go off of.
If nothing else, Mythbusters gets people interested in the process of examining life, not teaching how to use proper scientific method. If their only accomplishment is making people critically question things that are usually taken at face value, they'll have succeeded in my mind.
You're forgetting that we can assume the maximum possible measurement based on the scale included with the measuring device. Sig figs takes into account a wealth of information about the difficulty of reading a scale after it runs out of marks.
You're making the assumption that either we cannot measure anything to the precision guarenteed by a scale, or that we can measure past the precision guarenteed by a scale. In either case you'd be wrong. In the former case, you'd even be suggesting that measurement is useless.
Also, if you're in a laboratory setting collecting data for a scholarly work, you'd better damn well be collecting data to the maximum precision allowable by your instrument. What would be the point of the science if you didn't go as far as you could in obtaining a precise measurement?
Calling it a myth is pushing it.
SRSLY.
Unfortunately, he BSs about a vast number of other kinds of E, in a manor that is less entertaining and more, assholish, often insulting other cast members or the audience.
Can you be Even More Awesome?!
I dunno, the fact that he's willing to state the correlation coefficient so precisely makes me leary of his own statistical expertise.
The Fact that they use a standard deviation to test an Hypothesis, you know, instead of Hypothesis Testing makes me certain that he doesn't know jack about statistics.
you do _NOT_ use descriptive statistics to study samples!!!
I can't believe how wrong this analysis is... What you're supposed to test is that when seeded with a yawn, you're more susceptible then when not seeded, and this is a whole other set of calculations...
The only scientifically relevant statistic for Mythbusters is that a high percentage of the audience (most of the men, and probably a significant fraction of the women) want to see them do more experiments with Kari in a body suit (ala the butt-moulding exercise).
-Styopa