Slashdot Mirror
How To Better Verify Scientific Research
Hugh Pickens DOT Com writes "Michael Hiltzik writes in the LA Times that you'd think the one place you can depend on for verifiable facts is science but a few years ago, scientists at Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology and found only six could be proved valid. 'The thing that should scare people is that so many of these important published studies turn out to be wrong when they're investigated further,' says Michael Eisen who adds that the drive to land a paper in a top journal encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn't a safeguard because the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws. 'The journals want the papers that make the sexiest claims,' Eisen says. 'And scientists believe that the way you succeed is having splashy papers in Science or Nature — it's not bad for them if a paper turns out to be wrong, if it's gotten a lot of attention.' That's why the National Institutes of Health has launched a project to remake its researchers' approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience. 'The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole,' says Hiltzik. 'NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.'"
You can always believe what a scientist tells you. Science is infallible. Albert Einstein would kick Jesus Christ's ass.
So basically they want to introduce a Slashdot for scientists..
Prepare for a brand new style of flame-wars!
Reminds me of Eric Raymond's 'Given enough eyeballs, all bugs are shallow' law.
That evil for-profit capitalists drug company Amgen should be banished frmo science because all progressive "hate speech" buzzwords are encapsulated in one organization.
How do we know the new study is correct?
I think the real problem is that scientists aren't lending any prestige to reproducing experiments so nobody bothers. Journals want to publish new results, not confirmation. Advisors discourage students from reproducing experiments, which makes sense since they won't be published.
...... and I am happy its finally being acknowledged and tackled more openly.
Follow-up studies are where the validation/replication/testing happens. This is not new. Any decent scientist knows this. Peer review is a filter, but it's a pretty basic sanity check, not a comprehensive evaluation of the work. Once published, that opens a paper and the ideas within it to critique by ALL readers, not only the reviewers. Thus, post-publication is when the real scientific review happens. Peer review merely removes the stuff that isn't formulated, measured, and organized well enough to bother reading it in the first place (i.e. it gets rejected). It's an imperfect process, so sometimes stuff slips through anyway. That's what the follow-up papers are for.
One way to fix science is to stop the atheists pushing pseudoscience like the theory of evolution in direct defiance to the Word of God. Sorry atheists but you still haven't proven that two monkeys can give birth to a human being.
In any case it is not exact and all those studies they have depend on their interpretation of the statistics. There is usually a very low chance of repeatability and a large dependence on individual judgement. There is a large chance of bias and their error bars are probably way too small.
The best way of checking spurious, biased, or erroneous results is for someone else to independently do the same experiment. However there's no money or glory in replication. So nobody does it.
I wonder which will be most amusing, Fox's interpretation of this story or the tardbaggers' interpretation of that. I've already assigned "herp", "derp" and "6,000 years" to hotkeys.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
New Paper Idea: The study of light waves as they traverse the complex structure of a Brazilian volleyball team
It's important to remember that in vivo biology is not all of science. It's a lot harder to know what you're doing in biology. If you want excellent reproducible science, let's just roll balls down inclines, measure that and hope we don't get sick.
Well, it's good to see a major scientific institution waking up to a phenomenon Richard Feynman warned about in the 1970s. Yet it seems to me the proposed solution is a little ad hoc. If scientists want to restore integrity to their field(s) -- and I applaud their efforts to do so -- why aren't they using an experimental approach to do so? I think they should try several things and collect data to find out what actually works.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
Because you WILL see - like on Fox News - these articles and studies showing science is "full of it" and that things like Global Warming are false.
See what happened there? They'll take this information and instead of saying, "Hey look science is addressing a problem.", they'll turn it into science is full of shit and evolution, global warming, or whatever scientific discoveries that contradict their World view and narrative as being false.
We''l[ be able to
Whenever one of these stories is posted about inaccurate and falsified research papers, it's always a field related to biology. This doesn't seem to be nearly as much of a problem with the hard sciences (physics, chemistry). We should avoid rhetoric like "science has lost its way" since the problem is mostly isolated to one branch of science and such statements only serve as ammo for the anti-science crowd. Disclaimer: I'm a physicist.
So, this information source is infected the same as any other information source today. No one cares if they are right or true anymore, just if it gets views. I think we all know most science isn't sexy...
-----
Interestingly, the Economist's article on the same points this weeks notes that there is a group specifically devoted to doing replication- the Reproducibility Initiative from PLOS One. They've got a $1.3 million grant from the Arnold Foundation to look at 50 high profile papers in cancer research.
"Seven Deadly Sins? I thought it was a to-do list!"
It's almost as if behavior is evolving to maximize success at sucking on the tit of government.
Nah.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
...except about something like catastrophic anthropogenic global warming :)
It's so funny that the left wing can be so insightful about certain things, but then engage in utter, mind blowing hypocrisy counter to their rational, reasoned argument.
The answer in this case is simple - make every damn scientific paper start with an intro section called "necessary and sufficient falsifiable hypothesis statement". Too much speculative navel gazing and fuzzy study setup leaves the room open to fiction writing, rather than cold hard scientific scrutiny.
Remember, first and foremost, science is a way for us to prove ourselves *wrong* - it's a way of knocking down ideas, and only grudgingly giving acceptance to the ones that survive the contest. The best scientists ruthlessly try to find every possible hole in their ideas, rather than glossing over contradictory evidence or alternatives.
Science is a method of doing things. Some scientists in some research fields are failing to follow through with the science, which is to verify results.
Or "How Better To Formulate Subject Lines". But "How To Better Formulate Subject Lines"? Ugh. Have to read that three times to parse it.
Physics is not immune to parasitic and mercenary research phenomena either, especially in more exotic areas with great funding potential, such as quantum computing & crypto where exaggerations and self-puffery are common. One might say the whole field is of that kind, since their whole theorizing (which is all they got) rests on the speculative aspects of quantum measurement theory, the foundations of which are still awaiting unambiguous experimental demonstration (such as the "loophoole free" violations of Bell inequalities), for over half century already. Should the experimental failure to confirm the fundamental conjectures persist, the whole field will be recognized as fancily relabeled analog computing (such as D-Wave system).
Far too many people treat science like a religion. If a scientist says it, it must be true, which ironically is the exact opposite of science. As has already been pointed out, all science can do is tell you something DOESN'T work that way. Instead what happens is people latch on to stupid things as if its carved in stone, regardless of how many times over the years it gets proved to be untrue or not entirely correct, they'll latch on to current theory and treat it as if its a law, and won't even blink an eye when the current theory turns out to be wrong and needs modified, the next time around ... mysteriously, it can't be wrong now!!@%!
People who treat science like a religion are just as bad as religious nut jobs, arguably worse since at least the religious nut job is aware that they are basing their thoughts on faith rather than proof.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
What is not discussed is that in science as in life it's all about incentives. All you have to is look at who is paying for these studies, directly (through research grants) or indirectly (speaking or consulting fees), and things will become much clearer. The biomedical and life sciences are most vulnerable to corruption because the incentives are very high, successful drug/treatments are worth a lot of money. Even unsuccessful ones, given the proper appearance of effectiveness are worth money.
Other sciences are less susceptible because there is no incentive to hype the results, not because those scientists are more ethical. There is two solutions for the problem. One is to remove incentives, which would mean overhauling the whole system of scientific funding. The other is to mandate raw data sharing. This would make it easier for people to reanalyze the data without actually redoing the experimental parts.
A good example of this is Reinhart-Rogoff controversy in economics, where they claimed one thing in their widely publicized 2010 paper (high debt levels impede growth), but their statistical analysis was shown to be riddled with errors, skewing the data to the desired conclusion. This was discovered the when they shared their raw data with a University of Massachusetts grad student. While data sharing would not eliminate these issues it would make is harder to perform "statistical" analysis that introduces biases.
Really there is a simple reason for this. (Of course, it's not the only one but presumably the primary one.)
Tenure-track positions and funding are to a large extent determined on the basis of the number of publications weighted by the reputation of the journals, not by the quality of publications.
The idea is that good journals will reject bad papers, which doesn't work as well as is desirable due to the extreme amount of submissions the journals receive, which have to be reviewed by relatively small numbers of unpaid voluntary reviewers.
There are many ways this problem could be alleviated and I have no idea which would be the best one. For example, hiring comittees could be encouraged to only take a look at 10 papers chosen by the applicant and disregard all others including their total number. But it's doubtful they would follow this advice in practise. Or, "allowed" publications per average year could be limited to a minium of n and a maximum of m papers. So for example, to keep funding you need to publish (on average, over a larger period of time) at least 1 peer-reviewed article and no more than 3 per year in average. Sounds crazy and I don't know how to enforce this, but it would increase the quality of papers if m is chosen sufficiently low. Or, get more stringent peer reviewing, although it's a mistery how you'd obtain that in the current system. Perhaps open access journals with crowd reviewing/ranking and meta-moderation would work, as long as mechanisms are held in place to weed out sockpuppets and trolls - difficult, though.
Anyway, it's mostly the publication pressure, in terms of numbers, that causes bad publications.
Now back to work... I need to finish a hastily written paper.
The correct take-away from this kind of study is not that a specific field of science is "broken" (also, cancer research is not all of science), but rather that there is room for improvement.
There is no question whatsoever that cancer research has made leaps and bounds over the last few decades in terms of improving the lives of many people with cancer, both by helping them to live longer, and by helping them to live better. What this kind of study shows is that we can do even better still, if we can find ways to fix the flaws that remain in cancer research.
It has been said in other papers too that a lot of the literature is wrong (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124) and that this is more likely in higher impact journals and for papers with lower sample sizes (http://www.nature.com/nrn/journal/vaop/ncurrent/full/nrn3475-c6.html). The idea is that a smaller sample size is more likely to lead to a Type I error (incorrectly finding a statistically significant result) or over-estimating the size of an effect. Consequently, these smaller sample size studies find what looks like a stunning effect but what they're really seeing is an outlier. The paper looks awesome so it gets published somewhere high impact, where it is sensationalised. This effect is exacerbated by the "publish or perish" mentality, where researchers are pressured to produce many high impact papers in order to get grants. It's also a function of the fact that a lot of research is being done, so the high volume increases the odds of this shit happening. Cancer biology is particularly prone to this sort of effect because it's very competitive, there's a lot of interest in it and so it generates high impact papers, and there are a lot of big screening studies that depend heavily on statistics to confirm effects. In some branches of biology you hardly need a stats test because variables are few in significance is obvious. However, when you're screening vast numbers of drug targets then you have all sorts of problems with multiple comparisons and the like. You need elaborate stats tests and they have to be done right. Overall, however, whether the community as a whole believes something is determined by state of the literature in general and not just a single study. What we consider true or false is influenced by the politics of science as well as the data. This is nicely reviewed in the controversial book, "The Golem", by Collins and Pinch (http://www.amazon.com/The-Golem-Should-Science-Classics/dp/1107604656).
soylentnews.org
The problems that plague science today are much deeper than the simple, solvable problem of peer review. If you actually listen to the critics who have been speaking out on the issue for decades now, the problems start in grad school. See Jeff Schmidt's book, Disciplined Minds, which exposes the details of how consensus actually forms in science today. The public likes to imagine that consensus is decided by individuals who are aware of alternative options for belief. The truth is that the consensus is simply manufactured in the grad schools, through an over-reliance upon memorization (as opposed to checking for actual conceptual comprehension, like with force concept inventory tests) and the weeding out of students who stray from the technical details of the problems they are assigned to. The truth is that the features we desire in professionals -- obedient thinkers who can fit into large organizations without "getting political" -- is really quite different than the values we associate with thinking like a scientist (which necessarily includes open-mindedness and skepticism). The notion of "professional scientist" is actually an idea with internal conflicts. It's a contradiction out in the open which apparently few have put any thought into. But, once you look at the way we train professionals today, it becomes apparent that we are not training them to actually think like scientists.
We actually had an incredible chance to have this debate back in May of 2000 when Noam Chomsky stood up with around 700 researchers in support of Jeff Schmidt. Schmidt even won his case against the American Institute of Physics, but the AIP's purpose has always been to obscure this debate from national discourse.
The AIP realizes that the credibility of much of science is basically on the line. If consensus is largely manufactured, then the public cannot rely upon it as a guide in the more empirically challenged domains.
Aren't we already doing the same exact thing here?
It's true that the system can be gamed in the short run. And sometimes someone can be game it enough to get tenure. But without follow up and citations, they'll just end up in academic limbo of being an associate professor with no funding.
Fraud is an obvious problem but I think a more widespread problem is one that is harder to penalize: plausibly deniable hype.
Somebody writes a paper with an interesting but very specialized result. No-one calls it a revolution. But the authors give their result a sexy name, because Why not? So the technical literature acquires a cute new term: "miracles", say. Then a few years later somebody else writes a paper that bears tangentially on the now-established topic of so-called 'miracles'. The authors of this new paper are scrupulous in saying nothing technically invalid. But the caveats that admit that this is merely a small increment of progress are stated in a few brief words of dense jargon, while the abstract assures you that Einstein is smiling in heaven, because Miracles are Real!
Science and Nature aren't (yet) the National Enquirer, just like MTV isn't (yet) Spice; but sex sells everywhere, and Nature and Science kind of like to publish papers about "Miracles!" as long as the thong of technical rigor covers just enough. Plausibly deniable hype seems to be spreading through a co-evolution that makes everyone complicit. You only notice it when you look around after a decade or two and think, Hey, what about all those breakthroughs I've been reading about all my life? Why don't they seem to have changed much?
Get rid of the big money and political interests funding it.
(i.e. Exxon, BP Global, Al Gore Carbon Exchanges.)
The people are not interested in climate change man made or otherwise, they are interested in profits.
To avoid this, government in the past was usually employed to carry out science. In the golden age of scientific discovery, (50-60's) gigantic paces in scientific and technological progress were made, not because it was profitable to do so, but because one country in the world, the United States decided that the knowledge gained from such an exercise was the profit. This profit was to be applied to the human conditions of food, shelter and medical.
Now, we have a fascist state, and there can be no separation of government/profit in a corporate fashion. Science isn't even possible to do any more on the scale of what was done in the 50's-60's.
Now, the only science we do largely, is commercial and it is sick and twisted.
(i.e. Like the focus on symptomatic causes of disease because that is more profitable than a cure, man made climate change with people seriously about setting up carbon exchanges to save the planet and more investment in technological trinkets and more iShit.).
It is all going to fail, and it is going to fail harshly for the species who is apply it and is heading out the door and into the fossil record.
-Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
...none of this is applicable to Climate Change?
Right...?
The idea is that a smaller sample size is more likely to lead to a Type I error
If you do your statistics right, the likelihood of a Type I error is unaffected by sample size. When you use an arbitrary threshold for determining a significant effect (typically 5%), the number of papers attaining that threshold "by chance" (=5%) is independent of sample size.
Not a bad idea, just be sure to post lots of pictures to keep people interested :-P
"When information is power, privacy is freedom" - Jah-Wren Ryel
If you do your statistics right, the likelihood of a Type I error is unaffected by sample size. When you use an arbitrary threshold for determining a significant effect (typically 5%), the number of papers attaining that threshold "by chance" (=5%) is independent of sample size.
http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html
Maybe I phrased it badly. Also I gave the wrong link. The correct one is: http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html and is worth reading.
The point is that the error being made in research isn't purely a statistical one. It originates in various bad practices, such as "flexible" study design and an ignorance of statistical power. The smaller the sample size, the greater the standard error of the mean. Thus, studies with small sample sizes are more likely to produce an estimate of the population mean that is very different from the true value. Negative results aren't interested and don't get pursued, so we're left with a bias. The result is that under-powered studies are more likely to produce large, "interesting looking", effects which get published in top journals.
soylentnews.org
In part. But there are frequently hard to quantify systematic errors that can be corrected for by using a larger sample size with more diverse representation. For example, a study might pick up a spurious correlation due to focusing on white men of western European descent between the ages of 20-29, and that correlation may not hold when a larger study uses a more diverse group of people.
There's also the issue of smaller studies being quicker and easier, which reduces the desire to publish even if the findings are negative.
"The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study...."
The angle the popular media takes on the problems in science is not fair to scientists. We didn't choose the system; static or decreasing funding and uncertain budgets did. In fact, for young scientists like me, the system is very hard, and it takes a lot not to just give up in frustration. It's not what I imagined being a scientist would be like. I never imagined glory; but I did imagine being left alone to obsess over problems instead of forever having to fight grant battles, deal with mountains of admin paperwork, and march to the furious drumbeat of publish-or-perish. But I'm still here because I still love science itself, even if not the system of science in the U.S. I just wish we wouldn't then get the problems of the system pinned on us. We didn't choose it; it was forced on us!
"And scientists believe that the way you succeed is having splashy papers in Science or Nature "
Mind that any youtube video that gets leaked to MIT Tech review, Gizmodo, or even Venture Beat.
To any PhD nowadays, it's all about the benjamins aside from Silicon Valley trying to make anyone technical, a rock star. And it's to the point I'm sick and tired of watching some cool demo on youtube having my crazy CTO to "buy it" and finding out it's really vaporware in the end.
You verify research by reproducing the results. If you can get them, it's science. If you can't get them, it's bullshit. It's as easy as that.
Seven puppies were harmed during the making of this post.
Scientific publication is not science. It is journalism. Journist get degrees in liberal art, not science. Journalist don't give a damn about truth, only their story. Even the journalist that is reporting this story tries to confuse publication with science, for the sake of his story.
Far too much is published already, and much of it just isn't very good.
But who decides what is "good"? The temptation is for scientists to publish the results that support their theory and reject the rest of the results as "bad" data, leading to massive selection bias.
Surely it is better to publish all the data so that others can check the conclusions that the author has drawn from it? And in the case of publicly-funded research it seems right that all the data should be made publicly available (with the obvious exception of sensitive information such as personal medical details).
Funding agencies should pre-budget follow-up studies, and require that grant proposals include at least guidelines now how follow-up work might be conducted.
Many (most?) of the contributors to PubMed Commons are the same people who have succeeded using the existing system: scientists with flashy papers. The independence of such a system seems compromised and will suffer from the same politics that currently affects most disciplines. It also suffers from the fact that it is one more thing of which we need to keep abreast.
It seems like a case of treating the symptom when the root cause is scientists' desire for job security or excessive funding. Shouldn't we modify the funding benchmarks and career paths to treat the root causes?