Slashdot Mirror
Independent Labs To Verify High-Profile Research Papers
ananyo writes "Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results. The problem of irreproducible results has gained prominence in recent months. In March, a cancer researcher at Amgen pharmaceutical company reported that its scientists had repeated experiments in 53 'landmark' papers, but managed to confirm findings from only six of the studies. And last year, an internal survey at Bayer HealthCare found that inconsistencies between published findings and the company's own results caused delays or cancellations in about two-thirds of projects. Now, 'Reproducibility Initiative,' a commercial online portal is offering authors the chance of getting their results validated (albeit for a price). Once the validation studies are complete, the original authors will have the option of publishing the results in the open access journal PLoS ONE, linked to the original publication."
OMG!!! LIBURULLLS!!!11!!1
Negative results are sometimes just as interesting as positive ones. As you usually learn something.
That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done.
I'm puzzled by this sentence, though: "Once the validation studies are complete, the original authors will have the option of publishing the results in PLoS ONE, linked to the original publication. "
They're saying that the people who did the work replicating the experiment don't get the credit for their work, but instead the authors of the paper that they're trying to replicate do?
And, what if the new work doesn't replicate the results? Does it get published? Credited to whom?
http://www.geoffreylandis.com
http://heygirlitspaulryan.tumblr.com/
Check out the thread over at In The Pipeline
http://pipeline.corante.com/archives/2012/08/14/reproducing_scientific_results_on_purpose.php
and
http://pipeline.corante.com/archives/2012/08/15/more_on_reproducing_scientific_results_organic_chemistry_edition.php
If Slashdot were chemistry it would look like this:Cadaverine
I'm Spike Lee and I'm voting for Obama because he's black,and you should too.
http://www.breitbart.com/Breitbart-TV/2012/08/15/Spike-Lee-Im%20going-to-do-Whatever-Can-to-Get-Obama-a-second-term-So-he-can-finally-do-what-he-wants
The idea to reproduce important results is good and is part of the scientific method. In practice this is much harder to accomplish due to several constraints. I can only speak for my field but I think this applies to other fields as well that the reproduction is hard by itself.
This leads us to a bunch of problems. If it takes a graduate student a year to collect a data set on a custom made machine that is expensive and time consuming who has the resources to reproduce the results? In most branches we are limited by the available personnel. It is hard to imagine giving someone the task of 'just' reproducing someone else's result, as this does not generate high-impact publications nor can be used for grant applications.
The thought behind this would benefit the scientific progress, especially to weed out questionable results that can lead you far off track but someone needs to do it. And it better not be me, as I need time for my own research to publish new results. Any reviewer always asks him/herself whether this is really an achievement that it is worth publishing, which reviewer would accept a paper stating "We reproduced X and did not find any deviations from the already published results" ?
There is simply no way this would be effective for major research topics. They can't be experts across all fields, e.g., they would not have regulatory clearance to do medical studies. They would not have equipment or experience to do esoteric materials or particle physics etc. So yeah, call me extremely skeptical.
sigs are for losers (except to point out that sigs are for losers)
The article says that the "authors will pay for validation studies themselves" at first. This is a nice idea, but it is not practical in an academic setting. Academic labs would rather spend money on more people or supplies instead of paying an independent lab to replicate data for them. New ideas are barley able to get funding these days, so why would extra grand money be spent to do the same exact studies over again. There could be a use for this in industry, but they would probably pay their own employees to do this instead if it is worth it.
I can see this happening for some fairly small studies, but many very big studies simply can't be replicated. For example, a big public health study will possibly change the sampling population. What about the LHC? How could anyone realistically replicate that work? The deal is replication isn't really replication as you can't always copy what someone has already done. This idea just seems more like profiteering than anything else. What we really need are options for research groups to publish studies that failed but say something interesting about why they failed. This is much more useful. This way we all learn. Plus big labs aren't always free from suspicion themselves.
When they're all in cahoots (for grant money) based on falsified research.
Yes, the more I look at this, the more I see little chance for it to work.
A graduate student will typically spend ~2 years of dedicated study in a narrowly specialized field to learning enough lab technique to do a difficult experiment, often either building their own custom-made equipment, or using one-of-a-kind equipment hand-built by a previous graduate student, and do so with absurdly low pay, in order to produce new results. You can't just buy that; they're working for peanuts only because they are looking for the credit for an original contribution to the field. And then you're going to say "oh, by the way, the original authors get the publication credit for your work if it reproduces their results, and we won't publish at all if you don't."
And, frankly, why would the original researchers go for this? You're really asking institutions to pay in order to set up a laboratory at a rival institution, and then spend time and effort painstakingly tutoring their rivals in technique so as to bring them up to the cutting edge of research? And even if you can bring a team from zero up to cutting-edge enough to duplicate your work, what you get out of it is a publication in a database saying you were right in the first place?
http://www.geoffreylandis.com
>53 'landmark' papers, but managed to confirm findings from only six of the studies.
That's 89 percent crap. (88.7%)
Sounds about right.
And as I get older, it seems that this observation holds true more every day, in everything.
--
BMO
I think you are looking at this the wrong way. If anything it is more difficult to publish results that are consistent with other studies because there isn't much interest (unless it is controversial). Studies have a better chance of being published in high-impact (widely read) journals if they report something new that causes a change in the way the scientific field thinks.
Multi-variable medical studies need something like this as well. They also need to have 'real world' results to see if their study findings scale to the millions of people in the general population.
College students should be the ones doing this type of stuff. Universities have a budget for this, it will teach the kids what is current in their field, and get them exposure to the test equipment and process.
Also, isn't this what peer-reviewed is supposed to be, prior to getting published?
A lot of published work involves a very large number of experiments, sometimes done on very expensive instrumentation over a fairly long span of time. If the costs for reproducing the results are not scaled to the complexity of the work, this new lab won't be able to keep their lights on for long...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
In all computer related fields, that's pretty easy: give the code. It's often a pain in the ass to reproduce the results (and I talk only for my field), but as soon as you get the code, then you see what's the tricky part, and what's left to improve.
Video of some good progressive thrash music
If there's confirmation, they get to republish their results. If there isn't, they get to republish in the JIR
Pound! Bang! Bin! Bash! is this a shell script or a Batman comic?
The costs involved in performing research would preclude this working in most fields. However where there would be considerable value in this sort of 'out of house' service is in performing re-analysis of the raw data behind the publication. Stats is hard and unfortunately it's all too easy to make a bit of a hash out of it. Unfortunately the current peer review process doesn't always address this adequately - either because the reviewers aren't neccessarily any better at statistics themselves or the data as presented has been stepped through processing that may add unexpected bias. Having a career statistician run a leery eye over the analysis in the orginal Wakefield paper certainly wouldn't have hurt.
Python coder | PyQt Applications | Writer
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis.
That would be nice if things were that simple.
Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.
On my planet, stuff doesn't always work. When it doesn't work, it's not always easy to figure out why it doesn't work. When you don't get a result, it's hard to be confident that you didn't do something wrong-- and the people who are confident that they didn't do something wrong... are often wrong. It's not always trivial to say whether or not you did something wrong, or whether the experiment set up had a flaw, or there was something that turns out to be important that you didn't know was important, or whether the result you're trying to replicate was just wrong to start with.
http://www.geoffreylandis.com
There have been many hallmark publications on cancer research, which usually involve at the very least extensive animals studies. Many involve human subjects. The cost to test drugs or theories on humans is extensive. Most scientists don't have the funds to redo these experiments, and wouldn't want to either -- the grant money they receive would be towards building on previous results. No funding agency would give money to re-verify a study already published.
At the very least, the authors could make ALL data available for someone to check. Data can be misrepresented very easily, especially in statistics, so having an independent group to verify results would be very welcome.
IMHO the problem here is over-reliance on statistical significance as a sign of the validity of a hypothesis. Statistical signficance is simply an estimate of the probability of Type I error. Although it is necessary to find significance in order to provisionally reject the null hypothesis, significance by itself is not sufficient to support the experimental hypothesis. What is necessary is some measure of effect size, or even better a measure of predictive power on a separate cross-validated data set. If all publications were required to report effect size as well as significance, problems with replication failure would be reduced substantially.
Isn't the purpose of publishing your research to open it up to scrutiny, and yes validation of your results? In college I read a few dozen papers which merely validated the results of another paper.
Only the dead have seen the end of War. - Plato
For example, the Organic Syntheses Journal has been doing it for decades now, and for all submissions, not just "landmark" papers (how do you know this a priori that a paper will be extremely important in a reproducible and relaiable fashion, is unclear to me)
See: http://www.orgsyn.org/submission.html for details
Another optios is that papers that have been independently reproduced bear a tag or distinctive category than the ones that went *only* through peer review process (which might not entail reproducing the experimental work).
So all we need to do now is wait for someone to build another LHC, find the Higgs themselves, and confirm the CERN result. Then they get to publish!
If I can be modded down for being a troll, can I be modded up for being an orc, or a balrog?
I like the idea, but, how do you fund this effort? I don't see the article making any mention of this.
We all spend our time writing grants now to support our own research and have little enough time to do it. Now we're expected to do someone else's research? I suppose it's a bit like reviewing articles, if you want to publish, you should review. However, this takes much more time, effort and money to do.