Independent Labs To Verify High-Profile Research Papers

ananyo writes "Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results. The problem of irreproducible results has gained prominence in recent months. In March, a cancer researcher at Amgen pharmaceutical company reported that its scientists had repeated experiments in 53 'landmark' papers, but managed to confirm findings from only six of the studies. And last year, an internal survey at Bayer HealthCare found that inconsistencies between published findings and the company's own results caused delays or cancellations in about two-thirds of projects. Now, 'Reproducibility Initiative,' a commercial online portal is offering authors the chance of getting their results validated (albeit for a price). Once the validation studies are complete, the original authors will have the option of publishing the results in the open access journal PLoS ONE, linked to the original publication."

Comments

Always been a problem

[Geoffrey.landis]

That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done.

I'm puzzled by this sentence, though: "Once the validation studies are complete, the original authors will have the option of publishing the results in PLoS ONE, linked to the original publication. "

They're saying that the people who did the work replicating the experiment don't get the credit for their work, but instead the authors of the paper that they're trying to replicate do?

And, what if the new work doesn't replicate the results? Does it get published? Credited to whom?

Who will pay for this?

[Joe Torres]

The article says that the "authors will pay for validation studies themselves" at first. This is a nice idea, but it is not practical in an academic setting. Academic labs would rather spend money on more people or supplies instead of paying an independent lab to replicate data for them. New ideas are barley able to get funding these days, so why would extra grand money be spent to do the same exact studies over again. There could be a use for this in industry, but they would probably pay their own employees to do this instead if it is worth it.

Fail [Re:How can we implement this in practice?]

[Geoffrey.landis]

Yes, the more I look at this, the more I see little chance for it to work.

A graduate student will typically spend ~2 years of dedicated study in a narrowly specialized field to learning enough lab technique to do a difficult experiment, often either building their own custom-made equipment, or using one-of-a-kind equipment hand-built by a previous graduate student, and do so with absurdly low pay, in order to produce new results. You can't just buy that; they're working for peanuts only because they are looking for the credit for an original contribution to the field. And then you're going to say "oh, by the way, the original authors get the publication credit for your work if it reproduces their results, and we won't publish at all if you don't."

And, frankly, why would the original researchers go for this? You're really asking institutions to pay in order to set up a laboratory at a rival institution, and then spend time and effort painstakingly tutoring their rivals in technique so as to bring them up to the cutting edge of research? And even if you can bring a team from zero up to cutting-edge enough to duplicate your work, what you get out of it is a publication in a database saying you were right in the first place?

Ratios

[bmo]

>53 'landmark' papers, but managed to confirm findings from only six of the studies.

That's 89 percent crap. (88.7%)

Sounds about right.

I repeat Sturgeonâ(TM)s Revelation, which was wrung out of me after twenty years of wearying defense of science fiction against attacks of people who used the worst examples of the field for ammunition, and whose conclusion was that ninety percent of SF is crud. Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms. -- Theodore Sturgeon

And as I get older, it seems that this observation holds true more every day, in everything.

--
BMO