Independent Labs To Verify High-Profile Research Papers

ananyo writes "Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results. The problem of irreproducible results has gained prominence in recent months. In March, a cancer researcher at Amgen pharmaceutical company reported that its scientists had repeated experiments in 53 'landmark' papers, but managed to confirm findings from only six of the studies. And last year, an internal survey at Bayer HealthCare found that inconsistencies between published findings and the company's own results caused delays or cancellations in about two-thirds of projects. Now, 'Reproducibility Initiative,' a commercial online portal is offering authors the chance of getting their results validated (albeit for a price). Once the validation studies are complete, the original authors will have the option of publishing the results in the open access journal PLoS ONE, linked to the original publication."

Comments

Always been a problem

[Geoffrey.landis]

That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done.

I'm puzzled by this sentence, though: "Once the validation studies are complete, the original authors will have the option of publishing the results in PLoS ONE, linked to the original publication. "

They're saying that the people who did the work replicating the experiment don't get the credit for their work, but instead the authors of the paper that they're trying to replicate do?

And, what if the new work doesn't replicate the results? Does it get published? Credited to whom?

Re:Always been a problem

[Naffer]

It’s much worse than this. The burden of proof on people attempting to publish studies showing that work cannot be replicated is extremely high. Often many-fold more experiments and controls are required to show that it isn’t simply a failure on the part of the group attempting to repeat the experiment. Frequently these sorts of papers must offer an alternative hypothesis to explain both the original and new results as well. These sorts of studies are very difficult and time consuming, and can’t be given to junior graduate students who haven’t already proven themselves to be capable experimentalists. Thus to do something like this you need to assign one or more very capable senior students/postdoctoral workers, which costs money and time and takes away from original research.

Re:Always been a problem

[fermion]

Isn't this science? A paper is published and it is treated suspiciously until the work is repeated or applied to another result. An independent lab is not going to guarantee that a result is valid. In most fields, any lab that has the expertise to repeat a study is not going to be independent. All these scientist know each other. There wil always be a risk of collusion. If scientific dishonest is the issue, two dishonest scientist are not going to solve the issue, and if a random lab can't duplicate, the original scientist can just say they didn't have the expertise.

The reality is that any experiment is just a data point, with the conclusion not nearly as interesting as the process. For instance the Millikan oil drop experiment likely had some level of fraud and probably would not be reproducible in the modern sense. However the apparatus and process did give us a path to the electric charge that eventual lead to a something that was more defensible. Likewise Einstein's photoelectric effect experiment did not prove that light was a particle. The apparatus and process were vital to experiments done half a century later that did prove the nature of light.

So the problem is not that results are invalid or scientific dishonest. IMHO the problem comes from three other sources. First, is the idea that results, not process, is the key part of a scientific study. This comes from the bad way that science is taught, and the fraudulent presentation of the scientific method. Second, is the commercial journals and their desire for publicity and 'impact factor' to drive sales. This is where I think open journals might help. Third is the focus on soft sciences, like medicine, to represent all science. In medicine, the results are all that matter, which necessarily leads to the corruption and pervasion of the entire process.

Re:cool!

[Geoffrey.landis]

Negative results are sometimes just as interesting as positive ones. As you usually learn something.

You would think.

In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.

How can we implement this in practice?

[Vario]

The idea to reproduce important results is good and is part of the scientific method. In practice this is much harder to accomplish due to several constraints. I can only speak for my field but I think this applies to other fields as well that the reproduction is hard by itself.

This leads us to a bunch of problems. If it takes a graduate student a year to collect a data set on a custom made machine that is expensive and time consuming who has the resources to reproduce the results? In most branches we are limited by the available personnel. It is hard to imagine giving someone the task of 'just' reproducing someone else's result, as this does not generate high-impact publications nor can be used for grant applications.

The thought behind this would benefit the scientific progress, especially to weed out questionable results that can lead you far off track but someone needs to do it. And it better not be me, as I need time for my own research to publish new results. Any reviewer always asks him/herself whether this is really an achievement that it is worth publishing, which reviewer would accept a paper stating "We reproduced X and did not find any deviations from the already published results" ?

Dumb racket

[kencurry]

There is simply no way this would be effective for major research topics. They can't be experts across all fields, e.g., they would not have regulatory clearance to do medical studies. They would not have equipment or experience to do esoteric materials or particle physics etc. So yeah, call me extremely skeptical.

Who will pay for this?

[Joe Torres]

The article says that the "authors will pay for validation studies themselves" at first. This is a nice idea, but it is not practical in an academic setting. Academic labs would rather spend money on more people or supplies instead of paying an independent lab to replicate data for them. New ideas are barley able to get funding these days, so why would extra grand money be spent to do the same exact studies over again. There could be a use for this in industry, but they would probably pay their own employees to do this instead if it is worth it.

Well now

[symes]

I can see this happening for some fairly small studies, but many very big studies simply can't be replicated. For example, a big public health study will possibly change the sampling population. What about the LHC? How could anyone realistically replicate that work? The deal is replication isn't really replication as you can't always copy what someone has already done. This idea just seems more like profiteering than anything else. What we really need are options for research groups to publish studies that failed but say something interesting about why they failed. This is much more useful. This way we all learn. Plus big labs aren't always free from suspicion themselves.

Fail [Re:How can we implement this in practice?]

[Geoffrey.landis]

Yes, the more I look at this, the more I see little chance for it to work.

A graduate student will typically spend ~2 years of dedicated study in a narrowly specialized field to learning enough lab technique to do a difficult experiment, often either building their own custom-made equipment, or using one-of-a-kind equipment hand-built by a previous graduate student, and do so with absurdly low pay, in order to produce new results. You can't just buy that; they're working for peanuts only because they are looking for the credit for an original contribution to the field. And then you're going to say "oh, by the way, the original authors get the publication credit for your work if it reproduces their results, and we won't publish at all if you don't."

And, frankly, why would the original researchers go for this? You're really asking institutions to pay in order to set up a laboratory at a rival institution, and then spend time and effort painstakingly tutoring their rivals in technique so as to bring them up to the cutting edge of research? And even if you can bring a team from zero up to cutting-edge enough to duplicate your work, what you get out of it is a publication in a database saying you were right in the first place?

Ratios

[bmo]

>53 'landmark' papers, but managed to confirm findings from only six of the studies.

That's 89 percent crap. (88.7%)

Sounds about right.

I repeat Sturgeonâ(TM)s Revelation, which was wrung out of me after twenty years of wearying defense of science fiction against attacks of people who used the worst examples of the field for ammunition, and whose conclusion was that ninety percent of SF is crud. Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms. -- Theodore Sturgeon

And as I get older, it seems that this observation holds true more every day, in everything.

--
BMO

Statistical confirmation

[mutube]

The costs involved in performing research would preclude this working in most fields. However where there would be considerable value in this sort of 'out of house' service is in performing re-analysis of the raw data behind the publication. Stats is hard and unfortunately it's all too easy to make a bit of a hash out of it. Unfortunately the current peer review process doesn't always address this adequately - either because the reviewers aren't neccessarily any better at statistics themselves or the data as presented has been stepped through processing that may add unexpected bias. Having a career statistician run a leery eye over the analysis in the orginal Wakefield paper certainly wouldn't have hurt.

Re:cool!

[donaggie03]

Negative results are sometimes just as interesting as positive ones. As you usually learn something.

You would think.

In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.

What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis. Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.

I want to live on your planet [Re:cool!]

[Geoffrey.landis]

In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.

What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis.

That would be nice if things were that simple.

Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.

What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.

On my planet, stuff doesn't always work. When it doesn't work, it's not always easy to figure out why it doesn't work. When you don't get a result, it's hard to be confident that you didn't do something wrong-- and the people who are confident that they didn't do something wrong... are often wrong. It's not always trivial to say whether or not you did something wrong, or whether the experiment set up had a flaw, or there was something that turns out to be important that you didn't know was important, or whether the result you're trying to replicate was just wrong to start with.

Re:cool!

[Hatta]

This has nothing to do with negative results. This is to weed out false positives.