Slashdot Mirror
Independent Labs To Verify High-Profile Research Papers
ananyo writes "Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results. The problem of irreproducible results has gained prominence in recent months. In March, a cancer researcher at Amgen pharmaceutical company reported that its scientists had repeated experiments in 53 'landmark' papers, but managed to confirm findings from only six of the studies. And last year, an internal survey at Bayer HealthCare found that inconsistencies between published findings and the company's own results caused delays or cancellations in about two-thirds of projects. Now, 'Reproducibility Initiative,' a commercial online portal is offering authors the chance of getting their results validated (albeit for a price). Once the validation studies are complete, the original authors will have the option of publishing the results in the open access journal PLoS ONE, linked to the original publication."
Negative results are sometimes just as interesting as positive ones. As you usually learn something.
That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done.
I'm puzzled by this sentence, though: "Once the validation studies are complete, the original authors will have the option of publishing the results in PLoS ONE, linked to the original publication. "
They're saying that the people who did the work replicating the experiment don't get the credit for their work, but instead the authors of the paper that they're trying to replicate do?
And, what if the new work doesn't replicate the results? Does it get published? Credited to whom?
http://www.geoffreylandis.com
Check out the thread over at In The Pipeline
http://pipeline.corante.com/archives/2012/08/14/reproducing_scientific_results_on_purpose.php
and
http://pipeline.corante.com/archives/2012/08/15/more_on_reproducing_scientific_results_organic_chemistry_edition.php
If Slashdot were chemistry it would look like this:Cadaverine
The idea to reproduce important results is good and is part of the scientific method. In practice this is much harder to accomplish due to several constraints. I can only speak for my field but I think this applies to other fields as well that the reproduction is hard by itself.
This leads us to a bunch of problems. If it takes a graduate student a year to collect a data set on a custom made machine that is expensive and time consuming who has the resources to reproduce the results? In most branches we are limited by the available personnel. It is hard to imagine giving someone the task of 'just' reproducing someone else's result, as this does not generate high-impact publications nor can be used for grant applications.
The thought behind this would benefit the scientific progress, especially to weed out questionable results that can lead you far off track but someone needs to do it. And it better not be me, as I need time for my own research to publish new results. Any reviewer always asks him/herself whether this is really an achievement that it is worth publishing, which reviewer would accept a paper stating "We reproduced X and did not find any deviations from the already published results" ?
There is simply no way this would be effective for major research topics. They can't be experts across all fields, e.g., they would not have regulatory clearance to do medical studies. They would not have equipment or experience to do esoteric materials or particle physics etc. So yeah, call me extremely skeptical.
sigs are for losers (except to point out that sigs are for losers)
Sounds good. Anything that pisses off rednecks is good enough for me.
The article says that the "authors will pay for validation studies themselves" at first. This is a nice idea, but it is not practical in an academic setting. Academic labs would rather spend money on more people or supplies instead of paying an independent lab to replicate data for them. New ideas are barley able to get funding these days, so why would extra grand money be spent to do the same exact studies over again. There could be a use for this in industry, but they would probably pay their own employees to do this instead if it is worth it.
I can see this happening for some fairly small studies, but many very big studies simply can't be replicated. For example, a big public health study will possibly change the sampling population. What about the LHC? How could anyone realistically replicate that work? The deal is replication isn't really replication as you can't always copy what someone has already done. This idea just seems more like profiteering than anything else. What we really need are options for research groups to publish studies that failed but say something interesting about why they failed. This is much more useful. This way we all learn. Plus big labs aren't always free from suspicion themselves.
When they're all in cahoots (for grant money) based on falsified research.
Yes, the more I look at this, the more I see little chance for it to work.
A graduate student will typically spend ~2 years of dedicated study in a narrowly specialized field to learning enough lab technique to do a difficult experiment, often either building their own custom-made equipment, or using one-of-a-kind equipment hand-built by a previous graduate student, and do so with absurdly low pay, in order to produce new results. You can't just buy that; they're working for peanuts only because they are looking for the credit for an original contribution to the field. And then you're going to say "oh, by the way, the original authors get the publication credit for your work if it reproduces their results, and we won't publish at all if you don't."
And, frankly, why would the original researchers go for this? You're really asking institutions to pay in order to set up a laboratory at a rival institution, and then spend time and effort painstakingly tutoring their rivals in technique so as to bring them up to the cutting edge of research? And even if you can bring a team from zero up to cutting-edge enough to duplicate your work, what you get out of it is a publication in a database saying you were right in the first place?
http://www.geoffreylandis.com
>53 'landmark' papers, but managed to confirm findings from only six of the studies.
That's 89 percent crap. (88.7%)
Sounds about right.
And as I get older, it seems that this observation holds true more every day, in everything.
--
BMO
I think you are looking at this the wrong way. If anything it is more difficult to publish results that are consistent with other studies because there isn't much interest (unless it is controversial). Studies have a better chance of being published in high-impact (widely read) journals if they report something new that causes a change in the way the scientific field thinks.
Multi-variable medical studies need something like this as well. They also need to have 'real world' results to see if their study findings scale to the millions of people in the general population.
College students should be the ones doing this type of stuff. Universities have a budget for this, it will teach the kids what is current in their field, and get them exposure to the test equipment and process.
Also, isn't this what peer-reviewed is supposed to be, prior to getting published?
A lot of published work involves a very large number of experiments, sometimes done on very expensive instrumentation over a fairly long span of time. If the costs for reproducing the results are not scaled to the complexity of the work, this new lab won't be able to keep their lights on for long...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Yes that's just what Spike Lee is saying too. Fuck the country, screw what's best for our children and our future. Government cheese and food stamps for everyone!
Vote Obama man 'cause he's gonna give it to the man.
The good news is that for the most part all of this is just talk, most of you idiots are too stupid to even be registered and when it comes time to vote you are just as likely to be high and forget to actually do it.
Obama=Epic Fail
In all computer related fields, that's pretty easy: give the code. It's often a pain in the ass to reproduce the results (and I talk only for my field), but as soon as you get the code, then you see what's the tricky part, and what's left to improve.
Video of some good progressive thrash music
If there's confirmation, they get to republish their results. If there isn't, they get to republish in the JIR
Pound! Bang! Bin! Bash! is this a shell script or a Batman comic?
Why don't you just be honest and say your problem with Obama is he's an 'uppity' black man?
The costs involved in performing research would preclude this working in most fields. However where there would be considerable value in this sort of 'out of house' service is in performing re-analysis of the raw data behind the publication. Stats is hard and unfortunately it's all too easy to make a bit of a hash out of it. Unfortunately the current peer review process doesn't always address this adequately - either because the reviewers aren't neccessarily any better at statistics themselves or the data as presented has been stepped through processing that may add unexpected bias. Having a career statistician run a leery eye over the analysis in the orginal Wakefield paper certainly wouldn't have hurt.
Python coder | PyQt Applications | Writer
Because that is not my problem with Obama. I despise Marxists.
I wonder why you would say such a thing about me, I have never said anything even remotely close to attacking Obama's race.
You on the other hand opened right up with 'Anything that pisses off rednecks is good enough for me.'
Pot, meet kettle.
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis.
That would be nice if things were that simple.
Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.
On my planet, stuff doesn't always work. When it doesn't work, it's not always easy to figure out why it doesn't work. When you don't get a result, it's hard to be confident that you didn't do something wrong-- and the people who are confident that they didn't do something wrong... are often wrong. It's not always trivial to say whether or not you did something wrong, or whether the experiment set up had a flaw, or there was something that turns out to be important that you didn't know was important, or whether the result you're trying to replicate was just wrong to start with.
http://www.geoffreylandis.com
Obama isn't a marxist. He isn't even a socialist.
He is a moderately conservative pro corporate welfare capitalist.
Obama is a Marxist? Since when? What 'Marxist' laws has he signed into law? The one that required people to buy private health insurance or face fines? The ones he tried to get passed to favor the RIAA/MPAA corporations? The ones that gave the corporations who helped spy on US citizens immunity from lawsuit? The same guy whose campaign was financed by multi-billion dollar corporations? Yeah, that totally sounds like what a 'Marxist' would be like. Only in conservaturd land could an obvious crony capitalist and corporatist be called a 'Marxist'.
There have been many hallmark publications on cancer research, which usually involve at the very least extensive animals studies. Many involve human subjects. The cost to test drugs or theories on humans is extensive. Most scientists don't have the funds to redo these experiments, and wouldn't want to either -- the grant money they receive would be towards building on previous results. No funding agency would give money to re-verify a study already published.
At the very least, the authors could make ALL data available for someone to check. Data can be misrepresented very easily, especially in statistics, so having an independent group to verify results would be very welcome.
IMHO the problem here is over-reliance on statistical significance as a sign of the validity of a hypothesis. Statistical signficance is simply an estimate of the probability of Type I error. Although it is necessary to find significance in order to provisionally reject the null hypothesis, significance by itself is not sufficient to support the experimental hypothesis. What is necessary is some measure of effect size, or even better a measure of predictive power on a separate cross-validated data set. If all publications were required to report effect size as well as significance, problems with replication failure would be reduced substantially.
Stick to the subject drone, you posit that I am a racist and I reject that. You are the one making racist statements.
As to Obama's Marxism this is well known, the problem is you don't even know what the term means so you wouldn't know one if you saw one.
Read and learn drone.
http://www.amazon.com/exec/obidos/ASIN/1451698097/theamericansp-20/
I continually am amazed at the lack of understanding of these things around here. You truly think of Obama as conservative?
First of all lets get our terms straight, I am talking about the western definition of conservative and liberal, not the classical liberal.
Now I have two questions for you, you say Obama is 'moderately conservative', please name one of these conservative things he has done?
Secondly, you really have to explain what the hell you mean by 'pro corporate welfare capitalist'. Do you even understand what the term capitalist means? Are you actually thinking of crony capitalism, which is a real term and BTW has nothing whatsoever to do with capitalism (more accurately referred to as the free market)? Crony capitalism in reality is essentially a form of fascism in which the leaders of industry and those of the state work in collusion to manipulate the market; case in point would be Obama bailing out GM (and in doing so what he really was trying to do was bail out the unions - you understand this right? Unions are socialist organizations at their core - you understand this right?), then working in GM's favor with tax breaks, regulations targeted at their competitors, etc. All of these things are facts - you understand this right? These are not the actions of a conservative by any stretch of the imagination.
So do tell, what things has Obama done that qualify him as a conservative? And what the hell exactly is a 'pro corporate welfare capitalist'?
Address my points rather than trying to dodge them. How are any of the actions that I listed that Obama has done are even remotely considered 'Marxist'? The vast majority of the laws he signs and executive orders are basically corporation-enriching and basically further the fascist plice state setup by the Bush Administration. No Marxist would agree with the vast majority of the shit he's done.
Stick to the subject drone, you posit that I am a racist and I reject that. You are the one making racist statements.
As to the rest of your rant look up two posts, research this
http://obamaism.blogspot.com/
And then go fuck yourself.
Isn't the purpose of publishing your research to open it up to scrutiny, and yes validation of your results? In college I read a few dozen papers which merely validated the results of another paper.
Only the dead have seen the end of War. - Plato
For example, the Organic Syntheses Journal has been doing it for decades now, and for all submissions, not just "landmark" papers (how do you know this a priori that a paper will be extremely important in a reproducible and relaiable fashion, is unclear to me)
See: http://www.orgsyn.org/submission.html for details
Another optios is that papers that have been independently reproduced bear a tag or distinctive category than the ones that went *only* through peer review process (which might not entail reproducing the experimental work).
So all we need to do now is wait for someone to build another LHC, find the Higgs themselves, and confirm the CERN result. Then they get to publish!
If I can be modded down for being a troll, can I be modded up for being an orc, or a balrog?
Where'd you go drone? You make a statement, do you stand by what you have said or not? I think you have no idea what you are talking about.
Silence is acceptance.
I like the idea, but, how do you fund this effort? I don't see the article making any mention of this.
We all spend our time writing grants now to support our own research and have little enough time to do it. Now we're expected to do someone else's research? I suppose it's a bit like reviewing articles, if you want to publish, you should review. However, this takes much more time, effort and money to do.
So drone where did you go drone. You cannot support your statement. You are a coward.
I have read some of your other posts, frankly you are batshit crazy.
You are either very young or very stupid. Quite possibly both. Go ahead and try to support your contention that Obama is a conservative, you will only show everyone more fully how much I am right.
http://obamaism.blogspot.com/
You really are not worth replying to you anonymous coward. But I will inform you anyway.
Yes. Obama really is conservative.
The "American" definition of conservative is not conservative. It is a radical regressive movement attempting to overturn the liberties that we have fought and died for since the American and French revolutions. It is attempting to return to a dark age where a select class of gentry wields absolute authority and power through economic, political and superstitious means. In the past, this was done through the concept of the divine right of kings and through the threat of supernatural punishment/reward by various religious institutions. Now they use a perversion of the free market ideal instead of the divine right of kings to wield power through corporate proxies.
Obama is conservative. He is attempting to maintain the status quot and protect the institutions are currently in place. If you have actually read the wealth of nations by Adam Smith you would see that he makes very strong warnings about the inherent problems with his abstract model that require regulation to manage. So yes, Obama is a capitalist. However, he also plays authoritarian games and transfers wealth from the people to select interests through corporate tax breaks, grants and other programs. It is welfare for corporations, which only servers to enrich the wealthy.
He is not progressive, or liberal. And neither progressive, liberal or even socialist indicate Marxism at all.
I am not the ignorant one here. Anyone calling Obama a marxist is either stupid or ignorant.