Slashdot Mirror
Most Scientists 'Can't Replicate Studies By Their Peers' (bbc.com)
Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. From a report: This is frustrating clinicians and drug developers who want solid foundations of pre-clinical research to build upon. From his lab at the University of Virginia's Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies. "The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results." You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable. The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions. Sadly nothing, it seems, could be further from the truth.
If you can't reproduce it, it's either fake or you were just being sloppy. Either way, it's no wonder ordinary civilians have doubts.
Drug trials are limited in scope because there are restrictions on the patents of the studied compounds, which greatly limits the capacity to replicate the trial. Multiple studies have been done on climate, which is more open access.
As per the subject, this comes from the collision of two things that are completely counter to the process of science.
1) Patent theory. Since many more nations have access to patents than actually respect patents, it is self-destructive to put enough detail in a patent to actually build what the patent is for. For research papers, this has the benefit of being informed of any attempts to replicate the study, because the other labs will call in and ask questions to find out what was left out. This lets the guy who signed the paper know who is working in the same field and can decide whether to be helpful or antagonistic.
2) Fiction. "Publish or perish! " "No one actually reads the papers!" "Everyone else is too busy writing their own papers to even look at replicating the experiment, who cares if it doesn't actually work?" And other little labor-saving excuses that show scientists are just as dishonest and self-serving as politicians.
In the experimental protocols listed in a paper, it is not unusual to have a method's section that is more or less an executive summary rather than a very detailed account of the underlying protocol. This is for two reasons: to great a level of detail leads to a methods section as big as the publication that the paper appears in, and second because many protocols more or less boil down to using a particular series kit or out-sourced lab service. Most journals require data supplements where an author must share their datasets in electronic form as an online addendum to the publication. I would support a similar requirement for a long-form protocol for reproduction of the study.
That said, some protocols necessarily take a lot of money, special equipment, a carefully selected population of volunteers, and time. Reproducing some studies can be outright impractical.
In computational biology and other computational extensions of the physical science, the reproducibility basically comes in the form of requirements to provide the software and raw data for a study. It's easy for the individual that compiles this information to verify that they get the same result as the one they report in the article. The concern there boils down to the provenance of the source data, which may be from registries, public data sets, or some combination of public and private data.
getting the results you need means you can push a drug through trials and make a lot of money
if you hurt or kill someone it won't happen for 20-30 years and by that time you will be retired and the person in charge at the time will be legally responsible while you chill in your nice house
Many studies have been done on anthropic climate change, but almost no experiments.
The authors should have done it themselves before publication
In the rush to "publish or perish," you don't have time to re-run your experiment.
A "solution" would be "split publication" - publish results after the first experiment but call it "unverified." Then when you or another researcher reproduces the experiment, publish again.
The first researcher would receive the primary "credit" but only if the results held up under scrutiny.
Over time, researchers who accumulated a lot of "un-verified" initial publications would see their reputations suffer.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
If they have trouble reproducing studies maybe they need to go back to science school. Or look up "science" on wikipedia and do more learning.
Ah, because there's no way in hell that the initial experiments could have been fabricated to favor certain outcomes, especially within the trillion-dollar Cancer Treatment Complex, right?
Yeah, you're right. Over 65% of trained researchers must be stupid or something...
Scientists are not rewarded for reproducing/debunking previous work. You can't easily get it published, because it is not regarded as new. Honestly, I think grad student projects should be almost entirely reproducing other results. It would insure that every important result is reproduced, and increase the emphasis of doing the science correctly rather than finding some novel result (which is usually a 2 sigma result which again can't be reproduced).
The purpose of peer review is to identify incorrect theories and throw them out. This article says an awful lot more about the state of research today than it does about peer review which is doing what it's supposed to be doing.
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
This is medical research, not science. Medicine uses science because often the best way to cure something is to understand it but, very importantly, it has a very different motivation to science. Finding a "magic" pill which cures disease X without side effects but whose mechanism is completely unknown is great medicine but appalling science. Science is all about understanding how things work, medicine is all about treating human ailments.
This leads to a different approach using the tools of science. Medical researchers tend to focus far more on correlation over causation because that is what is most important to this. Unfortunately this approach leaves them open to random statistical effects which require a very good understanding of statistics to avoid and even then it can still be very easy to fool yourself e.g. the Monty Hall effect.
So lets call this problem what it is: a problem with medical research.
The human body is the most complex organism in the known universe so there's nothing to be sneezed at or be surprised by. For instance recent studies have shown that for a lot of people placebo works even when people have a perfect knowledge that they are given placebo.
As another confirmation, the brain has the ability to directly change/affect the chemical processes in the body as demonstrated by Wim Hof who can manage his body's temperature at will.
If I were publishing a paper on something that could lead to a serious pile of greenbacks, you can be damn sure my paper is going to exclude some details that would prevent others from monetizing off of my work...
After all, science these days is not solely for the pursuit of truth and knowledge. Research is bought and paid for, and like any venture capital, the investments are expected to pay off.
Many studies have been done on anthropic climate change, but almost no experiments.
The infrared absorption of carbon dioxide is experimentally measured in the laboratory. And there is a vast amount of measured data on the earth's atmosphere and climate, from surface, atmospheric, and orbital probes, not to mention probes of other planets; and we acquire terabytes of additional data every year.
The basics of Earth's energy balance are well understood, and they are understood, in part, because of this vast amount of experimental and observational data.
http://www.geoffreylandis.com
It's not heresy if the science is sound. Simply questioning isn't valid, though.
Questioning, of course, is always valid. But "questioning" is useless when the questioner has no interest in listening to anybody answering the question.
Far too much of the "questioning" about climate science is from people who have no interest in any of the science, the measurements, or the data, and won't bother to learn anything about it.
http://www.geoffreylandis.com
Many studies have been done on anthropic climate change, but almost no experiments.
There's one rather large experiment going on right now. Unfortunately we're all inside the test tube. So far it's turning out more or less how we expected.
When instrument calibration is examined and bad data thrown out, the researcher involved is accused of manipulation of the data to suit nefarious ends.
This of course presumes the experiment **can** be replicated.
As for "peerless", there have been researchers at top labs (Bell, for one) that have fabricated their research.
he infrared absorption of carbon dioxide is experimentally measured in the laboratory
No one rational doubts this. That has never been what the climate change debate was about. But the atmosphere is not a bottle of air, or even a bottle of air and water (any modern meteorological model treats modeling he ocean at least as importantly as modeling the air). The atmosphere+hydrosphere is a complex, evolving system with many feedback mechanisms, both positive and negative.
I mean, really, do you think a climate model is simply modeling a static stack of air with some CO2 in it? Really?
The question is: quantitatively, what rate of human CO2 emission with create what effects, in detail. This is not the sort of science that lends itself to reproducible experiments, but that's fine, neither does astronomy or cosmology. It is, like any science, required to make falsifiable quantitative predictions.
And, frankly, the best models aren't doing so well, giving about 2 sigmas of accuracy. If you generated hundreds of models at random, you'd expect a couple dozen to have 2 sigmas of accuracy. That doesn't mean the models are flawed in any fundamental way, but there's a big gap between "not fundamentally flawed" and "great, proven science".
Socialism: a lie told by totalitarians and believed by fools.
"Yes, but how FAST is it sinking?"
"We're not entirely sure, but we know it is because we keep drilling holes."
"Well, we better keep drilling until we know how fast, just to be safe."
No, that's the whole point of your debate. Sadly you are outshouted by the people who deny that its happening. I'm all for the debate you want to have.
Yet in this chaotic system, the record highs increasingly outweigh the record lows, suggesting an increasing upward trend.
Ezekiel 23:20
Calling something expensive doesn't make it less urgent. Humans will suffer (possibly) from the increased cost of building solar and wind plants (even though that will create a crap ton of jobs and reduce dependency on foreign energy sources.) But HUMANKIND will suffer for our shortsightedness if we delay indefinitely. And trust me, just because you don't know how soon, doesn't mean it won't be SOON. All in all, I think my point was salient. Already, in my life, glaciers here as long as humankind have just melted right a way. Assuming more adverse effects won't happen in your human lifetime is like standing in a storm shouting "It can't rain any more than it already has." You might want to meditate on that for a minute before you go around accusing people of being ignorant. Sometimes things are hard, that doesn't mean they aren't the right thing to do. You know there is a problem, put on your big boy breeches and help fix it.
I was just commenting on the premise that greed == always bad == corporations. Greed can also exist in individual scientists and bureaucrats and can be against the best interest of the corporation (or the funders of the project). Cancer cures are a good thing. Cancer cures requires work and investment capital. Scientists need to be paid (along with everyone else including HR and people mopping the floors) Investment capital needs to be repaid with dividends. All the above are good good things.
Common F. Sense agrees that all of the above are good things
The problem is Greed N. Corruption isn't really interested in curing jack shit anymore, and will always favor perpetual treatments to feed profits.
Treatments create unending profits.
Treatments create unending jobs.
Cures ultimately destroy jobs and severely limit perpetual revenue and profits, which does not pay the dividends that Wall Street now demands.
Those running counter to the best interests of those in Control will ultimately be removed from the equation.
Record high temps, record low temps. record rain, record drought.
That's actually what you'd expect with a chaotic system built of multiple random variables. It would be unnatural for weather to always be the same.
Actually it's not. It's a simple fact that in a stable system, as time goes on, there are fewer and fewer "record" events because each new record needs to be more extreme than all previously recorded events. Over time, record-breaking events decline significantly. So, an increase in record events is, by itself, evidence that the system is undergoing change.
Fanatically anti-fanatical
It will be tough to make a switch now but doing so will make it better and in the long run less painful.
How much of a switch? What's the benefit of the switch? What's the cost of the switch? No matter how frantically you wave your hands, you're not providing numbers.
Socialism: a lie told by totalitarians and believed by fools.