Slashdot Mirror
Positive Bias Could Erode Public Trust In Science
ananyo writes "Evidence is mounting that research is riddled with positive bias. Left unchecked, the problem could erode public trust, argues Dan Sarewitz, a science policy expert, in a comment piece in Nature. The piece cites a number of findings, including a 2005 paper by John Ioannidis that was one of the first to bring the problem to light ('Why Most Published Research Findings Are False'). More recently, researchers at Amgen were able to confirm the results of only six of 53 'landmark studies' in preclinical cancer research (interesting comments on publishing methodology). While the problem has been most evident in biomedical research, Sarewitz argues that systematic error is now prevalent in 'any field that seeks to predict the behavior of complex systems — economics, ecology, environmental science, epidemiology and so on.' 'Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves,' he adds. Do Slashdot readers perceive positive bias to be a problem? And if so, what practical steps can be taken to put things right?"
They're overstating precision. Which rather then a forgivable error is an elementary mistake no trained scientist should ever make.
A VERY basic concept they teach at the lowest level of science education is the distinction between accuracy and precision. This is science 101.
Accuracy is whether or not a given conclusion is correct.
Precision is to the degree of specificity.
Typically you run into problems on complex subjects because they overstate the precision of their data or their ability analyze the data.
This can boil down to simple thinks like significant digits.
For example, I'm measuring volume to two significant digits in a giant data set with thousands of measurements. When and if I average those numbers the final average can't have more then two significant digits. That sounds elementary but you see this error made on some big studies. You'll have a situation where something is being measured in a crude sense by many sources and then in the analysis a much higher degree of specificity is implied.
Often that degree of specificity is required to make certain conclusions which is why they break the rule. This is lazy and a breach of scientific ethics. What they need to do is collect the data all over again this time to the level of specificity they need.
Simply saying its too hard to collect the data properly so they're going to make assumptions is not reasonable or ethical. I suppose you could do it so long as you kept an asterisk next to the data and the findings to make it very clear throughout that the conclusion is a guess and not in any way empirical science since at some point people were guesstimating results.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
Positive bias works a little differently than you think. First, a few definitions. Positive result looks like this: "We tried X on Y and it works." Negative result looks like this: "We tried X on Y and it doesn't work." Which one looks more interesting? That's right, the positive one. Now remember that outside mathematics, science is stuck with probabilities. There's always a very small chance of false positives (and also false negatives, but those are less of a problem).
So let's suppose that we have an experiment which should come out negative with 99% certainty. There's 1% chance of false positive. Now let's have 100 scientists independently perform the experiment. Chances are that 1 scientist will get a false positive. This scientist will then publish a paper while the other 99 will give up and research something else without publishing because of the impression that negative results are not interesting.
This is how positive bias works. Those 99 negative outcomes need to be reported to show that the one false positive is a false positive but they aren't reported. It's not some kind of intentional fraud but merely a consequence of overt obsession with original and sensational results dictated by those who pay for research. It doesn't matter what the research actually means, only that it can be phrased as "We tried X on Y and it works."
To me it seems a perfect example of how NON SCIENTISTS have made the expectations that scientists are not able to live up to.
The report was basically "Hey, guys, this looks odd, but we've corrected for everything we could think of. You may want to see if you can replicate this".
Nobody else managed to replicate.
MEANWHILE the original posters were being told how they had it wrong. They changed their process to accord for this and retained some extra difference, despite this. Another attempt by another team showed no effect, so the original scientists CONTINUED to see what could be the cause if not FTL neutrinos.
They then found that the 50ns difference can be accounted for if the connectors were not tight.
NOT, at that time, that this was the cause of the discrepancy, but that
a) the couplings were loose NOW, but they didn't check THEN, could have loosened since then
b) an effect of this is a slight delay, enough to remove the difference they'd seen
All absolutely fine and no "positive bias" AT ALL.
But what is the public saying about it? What were the press saying about it?
THAT debacle (and how it's ignored by the parties involved in it) is a perfect example of the problems science has with people who aren't scientists.