Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
When you're talking about scientific papers, a "bad apple" reviewer may be able to skew the record in terms of 1-10 scales, but reviewers also do a qualitative write-up of the material. That's really the only important part and if one or two people fall outside the line of general consensus, they'll just be ignored.
A referees rejection can be overruled by the editor. It's his job to choose referees that will understand the research and make sure they are just.
It is done as you've guessed, but it's still often obvious who the author is. Don't forget that sometimes a bad review has nothing to do with knowing who the author is. If you come across a paper that's done almost exactly the same work as you have done, or criticises your work, you could choose to give it a false bad review to try to prevent it from being published. I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.
jh
Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.
If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!
Broken record time, but yes. Such subversion of the peer review process did show up. The culprits weren't the ones you expect.
In general however, I think that this study is rather pessimistic. And anyway, it hasn't been peer reviewed, so who knows... ;-)
(yes, I did read TFA, but not the paper
Social Text is emphatically not prominent nor was peer-reviewed at the time of the affair.
http://en.wikipedia.org/wiki/Sokal_affair
While the Sokal affair is interesting, it has nothing to do with the matter at hand.
entropy happens
I have several publications that were significantly improved through the peer review process. When I review papers my goal is not to shoot down the work, rather I try find ways to improve it. Of course there are 'bad' reviewers, who think that reviewing a paper is shredding it to pieces. These are actually easy to spot, because they rarely suggest anything useful and are often ignored by the journal editors. Speaking of which, journal editors are yet another part of the peer review process that is missing from their model
Speaking as someone who's at least read up on this stuff (not an expert, but definitely an informed layman), such large-scale adoption of nuclear power comes with its own problems. For one, building such plants is going to be extremely costly, and probably can't be done in time to make a useful difference.
You talk about reprocessing, but even after that, you eventually end up with some radioactive waste products, to say nothing about radiation leakage into the environment.
Finally, I think it's just yet another "all our eggs in one (radioactive) basket" solution. I'd rather have a wide range of options, from renewables like wind, solar or geothermal, to, yes, nuclear power where that's appropriate.
It's difficult to comprehend why a place with ample local generation capability (say, solar power in the Thar desert in India) should go with an expensive nuclear power plant, when the alternative is cheaper, and a more efficient use of resources readily available (as opposed to resources mined from the ground a few thousand kilometers away in another continent), as you "nuclear only" types keep coming up with.
You left one out: ignorant fools who conflate highly radioactive waste products with wastes with long half-lives. If you listen to them, you come away with the impression that the wastes from a reactor stay Highly Radioactive for thousands of years, ignoring the fact (if they're even aware of it) that unstable isotopes are either highly active or have long half lives, never both, because the two qualities are mutually incompatable.
Good, inexpensive web hosting
As usual, the strong caveat at the end of the article goes unnoticed:
But Tim Smith, senior publisher for New Journal of Physics at IOP Publishing, which also publishes physics world.com, feels that the study overlooks the role of journal editors. "Peer-review is certainly not flawless and alternatives to the current process will continue to be proposed. In relation to this study however, one shouldn't ignore the role played by journal editors and Boards in accounting for potential conflicts of interest, and preserving the integrity of the referee selection and decision-making processes,"
IRL the reviewers are not chosen at random. Which burns the straw men built by the summary, most of the article, and the skeptics.
One of these days I'm going to cut you into little pieces. - PF