Peer Review Highly Sensitive To Poor Refereeing

$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."

Comments

Re:Highly political subjects?

[prefect42]

It is done as you've guessed, but it's still often obvious who the author is. Don't forget that sometimes a bad review has nothing to do with knowing who the author is. If you come across a paper that's done almost exactly the same work as you have done, or criticises your work, you could choose to give it a false bad review to try to prevent it from being published. I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.

Bit of an arbitrary model

[gnutrino]

First off in case anyone is in doubt this study use a model of peer review - no experiment or observation of an actual peer review process was done. That's not to say interesting and enlightening things can't come from modeling but in this case the moldel they use seems very questionable and highly arbitrary. This part in particular is highly dubious:

Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.

If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!

Re:Climategate for example

[oiron]

Broken record time, but yes. Such subversion of the peer review process did show up. The culprits weren't the ones you expect.

In general however, I think that this study is rather pessimistic. And anyway, it hasn't been peer reviewed, so who knows... ;-)

(yes, I did read TFA, but not the paper

Re:The Social Text Affair

[iris-n]

Social Text is emphatically not prominent nor was peer-reviewed at the time of the affair.

http://en.wikipedia.org/wiki/Sokal_affair

While the Sokal affair is interesting, it has nothing to do with the matter at hand.

I call shenanigans on this one

[pesho]

Chances are that this paper is not going to pass peer review. A brief read paper shows that they don't even attempt to validate the model with real data (too lazy for real research I guess). Their model is also overly simplistic to the point of stacking the deck towards proving that peer review is bad. The reviewer role is simplified to an accept/reject decision, which has nothing to do with reality. They completely eliminate the revision step in the peer review process, where authors address either through new experiments or through argument the comments of the reviewers. If you look at the 'characters', what they call a 'rational' reviewer looks more like a 'bastard' reviewer. They completely ignore the possibility that a reviewer can make suggestions that improve the paper.

I have several publications that were significantly improved through the peer review process. When I review papers my goal is not to shoot down the work, rather I try find ways to improve it. Of course there are 'bad' reviewers, who think that reviewing a paper is shredding it to pieces. These are actually easy to spot, because they rarely suggest anything useful and are often ignored by the journal editors. Speaking of which, journal editors are yet another part of the peer review process that is missing from their model