Peer Review Highly Sensitive To Poor Refereeing

$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."

Comments

The climate skeptics will have a field day

[Anonymous Coward]

This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.

I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.

Highly political subjects?

[AnonymousClown]

"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.

So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?

Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.

That's the way I'm interpreting this study.

Re:Highly political subjects?

[MozeeToby]

Can anyone give me a good reason why the reviewers get information about the author in the first place? Granted, there are disciplines that are closed knit to the point that the reviewers would recognize the author based on their past work, but in most cases I would think not knowing who the author is would address at least some of the issues that they highlighted here. It's hard to obscure the rest of the review process but limiting nepotism should be relatively simple.

Re:Highly political subjects?

[prefect42]

It is done as you've guessed, but it's still often obvious who the author is. Don't forget that sometimes a bad review has nothing to do with knowing who the author is. If you come across a paper that's done almost exactly the same work as you have done, or criticises your work, you could choose to give it a false bad review to try to prevent it from being published. I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.

Re:Highly political subjects?

[Trepidity]

As a computer scientist, my impression is that the program committees really are pretty random, or at least based on some sort of preference other than a widely agreed "quality" standard. Try it sometime: resubmit a paper rejected from a top CS conference verbatim to another top CS conference. The correlation between the reviews is usually quite low, both in terms of the numerical scores, and especially in terms of what they liked / complained about.

Re:Highly political subjects?

[DriedClexler]

Granted, there are disciplines that are closed knit to the point that the reviewers would recognize the author based on their past work

Pardon my naivete, but I don't think such fields should exist. They present an extremely hazard for groupthink and inbred rubber-stamping.

Any speciality should bend over backwards to maintain close ties with the surrounding fields of research so that others will understand how it relates and better be able to detect when bad practices are becoming standard. And it is vanishingly unlikely that this super sub-speciality will *never* stumble upon a problem isomorphic to a well-studied one in a distant field.

It's because of this "oh this is a hard subfield, stay off my turf" mentality that causes things like ecologists *just now* starting to use the method of adjacency matrix eigenvectors (i.e. PageRank) to identify critical spiecies, despite the method having been known to mathematicians for 40 years.

Hey scientists: science is a group process. You're special, but you're not that special. Please build off of the existing work. Don't compartmentalize. Good science connects, and connects deeply. Yours should too.

It's part of automating the process.

[Remus Shepherd]

Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.

I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.

Re:It's part of automating the process.

[starless]

Except that I've heard of people deliberately adding people to acknowledgements to try to make sure they don't get those people as referees (and it hasn't worked)!

Re:Review content matters

[Shrike82]

Well that's not always the case. Different journals have different review processes. Some ask for numerical choices on a scale, others want choices in terms of "strongly agree", "somewhat agree" etc. for specific questions, others want only written comments and a final choice. Even this final choice is different in many cases, sometimes restricted to Accept, Accept with minor corrections, Accept with major corrections, Invite for resubmission and simply Reject, while others take the final choice as an aggregate of multiple choice responses or numerical averages. Some systems are obviously easier to be biased with than others.

Regardless of all this though, sometimes you'll find out that only two of three reviewers responded, and at least one of those probably got one of their postdocs or even a PhD student to do the review. Some reviews will have empty parts where a reviewer was supposed to write a paragraph but couldn't be bothered, or because they didn't want to reveal the fact that they were totally unfamiliar with the subject matter. Getting a journal paper published is more hit and miss than you'd think. I used to think that a good paper with good ideas was enough, but it's not always the case.

Re:Just like the Slashdot moderation system

[spikenerd]

Slashdot let's you publish first, and be reviewed later. The peer-review system used by scientists forces them to work on their papers until someone finally "mods" it acceptable. Imagine how much faster science could advance if we had a system that actually let scientists focus on research, let people trained in technical writing do the reporting, and let Google design a post-publication moderation system to sort out the useful advances from the career posturing. Science could learn a lot from Slasdot. It is simply ignorant that we continue to put up huge barriers to publication.

Re:just like /.?

[alvinrod]

As much crap as /.'s moderation system gets, it actually tends to be one of the better systems I've seen on the net. First of all, it's highly customizable so if I wanted to I can easily set it to add or remove value from certain types of moderations. I usually bump flamebait and troll up a few pegs just so I can see the posts that do occasionally get unfairly moderated. I can also add other posters I find interesting to a list and bump up their post value so if I'm interested in what they have to say I can always make sure I'll see it.

I also think that the community goes a long way towards making the system work well. Sure there will always be people who abuse the system and moderate posts with which they disagree as flamebait, etc. but the community as a whole does a good job of promoting interesting lines of conversation and for any given topic there are probably a few people in the community who specialize in that area and can provide some excellent commentary.

It's not perfect, but it's probably one of the best systems in actual practice that's currently being used.

Bit of an arbitrary model

[gnutrino]

First off in case anyone is in doubt this study use a model of peer review - no experiment or observation of an actual peer review process was done. That's not to say interesting and enlightening things can't come from modeling but in this case the moldel they use seems very questionable and highly arbitrary. This part in particular is highly dubious:

Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.

If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!

Re:Climategate for example

[oiron]

Broken record time, but yes. Such subversion of the peer review process did show up. The culprits weren't the ones you expect.

In general however, I think that this study is rather pessimistic. And anyway, it hasn't been peer reviewed, so who knows... ;-)

(yes, I did read TFA, but not the paper

Doing something unprecedented

[oiron]

I actually read the comments on TFA, and down at , there's a particularly interesting one:

This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.

There's actually a reasonably intelligent discussion going on in there...

Using Professors' Egos to your advantage...

[happy_place]

My dad (has PhD in a scientific field from Cornell) told me that when submitting a thesis to a review board of professors, it really doesn't matter how "Tough" a professor is as long as that professor in your committee has a rival. Take advantage of their ego with an equally assertive ego. You purposefully choose the rival professor to join your committee as well. Then they'll spend all the review board discussions and presentations contradicting and arguing one with another, and in the end they'll both be so incensed, that they cancel each other out, and it doesn't matter what you presented... I guess the TFA is only pointing out that this occurs at the publishing level as well.

Re:The Social Text Affair

[iris-n]

Social Text is emphatically not prominent nor was peer-reviewed at the time of the affair.

http://en.wikipedia.org/wiki/Sokal_affair

While the Sokal affair is interesting, it has nothing to do with the matter at hand.

Re:just like /.?

[MozeeToby]

I think it comes down to two things.

First, the relative rarity of mod points encourages people to take it seriously. It also encourages the most active, most interesting posters to give up posting once in a while to moderate. Most other sites use a mod system that allows so many votes per article but still don't allow you to post and mod the same article. That means that the most frequent posters will seldom mod and that there can be people who only ever mod articles without ever commenting on them.

Second, attaching a reason to the mods encourages people to actually think about why they are modding the way that they are. As many people say, there is no "-1 I Disagree" mod, in order to mod someone down you have to be saying that they are actively trying to derail the conversation. Of course, lots of modders will ignore that and mod however they want, but I think that it does make at least some people stop and think before they accuse someone else of being flamebait or a troll.

Re:You mean whine when a POS paper is printed

[oiron]

I agree with all that you said, but for heaven's sake, use full-forms at least once

For those as confused as I, the paper the parent refers to is "Gerhard Gerlich, Ralf D. Tscheuschner. Falsification of the atmospheric CO2 greenhouse effects within the frame of physics.".

Contains such howlers as "There's no such thing as average temperature"... RC Wiki page

Re:just like /.?

[Trepidity]

I've long thought there should be a "-1, Disagree" option in the drop-down box that takes a mod point but has no effect.

Re:You mean whine when a POS paper is printed

[radtea]

Contains such howlers as "There's no such thing as average temperature"...

Why is that a howler? It seems like uncontroversial physics if by "no such thing" you mean it has no physical meaning.

If I take two bodies, one with temperature T1 and one with temperature T2, what is their average temperature? If you say (T1+T2)/2 you are mathematically correct, but thermodynamically incoherent.

There is in general no thermodynamically meaningful way of averaging temperatures, which is why we should be talking about atmospheric heat content, not temperature, and why ocean temperatures--which are rising--are by far the most plausible evidence for increasing heat content in the troposphere.

I call shenanigans on this one

[pesho]

Chances are that this paper is not going to pass peer review. A brief read paper shows that they don't even attempt to validate the model with real data (too lazy for real research I guess). Their model is also overly simplistic to the point of stacking the deck towards proving that peer review is bad. The reviewer role is simplified to an accept/reject decision, which has nothing to do with reality. They completely eliminate the revision step in the peer review process, where authors address either through new experiments or through argument the comments of the reviewers. If you look at the 'characters', what they call a 'rational' reviewer looks more like a 'bastard' reviewer. They completely ignore the possibility that a reviewer can make suggestions that improve the paper.

I have several publications that were significantly improved through the peer review process. When I review papers my goal is not to shoot down the work, rather I try find ways to improve it. Of course there are 'bad' reviewers, who think that reviewing a paper is shredding it to pieces. These are actually easy to spot, because they rarely suggest anything useful and are often ignored by the journal editors. Speaking of which, journal editors are yet another part of the peer review process that is missing from their model