Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.
I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.
I mean scientists who publish among themselves, i.e. inside their narrow specialty, in their own journals, without checking whether the problem at hand has been solved elsewhere. This is more and more common as people get more specialized, and can lead to very basic errors propagated inside the whole community, like rheologists believing in the existence of pure elongational flow (a trivial misunderstanding of tensor algebra). Since the peers reviewing the papers are members of the same community, those errors usually get unnoticed.
Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.
I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
Well that's not always the case. Different journals have different review processes. Some ask for numerical choices on a scale, others want choices in terms of "strongly agree", "somewhat agree" etc. for specific questions, others want only written comments and a final choice. Even this final choice is different in many cases, sometimes restricted to Accept, Accept with minor corrections, Accept with major corrections, Invite for resubmission and simply Reject, while others take the final choice as an aggregate of multiple choice responses or numerical averages. Some systems are obviously easier to be biased with than others.
Regardless of all this though, sometimes you'll find out that only two of three reviewers responded, and at least one of those probably got one of their postdocs or even a PhD student to do the review. Some reviews will have empty parts where a reviewer was supposed to write a paragraph but couldn't be bothered, or because they didn't want to reveal the fact that they were totally unfamiliar with the subject matter. Getting a journal paper published is more hit and miss than you'd think. I used to think that a good paper with good ideas was enough, but it's not always the case.
You can advertise in this sig from as little as £99.99 a month!
Slashdot let's you publish first, and be reviewed later. The peer-review system used by scientists forces them to work on their papers until someone finally "mods" it acceptable. Imagine how much faster science could advance if we had a system that actually let scientists focus on research, let people trained in technical writing do the reporting, and let Google design a post-publication moderation system to sort out the useful advances from the career posturing. Science could learn a lot from Slasdot. It is simply ignorant that we continue to put up huge barriers to publication.
I actually read the comments on TFA, and down at , there's a particularly interesting one:
This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.
There's actually a reasonably intelligent discussion going on in there...
My dad (has PhD in a scientific field from Cornell) told me that when submitting a thesis to a review board of professors, it really doesn't matter how "Tough" a professor is as long as that professor in your committee has a rival. Take advantage of their ego with an equally assertive ego. You purposefully choose the rival professor to join your committee as well. Then they'll spend all the review board discussions and presentations contradicting and arguing one with another, and in the end they'll both be so incensed, that they cancel each other out, and it doesn't matter what you presented... I guess the TFA is only pointing out that this occurs at the publishing level as well.
http://www.beanleafpress.com
As a computer scientist, my impression is that the program committees really are pretty random, or at least based on some sort of preference other than a widely agreed "quality" standard. Try it sometime: resubmit a paper rejected from a top CS conference verbatim to another top CS conference. The correlation between the reviews is usually quite low, both in terms of the numerical scores, and especially in terms of what they liked / complained about.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
I've long thought there should be a "-1, Disagree" option in the drop-down box that takes a mod point but has no effect.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
But the proper way to communicate disagreement should be to respond with a reasoned counterargument, the goal being to show the justification for your disagreement and allow future readers the benefit of seeing your reasoning.
A "-1 Disagree" mod is anonymous censorship at it's worst. It adds nothing to the discourse. If all that you can add to the discussion is "I Disagree", then you can't add anything of merit to the discussion.
Contains such howlers as "There's no such thing as average temperature"...
Why is that a howler? It seems like uncontroversial physics if by "no such thing" you mean it has no physical meaning.
If I take two bodies, one with temperature T1 and one with temperature T2, what is their average temperature? If you say (T1+T2)/2 you are mathematically correct, but thermodynamically incoherent.
There is in general no thermodynamically meaningful way of averaging temperatures, which is why we should be talking about atmospheric heat content, not temperature, and why ocean temperatures--which are rising--are by far the most plausible evidence for increasing heat content in the troposphere.
Blasphemy is a human right. Blasphemophobia kills.
The poor review assignment at large conferences contributes to that effect as well, I think. I almost always have at least one of three reviewers, and sometimes even two of three, give a noncommittal review along the lines of, "well this isn't really my area, but it seems pretty good". Those reviews basically are non-reviews, so the acceptance decision is then entirely up to the remaining one or two reviewers. So it often comes down to: did the one person who actually provided an opinion on your paper like it or not like it?
In my experience that's often pretty subjective, especially for conferences with tight length limits (standard in AI is six pages). If the reviewer personally found the paper to be on an interesting subject with an interesting approach that he/she felt should be investigated, almost any shortcomings can be excused, and the reviewer will conclude that "Overall, this paper provides a valuable contribution to an important ongoing discussion in this area." But if the reviewer doesn't like it, finds it boring, dislikes the approach, etc., it's easy to find something that had insufficient detail, didn't sufficiently distinguish from related work, didn't sufficiently motivate the problem or investigate/validate the applications, etc., etc., since you really can't fit that much in six pages.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10