Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
I can't quite remember what it was, but I seem to remember seeing it everywhere. It was exactly like TFA article, though. Damn, what was that place called again?
Living With a Nerd
This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.
I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.
"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.
So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?
Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.
That's the way I'm interpreting this study.
RIP America
July 4, 1776 - September 11, 2001
When you're talking about scientific papers, a "bad apple" reviewer may be able to skew the record in terms of 1-10 scales, but reviewers also do a qualitative write-up of the material. That's really the only important part and if one or two people fall outside the line of general consensus, they'll just be ignored.
I mean scientists who publish among themselves, i.e. inside their narrow specialty, in their own journals, without checking whether the problem at hand has been solved elsewhere. This is more and more common as people get more specialized, and can lead to very basic errors propagated inside the whole community, like rheologists believing in the existence of pure elongational flow (a trivial misunderstanding of tensor algebra). Since the peers reviewing the papers are members of the same community, those errors usually get unnoticed.
Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.
I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
The great anecdote demeaning peer-reviewed journals is The Social Text Affair, where a prominent peer-reviewed journal published with enthusiasm the article "Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity", only to be informed it was, in fact, computer-generated gibberish submitted as a joke.
Can we get a "-1 Wrong" moderation option?
There are a couple of significant and important limitations in the model:
a) It assumes only two reviews per paper, and that the reviews are pure boolean, and that reviewer types are also pure and reviewers are randomly selected (when two of the classes of reviewers, 'mythantropes' (always reject) and 'altruists' (always accept) are specifically selected against by editors and PC chairs based on reputation).
b) It does not consider the cases (such as conferences) where there is a program committee meeting and the papers are not just considered on their own, but gone through a relative ranking process.
Test your net with Netalyzr
Slashdot let's you publish first, and be reviewed later. The peer-review system used by scientists forces them to work on their papers until someone finally "mods" it acceptable. Imagine how much faster science could advance if we had a system that actually let scientists focus on research, let people trained in technical writing do the reporting, and let Google design a post-publication moderation system to sort out the useful advances from the career posturing. Science could learn a lot from Slasdot. It is simply ignorant that we continue to put up huge barriers to publication.
There is an aforism: Democracy as a form of government is riddled with problems. However we are yet to invent anything better.
Same with the peer review. It has its problems. However, we are yet to invent anything better
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
As much crap as /.'s moderation system gets, it actually tends to be one of the better systems I've seen on the net. First of all, it's highly customizable so if I wanted to I can easily set it to add or remove value from certain types of moderations. I usually bump flamebait and troll up a few pegs just so I can see the posts that do occasionally get unfairly moderated. I can also add other posters I find interesting to a list and bump up their post value so if I'm interested in what they have to say I can always make sure I'll see it.
I also think that the community goes a long way towards making the system work well. Sure there will always be people who abuse the system and moderate posts with which they disagree as flamebait, etc. but the community as a whole does a good job of promoting interesting lines of conversation and for any given topic there are probably a few people in the community who specialize in that area and can provide some excellent commentary.
It's not perfect, but it's probably one of the best systems in actual practice that's currently being used.
Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.
If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!
Broken record time, but yes. Such subversion of the peer review process did show up. The culprits weren't the ones you expect.
In general however, I think that this study is rather pessimistic. And anyway, it hasn't been peer reviewed, so who knows... ;-)
(yes, I did read TFA, but not the paper
The peer review system is great for regulation, standardization and unification. However, all scientists that I've worked with/researched with/spoken with much about this topic admit that the system can be annoyingly flawed by group think and conformity. One bad apple ruins the bunch, right?
The good news? While this part of the scientific community is not immune to problems, the slack is picked up elsewhere: As long as methods, data and results are transparent, reproducible and published we can actually have quality science.
I often speak to people about scientific research and they're shocked that it's not full proof. This is kind of like buying software (perhaps even a Microsoft product) and finding that it's not perfect. Science is done by committee and progresses slowly. "If we know what we were doing, it wouldn't be called research" ~Albert Einstein
Then again, I'm an idiot....
...... and idiots rule the world....
I actually read the comments on TFA, and down at , there's a particularly interesting one:
This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.
There's actually a reasonably intelligent discussion going on in there...
My dad (has PhD in a scientific field from Cornell) told me that when submitting a thesis to a review board of professors, it really doesn't matter how "Tough" a professor is as long as that professor in your committee has a rival. Take advantage of their ego with an equally assertive ego. You purposefully choose the rival professor to join your committee as well. Then they'll spend all the review board discussions and presentations contradicting and arguing one with another, and in the end they'll both be so incensed, that they cancel each other out, and it doesn't matter what you presented... I guess the TFA is only pointing out that this occurs at the publishing level as well.
http://www.beanleafpress.com
Peer review only works if the reviewers can be trusted and don't form a clique to get their work in and keep other people out. Surely anyone with even basic knowledge of human psychology would understand this?
Because this is an important question for serious people, but has no bearing on why various cranks (Intelligent Design people, climate change "skeptics", Time Cube, etc.) may have trouble getting their work in print. Papers by such people generally don't end up in the peer review phase - they aren't sent out for evaluation by the journal, so peer review doesn't matter.
That said, peer review provides substantially the same benefit as those "shoplifters will be prosecuted" signs you see in department stores.
Shoplifters are very seldom if ever actually prosecuted - but the threat, even the vaguest menace - of public scrutiny has an impact on behavior. I'm not talking about scientific fraud (which peer review will seldom catch,) but about quality of reasoning, doing the needed controls, etc. We may have a system that rewards good research little-better than an unbiased coin, but the <b>perception</b> that it works, or that it might work for you, motivates people to do the work needed to survive peer-review.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
And it would also help the readers understand the article, a good referee report is quite illuminating. However, this has already been tried out by Nature in 2006
http://www.nature.com/nature/peerreview/
and didn't work so well. Apparently, scientists are somewhat reluctant to openly criticise each other's work. But there's PLoS ONE that is alive and well, giving us some hope.
Michael Nielsen has a fine essay about this in his blog:
http://michaelnielsen.org/blog/the-future-of-science-2/
entropy happens
I think it comes down to two things.
First, the relative rarity of mod points encourages people to take it seriously. It also encourages the most active, most interesting posters to give up posting once in a while to moderate. Most other sites use a mod system that allows so many votes per article but still don't allow you to post and mod the same article. That means that the most frequent posters will seldom mod and that there can be people who only ever mod articles without ever commenting on them.
Second, attaching a reason to the mods encourages people to actually think about why they are modding the way that they are. As many people say, there is no "-1 I Disagree" mod, in order to mod someone down you have to be saying that they are actively trying to derail the conversation. Of course, lots of modders will ignore that and mod however they want, but I think that it does make at least some people stop and think before they accuse someone else of being flamebait or a troll.
Letting technical writing people doing the writing won't work. A large part of the scientific writing is the discussion of the experiment, which not only helps the scientist clarify his or her own thoughts and gives insight into future experiments, but also really only is worth reading if the scientist or members of the experimental team do it themselves. Technical writers really only would have the ability to write the experimental procedure, and even then it would be hard. Since science is so specialized you'd have to have technical writers for thousands of subdisciplines, etc. This goes especially true for mathematics, where the writing procedure is very closely related to doing mathematics.
Already because of this, no time for the scientist would be saved. A Google moderation system would have two problems. First, it wouldn't save any time because you still have to have some person doing the reviewing, and secondly you have to have someone qualified doing the reviewing whom you can trust to some extent to review in confidence, for otherwise if there are certain major problems with the paper but a few good ideas, they can be "stolen" by others, which may become a problem.
"What lies behind us, and what lies before us are tiny matters compared to what lies within us." Ralph Waldo Emerson
Well, I thought i had developed a better system but the thesis was shot down in peer review.
Ice Cream has no bones.
The government uses peer review to evaluate proposals for science and engineering grants. The same issues probably apply to those evaluations.
I have experienced a situation in which one reviewer recommended turning down a grant for reasons that could be considered as biased, although the bias was groupthink rather than individual. The other reviewers were enthusiastic about funding the grant and regarded it as a potential game-changer. It didn't get funded. A few years later the game-changing nature of the technology was recognized, but it was too late for the original applicant.
I agree with all that you said, but for heaven's sake, use full-forms at least once
For those as confused as I, the paper the parent refers to is "Gerhard Gerlich, Ralf D. Tscheuschner. Falsification of the atmospheric CO2 greenhouse effects within the frame of physics.".
Contains such howlers as "There's no such thing as average temperature"... RC Wiki page
I've long thought there should be a "-1, Disagree" option in the drop-down box that takes a mod point but has no effect.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
But the proper way to communicate disagreement should be to respond with a reasoned counterargument, the goal being to show the justification for your disagreement and allow future readers the benefit of seeing your reasoning.
A "-1 Disagree" mod is anonymous censorship at it's worst. It adds nothing to the discourse. If all that you can add to the discussion is "I Disagree", then you can't add anything of merit to the discussion.
Contains such howlers as "There's no such thing as average temperature"...
Why is that a howler? It seems like uncontroversial physics if by "no such thing" you mean it has no physical meaning.
If I take two bodies, one with temperature T1 and one with temperature T2, what is their average temperature? If you say (T1+T2)/2 you are mathematically correct, but thermodynamically incoherent.
There is in general no thermodynamically meaningful way of averaging temperatures, which is why we should be talking about atmospheric heat content, not temperature, and why ocean temperatures--which are rising--are by far the most plausible evidence for increasing heat content in the troposphere.
Blasphemy is a human right. Blasphemophobia kills.
Damn, you're an asshole.
Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
"Imagine how much faster science could advance if we had a system that actually let scientists focus on research, let people trained in technical writing do the reporting, and let Google design a post-publication moderation system to sort out the useful advances from the career posturing."
/. about some massive new breakthrough that is going to cure cancer and halitosis but instead is just another small-medium technical advance and everyone demands to know who the jackass scientists were that overinflated their results, how dare those overfunded eltist blowhard frauds? Yeah, that's what almost happened to me. How exactly will your system avoid massively amplifying this?
A few things:
1. Where is this money to hire 10,000 technical writers going to come from when, excluding tenured faculty, all academic scientists are threatened with losing their jobs due to lack of funds at least one year in three?
2. What exactly is a technical writer going to be doing? They will never have the necessary background to write a review article (seriously--a review article can have anywhere from 150 to well over 300 referenced papers, selected from an even larger pool). A research paper is the description of what 1-30 people have spent the last couple of years doing, plus a small literature review, plus discussion and future directions. Is the technical writer going to go through a few to a few dozen notebooks from all of these people and assemble it, despite not knowing what's going on and somehow predict where it's going to go? Or are they there to check over a rough draft and polish it? If they're the polisher, how much time do I have to spend getting them to understand the terminology, which can be and often is extremely precise? How much time do I have to spend going over the paper, which will have my name on it and not theirs, and which could (especially in the case of a very poorly written or wrong paper) have a huge impact on my career, checking to make sure that the wording is correct? Nuance can be critical in a scientific paper, especially if you're attacking somebody else's results. Communities are small and egos can be large.
3. Your "technical writers write it and stick it on Google model" sounds like science reporting to me. I've had research described by the local paper. Thankfully the reporter was extremely diligent and emailed us what was about to be final copy. The boss was livid with the changes introduced by the reporter's editor to make the work "punchier." Had it gone out as it was it would have been a major embarrassment to the lab but we managed to do triage on it. You know all those stories that get written up on
4. Google-based moderation will increase the incidence of posturing. Ever seen a website pushed to the top of the search heap by artificial means?
5. Lots of scientists read slashdot. We're well aware of the crapfest that is the slashdot moderation system. See any post having anything to do with global warming or evolution. Hell see any post having anything to do with biology and there's some asshole tagging it "whatcouldpossiblygowrong" and at least a dozen highly modded comments demanding that the scientist should be prevented from playing God...when it's poking about in a few systems in a highly benign fashion.
6. There are barriers to publication. Some of them are a good thing. Any ignorant crank can spew crap online, as is their right. A paper in a peer-reviewed journal ideally, and in fact normally, means more. It means that it has been read by peers of the authors who should, and usually do, spot outright bullshit. It means that it has gone through a process of criticism--not necessarily 100% conductive but papers are usually made a little bit better; think add/remove/modify figure X or do this one extra experiment. Is peer review perfect? Hell no! Anyone who's published more than two papers has gotten back comments that are useless and we're all familiar with the occasional 100% bull
Average temperature between two different (widely separated) points might be meaningless, but the average of a continuous measurement is definitely significant. Even spatially, average temperature has a physical meaning. For example, the average surface temperature of the sun is 6500 K, though if you measure at various points, you may get more or less than that.
In any case, that's only one of the many "interesting" ideas in that paper...
Small correction if I may: Peer reviewed => not necessarily credible science, but you never ever base conclusions on only one paper.
In most cases, a good rebuttal or three will counter the effects of any completely wacko paper that slipped through. In the case I cited, not only were there rebuttals, many of the other editors involved resigned from the journal because they felt that it was a wrong that that paper got published. I think the (larger) system more or less corrected itself there, at least until people started quote-mining stolen e-mails.
Is this study peer reviewed?
I have several publications that were significantly improved through the peer review process. When I review papers my goal is not to shoot down the work, rather I try find ways to improve it. Of course there are 'bad' reviewers, who think that reviewing a paper is shredding it to pieces. These are actually easy to spot, because they rarely suggest anything useful and are often ignored by the journal editors. Speaking of which, journal editors are yet another part of the peer review process that is missing from their model
Sometimes after you've already used a couple mod points in a thread, you come across something that's +5 Insightful, yet you know is maliciously incorrect. People are replying to it as if the person was correct (they're +5 insightful must be right).
What do you do? Some people get mod points more than others... if someone only gets five points once every other month or so, at best, are they going to throw them all away by commenting in the thread? What chance does their comment even have of getting read, since a few people have replied already? Better to mod down and hope that either other people with mod points or people who are commenting pick up on it and realize the guy's wrong.
On the other hand, this is a great situation if other people who have replied to the +5 insightful yet wrong guy also realize the guy's wrong, because then you can mod those people up. But it doesn't always work that way.
I'm not saying it's the right way to use the system, but that's a common situation. You sometimes see people say "I'm giving up my mod points because you're so wrong", but you have to imagine most of the time they just down-mod instead. I often choose to use mod points in threads on a subject I know about rather than something I don't, which makes sense, but those are the threads where I'm most likely to have something to say as well; it's a bit of a conflict. I have some mod points now and was thinking of modding up a couple posts in here, but decided to respond to you... and the article is "old" enough that I'll probably not get modded up (or even read by anyone) so it's a wasted opportunity ;)
Also in case you mods haven't noticed, making me invisible doesn't work. I just repost the exact-same thing tomorrow. I refuse to be censored
I think Slashdot's system has the exact same flaw as Web Of Trust (WOT) which blocks foxnews.com on my browser. The system is abused by people pushing an agenda, with the intent of censoring what they don't like. It really needs to be abolished since it serves to SUPPRESS ideas rather than encouraging the sharing of thoughts.
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
Note also that ideas don't tend to be repeatedly suppressed unless they are truly out-there radical. I sometimes see posts promoting things that to me smell of the crank-shafting kookery that is regularly debunked as crap, and yet it doesn't get downmodded. Why? Probably because the opinion-modders simply couldn't be bothered then. I also tend to see posters loudly complaining that their opinions are being systematically downmodded, when it's really their arseholyness that is being systematically downmodded, with the dissenting opinion being the final straw.
Sure, that is a kind of suppression of opinion, but polite, clear and coherent posts should also be promoted (that's how I mod).
True confidence comes not from realising you are as good as your peers, but that your peers are as bad as you are.
In other words, when the prevailing scientific opinion goes against the prejudices of the "public" (read: bloggers with delusions of grandeur), they will loudly claim that "peer review doesn't work", without understanding what it entails, and what it's for.
A reviewer hasn't necessarily checked the paper for accuracy, or tried to reproduce the results. What they are supposed to do is to ensure that the paper isn't unmitigated trash, and that the person who wrote it at least understands something about the subject. When a scientist says that they don't want to look at an unreviewed or unpublished work, what they're saying is essentially, "Don't waste my time with stuff that I don't even know is worth looking at".
Accuracy is what rebuttals and counters are for, and they're peer reviewed to ensure that the person writing them really does understand the subject at hand, too. Review is more of a noise filter than an accuracy measure.
Like I said before, crowdsourcing or "mob review" as you put it has its place in messaging and interpretation for the lay public. It doesn't in any way replace the scientific method or the peer review process. Please get that into your head.