Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.
So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?
Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.
That's the way I'm interpreting this study.
RIP America
July 4, 1776 - September 11, 2001
As much crap as /.'s moderation system gets, it actually tends to be one of the better systems I've seen on the net. First of all, it's highly customizable so if I wanted to I can easily set it to add or remove value from certain types of moderations. I usually bump flamebait and troll up a few pegs just so I can see the posts that do occasionally get unfairly moderated. I can also add other posters I find interesting to a list and bump up their post value so if I'm interested in what they have to say I can always make sure I'll see it.
I also think that the community goes a long way towards making the system work well. Sure there will always be people who abuse the system and moderate posts with which they disagree as flamebait, etc. but the community as a whole does a good job of promoting interesting lines of conversation and for any given topic there are probably a few people in the community who specialize in that area and can provide some excellent commentary.
It's not perfect, but it's probably one of the best systems in actual practice that's currently being used.
The peer review system is great for regulation, standardization and unification. However, all scientists that I've worked with/researched with/spoken with much about this topic admit that the system can be annoyingly flawed by group think and conformity. One bad apple ruins the bunch, right?
The good news? While this part of the scientific community is not immune to problems, the slack is picked up elsewhere: As long as methods, data and results are transparent, reproducible and published we can actually have quality science.
I often speak to people about scientific research and they're shocked that it's not full proof. This is kind of like buying software (perhaps even a Microsoft product) and finding that it's not perfect. Science is done by committee and progresses slowly. "If we know what we were doing, it wouldn't be called research" ~Albert Einstein
Then again, I'm an idiot....
...... and idiots rule the world....
Except that I've heard of people deliberately adding people to acknowledgements to try to make sure they don't get those people as referees (and it hasn't worked)!
Peer review only works if the reviewers can be trusted and don't form a clique to get their work in and keep other people out. Surely anyone with even basic knowledge of human psychology would understand this?
Because this is an important question for serious people, but has no bearing on why various cranks (Intelligent Design people, climate change "skeptics", Time Cube, etc.) may have trouble getting their work in print. Papers by such people generally don't end up in the peer review phase - they aren't sent out for evaluation by the journal, so peer review doesn't matter.
That said, peer review provides substantially the same benefit as those "shoplifters will be prosecuted" signs you see in department stores.
Shoplifters are very seldom if ever actually prosecuted - but the threat, even the vaguest menace - of public scrutiny has an impact on behavior. I'm not talking about scientific fraud (which peer review will seldom catch,) but about quality of reasoning, doing the needed controls, etc. We may have a system that rewards good research little-better than an unbiased coin, but the <b>perception</b> that it works, or that it might work for you, motivates people to do the work needed to survive peer-review.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
I think it comes down to two things.
First, the relative rarity of mod points encourages people to take it seriously. It also encourages the most active, most interesting posters to give up posting once in a while to moderate. Most other sites use a mod system that allows so many votes per article but still don't allow you to post and mod the same article. That means that the most frequent posters will seldom mod and that there can be people who only ever mod articles without ever commenting on them.
Second, attaching a reason to the mods encourages people to actually think about why they are modding the way that they are. As many people say, there is no "-1 I Disagree" mod, in order to mod someone down you have to be saying that they are actively trying to derail the conversation. Of course, lots of modders will ignore that and mod however they want, but I think that it does make at least some people stop and think before they accuse someone else of being flamebait or a troll.
Letting technical writing people doing the writing won't work. A large part of the scientific writing is the discussion of the experiment, which not only helps the scientist clarify his or her own thoughts and gives insight into future experiments, but also really only is worth reading if the scientist or members of the experimental team do it themselves. Technical writers really only would have the ability to write the experimental procedure, and even then it would be hard. Since science is so specialized you'd have to have technical writers for thousands of subdisciplines, etc. This goes especially true for mathematics, where the writing procedure is very closely related to doing mathematics.
Already because of this, no time for the scientist would be saved. A Google moderation system would have two problems. First, it wouldn't save any time because you still have to have some person doing the reviewing, and secondly you have to have someone qualified doing the reviewing whom you can trust to some extent to review in confidence, for otherwise if there are certain major problems with the paper but a few good ideas, they can be "stolen" by others, which may become a problem.
"What lies behind us, and what lies before us are tiny matters compared to what lies within us." Ralph Waldo Emerson
I agree with all that you said, but for heaven's sake, use full-forms at least once
For those as confused as I, the paper the parent refers to is "Gerhard Gerlich, Ralf D. Tscheuschner. Falsification of the atmospheric CO2 greenhouse effects within the frame of physics.".
Contains such howlers as "There's no such thing as average temperature"... RC Wiki page
Well, peer-review the junk ideas out (aka, vote on the economic aspects). The science is pretty well settled, but the economics is not so clear (as if it ever really is).
Come up with better solutions, implement them if you can, support good solutions if you can't. The problem isn't going away by denying it because you don't like the currently proposed solutions.
That, we can discuss. "Global cooling in the 1970s" is just noise in the channel.
The thing is that the "fix" is plain and simple, it's just rejected out of hand by the Envirowhackos because it doesn't involve government running our lives, a reduction in the standard of living, and allows for more growth and prosperity: Nuclear power.
Now comes all the posts about peak uranium that ignore technology like breeders and thorium.
Then comes everyone who thinks all reactors are built like Chernobyl.
Next are high level waste folks who don't understand what reprocessing does.
Last, but not least, are all the people who equate a nuclear reactor with a nuclear bomb.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Oh please. Let's face it, it's easy for the government to ignore environmental concerns; they've been doing that for years. The real barrier is the general public that's okay with nuclear power as long as the power plant isn't near their neighborhood, as long as trains carrying fuel or waste don't go anywhere near their house. They'd love them some cheap electricity, sure, but just build it near some other people.
I've upped my standards, so up yours.