Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.
So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?
Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.
That's the way I'm interpreting this study.
RIP America
July 4, 1776 - September 11, 2001
The funny thing is, the skeptics suffer from the same problem.
I hope the moderates don't, otherwise were borked.
Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
There is an aforism: Democracy as a form of government is riddled with problems. However we are yet to invent anything better.
Same with the peer review. It has its problems. However, we are yet to invent anything better
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
As much crap as /.'s moderation system gets, it actually tends to be one of the better systems I've seen on the net. First of all, it's highly customizable so if I wanted to I can easily set it to add or remove value from certain types of moderations. I usually bump flamebait and troll up a few pegs just so I can see the posts that do occasionally get unfairly moderated. I can also add other posters I find interesting to a list and bump up their post value so if I'm interested in what they have to say I can always make sure I'll see it.
I also think that the community goes a long way towards making the system work well. Sure there will always be people who abuse the system and moderate posts with which they disagree as flamebait, etc. but the community as a whole does a good job of promoting interesting lines of conversation and for any given topic there are probably a few people in the community who specialize in that area and can provide some excellent commentary.
It's not perfect, but it's probably one of the best systems in actual practice that's currently being used.
The peer review system is great for regulation, standardization and unification. However, all scientists that I've worked with/researched with/spoken with much about this topic admit that the system can be annoyingly flawed by group think and conformity. One bad apple ruins the bunch, right?
The good news? While this part of the scientific community is not immune to problems, the slack is picked up elsewhere: As long as methods, data and results are transparent, reproducible and published we can actually have quality science.
I often speak to people about scientific research and they're shocked that it's not full proof. This is kind of like buying software (perhaps even a Microsoft product) and finding that it's not perfect. Science is done by committee and progresses slowly. "If we know what we were doing, it wouldn't be called research" ~Albert Einstein
Then again, I'm an idiot....
...... and idiots rule the world....
What I find sad about the /. system is that it works very well as you describe, it was one of the first, and now, 10+ years later, most sites have no moderation system, or terrible moderation system, or terrible and very complex moderation systems.
Because who would want to use what works and not reinvent their own!?
For any journals which use the model in the paper
- 2 reviewers;
- only accept or reject decisions; and
- no editorial board member involved
they might do something.
However, the author's model does not fit ANY credible scientific journal.
In general at least 3 reviewers are used. Further, the editorial member checks the reviews and is an expert in the field.
In other words, the normal journals will only publish junk papers if ALL 3 reviewers AND the editorial board member are poor.
Actually, I can't think of a single journal in science which uses the author's model.
Except that I've heard of people deliberately adding people to acknowledgements to try to make sure they don't get those people as referees (and it hasn't worked)!
Peer review only works if the reviewers can be trusted and don't form a clique to get their work in and keep other people out. Surely anyone with even basic knowledge of human psychology would understand this?
I think I'll point some skeptics at this paper
Let me know when you find some. I mostly meet deniers, with a deep ignorance of climatology or any other science and a deep conviction of a conspiracy.
If you locate some actual skeptics, people capable of analyzing the evidence, who have come to the opposite conclusion of the vast majority of actual climatologists, I'd love to hear from them.
Because this is an important question for serious people, but has no bearing on why various cranks (Intelligent Design people, climate change "skeptics", Time Cube, etc.) may have trouble getting their work in print. Papers by such people generally don't end up in the peer review phase - they aren't sent out for evaluation by the journal, so peer review doesn't matter.
That said, peer review provides substantially the same benefit as those "shoplifters will be prosecuted" signs you see in department stores.
Shoplifters are very seldom if ever actually prosecuted - but the threat, even the vaguest menace - of public scrutiny has an impact on behavior. I'm not talking about scientific fraud (which peer review will seldom catch,) but about quality of reasoning, doing the needed controls, etc. We may have a system that rewards good research little-better than an unbiased coin, but the <b>perception</b> that it works, or that it might work for you, motivates people to do the work needed to survive peer-review.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
I think it comes down to two things.
First, the relative rarity of mod points encourages people to take it seriously. It also encourages the most active, most interesting posters to give up posting once in a while to moderate. Most other sites use a mod system that allows so many votes per article but still don't allow you to post and mod the same article. That means that the most frequent posters will seldom mod and that there can be people who only ever mod articles without ever commenting on them.
Second, attaching a reason to the mods encourages people to actually think about why they are modding the way that they are. As many people say, there is no "-1 I Disagree" mod, in order to mod someone down you have to be saying that they are actively trying to derail the conversation. Of course, lots of modders will ignore that and mod however they want, but I think that it does make at least some people stop and think before they accuse someone else of being flamebait or a troll.
Letting technical writing people doing the writing won't work. A large part of the scientific writing is the discussion of the experiment, which not only helps the scientist clarify his or her own thoughts and gives insight into future experiments, but also really only is worth reading if the scientist or members of the experimental team do it themselves. Technical writers really only would have the ability to write the experimental procedure, and even then it would be hard. Since science is so specialized you'd have to have technical writers for thousands of subdisciplines, etc. This goes especially true for mathematics, where the writing procedure is very closely related to doing mathematics.
Already because of this, no time for the scientist would be saved. A Google moderation system would have two problems. First, it wouldn't save any time because you still have to have some person doing the reviewing, and secondly you have to have someone qualified doing the reviewing whom you can trust to some extent to review in confidence, for otherwise if there are certain major problems with the paper but a few good ideas, they can be "stolen" by others, which may become a problem.
"What lies behind us, and what lies before us are tiny matters compared to what lies within us." Ralph Waldo Emerson
If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.
I don't think it's a massive problem, it just relies on people being ethical about declining to review something if they have an interest (in the legal sense) in the work. People have a lot to lose if they try and cheat the system and get caught.
You can advertise in this sig from as little as £99.99 a month!
I agree with all that you said, but for heaven's sake, use full-forms at least once
For those as confused as I, the paper the parent refers to is "Gerhard Gerlich, Ralf D. Tscheuschner. Falsification of the atmospheric CO2 greenhouse effects within the frame of physics.".
Contains such howlers as "There's no such thing as average temperature"... RC Wiki page
Well, peer-review the junk ideas out (aka, vote on the economic aspects). The science is pretty well settled, but the economics is not so clear (as if it ever really is).
Come up with better solutions, implement them if you can, support good solutions if you can't. The problem isn't going away by denying it because you don't like the currently proposed solutions.
That, we can discuss. "Global cooling in the 1970s" is just noise in the channel.
Damn, you're an asshole.
Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
The thing is that the "fix" is plain and simple, it's just rejected out of hand by the Envirowhackos because it doesn't involve government running our lives, a reduction in the standard of living, and allows for more growth and prosperity: Nuclear power.
Now comes all the posts about peak uranium that ignore technology like breeders and thorium.
Then comes everyone who thinks all reactors are built like Chernobyl.
Next are high level waste folks who don't understand what reprocessing does.
Last, but not least, are all the people who equate a nuclear reactor with a nuclear bomb.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Oh please. Let's face it, it's easy for the government to ignore environmental concerns; they've been doing that for years. The real barrier is the general public that's okay with nuclear power as long as the power plant isn't near their neighborhood, as long as trains carrying fuel or waste don't go anywhere near their house. They'd love them some cheap electricity, sure, but just build it near some other people.
I've upped my standards, so up yours.
Small correction if I may: Peer reviewed => not necessarily credible science, but you never ever base conclusions on only one paper.
In most cases, a good rebuttal or three will counter the effects of any completely wacko paper that slipped through. In the case I cited, not only were there rebuttals, many of the other editors involved resigned from the journal because they felt that it was a wrong that that paper got published. I think the (larger) system more or less corrected itself there, at least until people started quote-mining stolen e-mails.
In other words, when the prevailing scientific opinion goes against the prejudices of the "public" (read: bloggers with delusions of grandeur), they will loudly claim that "peer review doesn't work", without understanding what it entails, and what it's for.
A reviewer hasn't necessarily checked the paper for accuracy, or tried to reproduce the results. What they are supposed to do is to ensure that the paper isn't unmitigated trash, and that the person who wrote it at least understands something about the subject. When a scientist says that they don't want to look at an unreviewed or unpublished work, what they're saying is essentially, "Don't waste my time with stuff that I don't even know is worth looking at".
Accuracy is what rebuttals and counters are for, and they're peer reviewed to ensure that the person writing them really does understand the subject at hand, too. Review is more of a noise filter than an accuracy measure.
Like I said before, crowdsourcing or "mob review" as you put it has its place in messaging and interpretation for the lay public. It doesn't in any way replace the scientific method or the peer review process. Please get that into your head.
I don't know what an "Envirowhacko" is, but I don't know any environmentalists who want government running our lives or a reduction in the standard of living. In fact, in terms of big government, most want an end to the huge subsidies, in the form of loan guarantees, that make possible the construction of nuclear power plants. Nuclear power is inherently centralized and requires strong government oversight for both safety (from accident and terrorism) and non-proliferation concerns; decentralized power production from millions of solar panels, wind turbines, and biofuel-run generators, goes well with the idea of decentralized government.
I'm all for prosperity, but "growth", in and of itself, is the ideology of the cancer cell. The planet is limited. The human species is just about wrapping up its adolescence, it's time for physical growth to stop -- and for those energies to go into mental, emotional, and dare I say, "spiritual" growth.
I think the possibilities of fusion and of energy-amplifier reactors using thorium are interesting, but they are not here yet. Solar and wind are here now, as are meaningful improvements in energy efficiency. Plus, the U.S. and Israel won't threaten to bomb countries that build solar arrays or wind farms.
All reactors? No, but if you want a solution to the world's energy needs, you need to think about reactors being built by the same sort of companies that paint kids toys with lead pain and put melamine in pet food.
So, in short: nuclear power is not a "plain and simple" solution -- and most likely there is no single "plain and simple" solution.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood