Slashdot Mirror
Peer Review Highly Sensitive To Poor Refereeing
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
I can't quite remember what it was, but I seem to remember seeing it everywhere. It was exactly like TFA article, though. Damn, what was that place called again?
Living With a Nerd
thats what always happens to my comments on here... frosty pist!
I saw that last sentence coming. I can imagine $RANDOMLUSER's supervisor won't accept his excuse for said supervisor's paper failing peer review.
Does this remind anyone of Climategate?
Anyone?
Bueller?
Bueller?
Result: -1 Troll
Now mod me up for factual accuracy or face infinite recursion
This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.
I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.
"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.
So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?
Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.
That's the way I'm interpreting this study.
RIP America
July 4, 1776 - September 11, 2001
Just as the Climategate e-mails showed, a corrupted peer reviewed process produces all kind of junk passing as science.
A small group of buddies agree to let their own articles through, and delay or thwart any paper that contradicts or challenges their agenda.
(no text)
Ok, mod me down.
Very often, people confuse simple with simplistic. The nuance is lost on most. - Clement Mok
Peer Review Highly Sensitive To Poor Refereeing
Much like
Trolling is a art,
When you're talking about scientific papers, a "bad apple" reviewer may be able to skew the record in terms of 1-10 scales, but reviewers also do a qualitative write-up of the material. That's really the only important part and if one or two people fall outside the line of general consensus, they'll just be ignored.
Since the scientific community is so very obsessed with peer review, will this study actually modify the standard procedure?
Of course, all I can think of is: gee, I wonder if this has had any impact on all the climate change studies that are constantly contradicting each other...
I can't decide if I should care whether this was peer reviewed or not.
Almost always - no, that's not a scientific deduction, it's just coming from skewed subjective personal experience - the ones who most complain about problems with article peer review systems are those who have the most problems publishing decent articles at decent places. Also, nepotism? Ever heard of [single/double] blind reviews? I guess this must be one of those slow news days.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
I mean scientists who publish among themselves, i.e. inside their narrow specialty, in their own journals, without checking whether the problem at hand has been solved elsewhere. This is more and more common as people get more specialized, and can lead to very basic errors propagated inside the whole community, like rheologists believing in the existence of pure elongational flow (a trivial misunderstanding of tensor algebra). Since the peers reviewing the papers are members of the same community, those errors usually get unnoticed.
Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.
I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
The great anecdote demeaning peer-reviewed journals is The Social Text Affair, where a prominent peer-reviewed journal published with enthusiasm the article "Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity", only to be informed it was, in fact, computer-generated gibberish submitted as a joke.
Can we get a "-1 Wrong" moderation option?
There are a couple of significant and important limitations in the model:
a) It assumes only two reviews per paper, and that the reviews are pure boolean, and that reviewer types are also pure and reviewers are randomly selected (when two of the classes of reviewers, 'mythantropes' (always reject) and 'altruists' (always accept) are specifically selected against by editors and PC chairs based on reputation).
b) It does not consider the cases (such as conferences) where there is a program committee meeting and the papers are not just considered on their own, but gone through a relative ranking process.
Test your net with Netalyzr
just tweak arXiv so that reviews of posted articles are linked to in the abstract page. or something like that. why the need for anonymous reviewers? as long as the discussion is objective and to the point, there's no need for anonymity. and we are far past the point where it costs money to attach comments to an article...
new sig
After reading the article, you can add these terms to George Carlin's list of oxymoronic phrases.
"Flyin' in just a sweet place,
Never been known to fail..."
Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.
If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!
The peer review system is great for regulation, standardization and unification. However, all scientists that I've worked with/researched with/spoken with much about this topic admit that the system can be annoyingly flawed by group think and conformity. One bad apple ruins the bunch, right?
The good news? While this part of the scientific community is not immune to problems, the slack is picked up elsewhere: As long as methods, data and results are transparent, reproducible and published we can actually have quality science.
I often speak to people about scientific research and they're shocked that it's not full proof. This is kind of like buying software (perhaps even a Microsoft product) and finding that it's not perfect. Science is done by committee and progresses slowly. "If we know what we were doing, it wouldn't be called research" ~Albert Einstein
Then again, I'm an idiot....
...... and idiots rule the world....
I actually read the comments on TFA, and down at , there's a particularly interesting one:
This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.
There's actually a reasonably intelligent discussion going on in there...
My dad (has PhD in a scientific field from Cornell) told me that when submitting a thesis to a review board of professors, it really doesn't matter how "Tough" a professor is as long as that professor in your committee has a rival. Take advantage of their ego with an equally assertive ego. You purposefully choose the rival professor to join your committee as well. Then they'll spend all the review board discussions and presentations contradicting and arguing one with another, and in the end they'll both be so incensed, that they cancel each other out, and it doesn't matter what you presented... I guess the TFA is only pointing out that this occurs at the publishing level as well.
http://www.beanleafpress.com
The outliers might not be due to conscious suppression of competing research. People just have some ways of thinking that make their subjective opinions sometimes contrast with what an objective observer would think.
My first anonymous coward...
My boss (a leading authority in his area, a serious person) has found around 10 errors in epidemiological modelling papers (papers that, by they way, influence WHO policies that influences public health). These are proven errors (in mathematical formulas). They vary from minor to massive.
He informed all authors of the respective problems. How many corrected or retracted the manuscripts? ZERO.
Will he write a paper documenting all the errors (as they are hard-science type of errors, in formulas)? Are you crazy? Remember that the other authors in the area peer review his papers. Therefore we will not risk their wrath.
But note that this varies a lot from area to area. I work in another area (related to genetics) and 90%+ of the people that I know are honest.
But yes, some areas need a cleaning up.
Welcome to the real world intellectuals....this is how things are done. I'm not saying its right, but knowing the right people is how things function. I know the University and intellectual world like to think that they are removed from such things but that is just not the case.
I remember a news show highlighting a "famous" professor that did some amazing work in his early carrier and then proceeded to build a bigger carrier while reusing fake data and bogus information. He WOULD NOT have been able to do so if he had not known the right people and had the "right amount of fame". No one doubted him...however if his papers where published by a "nobody" they would have been caught QUICKLY.
-- Disclaimer: I can't really back up anything I post on
Peer review only works if the reviewers can be trusted and don't form a clique to get their work in and keep other people out. Surely anyone with even basic knowledge of human psychology would understand this?
Because this is an important question for serious people, but has no bearing on why various cranks (Intelligent Design people, climate change "skeptics", Time Cube, etc.) may have trouble getting their work in print. Papers by such people generally don't end up in the peer review phase - they aren't sent out for evaluation by the journal, so peer review doesn't matter.
That said, peer review provides substantially the same benefit as those "shoplifters will be prosecuted" signs you see in department stores.
Shoplifters are very seldom if ever actually prosecuted - but the threat, even the vaguest menace - of public scrutiny has an impact on behavior. I'm not talking about scientific fraud (which peer review will seldom catch,) but about quality of reasoning, doing the needed controls, etc. We may have a system that rewards good research little-better than an unbiased coin, but the <b>perception</b> that it works, or that it might work for you, motivates people to do the work needed to survive peer-review.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
You mean whine when a POS paper is printed.
G&T's paper in E&E refuses to accept that the presence of another object nearby at temperature will cause the temperature of a hotter object to be higher than it would without. Making a mockery of the laws of thermodynamics which REQUIRE that this be true.
E.g. If you have one end at 300C and the other end at 200C, at some time later, say, 1 minute, the hotter end will have cooled to, say, 280C. If the other end was at 100C rather than 200C, G&T would have the temperature of the hotter end at 280C still.
But that makes a mockery of the thermal conduction laws which include the temperature DIFFERENCE between two ends.
It also makes a mockery of the radiative loss equations for an object in a hot enclosure:
t1 = temperature of hot body
t2 = temperature of enclosure
power loss = sigma * (t1^4 - t2^4)
Yet G&T's paper in E&E would say that this is impossible! Since the temperature drop of the hot body cannot, to them, change if the enclosure is hotter.
Think also that according to G&T, you end up with the hot end colder than the cool end, because it cannot be affected by the temperature of the cool end, therefore it must cool to the same temperature as the situation of absolute zero at the other end. That means it cools to absolute zero. Which is colder than the other end.
How does G&T's paper work even at teenager school physics level of understanding?
It doesn't.
But it got through to be published?
Yet so many are bleating here about how it must be the IPCC "warmists" suppressing papers opposing them. Here we have
a) no opression, the paper is printed
b) the paper is OBVIOUSLY crap
yet somehow, (b) doesn't mean that denialists promote crap papers to bolster their position. According to some, anyway.
How does THAT work?
The government uses peer review to evaluate proposals for science and engineering grants. The same issues probably apply to those evaluations.
I have experienced a situation in which one reviewer recommended turning down a grant for reasons that could be considered as biased, although the bias was groupthink rather than individual. The other reviewers were enthusiastic about funding the grant and regarded it as a potential game-changer. It didn't get funded. A few years later the game-changing nature of the technology was recognized, but it was too late for the original applicant.
the problems of nepotism and tribalism are everywhere.
from internet message boards and professional office environments to national government and international politics.
here's a paradox for you,
someone could do a study on how to eleminate nepotism and tribalism.
then they can put it up for peer review.
In my humanities field, where the vast majority of publications in the last 100+ years were not peer-reviewed, we're seeing pressure from administrators to publish in "peer-reviewed" journals. And, yes, we're seeing the predictable results:
A. Any half-competent referee can figure out pretty quickly the author.
B. While there are data and interpretation in the articles, the author and referee are supposedly skilled at argumentation, and, even when they're not, "nothing new here" will sink any article the ref. doesn't like.
C. Nobody pays us to referee articles; it takes a lot of time, and does nothing for our personal careers.
We had a fun case recently, where one professor (known for shooting down his enemies with the "nothing new" response) ironically turned out to have built his career on plagiarizing other's (non-peer-reviewed, usually) material and submitting them to peer-reviewed journals, with a first page of his own that was sufficient to identify him as the author. He made professor pretty quick. In one case, he plagiarized some guy's published Ph.D. thesis, and submitted it to a journal. The journal sent it to (rumour has it) the victim's Ph.D. advisor to be refereed, and this advisor recommended it for publication without noticing that it was a wholesale rip-off.
In another case, as relatively unknown scholar complained to a peer-reviewed journal that this dude had plagiarized his thesis. From what I understand, the journal submitted the guy's complaint, the plagiarizer's response, and the evidence (which was conclusive) to blind peer review, and the referee sided with the plagiarist. After the extent of the professor's copying has been published, I suppose that journal will bypass peer review and retract the article.
In the sciences, you can argue that a "few bad apples" will cause problems, but in the humanities it's outright broken, and basing hiring and promotions purely on the prestige of the journal or the presence of peer review demonstrably leads to failure.
...and then curse again. Let the recursion begin!
Set your phasers on "funky"!
Yup, I've seen exactly the same thing in several sub-fields of computer science. You read about solutions in textbooks published in the '80s, describing established knowledge, and then see brand new research papers in a different area where they ignore all of this prior research and come up with inferior solutions. About 90% of the papers I've read on Grid Computing fit in this description.
I am TheRaven on Soylent News
Anyone having an open eye in academia could confirm this.
And yet at some point I thought that the emergence of a huge, readily-accessible, and often high-quality literature on the internet would somehow make for the growing lack of general scientific culture among students and researchers. Indeed, it's amazing to see that (at least in my field) a large proportion of new and recent papers are now directly posted online in spite of desperate efforts by Wiley, Springer and such. I think this is quite recent and hugely beneficial to everyone. Provided people really use that information, of course...
"a good paper of theirs gets turned down, or when a bad paper they disagree with gets published"
This basically says it all. No one ever submits a bad paper, in their own opinion, and bad papers are limited to the ones that they disagree with.
In case you missed it, peer review is dead. Get over it. Don't be sad, you can still have your journals and pomp and circumstances. You can drag out the rotted corpse and parade around with your scientist friends. It just carries no weight with the public. We've seen how twisted it can get.
That is why mob review has supplanted peer review. You can call it crowd sourcing or whatever, but it is not going to go away. Any paper that makes it thru peer review now has to make it past the non-peer reviewers, that is, everyone else. If you thought your peers were tough reviewers, then you are going to really hate mob review.
Using a model? Guess what, there are people who love to dissect models. Using statistics, lots of statisticians running around that are going to poke at yours.
My advice, don't fight it. The majority of people just want to get to the truth.
Peer review is often well established scientists shitting on competition to their method, or dissenting views. Each topic has it's fiefdoms battling to protect their own interests. Time after time I have seen good papers sunk because they are competition, and mediocre papers published after the authors simply add citations suggested by the referee – of course, those papers are the referees' and their friends'. I haven't directly been involved in the granting process, but I get the impression that the review boards operate in a similar manner. And through all of this science progresses. Bizarre.
46 & 2
They are established scientists with their own self interest. If a small number of referees can have a large effect, imagine what a small number of poor editors could do.
46 & 2
Is this study peer reviewed?
In what way were they made one body? If they were merged, then that's simple.
Or do you not manage to cool your bath down by pouring some cold water into the too-hot water?
Your assertion that there is no average temperature is itself a howler.
I have several publications that were significantly improved through the peer review process. When I review papers my goal is not to shoot down the work, rather I try find ways to improve it. Of course there are 'bad' reviewers, who think that reviewing a paper is shredding it to pieces. These are actually easy to spot, because they rarely suggest anything useful and are often ignored by the journal editors. Speaking of which, journal editors are yet another part of the peer review process that is missing from their model
There is a variant on this, when some researchers dig up physics, biology, etc journals, take models/approaches from them and apply them to CS problems. Because in the CS field these results are not really known, the paper is usually considered interesting. Another effect is that while all of the theories are properly referenced (so no plagiarism happens) the reviewers usually assume more work done by the researcher than what actually was done (reviewers rarely read the cited papers).
The Precogs sent a ball rolling. They want their Minority Report back!
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Some ignoramus put forward a regressive edit on an article lacking citations as evidence of process problems at Wikipedia. Yes, the edit was poorly judged, but an article lacking citations is a cork in a windstorm.
I tend to defend Wikipedia since the interesting question is "Why does it work at all?" One lesson we've learned is that metrics of irreproachability constitute a poor utility function with respect to what most people need most of the time. It falls short when the aim is to slap your name on a document purporting to contain original work adding value to the community.
So yes, Wikipedia falls vastly short of providing a credible foundation for promulgating reputation. The niche it properly occupies is halfway between the card catalog and the dusty tomes of eminence. It's a secondary transfer station in the subway system of knowledge.
When I was eight years old my father showed me how to wind a wire around a large nail and make an electromagnet. It wasn't long before I looked up electromagnetism in an edition of the Encyclopedia Britanica from the 1960s. This was heavy going for a nine year old. I learned practically nothing, the eminence and authority of the text flying completely over my head.
Sugata Mitra: The child-driven education
How did it go? SPOILER ALERT
Going back to my early experience with Britannica, "apart from understanding that the ratio of windings in a transformer established the ratio of voltage and the inverse ratio for current, I've understood hardly anything else." That's not high praise. I'd have made more progress with an internet full of unreliable napkin diagrams explicated in a foreign language.
The problem is that all too often irreproachable equates to inaccessible. You can even find this relationship within the cloisters of the peer review process.
From the font of no knowledge:
My recent insight into peer review is that it operates at two distinct time scales. The short time scale is career advancement with tau = 5 years. The longer time scale is reliable scientific consensus which a time scale of tau = 30 years. (It sometimes takes seven tau periods before the last ripple is smoothed out.) There is no question that peer review is a excellent mechanism to achieve scientific consensus over century time scales.
The scientists themselves are caught up in the dual function of peer view to gate career advancement and to establish the accurate, long term consensus, and thereby sometimes fall into the trap of conflating power with authority.
Within the context of scientific peer review, over the generational time frame, the cliques and rivalries fall by the wayside, the old arguments lose their sex appeal, and the bad actors move onto more topical debates.
Peer review fails to transcend human politi
That's because the period is too short. Statistical significance means "unlikely to have been just "luck"" and that requires a significant number of values.
30 years for climate, though sometimes 15 years is (barely) enough.
So "no significant trend" != "no trend" and 4 years does not a trend make.