Slashdot Mirror
Improvements on the Scientific Review Process?
"Another really frustrating point is many people feel that some papers are accepted on the basis of the reputation of the senior author and not purely on scientific merit (i.e., the burden of proof is lessened for established scientists), while the opposite is true for younger investigators (i.e., the burden of proof for novel findings may be higher, sometimes unreasonably so). One's scientific pedigree also helps, to a certain extent (if you trained with a big name, you are more likely to get published).
In the examples above, 2-way communication via anonymous email between the authors and reviewers might solve this particular problem; also, I think that if the identities of the reviewers are protected, why not the authors? Perhaps a solution to my second issue would be to have the author's names hidden from the reviewers until after the review process is complete."
"When the reviewers encounter something that might be confusing to them in the manuscript, they take it as a negative, when really a very simple explanation from the authors would clear it up. After all, its the scientific content, not 1337 wr171ng skillz that is being reviewed. "
Well actually two things are important. The main material, and the other is your ability to communicate to others what's important. What good is the greatest idea ever discovered if people can't understand you?
The other is true, and a bit regretable, if not at least understandable.
Two thoughts come to mind:
First impression is that this could be a good application of the "open" community / wiki style of document management and review. The problem being that you would have to prevent the good informaton from being "polluted" by bias sources while still being able to filter out the bunk science.
Second, as someone mentioned in the previous story, maybe it would be a good idea to cast off the anonymous review anyway!
=Smidge=
When the reviewers encounter something that might be confusing to them in the manuscript, they take it as a negative, when really a very simple explanation from the authors would clear it up. After all, its the scientific content, not 1337 wr171ng skillz that is being reviewed.
I'm not so sure you can separate the two so easily. First, consider the fact that if the reviewer cannot understand what you are saying by reading your paper, many other people who will read your paper once it's been published won't understand it either. I can just see it: the reviewer asks the author a question using this email system you describe. The author explains their intent and the reviewer understands and accepts the paper. The confusing wording in the paper is never cleared up. Expecting all those people who have to read the thing once it's published to make heads or tails out of something worded poorly is just not fair. And don't give me the "once the author sees the reviewer's confusion, s/he will take it upon themselves to fix up the wording" bit. That would be really nice but don't count on it.
Second, you seem to make the distinction between having great scientific thoughts and the ability to communicate them effectively. Brilliant physists like Richard Feynman were able to explain very complicated concepts to a wide audience. If you really, truly understand something you should be able to make it understandable to others. If you cannot describe something without falling back on a lot of jargon, then perhaps you don't really have that deep an understanding after all.
Third, writing really isn't that difficult. Let's face it: a lot of science-types look down on those who majored in literature and humanities stuff in college. The feeling among scientists is that all the writing stuff is easy compared to the hardcore technical stuff that they do. If that's so, then prove it. It really shouldn't be too difficult for them to produce something reasonably easy to read. No one is going to kill you if you don't get all the rules for comma usage correct. But you should really have the ability to communicate your thoughts clearly to others in your field.
GMD
watch this
In astronomy, the review process is straightforward-- I've had reviewers email me directly, or have emails forwarded by editors. An email being forwarded keeps anonymity and, if the editor is professional, adds little drag to the process. Perhaps a 1-day delay on what is typically a 2-week process of review.
That said, since eBay, half.com, etc all have "send question to seller" that maintains anonymity but allows direct communication, I'd think a peer-review journal site could easily manage that.
Already, things like ApJ prefer/almost require electronic submission, and have a page for authors to check on their progress. It'd be easy to implement.
Finally, on anonymity-- the best reviewer comments I've received were from reviewers who declined to be anonymous. They were comfortable enough to communicate directly, and the work was better for it. So perhaps the very requirement of anonymity could be reconsidered.
So, in short:
1) Use electronic marketplace ideas to streamline and improve the review process,
2) All anonymity to (like on slashdot) be optional.
A.
Of course, the process could be faster, don't we all know. It can be upwards of 6 months to years to get an article published, depending on the journal and diligence of reviewers. But who has time to review? It's an unpaid, thankless job!
Electronic journals, open, double blind review processes. Open archive (arxiv.org) for everything. That is the way this should go.
-- Bob
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
1. If anything I think anonimity of reviewers needs
to be strengthened. Currently, you can guess who
your reviewer is by the style of response, what
issues the person raises etc. This allows the
submitter to taylor response to the reviewer, i.e.
it becomes a game of salesmanship. This is hard to
fix but the point I am making is that it should
go the other way: less personal more objective.
2. Scientific logic is that it is better to not
publish than publish something uncertain. So the
assinine reviewers are the price you pay for a
working peer review system. Ain't nuthin you can
do about it, much like getting off on a technicality
is a feature of our justice system.
3. There are so many journals that the issue of
stupid or stubborn reviewers should not deter a
good paper from being published, you just have to
try a few times. It is also possible to publish
your work in conference proceedings where the
review is a lot more lax. In short, insofar as we
ignore the career-building aspect of publishing
in a prestigeous journal, a good paper can be made
public in so many ways as to be almost
irrepressible.
4. If your goal is career-building, then you have
to deal with gatekeepers, no matter what the
system. I suspect the current system is not too
bad.
How about learning to write better. You are looking for a complex solution to a simple problem.
I also don't think that publishing is too difficult, or that reviewers emphasize style too much.
Publishing is too easy; there are way too many expensive journals out there, and way too many papers in them that never get referenced even once in the later literature.
The style of most scientific papers is agonizingly bad compared to the standards of other fields. As a reviewer, one of the big problems I often encountered was papers submitted by people whose English was so bad that you just couldn't understand what they were saying. A lot of those papers ended up getting published anyway. Once in a while my wife, who teaches French, leaves a literary journal in the bathroom, and I read an article just for grins. Although I often don't understand much of it, I'm always struck by how lively and entertaining the writing is.
Find free books.
Have you ever considered that the article will be published in a journal and read by people without the benefit of two-way communication with the author(s)? If, for instance, reviewer confusion can be fixed with a simple explanation then why isn't that explanation in the article? Reviewer comments are just that: comments that will help you improve the article for resubmission, not some sort of scientific hazing ritual and anonymous communication may only serve to degrade the quality of publication.
If it really bothers you that much, have colleagues not involved in the project review papers before submission and do the same for them.
In the examples above, 2-way communication via anonymous email between the authors and reviewers might solve this particular problem;
Actually all the journals I know of in computer science allow two-way communication anonymized via the editor.
I post to slashdot and recently have run into a slew of frustrations trying to get my posts modded up. For instance, one really frustrating thing is, moderators slap me with a -1 when they encounter something that might be confusing to them in the post. They take it as a negative, when really a very simple follow up post from the author would clear it up. After all, its the revevance of the content, not 1337 3ngr15h skillz that is being moderated.
What are some of the frustrations you have come across when posting to slashdot, and have you any ideas on how to improve on the moderation process?
Another really frustrating point is many people feel that some posts get an automatic +2 on the basis excellent karma, not purely on the merit of the post itself (i.e., the burden of originality is lessened for established slashdotters), while the opposite is true for posters with bad karma or a high account number (i.e., the burden of proof for creative posts may be higher, sometimes unreasonably so).
One's list of friends and fans also helps, to a certain extent (if you are a friend of a slashdot Rock Star, you are more likely to get modded up).
In the examples above, a follow-up anonymous coward post by the reviewer might solve this particular problem (lest the reviewer lose the ability to moderate by posting with his regular account); also, I think that if the identities of the moderators are protected, why not the authors? Perhaps a solution to my second issue would be to have the author's names hidden from anyone with mod points (as in metamoderation).
-jim
My first attempt at a scientific publication was a total eye-opener into how bad the process can be. I submitted a paper, and the editor forwarded it on to a referee. No word back for a month, so we complained. Eventually the editor passed it on to another referee. That one took about 3 weeks to respond, at which point they said that we weren't doing anything new --- someone else had done the same thing a month ago! (It wasn't even published yet.) If not for the first referee refusing to respond, we would have been the first. Very frustrating. They should have a 1-week time-limit for referees to respond.
We don't have a better alternative to peer review, but it's well known that you can send the same paper or grant proposal to several different groups of qualified peer reviewers and get completely different reviews.
This is gnarly enough for publication, but if public policy depends on it (e.g., peer review of the scientific basis of environmental regulations at EPA and FDA) peer review can go completely to hell.
I used to work for AAAS, the publishers of Science Magazine. Science is the premier and oldest peer reviewed general science journal (about 150 years). One of my projects was working on their Manuscript tracking system, including making the submission process electronic through Submit To Science.
Science has a greater-than 80% rejection rate, because there are only so many pages in the magazine. Many people are frustrated that they've been rejected five and six times (they have about 20 years of author submission history online), but with a general science journal you have to pick the best of the best astronony and biology and signal transduction papers to print.
The review process is laborious, it involves a lot of people, but it works pretty much flawlessly. Certain aspects have evolved over time, but frankly the people who get published thing the process works and it's great, the people who don't are the ones who grouss. There are constant arguments (almost every week) about what concessions can and cannot be done without risk to watering the quality of the journal down.
"All I ever wanted was to see Larry Wall give Bill Gates a Perl necklace."
http://www.eisenschmidt.org/jweisen
Anonymous email wouldn't help manuscript review much, but it could help grant review. If a reviewer misunderstands a manuscript, you should revise the manuscript, as others have noted. But revising a grant proposal often means waiting a year. Grant reviewers should have the option of contacting PIs anonymously by email. As a reviewer, I wouldn't bother doing this for a sloppy proposal, but might if I had one or two specific concerns about an otherwise excellent proposal. Unfortunately, half of proposals are good and only 10% can be funded, so some good proposals will always be rejected.