Improvements on the Scientific Review Process?

myc asks: "A recent story has discussed the pros and cons of the publishing end of scientific literature, but what about the review end? I work in biology and recently have run into a slew of frustrations trying to get my work published. For instance, one really frustrating thing is, because of the need to keep the reviewers' identify confidential, communication between authors and reviewers is essentially one-way via the editorial office. When the reviewers encounter something that might be confusing to them in the manuscript, they take it as a negative, when really a very simple explanation from the authors would clear it up. After all, its the scientific content, not 1337 wr171ng skillz that is being reviewed. What are some of the frustrations you have come across when trying to publish your work, and have you any ideas on how to improve on the review process?"

"Another really frustrating point is many people feel that some papers are accepted on the basis of the reputation of the senior author and not purely on scientific merit (i.e., the burden of proof is lessened for established scientists), while the opposite is true for younger investigators (i.e., the burden of proof for novel findings may be higher, sometimes unreasonably so). One's scientific pedigree also helps, to a certain extent (if you trained with a big name, you are more likely to get published).

In the examples above, 2-way communication via anonymous email between the authors and reviewers might solve this particular problem; also, I think that if the identities of the reviewers are protected, why not the authors? Perhaps a solution to my second issue would be to have the author's names hidden from the reviewers until after the review process is complete."

Comments

Two things

[Smidge204]

Two thoughts come to mind:

First impression is that this could be a good application of the "open" community / wiki style of document management and review. The problem being that you would have to prevent the good informaton from being "polluted" by bias sources while still being able to filter out the bunk science.

Second, as someone mentioned in the previous story, maybe it would be a good idea to cast off the anonymous review anyway!

=Smidge=

Not much to be done

[Compuser]

1. If anything I think anonimity of reviewers needs
to be strengthened. Currently, you can guess who
your reviewer is by the style of response, what
issues the person raises etc. This allows the
submitter to taylor response to the reviewer, i.e.
it becomes a game of salesmanship. This is hard to
fix but the point I am making is that it should
go the other way: less personal more objective.

2. Scientific logic is that it is better to not
publish than publish something uncertain. So the
assinine reviewers are the price you pay for a
working peer review system. Ain't nuthin you can
do about it, much like getting off on a technicality
is a feature of our justice system.

3. There are so many journals that the issue of
stupid or stubborn reviewers should not deter a
good paper from being published, you just have to
try a few times. It is also possible to publish
your work in conference proceedings where the
review is a lot more lax. In short, insofar as we
ignore the career-building aspect of publishing
in a prestigeous journal, a good paper can be made
public in so many ways as to be almost
irrepressible.

4. If your goal is career-building, then you have
to deal with gatekeepers, no matter what the
system. I suspect the current system is not too
bad.

Re:Clarity of exposition

[myc]

I'm not so sure you can separate the two so easily. First, consider the fact that if the reviewer cannot understand what you are saying by reading your paper, many other people who will read your paper once it's been published won't understand it either. I can just see it: the reviewer asks the author a question using this email system you describe. The author explains their intent and the reviewer understands and accepts the paper. The confusing wording in the paper is never cleared up. Expecting all those people who have to read the thing once it's published to make heads or tails out of something worded poorly is just not fair. And don't give me the "once the author sees the reviewer's confusion, s/he will take it upon themselves to fix up the wording" bit. That would be really nice but don't count on it.

Why not? In publishing fiction and literature, is this not the realm of the editor's work? I submit that what you just described is exactly how scientific literature should be published. Referees judge on scientific merit, editors should facilitate communication to ensure that no misunderstandings are made.

Second, you seem to make the distinction between having great scientific thoughts and the ability to communicate them effectively. Brilliant physists like Richard Feynman were able to explain very complicated concepts to a wide audience. If you really, truly understand something you should be able to make it understandable to others. If you cannot describe something without falling back on a lot of jargon, then perhaps you don't really have that deep an understanding after all.

Not everyone is Feynman. and probably not every physicist can understand Feynman. I dunno, IANAP so I can't speak to that. It is clear that since biology is so specialized that people often working in labs next door to each other have no idea what each are up to. It's common for biology papers to be fairly concise and contain reasonable amounts of jargon, making it difficult for people not directly in the field. Of course there are the occasional ground breaking papers that touches on a subject so central that EVERYONE gets it; most papers are not of that caliber and are geared towards specialists.

>Third, writing really isn't that difficult. Let's face it: a lot of science-types look down on those who majored in literature and humanities stuff in college. The feeling among scientists is that all the writing stuff is easy compared to the hardcore technical stuff that they do. If that's so, then prove it. It really shouldn't be too difficult for them to produce something reasonably easy to read. No one is going to kill you if you don't get all the rules for comma usage correct. But you should really have the ability to communicate your thoughts clearly to others in your field.

again, how do you deal with incompetent reviewers? you basically get one chance at revision. if they are failing to comprehend at a very basic level, is it REALLY my fault as an author for failing to communicate my findings, or is it the editor's fault for not finding a more suitable reviewer?

A recent reviewer asked us to do an experiment to prove/disprove a point, but what he proposed DID NOT ADDRESS HIS CONCERNS AT ALL. I won't go into the gory details, but if you know anything about biology, the analogy is that he/she asked for a Northern blot to determine where the RNA is subcellularly localized. It made ABSOLUTELY NO SENSE TO ANY REASONABLE PERSON. The other reviewer of our manuscript gave us enthusiastic support, and the work at a technical level was strongly supported by another expert in the field when we presented it at a meeting. I find it really hard to believe that we failed at either a technical or communicative level, we just go really unlucky with a reviewer.

I don't know that constant anonymous email communication w/ referees is the answer. I do know that uninformed referees are a problem. They are supposed to provide oversight to the author's work; who is overseeing theirs? Certainly not the journal editors. They are so bogged down in submissions that out of practicality they must defer to the referee's decisions.