Peer-Reviewed Research Over The Web

bhoman writes "The San Francisco Chronicle (sfgate.com) has an article today about Stanford biochemist Patrick O. Brown, who helped develop low-cost DNA microarrays for gene research. He is seeking $20M to start a foundation that would fund peer-review of research papers and then make them available for free over the web, thereby avoiding the high-cost of subscriptions common in existing research publications. Predictably, some publishers seem to be warning that their publishing model is hard to improve upon. The article mentions that a previous effort by Brown and others, The Public Library of Science garnered the signatures of 30,000 supporters, but then implies that it basically failed, suggesting that academics need the journals more than vice versa. Sounds like Brown's idea is exactly what the web is made for."

Comments

Won't Work

[Egoine]

Good idea, but the article says it all:

"It's publish or perish," Stern said. "As long as we have promotion and tenure tied to publishing, change won't work."

sadly.

That would have been great.

Changing the publishing system

[CTib]

Yes!! I heard many years ago about efforts to replace the completely broken journal/subscription of today with a peer-review/web-of-trust. Problem is, there's a market of sci. journals for a reason! Scientific excellence became a screwed notion immediately after the WWII years, when the Iron Courtain broke in two (reasonably equal halves) the older web of trust. Scientific excellence has now to be quantitatively defined, by number of articles published, especially. This is very wrong, of course.

Thus, we first need to change our perception of scientific excellence and _then_ put in place a peer-review mechanism. And the new perceptions needs a peer-review mechanism in order to be reformed properly. Hen or egg? :-/

GPL for research papers!

[Knacklappen]

I agree, that the journal publishers provide nothing more than a peer-mating service and copyediting.
That's why I find it a scandal that they charge that much for a copy. It is not, that they actually have pay to much for, because AFAIK peer-review is done for the fame alone. As I see it, they get copyright on articles, written by tax-funded researchers... and earn quite well on it.

I would be very much on favour for kind of a GPL for research papers. Anyone knows about such tendencies?

(By the way: I always found this link quite useful)

Re:Influence and Prestige

[Raiford]

Most researchers will attempt to publish whatever they can whether it be good or bad work. This is the nature of keeping your job in academia. That magical line of integrity can move around a lot when going for tenure. There is no lack of bad, trivial and non-substantive research sitting on the library shelves. The anonymous peer-review process is as lengthy as it is flawed. Next time you visit your university library take a look at the size of a volume of something like Phys Rev or J. Phys. and compare it to the size of 20 years ago. The increase in scientific production does not necessarily translate to mostly good or significant science.

Pedigree

[BWJones]

Speaking as someone who just submitted an article to Science for review, I would have to say the primary hurdle that Brown will have to overcome is pedigree. Journals like Science and Nature have a history and an editorial board that ensure a rigorous review process that ultimately presents the best of the best and the most significant science to the scientific community. Publication of ones work in journals of this caliber are important to your career, and given their wide distribution can be critical in obtaining funding.

The implications of this are far more than simple "peer mating" and "copy editing" as one other poster suggested. Granted, there is nothing that can keep an online journal from eventually becoming the place to publish, but it will take time and a commitment to excellence that will have to be maintained for scientists to become comfortable in submitting their hard earned results to. Publication of observational science will not cut it. The implication of this is that since most scientists view Science and Nature (among a select few) as the pre-eminent journals, they will be concerned about submitting the most significant scientific results to a new online journal. Typically from what I have seen, when one gets rejected from the more prestigious journals, you start moving down your ladder of preference until somebody accepts your article. Of course results targeted for specific journals with a readership that would be interested in your results always matters and this is where online journals stand the best chance of making it as opposed to large pre-eminent general interest scientific journals such as Science and Nature.

Re:Pedigree

[Foamy]

I agree that Science and Nature are the place to publish, but the quality of the many of the articles in the biological sciences is shoddy at best. Granted there are a good deal of seminal papers that come out in Science or Nature, but there are an equal, if not greater, number of 'flash in the pan' papers which are never corroborated, or are plain wrong.

The reasons for this are many IMO. First there is a huge amount of politics that go into getting past the first hurdle when submitting to these two journals--that is, not getting immediately rejected as "not of sufficient general interest or scientific impact."

Second, often the reviews are, as the AC put it, "fast-tracked" once the first hurdle is overcome. I would like to learn what the acceptance rate is for acticle that receive peer review...I bet it is very high.

Third, the extremely limited space allowed for articles in these two journals precludes the possiblity of (i) a thorough review, (ii) an in depth presentation and discussion of the data, and (iii) a complete and utter inability for other researchers to test and replicate the findings.

I always take articles in Science and Nature with a huge grain of salt. After citing a recent nature article as evidence for pursuing a specific line of research, one of my graduate mentors told me, "Don't believe anything you read in Science of Nature" until it shows up in one of the less prestigious, but more focused journals.

That said, publishing in these two will definitely get your CV noticed when you apply for that faculty position and I wouldn't hesitate a second to publish there if I could...unless of course I thought my work would get into Cell.

Modern science is in a sorry state, IMHO

[kcbrown]

Bear with me on this, please, and please correct any errors I might make here.

The whole point behind science, its entire reason to exist, is to provide us with a predictive explanation of the world around us. It needs the "many eyes" approach more than just about any other human endeavor, because the entire point is to model the real world and you can't do that without a lot of observation.

Of course, science has also proven to be useful, and that's been something of an anathema to it. The reason is that things which are useful are things which people (corporations in particular) want to capitalize on in an exclusive way. It seems to me that there was a time when everyone recognized the truth that public disclosure and widespread collaboration is necessary for science to advance.

That no longer seems to be the case from where I sit. Today, corporations fund a great deal of research at the university level, and there is a great deal of pressure from both corporations and from the universities themselves to keep ongoing research under wraps as much as possible, in order to maximize the chances not just of publishing but also of getting patents on the results (which are probably then transferred to the corporations that funded the research).

Those people in the scientific community that I've spoken to believe, to a man, in collaboration with their peers in order to further science. They're held up by the people that fund their research.

How does this relate to publishing on the web? Well, publishing on the web removes a lot of the exclusivity that currently exists, so there will naturally be opposition to it from those who benefit from the control they have over scientific publishing right now. And my cynical mind tells me that there's a good chance that those who fund research exert an additional level of control through the current publishers (it would make sense, right?). It's my hope that research over the web will help in reducing the amount of exclusivity that seems to exist currently in the scientific community. But then, that's probably wishful thinking.

As long as that level of exclusivity exists, our understanding of the universe won't advance as quickly as it might otherwise. Perhaps things have always been this way and I'm just pining for better days that have never existed. But if there's even a chance that publishing on the web will improve the amount of collaboration and peer review, I think it's worth doing.

But this proposal doesn't do much to help with that, because it still concentrates the power of peer review and publishing into the hands of a few. What prevents researchers from collaborating with each other, getting peer review from each other, and publishing on the web directly, instead of going through middlemen like they do now? Seems to me that they're being held up by those that fund the research. And unfortunately, this proposal wouldn't change that.

Yes, it's a step in the right direction, and the current scientific publishers need some competition. But it shouldn't be seen as the end goal.

Our experience

[apsmith]

I work for the Physical Review journals at the American Physical Society, and I've been somewhat involved in these debates from the physics publisher side of things for the last 8 or so years - for example in the American Scientist Forum discussion that's been going on since 1998...

Anyway, I wish Brown all the best success, but as others have mentioned, it's a somewhat harder problem than it first seems (at least he's asking for $20 million, which is somewhat realistic for handling real peer review for a substantial number of articles - 10's of thousands at least).

What's behind this nebulous "peer review" concept, at least for us, is a complex and historically based system of checks and balances involving communications between authors, editors, and (anonymous and non-anonymous) reviewers; we're essentially a legal/court system for scientific articles. There's a lot of information-related issues in there, and information technology helps a lot (that's the part I'm involved in). But fundamentally, at least the way we do it, there needs to be a paid, responsible human being reading most communications and monitoring the process, and as far as we've been able to work out, you can't get the cost under about $500 or so per article.

Now, just distributing the papers can be done essentially for free (to as many people as would want to read for about $1-5 per article, for hardware, software, disk, network, etc.) which is what the famous physics e-print archive does so well. Of course it doesn't cost publishers any more than that to distribute articles online either - the costs are in the review part (and whatever copyediting they do), not in distribution.

You'll hear about journals now that are essentially free - this is almost always for one of two reasons:

1. The journal is very small, and some institution is picking up all the salary and incidental costs - $500/article works out to just $50,000/year for a 100 article/year journal.
   
2. The journal is heavily skimping on the "peer review" side of things - publishing conference proceedings papers for example with no review beyond the acceptance of the paper at the conference. Nothing wrong with that, but it's not what we normally mean by peer review.
   

Given the $500/article cost, the other question is does science really need this level of peer review, or can it get by with less? Well, we've already seen a couple of instances of scientific fraud that slipped by in physics in the last few months even with the current level of review - is skimping really a good idea? And is the $500 minimal cost or even $1000-$2000 typical cost per article now all that bad, compared to the typical $50,000-$100,000 research grant that generally funded such research?

Yet another proposed solution has been to publish fewer papers in those journals that receive the full peer-review treatment. Unless authors miraculously constrain themselves somehow, the only way that would save us money would be to reject a lot of things without review (because the costs are in the review process itself) - but then you've thrown out the whole "peer" process you're using to determine what's published!

So, maybe Brown has found a way through this morass - but the scientific system has a complex, little studied dynamic in which peer review as it currently stands plays an important role... if we really can't afford it (the old way) any more, we're headed into some uncharted waters...

Re:Our experience

[apsmith]

Well :-)

Are we a "near monopoly"? There are hundreds of physics journals out there, in competition for authors and readers. It's a pretty free market, in my opinion, though you'll hear all sorts of arguments on that front. I assume we receive so many papers because we do our job well. Even though we publish a lot, I believe the total number of physics articles published worldwide (and we receive about 70% of our submissions from outside the US) is about five times our volume. And there's always Science or Nature!

Now the economics are a little odd because we sell subscriptions primarily to libraries, which are sort of a captive market. However, you'd be surprised at the number of small colleges and public libraries and such that subscribe - if they weren't happy, what reason would they have to continue?

The key point is your #4:

This simply begs the question of whether there is anything inherently good about "what we normally mean by peer review." Your colleagues are going to form opinions of the quality and reliability of your work anyway. A lot of bad and/or unimportant stuff gets published in journals, so peer reviewing simply doesn't fulfill the whole function of quality control. Since we need further quality-control beyond traditional peer-reviewing, why do we need traditional peer-reviewing?

I don't know that we do. Perhaps we don't. It's an experiment worth trying. But is it an experiment that's sufficiently important to have the government force it on us (as some have suggested)?

Here's a triumph of illogic: traditional peer-reviewing is works badly; therefore we must continue to use it.

Ummm. So you're in favor of more rigorous review? Wouldn't that be more expensive? It seems to me the current system works reasonably well, is improving in speed and efficiency, and at only a couple of percent of research dollars, is reasonably affordable.

The main complaint of the public library of science people etc. seems to have been that the research isn't available free online. Well, in our happy medium in physics we have no problem with researchers posting their research on their own web sites, or on the e-print server, etc. Go ahead and do it! That makes it available free - but don't expect the journal publishers to make everything free for you; we're doing a different job here.

publish or perish

[rnd()]

The article doesn't mention who would be the peer review board in the online journal system.

I think Brown could learn a lot from the open discussion forum used by /.

Anyone could "publish" an article. People would receive alerts when an article was published in a topic area of their interest. Readers would be able to rate the article on several points, and would be able to add commentary, notes, etc.

Commentary, ratings, etc., could be sorted according to the evaluators' verified academic credentials (maybe I only care about what Harvard academics think of article X on particle physics, but someone else may be interested in what the general public, or for that matter 8th graders think of article X).

Any new system would have to preserve the aspect of the status quo that generally dictates that unless the big shots in your field think you are onto something, you don't get recognition.

Open Source Literature is the real paradigm shift

[dychiang]

As I've been following the Public Library of Science (PLoS) initiative for quite some time, I find that the discussion tends to get side-tracked around peripheral issues. People seem generally supportive but skeptical of the initiative for a couple of reasons. First is the issue of cost - is it feasible to publish high-quality articles for just a few hundred dollars per article? Secondly it is generally assumed that the brand reputation of big-name journals would be impossible to crack.

Take note that the real goal of this initiative is not to overthrow the time-tested process of peer review. Rather PLoS supporters are vested in changing the publishing process - away from the pay-per-view mentality and towards an open source type of license for scientific literature, where FULL TEXT articles can be viewed and re-distributed.

Of course the marginal costs for publishing and peer review remain. The PLoS leaders propose shifting the cost burden from readers to authors - by charging a certain fee to publish an article. Their reasoning is that since government agencies such as the NIH already pay millions of dollars for journal subscriptions within research grants, those funds could be used to subsidize the author's fees instead.

In case this sounds like "selling out" quality for profit, consider that it's in a journal's best interests to achieve prominence through a high citation rate. So quality would be ensured by recruiting high-profile scientists on editorial boards. Some journals are starting to adopt this paradigm, most notably the Journal of Biology and Genome Biology

How would journals reap profits then? By charging subscriber fees for insightful commentaries and research reviews - but still allowing free access to the fruits of publicly-funded scientific research.

Can this new crop of open source journals rival the industry behemoths? Such revolutions have already rippled through the CS, physics, and math communtities, thanks to the strong support among authors. A $20 million investment, along with a firm commitment from biomedical researchers, sounds like the kick-start needed.