Slashdot Mirror
Hoax-Proofing the Open Access Journals
Harvard biologist John Bohannon wrote about his experiment in an article published by Science Magazine. He submitted his deliberately bogus paper to 304 open-access publishers, including 183 that were listed in the Directory of Open Access Journals (DOAJ), which Bohannon calls the "Who's Who of credible open-access journals", and whose quality is supposedly vetted by the DOAJ staff.
Of the 304 open-access journals targeted by the sting, 60% published the paper. I think this mainly just shows that the average quality of open-access journals may always be low, but that's not surprising since anyone in the world can set up an "open access journal". That shouldn't be relevant to the reputation of the best open-access journals. If the best open-access journals acquire a reputation for high standards and proper peer review, then that will attract high-quality papers, whose publication will reinforce the reputation of the journal, which enables it to confer prestige on the papers it publishes, which in turn will continue to attract high-quality papers. The existence of other open-access journals with crummy standards, should be irrelevant.
What's more disturbing, is that of the 183 journals listed in the Directory of Open Access Journals, 45% of those published the paper -- which, according to Bohannon's article, surprised and disappointed the DOAJ founders. But perhaps if you're maintaining a database of thousands of allegedly reputable open-access journals, there's no way to make sure that they're all telling the truth about their standards and their practices. At a quick glance, all you can really say is that they would be good-quality journals if they're telling the truth about how they operate, but it's hard to tell from the outside whether they're being honest.
So perhaps a different solution is that we don't really need a huge number of good open-access journals. Rather, in each field, you could get by with a small number of "super-journals" which have a lot of reviewers on file, and which publish a high number of papers but apply uniformly high standards across all of them.
Consider: you have two journals, A and B. Each has their own non-overlapping database of 20 reviewers. When they receive a paper, the standard practice for each of them is to send the paper to 3 randomly chosen reviewers in their database. Each one receives 10 submissions per month.
Now combine A and B to form one single journal which has 40 reviewers and gets 20 submissions per month, and still sends out each submitted paper to three randomly chosen reviewers. The total amount of work performed by the reviewers, doesn't change. But now, if you're auditing the quality of a journal according to its adherence to its own practices, you only have to audit one journal instead of two. By the same logic there's no reason in principle that any number of journals in one field couldn't be subsumed into a few behemoths, which apply uniform standards across all their papers.
You could do this without waiting for the traditional system to be dismantled. Somebody in the field just assembles a list of people to be peer reviewers for the "virtual super-journal". That list is public, so that anybody can audit it and see that it consists of people with a credible reputation in their field. Anyone who pays the (nominal) fee can submit a paper to the VSJ, which sends the paper to a random selection of n reviewers from that list. If the paper "passes" the test, then it gets the stamp of approval of the VSJ, which says, "This paper was judged to be good by a majority of a random sample of reviewers on our list, and you can see from this list that the quality of our reviewers is pretty good."
And suppose someone wants to publish their paper in some other journal XYZ, and they also want to publish it in the VSJ just to get a certification of its quality, but journal XYZ doesn't allow them to simultaneously submit it to another journal for publication? In that case, you can still submit your paper to get the stamp of approval from the VSJ -- just pay the normal reviewing fee, and if it passes the VSJ's review process, they can list the paper on their website, saying, "This paper was judged to be good by a majority of our reviewers. We can't actually publish the paper here, because some other journal XYZ has exclusive publication rights, but you can view the paper at this link in this other journal." You still have the self-reinforcing cycle where the VSJ's stamp of approval maintains high standards, which attracts high quality papers, which reinforce the reputation of the VSJ's stamp of approval. There's no part of that cycle that requires the VSJ to actually "publish" the paper itself.
And people could subscribe to the VSJ's "stamp of approval" feed the way they subscribe to any other publication -- the VSJ can send out the papers themselves that they have the right to publish, or links to papers in other journals, saying, "This paper got our stamp of approval, and follow the link to read it here."
You could even use this process to do a "hit job" on someone else's paper that got published in another journal, but which you think is too low-quality to have been published. You can submit it to the VSJ and if the VSJ rejects it, you can ask them to list it as a paper that failed their review process. (Whether or not the VSJ would give you the option of doing this, may depend on their policies. It "sounds mean", yes, but academics are supposed to keep each other honest. I've never heard of a traditional journal doing that -- calling out a paper published somewhere else and saying, "This sucked, we never would have published it.")
There should probably be multiple open-access journals (or Virtual Super-Journals) within each field, so that the competition between them keeps them honest. But there's no reason to have such a huge number of them that the Directory of Open Access Journals can't keep track of what they're doing.
Surely the solution is to have people who understand the papers actually reading them. And if nobody among you understands them then you don't accept them.
And if you don't do that, you don't really have an academic "journal", just a blog.
It works for /.
Set your phasers on "funky"!
Perhaps the title of this article should be: Harvard biologist commits Fraud on Open Access Journals. The Journals all assume that the author is acting in good faith and believes his result. Not that he is trying to trick them into publishing an incorrect paper. They are 'peer' reviewers, not Police. Just because they got tricked by a con artist doesn't automatically lead to the conclusion that the system is broken.
A major problem with open-access journals is that there is no motivation for them to reject submissions, If anything, the more they publish the more money they make. Likewise, peer reviewers (at least in my field --natural language processing and machine learning) are never paid to review them. This is not a good combination. I cannot see any reason for journals nowadays. Either publish in conferences (which in some fields are competitive and very tightly reviewed) or better still publish them on arvXiv and have some kind of citation / comment system as a way to crowd-source quality control.
The established publications — often denounced as "greedy" for having the audacity of wanting to get paid — do add value, after all?
Next in the news: a private farm's crop beats the yield of a communal field.
In Soviet Washington the swamp drains you.
It doesn't matter if it is an "exclusive" journal or one that is open access. If a scientist submits fake data to a publication, shouldn't the scientific community take the time to verify his results? I'm pretty sure that had he just made up his findings someone somewhere would have called him on it and his cred amongst his colleagues will go down the toilet. Isn't this how cold fusion was proven false?
Slashdot engine should be used to maintain author and paper karma points, moderation and meta-moderation! Yeah?
Notice he only submitted his fake papers to open access journals. As a scientist, and especially as a biologist, he's perfectly aware of the importance of control groups. If he were honest, he would have submitted the same papers to closed, for-profit journals as well, even if it cost him money to do so.
you have fewer journals that have to be audited for procedural honesty
Taking this to its logical conclusion, a monopoly is the most honest organization, right?
Once one of these "fewer" journals has an established reputation, it can obscure its procedures and refuse to be audited, while it turns corrupt for profit. Since it's still a well-known journal (because who really has time to monitor the procedural audits, anyway?) it will still get the submissions and readers, and it will stay relevant for many years after "everybody knows" its' corrupt.
You do not have a moral or legal right to do absolutely anything you want.
Modern journals are all online. So, it would be easy to provide a traffic light system that indicates if a reference is found to be fraudulent or incorrect. In a correct paper, all references would have a green background. If one of those papers was found to be incorrect by a 3rd party, it would be flagged and its colour changed to amber. This would propagate to all papers that make a reference to that paper and the entire chain would become amber. This would force all authors to update their papers, or the author of the original flagged paper to correct their work. If a paper is just flat out wrong, discovered to be a fake, or fails to be updated after a period of time in amber, it would become red, which again would propagate to every paper that uses it as a reference.
This will keep the chain of dependencies clean throughout the entire scientific world and minimise the impact of improperly peer-reviewed work.
This paper has already been extensively critiqued. To me the biggest problem is that he didn't include any subscription journals.
Many intentionally flawed or nonsense papers have been submitted -- and published! -- to reputable journals in the past.
This latest demonstration by Bohannan just shows that the peer review system needs improvement. It does not show whether Open Access journals are better or worse than subscription journals in terms of quality and reliability of content.
Is this information really meaningful without a similar test on the paid journals?
The Journals all assume that the author is acting in good faith and believes his result. They are 'peer' reviewers, not Police.
Meaningful peer review demands an intelligent evaluation of the author's arguments and evidence and the clarity with which they are presented. Good faith does not imply good science. Belief does not imply good science. Neither faith or belief implies good writing and sound editing.
As crappy researchers will not get their shitty papers accepted at open access journals and will stop publishing.
If ten a month is the standard load for a reviewer, I think there's already a problem. Reading an article should probably be allotted at least an hour. Any fact-checking will take more. I read articles for the humanities, and that's pretty easy. You can spot a bullshitter pretty quickly, in a page or two. But I'd imagine science can be trickier. So, the half-hour or so it might take me to be sure that it's crap would probably double for science. At a minimum. I say minimum because reading a stack of a dozen poetry submissions can easily take me over an hour, and that's really not very much text. Then you have the separate but connected problem of being rushed or just feeling sick and tired of the stack and rushing through it. It seems to me like it's a recipe for rubber-stamping and carelessness. I know that a science journal my ex-wife worked for sent out far fewer articles a month. But it was a small journal on a narrow topic. I think that it will boil down to, this whole issue, to the fact that you'll always be able to game the system. The process of peer-review doesn't end with publication. For good reasons.
Peer review was a necessary filter when there were limited options for publishing to a wide community, mostly due to expense.It wasn't necessarily a golden age, and there were tradeoffs in that possibly many deserving papers weren't published for various reasons. Obviously it's different today and quality of content e.g. as measured by citations is starting to replace quantity. An ignored paper today is the same as a rejected paper yesterday.
for some reason, shit like this makes it to the front page. I think that reason is: timothy.
Anybody who expects peer review to let through only correct papers doesn't understand peer review. Science editors, of all people, should understand that, given the many bogus and fraudulent papers they have published over the years.
In different words, there's nothing to fix here.
The publication process has gone so far downhill it's basically not recognizable as science any more. This is driven by the university tenure process. Being a tenured professorship is a sweet job. The hours are short and flexible and the work is interesting and varied. Pay is less than industry, but once tenured the pay is guaranteed. Benefits are usually top-notch. That's an appealing package for anyone of reasonable intellect, middling ambition, and a desire for ironclad security. Not surprisingly, the supply of would-be professor labor greatly outstrips demand.
So who gets that cushy seat? Well, it's all based on publications and grant money. Grant money is based mostly on publications. So what you, would-be professor, need is a pile of publications. This is a huge change from the scientific publications of yore, which were by and large written for the benefit of the reader. These papers are written for the benefit of the WRITER, and that makes all the difference.
Most are on insanely obscure topics. The writer needs novelty (which is easiest achieved by obscurity) to get past peer review and no one cares if anyone else actually wants to know about the topic. Organization and clarity are for the birds - as long as the reviewers can't prove you're wrong per say it will get accepted somewhere eventually, especially at a pay journal. Reproducibility is actually undesirable - the last thing you want is scrutiny. It can't get you another publication, but it could force you to retract one. The problem being addressed by a given paper is typically very easy, but made to look very hard. Solve a hard problem, get one paper. Solve an easy problem that looks hard, get one paper. It's a no brainer.
Think these papers won't get past peer review? Think again. Mostly the journals don't REALLY read them. Just sort of skim. Think tenure committees will evaluate the papers on their merits? Think again. They don't have time, and in most universities the ultimate arbiter of tenure is the whole body of professors, most from different fields. TAre they're going to parse your obscure minutia? Heck no. They weigh it.
This can't be changed by fixing the journals. The real problem is that many of those publishing are doing so in bad faith. Right now scientists have exactly the journals they deserve.
It's about time to abandon the concept of "journals" altogether. No one wants to read journals. They want to read individual papers that pertain to the subject of their interest. Post individual papers, open to review by others who are qualified to comment on the content. The papers and comments are open access worldwide, after the specified comment period.
At present, peer review for most journals is done by junior faculty who are inexperienced and eager to advance their career, not be the real experts. It is a broken system. And there is no justification for maintaining a publishing model that was designed to share knowledge among a limited number of people during the print-on-paper era. Now, journal publishing is merely a mechanism for furthering the economic interests of publishers.
If StackOverflow made open access journals, he would now have damaged his reputation because other users of the site would vote his publication down. This would have a knock on effect on 1) How his future publications are ranked 2) How seriously the feedback he leave for others is taken 3) And ultimately the grant money he would receive. Basically, he would be shotting himself in the foot.
I think that the faster we move to open access journals with a single sign in, voting and comments - the sooner crap will be filtered out.
How do you verify the results?
Reviewers just look at the data, look at the assumptions and statements and see if it "makes sense".
Generally they're not capable of rerunning the study.
... then you would be able to vote his paper down for being nonsense. This would then reduce his reputation and would: 1) Reduce the rating of future publications 2) Reduce the affect of feedback that he leaves others 3) Reduce his funding. I.e. Post bad work and reduce the chances of succeeding as an academic.
Ok, maybe "fraud" is a slightly strong term, but it's pretty close. There are thousands of "open access journals" created only to make money by sounding as if they were legitimate journals, getting people to send them articles, and then charging to publish them. I get spam from them on a daily basis. They aren't legitimate, they have no interest in quality, and they have no reason ever to reject an article.
Please don't confuse legitimate open access journals like PLoS with these scams.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
Disclosure: I'm the co-founder of Publons.com
Really good post. I think you've hit on the key issue, which is that peer review can be done better.
One problem is that peer review probably is done well in a lot of cases; we only hear about it when the system breaks down. From that perspective the most obvious solution is to start focusing on peer review and giving reviewers credit for the times where they do a good job. One way we're doing that is by assigning DOIs to post-publication reviews that the community decide are a valuable contribution to science. This turns reviews into citable, indexable publications in their own right. (See e.g., http://dx.doi.org/10.14322/publons.r38)
Isn't much of the supposedly good research seriously faulty and impossible to reproduce? Fix that first?
Peerage of Science got quite a bit of publicity earlier this year. The idea is similar to what was suggested in the OP: You send you manuscript to PoS, who forward it to peer reviewers. When the reports come back in, journal editors can bid for the article. If the authors are interested in publishing the work in a journal that show the green light, they can. Suppose the authors' preferred journal(s) does not want to publish the paper, the authors can withdraw their paper from PoS and pursue more traditional routes. What you will get is more referee reports (I think I read ~5), and the referee's reports will be later peer review, too, so editors get a good sense of which referee to trust.
Before you say that 5 referee's is a waste: the average paper probably goes through a couple of different journals, being rejected once, twice, each time requiring 2-3 referee reports. Personally what I think is the biggest problem in science is inadequate peer review: Not enough people read the manuscript before it is published, and when it finally is published, no major changes can take place. There is very little discussion after, too, as comments in PLoS journals are rather informal, and one would onyl send a letter to the editor if there's something major wrong with the paper.
If you submit a paper that's crap, you get killed. If you're dead when your crap gets discovered, your closest relative gets killed, and so on.
I think this would work fine.