Science Magazine "Sting Operation" Catches Predatory Journals In the Act

sciencehabit writes "A sting operation orchestrated by Science's contributing news correspondent John Bohannon exposes the dark side of open-access publishing. Bohannon created a spoof scientific report, authored by made-up researchers from institutions that don't actually exist, and submitted it to 304 peer-reviewed, open-access journals around the world. His hoax paper claimed that a particular molecule slowed the growth of cancer cells, and it was riddled with obvious errors and contradictions. Unfortunately, despite the paper's flaws, more open-access journals accepted it for publication (157) than rejected it (98). In fact, only 36 of the journals solicited responded with substantive comments that recognized the report's scientific problems. The article reveals a 'Wild West' landscape that's emerging in academic publishing, where journals and their editorial staffs aren't necessarily who or what they claim to be."

Comments

Click

[mynamestolen]

How many of the open access journals rely on click through advertising? Follow the money, I say.

Re:Click

[ATMAvatar]

In this "experiment", what was the control group?

Re:Click

[girlintraining]

Real scientists create experiments that can be reproduced and independently verified and they did not. Q.E.D.

This is less about a failing of science and open publishing journals than the fact that on the internet, reputations can be shed like a snake sheds its skin -- you're just a few clicks away from a new account and a new identity. This has been a long-studied problem in cryptography -- how to create trust networks in public key crypto with key signing parties, etc. That the lessons learned there apply to social networking sites and open publication journals as well requires only the smallest amount of creativity to see.

If you want honesty, you need to have some way of punishing people who are dishonest. It really is that simple; You need a way to saddle them with a cost that can't be shed by simply switching identities. And the best way to do that, for better or for worse, is a central authority in the real world that matches online identities to real-world ones. Everything else is varying degrees of broken.

Create a blacklist of people who have lied and although you may be able to overwhelm the system for awhile, it is self-correcting... eventually it will run out of people willing or able to get blacklisted, and the quality will then start to rise as people are forced to be responsible for what they say and do.

Re:Click

[Kijori]

What would you like to control for?

The null hypothesis is that the journals have sufficiently good review processes to avoid publishing papers with obvious and fatal flaws. If you submit a paper with obvious and fatal flaws and it is published, that hypothesis is not true. It's proof by counter-example, and no control is required for it to be valid.

Controls?

[Anonymous Coward]

Did he send the bogus articles to closed publishers too? How did the rates compare? I tried to RTFA, but didn't see anything about controls.

Re:Controls?

[godrik]

No that is not the point. The point of OpenAccess papers is to allow a larger communicatino of the papers by removing the barrier of ridiculously high access fees. Accessing a single paper can cost $50 for a researcher that do not have the proper subscription. OpenAccess journals are mainly designed to take the editors and publishers which ask for a ridiculously high publication fee. or cost of access.

Open Access does not mean that anything get published in there. Though as a reviewer for many computer science journal, I can guarantee you that everybody can publish in there... assuming the level of contribution and style are up to standard of scientific method and writing. That is a very difficult thing to achieve for a non academic because of the time comitment in "learning" how to write these papers.

Re:Controls?

[TheGavster]

There is a certain amount of irony in someone attempting to prove that open access journals publish bad science through the use of bad science. I read the article, and his only mention of testing closed publications is in his conclusion, quoting a colleague who suggested just such a step. He discounts this by restating his thesis (that open access journals are more numerous and publish more papers than closed ones) before shifting topics.

Democratization

[SecurityTheatre]

A lot of people cite the democratizing power of "open access" and "crowd sourcing". I feel this is an example of the same principle at work.

On one hand, it is easier for those that are not entrenched within the bastions of power to be heard, but on the other hand, all data received from these sources must be treated much more cautiously.

In the past "being published" was a big deal, as it required a fairly high bar of factual accuracy, and that is still the case of many prestigious journals, but in the rush to Twitter-ize research and accept as many publishable details as rapidly as possible in the name of profit and prestige, the barriers to entry have eroded.

In much the same way that hard investigative journalism with strong ethical guidelines, verifiable sources and solid editing will always have a place in my heart, these reputable journals can serve to establish a foundation of trust in the scientific arena. And now, in much the same way that one should treat any writing within the "blogosphere" as suspect until verified, many open access journals must now be treated with the same level of suspicion until it is proven otherwise that they hold themselves to a higher standard.

TLDR: Democratization is not always a good thing.

Re:Democratization

[fermion]

First the disclaimer. I do believe that professionally peer reviewed journals and reporting still has a place. I pay significant sums of money to subscribe to a newspaper, a few top magazines, as well as Science and Nature. They serve a purpose and, to me, are worth the costs.

That said Science is not beyond reproach on accuracy. Both journals has had a very scandalous path over the past few years with their accepting clearly fraudulent papers. In July, evidently, Alirio Melendez had a paper retracted. This researcher fooled many major journals with at least 13 papers. Science also published the paper on bacteria living on arsenic, which is generally seen as having major issues. I recall reading a paper related to dancing and sexual attraction, maybe in Nature, being retracted due to fabricated data.

That said, there is little wrong with a single suspect paper being published. This is how scientists communicate. There is little protection against fraud such as occurred in this case because it is so patently silly. Building a system to protect against such silliness would mean that we would no longer be focused on science. The real problem here is that the popular media does not understand the difference between a single piece of research and the process of research. Places like /. should know better, but they don't. The process of science is to reproduce and extend results. When a bad paper corrupts the process, as has happened when Science and Nature has published suspect paper, that is a problem. These journals, having high impact factors, have a responsibility to proctor what they publish. A backwater online journal does no necessarily have such responsibility, rather relying on the ethics of the researcher and a faith in the process of science to ferret out unethical and silly people like these.

What is truly alarming is the simple bad science present in this research project. This experiment has no control group and does not try to match the target journals to an equivalent paper journals.

If the research was done properly the open access journals would be matched with closed journals on the basis of several relevant criteria, like impact factor, cost to publish, region predominately served, or the like. This is the way research is done. One can't just go out onto the street, ask 10 people who you don't like if they ever thought of killing someone, then claim that everyone in this group are murderers if 7 say yes.

The paper would then be submitted to all the journals, the results generated using well known statistical methods, and then, if there is some degree of confidence, the results published.

My prediction is that if you were paying a closed source low ranking journal to publish a paper asserting that the moon was composed of coagulated casein in a mesh of lipids they would not blink at printing it.

At the end of the day, in this case Science is no better than your average corrupt advertising agent.

Mediocrity in Academics

[EMG at MU]

For years we have known that there is a glut of graduates in the system. I remember my freshman year at university, the attitude of a lot of students was "the Masters is the new Bachelors, you have to have one to get an entry level job" or when I got closer to graduating it was "well I don't want to be done with school and my parents are helping me out so I'm going to go for my Masters". While education is awesome, the fact is that you don't have to be all that smart anymore to get a Masters or PhD.

Even as an undergrad I was pressured to publish. I didn't have the time nor the resources to do anything meaningful but my professors all said that I had to publish to even consider going to graduate school. They said that pretty much no matter what I do, even if its not novel or valuable to the academic community there will be a journal that will publish it. That's the current state of academics now.

Lets be clear: I'm not talking about MIT or Berkley. I'm talking about the thousands of research institutions across the country that while also doing amazing research, churn out Masters and PhDs like a printing mill. When you dilute the pool of researchers there is going to be subpar research. When there is a glut of subpar research there will be journals that see the business opportunity and publish anything you pay them to publish. This is not new.

Not submitted to proprietary journals?

[mspohr]

Science has an axe to grind here, obviously, and this "experiment" is seriously biased.
It does not appear that it was submitted to any closed, for-profit journals (like Science). It would have been much more interesting to see how many of them would have accepted the paper.

Re:Bias

[Anonymous Coward]

Actually, it's really really badly done.

To actually make any of the conclusions (or inferences) about the quality or rigor of open-access journals REQUIRES a control group of traditional journals to be operated on in a similar manner. In other words, there needs to be a sting on both open-access and traditional journals simultaneously.

Without that, no claims can be made. None. Not even one. Because we DO NOT KNOW how many traditional journals, like Science, would also have accepted their falsified paper(s). It's possible the traditional journals could have lower standards of quality and rigor than the open-access group.

Science and AAAS (of which I'm presently ashamed to be a member) should be blasted for publishing this tripe. It needs to be retracted, immediately. If they want to have the slightest shred of credibility here, they should at least conduct scientifically rigorous stings.

Disgusting.

Re:The total number of these journals is irrelevan

[goodmanj]

The problem is that serious decisions are made by people who have no idea which journals are top quality. Bad tenure decisions, bad engineering choices, and god forbid bad medical decisions are being made daily on the basis of nothing more than "hey, the European Journal of Chemistry sounds legit."

Re:Bias

[goodmanj]

Yes, but. This isn't entirely a binary scientific question. If the question were "are open-access journals worse than traditional journals?", you'd obviously need a control. But "Is the peer review process at open-access journals acceptable?" is not a scientific question, but one of values and personal preference. Most people would decide that a 50% failure rate is not acceptable, control or no control.

Now, we're all *very* curious to know whether traditional journals fare better than open ones, and Science is showing bias and intellectual dishonesty by avoiding that question, BUT that doesn't mean that this study has no value.

several valid pushbacks from this article

[drDugan]

This article is being widely panned as lacking controls, published without any critical review, and driven by self-interest from a traditional publisher with the most to lose from Open Access taking off (as it is). Some have gone so far to assert it's an over-reach for how badly it was done, and will make Science as a journal look partisan.

For example, quick scan brought up these three scathing responses:

Mike Eisen (HHMI Berkeley Professor)
http://www.michaeleisen.org/blog/?p=1439

Peter Suber (Author of the book "Open Access", Director of the Harvard Open Access Project, Faculty Fellow at the Berkman Center)
https://plus.google.com/u/0/109377556796183035206/posts/CRHeCAtQqGq

Mike Taylor (programmer with Index Data and a research associate at the department of earth sciences, University of Bristol)
http://svpow.com/2013/10/03/john-bohannons-peer-review-sting-against-science/

I'm sure this will heat up some much needed debate about poor quality journals and the failings of peer review, but with the lack of any controls at all, it says basically nothing about open access as a model for publishing.

Re:The total number of these journals is irrelevan

[semi-extrinsic]

In Norway, we have a "level" system that is used in academia throughout the country. It is used for evaluating researchers and research groups when it comes to employment, tenure, funding etc. Your "point score" is summed up, 2 points for publication in a "level 2" journal, 1 point for "level 1".

A journal is either "level 2", "level 1" or "level 0". "level 2" is a selection of top journals from each field in science, 2000 in total (for all of science, from computational physics to the sociology of music). "level 1" means the remaining serious peer-reviewed journals. "level 0" either means "bullshit journal" or "journal that was founded just last year".

Researchers may nominate journals for a change in status, e.g. 2->1, 0->, etc. The decisions are made by a government-appointet body on a yearly basis. It's nowhere near perfect, but it's a lot better than nothing.

Re:The total number of these journals is irrelevan

[Obfuscant]

The total number of these journals is perhaps the more relevant part of this article. There are 304 journals that are potential relevant places for that one submission? How can anyone keep up with the current science in any field when there are 304 places to look? Never mind that many of those aren't sufficiently vetting the product.

And if you are just writing them off and basing your reading on the "top ones", of what value are these?

While science journals are often used by researchers to find out what their colleagues are doing and can thus be vetted by the reader, they are quite often the bases for undergraduate and graduate educations, and putting deliberate crap in front of them is a Bad Thing.

Re:The total number of these journals is irrelevan

[dgatwood]

A journal is either "level 2", "level 1" or "level 0". "level 2" is a selection of top journals from each field in science, 2000 in total (for all of science, from computational physics to the sociology of music). "level 1" means the remaining serious peer-reviewed journals. "level 0" either means "bullshit journal" or "journal that was founded just last year".

Here's the problem with doing that so systemically: it is fundamentally anticompetitive, and leads to stagnation. Nobody would bother submitting to a "level 0" journal because it won't earn them any props at all, which means that the journal can never become anything more than a "level 0" journal. This means that you don't get fresh blood with new ideas on the review boards, so progress moves at a snail's pace. There's something to be said for disruptive innovation, even in academic publishing circles.

Also, the entire notion of judging the value of your scientific contribution based on what journal agreed to publish it is as absurd as judging the value of a car based on what dealer sold it. A paper should stand or fall on its own merits. A good article that pushes science forward, even if published in a minor journal, should weigh significantly in your favor for tenure, and a lousy article, even if published in a major journal, should not. A system that does the opposite is abject stupidity, pure and simple.

what did they expect???

[l3v1]

In all seriousness, I - as a researcher myself - understand the need of easy access to publications. However, I never supported the open access models that came into existence and are being built and pursued today. Why? Because it's all about the money and a lot of such journals absolutely do not care about quality, or about having big name editors who'd perform very thorough revision of reviews and make proper decisions about paper acceptances. Big journals have good editorial and review staff, and they simply can't allow them to be bad and irresponsible, because they actually care about their reputation and credibility. New breed open access journals on the other hand only care about revenue.

The instititue I work at has mandated open access publication as well as others did, however, they did not provide funding for us to actually publish open access versions at big name journals, so we try to play the system whenever we can, and publish in traditional journals with traditional publication schemes. I do not care about some politician-flavored scientists' (most of them not even publishing) dreams about some utopistic open access world. I care about publications appearing in credible journals, reviewed by credible people, producing quality publications - even if they are only attainable for money.

Umm no

[Weezul]

Clicks are not the problem. Journals don't get any money from advertisement clicks. Real problem is :

At present, "Open Access Publishing" mostly means "Author Pays". If the author is your customer, then obviously you publish whatever they want. We must abandon the extortionate academic publishers like Elsevier all together by building an arXiv overlay filters that take over the journal's role of reviewing and declaring papers important. And these must be paid for by tax money because the customer should be society.

Just like with universities, Britain has rampant grade inflation because the students all pay 15k USD per year (9k GBP). St Andrews has a 98% graduation rate. A 98% graduation rate tells me the university did basically no "selection" on their admitted students, all selection occurred when an admissions person read their test scores from high school. In other words, the student is the customer and the product is a little piece of paper. This is why Britain sucks so bad at engineering and must create that blatantly bullshit ranking system by THES to make themselves look good.

In continental europe, almost everyone who finishes high school can attend university without paying, but the universities select students by failing out the shitty ones, well society is the customer and the students are the product. It's infinitely more fare because gaming the system in high school does nothing and people who never really hit their stride until the find challenging material do well.