Science Magazine "Sting Operation" Catches Predatory Journals In the Act

sciencehabit writes "A sting operation orchestrated by Science's contributing news correspondent John Bohannon exposes the dark side of open-access publishing. Bohannon created a spoof scientific report, authored by made-up researchers from institutions that don't actually exist, and submitted it to 304 peer-reviewed, open-access journals around the world. His hoax paper claimed that a particular molecule slowed the growth of cancer cells, and it was riddled with obvious errors and contradictions. Unfortunately, despite the paper's flaws, more open-access journals accepted it for publication (157) than rejected it (98). In fact, only 36 of the journals solicited responded with substantive comments that recognized the report's scientific problems. The article reveals a 'Wild West' landscape that's emerging in academic publishing, where journals and their editorial staffs aren't necessarily who or what they claim to be."

Comments

Click

[mynamestolen]

How many of the open access journals rely on click through advertising? Follow the money, I say.

Re:Click

[moteyalpha]

Science Magazine did a bad experiment about submitting a spoof scientific report so that you would click on them! How can you trust a science magazine that uses bad scientific methods to make a point. Real scientists create experiments that can be reproduced and independently verified and they did not. Q.E.D.

Re:Click

[moteyalpha]

It would be a de novo experiment as the first was not independent. It seems much like the Microsoft "Scroogle" ads. It made me think they must be desperate to employ such methods. "Coke says Pepsi sucks, Coke confirms it" It was not intended to be a real serious poke at them as I really like their magazine and they do have good articles in my opinion. It sounds like marketing department logic at work here.

Re:Click

[girlintraining]

Real scientists create experiments that can be reproduced and independently verified and they did not. Q.E.D.

This is less about a failing of science and open publishing journals than the fact that on the internet, reputations can be shed like a snake sheds its skin -- you're just a few clicks away from a new account and a new identity. This has been a long-studied problem in cryptography -- how to create trust networks in public key crypto with key signing parties, etc. That the lessons learned there apply to social networking sites and open publication journals as well requires only the smallest amount of creativity to see.

If you want honesty, you need to have some way of punishing people who are dishonest. It really is that simple; You need a way to saddle them with a cost that can't be shed by simply switching identities. And the best way to do that, for better or for worse, is a central authority in the real world that matches online identities to real-world ones. Everything else is varying degrees of broken.

Create a blacklist of people who have lied and although you may be able to overwhelm the system for awhile, it is self-correcting... eventually it will run out of people willing or able to get blacklisted, and the quality will then start to rise as people are forced to be responsible for what they say and do.

Awesome, now let's test schools the same way

[intermodal]

Seems like degree-mills are more common than actual universities by the same token.

Democratization

[SecurityTheatre]

A lot of people cite the democratizing power of "open access" and "crowd sourcing". I feel this is an example of the same principle at work.

On one hand, it is easier for those that are not entrenched within the bastions of power to be heard, but on the other hand, all data received from these sources must be treated much more cautiously.

In the past "being published" was a big deal, as it required a fairly high bar of factual accuracy, and that is still the case of many prestigious journals, but in the rush to Twitter-ize research and accept as many publishable details as rapidly as possible in the name of profit and prestige, the barriers to entry have eroded.

In much the same way that hard investigative journalism with strong ethical guidelines, verifiable sources and solid editing will always have a place in my heart, these reputable journals can serve to establish a foundation of trust in the scientific arena. And now, in much the same way that one should treat any writing within the "blogosphere" as suspect until verified, many open access journals must now be treated with the same level of suspicion until it is proven otherwise that they hold themselves to a higher standard.

TLDR: Democratization is not always a good thing.

Re:Democratization

[fermion]

First the disclaimer. I do believe that professionally peer reviewed journals and reporting still has a place. I pay significant sums of money to subscribe to a newspaper, a few top magazines, as well as Science and Nature. They serve a purpose and, to me, are worth the costs.

That said Science is not beyond reproach on accuracy. Both journals has had a very scandalous path over the past few years with their accepting clearly fraudulent papers. In July, evidently, Alirio Melendez had a paper retracted. This researcher fooled many major journals with at least 13 papers. Science also published the paper on bacteria living on arsenic, which is generally seen as having major issues. I recall reading a paper related to dancing and sexual attraction, maybe in Nature, being retracted due to fabricated data.

That said, there is little wrong with a single suspect paper being published. This is how scientists communicate. There is little protection against fraud such as occurred in this case because it is so patently silly. Building a system to protect against such silliness would mean that we would no longer be focused on science. The real problem here is that the popular media does not understand the difference between a single piece of research and the process of research. Places like /. should know better, but they don't. The process of science is to reproduce and extend results. When a bad paper corrupts the process, as has happened when Science and Nature has published suspect paper, that is a problem. These journals, having high impact factors, have a responsibility to proctor what they publish. A backwater online journal does no necessarily have such responsibility, rather relying on the ethics of the researcher and a faith in the process of science to ferret out unethical and silly people like these.

What is truly alarming is the simple bad science present in this research project. This experiment has no control group and does not try to match the target journals to an equivalent paper journals.

If the research was done properly the open access journals would be matched with closed journals on the basis of several relevant criteria, like impact factor, cost to publish, region predominately served, or the like. This is the way research is done. One can't just go out onto the street, ask 10 people who you don't like if they ever thought of killing someone, then claim that everyone in this group are murderers if 7 say yes.

The paper would then be submitted to all the journals, the results generated using well known statistical methods, and then, if there is some degree of confidence, the results published.

My prediction is that if you were paying a closed source low ranking journal to publish a paper asserting that the moon was composed of coagulated casein in a mesh of lipids they would not blink at printing it.

At the end of the day, in this case Science is no better than your average corrupt advertising agent.

Re:Democratization

[Anonymous Coward]

"And now, in much the same way that one should treat any writing within the "blogosphere" as suspect until verified, ..."

Oddly enough, the entire point of science is that all claims are suspect until verified. And I applaud anyone that approaches any testimonials in that fashion. Whether it's in the blogosphere of the sciencesphere.

Not submitted to proprietary journals?

[mspohr]

Science has an axe to grind here, obviously, and this "experiment" is seriously biased.
It does not appear that it was submitted to any closed, for-profit journals (like Science). It would have been much more interesting to see how many of them would have accepted the paper.

Re:Bias

[Anonymous Coward]

Actually, it's really really badly done.

To actually make any of the conclusions (or inferences) about the quality or rigor of open-access journals REQUIRES a control group of traditional journals to be operated on in a similar manner. In other words, there needs to be a sting on both open-access and traditional journals simultaneously.

Without that, no claims can be made. None. Not even one. Because we DO NOT KNOW how many traditional journals, like Science, would also have accepted their falsified paper(s). It's possible the traditional journals could have lower standards of quality and rigor than the open-access group.

Science and AAAS (of which I'm presently ashamed to be a member) should be blasted for publishing this tripe. It needs to be retracted, immediately. If they want to have the slightest shred of credibility here, they should at least conduct scientifically rigorous stings.

Disgusting.

Re:The total number of these journals is irrelevan

[goodmanj]

The problem is that serious decisions are made by people who have no idea which journals are top quality. Bad tenure decisions, bad engineering choices, and god forbid bad medical decisions are being made daily on the basis of nothing more than "hey, the European Journal of Chemistry sounds legit."

Re:Controls?

[godrik]

No that is not the point. The point of OpenAccess papers is to allow a larger communicatino of the papers by removing the barrier of ridiculously high access fees. Accessing a single paper can cost $50 for a researcher that do not have the proper subscription. OpenAccess journals are mainly designed to take the editors and publishers which ask for a ridiculously high publication fee. or cost of access.

Open Access does not mean that anything get published in there. Though as a reviewer for many computer science journal, I can guarantee you that everybody can publish in there... assuming the level of contribution and style are up to standard of scientific method and writing. That is a very difficult thing to achieve for a non academic because of the time comitment in "learning" how to write these papers.

Re:Science is the new religion

[HornWumpus]

The soft sciences were never rigorous. Nothing has been lost or gained.

Very few are fooled. Sociologists/Psychologists/Economists can say they've 'proved' something till they're blue in the face. Nobody will take them seriously.

Re:Bias

[goodmanj]

Yes, but. This isn't entirely a binary scientific question. If the question were "are open-access journals worse than traditional journals?", you'd obviously need a control. But "Is the peer review process at open-access journals acceptable?" is not a scientific question, but one of values and personal preference. Most people would decide that a 50% failure rate is not acceptable, control or no control.

Now, we're all *very* curious to know whether traditional journals fare better than open ones, and Science is showing bias and intellectual dishonesty by avoiding that question, BUT that doesn't mean that this study has no value.

The problem is more that of peer review

[WillAffleckUW]

It's more who is reviewing the material, and to what level, that matters.

A lot of high-grade peer reviewed journals, like Science and Nature, have been hoodwinked by researchers from South Korea and China, where cheating is more endemic than here.

Or, as most of us say, Wait Until The Second Journal Article.

Re:The total number of these journals is irrelevan

[Obfuscant]

The total number of these journals is perhaps the more relevant part of this article. There are 304 journals that are potential relevant places for that one submission? How can anyone keep up with the current science in any field when there are 304 places to look? Never mind that many of those aren't sufficiently vetting the product.

And if you are just writing them off and basing your reading on the "top ones", of what value are these?

While science journals are often used by researchers to find out what their colleagues are doing and can thus be vetted by the reader, they are quite often the bases for undergraduate and graduate educations, and putting deliberate crap in front of them is a Bad Thing.

Re:Bias

[stenvar]

In fact, over the years, Science has published numerous scientifically fraudulent papers, some of which were pretty blatant. So, in a sense, we already have a control. In addition to control experiments, it needs three more things experiments usually need: a statistically representative data set, a justification, and a clearly defined hypothesis. It lacks all of those.

Peer review isn't meant to eliminate all errors from scientific papers, it's simply intended to make life a little easier for readers by weeding out papers they are probably not interested in. So, if the hypothesis is that "lower cost journals have less stringent peer review", that doesn't require any testing: it's almost certainly true, but it doesn't matter to anybody. Publishing a bad paper in a peer reviewed journal doesn't hurt anybody, except maybe the reputation of the journal.

Re:Controls?

[uslurper]

Maybe the cost is not so rediculously high if you consider that serious journals pay for the staff and resources to actuially read what is being submitted.

Umm no

[Weezul]

Clicks are not the problem. Journals don't get any money from advertisement clicks. Real problem is :

At present, "Open Access Publishing" mostly means "Author Pays". If the author is your customer, then obviously you publish whatever they want. We must abandon the extortionate academic publishers like Elsevier all together by building an arXiv overlay filters that take over the journal's role of reviewing and declaring papers important. And these must be paid for by tax money because the customer should be society.

Just like with universities, Britain has rampant grade inflation because the students all pay 15k USD per year (9k GBP). St Andrews has a 98% graduation rate. A 98% graduation rate tells me the university did basically no "selection" on their admitted students, all selection occurred when an admissions person read their test scores from high school. In other words, the student is the customer and the product is a little piece of paper. This is why Britain sucks so bad at engineering and must create that blatantly bullshit ranking system by THES to make themselves look good.

In continental europe, almost everyone who finishes high school can attend university without paying, but the universities select students by failing out the shitty ones, well society is the customer and the students are the product. It's infinitely more fare because gaming the system in high school does nothing and people who never really hit their stride until the find challenging material do well.