Slashdot Mirror
Science Magazine "Sting Operation" Catches Predatory Journals In the Act
sciencehabit writes "A sting operation orchestrated by Science's contributing news correspondent John Bohannon exposes the dark side of open-access publishing. Bohannon created a spoof scientific report, authored by made-up researchers from institutions that don't actually exist, and submitted it to 304 peer-reviewed, open-access journals around the world. His hoax paper claimed that a particular molecule slowed the growth of cancer cells, and it was riddled with obvious errors and contradictions. Unfortunately, despite the paper's flaws, more open-access journals accepted it for publication (157) than rejected it (98). In fact, only 36 of the journals solicited responded with substantive comments that recognized the report's scientific problems. The article reveals a 'Wild West' landscape that's emerging in academic publishing, where journals and their editorial staffs aren't necessarily who or what they claim to be."
How many of the open access journals rely on click through advertising? Follow the money, I say.
work in progress
Seems like degree-mills are more common than actual universities by the same token.
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
Did he send the bogus articles to closed publishers too? How did the rates compare? I tried to RTFA, but didn't see anything about controls.
A lot of people cite the democratizing power of "open access" and "crowd sourcing". I feel this is an example of the same principle at work.
On one hand, it is easier for those that are not entrenched within the bastions of power to be heard, but on the other hand, all data received from these sources must be treated much more cautiously.
In the past "being published" was a big deal, as it required a fairly high bar of factual accuracy, and that is still the case of many prestigious journals, but in the rush to Twitter-ize research and accept as many publishable details as rapidly as possible in the name of profit and prestige, the barriers to entry have eroded.
In much the same way that hard investigative journalism with strong ethical guidelines, verifiable sources and solid editing will always have a place in my heart, these reputable journals can serve to establish a foundation of trust in the scientific arena. And now, in much the same way that one should treat any writing within the "blogosphere" as suspect until verified, many open access journals must now be treated with the same level of suspicion until it is proven otherwise that they hold themselves to a higher standard.
TLDR: Democratization is not always a good thing.
I've made comments before comparing science and religion, and too often people think that I'm a religious person trying to belittle a genuine quest for knowledge. On the contrary, I think the genuine quest for knowledge is an amazingly worthwhile thing. However, science has become a method for the "practitioners" and "priests" to exert social, economic, and institutional influence by swaying the beliefs of those who are not educated enough or informed enough to differentiate between genuine knowledge and blind dogma.
It's less that I'm a backwoods book-hating theist. It's that I've "lost the faith" and don't believe in what we call "science". We've gotten into muddy waters, studying soft sciences in ways that will never reach definitive answers, and allowed politics and media to have too much sway. We've gotten better at engineering, and worse at knowledge.
Go ahead. Mod me as flamebait.
For years we have known that there is a glut of graduates in the system. I remember my freshman year at university, the attitude of a lot of students was "the Masters is the new Bachelors, you have to have one to get an entry level job" or when I got closer to graduating it was "well I don't want to be done with school and my parents are helping me out so I'm going to go for my Masters". While education is awesome, the fact is that you don't have to be all that smart anymore to get a Masters or PhD.
Even as an undergrad I was pressured to publish. I didn't have the time nor the resources to do anything meaningful but my professors all said that I had to publish to even consider going to graduate school. They said that pretty much no matter what I do, even if its not novel or valuable to the academic community there will be a journal that will publish it. That's the current state of academics now.
Lets be clear: I'm not talking about MIT or Berkley. I'm talking about the thousands of research institutions across the country that while also doing amazing research, churn out Masters and PhDs like a printing mill. When you dilute the pool of researchers there is going to be subpar research. When there is a glut of subpar research there will be journals that see the business opportunity and publish anything you pay them to publish. This is not new.
Science has an axe to grind here, obviously, and this "experiment" is seriously biased.
It does not appear that it was submitted to any closed, for-profit journals (like Science). It would have been much more interesting to see how many of them would have accepted the paper.
I don't read your sig. Why are you reading mine?
And this is different from the average Science paper... how?
Actually, it's really really badly done.
To actually make any of the conclusions (or inferences) about the quality or rigor of open-access journals REQUIRES a control group of traditional journals to be operated on in a similar manner. In other words, there needs to be a sting on both open-access and traditional journals simultaneously.
Without that, no claims can be made. None. Not even one. Because we DO NOT KNOW how many traditional journals, like Science, would also have accepted their falsified paper(s). It's possible the traditional journals could have lower standards of quality and rigor than the open-access group.
Science and AAAS (of which I'm presently ashamed to be a member) should be blasted for publishing this tripe. It needs to be retracted, immediately. If they want to have the slightest shred of credibility here, they should at least conduct scientifically rigorous stings.
Disgusting.
The problem is that serious decisions are made by people who have no idea which journals are top quality. Bad tenure decisions, bad engineering choices, and god forbid bad medical decisions are being made daily on the basis of nothing more than "hey, the European Journal of Chemistry sounds legit."
I think he's building a railroad...?
Yes, but. This isn't entirely a binary scientific question. If the question were "are open-access journals worse than traditional journals?", you'd obviously need a control. But "Is the peer review process at open-access journals acceptable?" is not a scientific question, but one of values and personal preference. Most people would decide that a 50% failure rate is not acceptable, control or no control.
Now, we're all *very* curious to know whether traditional journals fare better than open ones, and Science is showing bias and intellectual dishonesty by avoiding that question, BUT that doesn't mean that this study has no value.
This article is being widely panned as lacking controls, published without any critical review, and driven by self-interest from a traditional publisher with the most to lose from Open Access taking off (as it is). Some have gone so far to assert it's an over-reach for how badly it was done, and will make Science as a journal look partisan.
For example, quick scan brought up these three scathing responses:
Mike Eisen (HHMI Berkeley Professor)
http://www.michaeleisen.org/blog/?p=1439
Peter Suber (Author of the book "Open Access", Director of the Harvard Open Access Project, Faculty Fellow at the Berkman Center)
https://plus.google.com/u/0/109377556796183035206/posts/CRHeCAtQqGq
Mike Taylor (programmer with Index Data and a research associate at the department of earth sciences, University of Bristol)
http://svpow.com/2013/10/03/john-bohannons-peer-review-sting-against-science/
I'm sure this will heat up some much needed debate about poor quality journals and the failings of peer review, but with the lack of any controls at all, it says basically nothing about open access as a model for publishing.
In Norway, we have a "level" system that is used in academia throughout the country. It is used for evaluating researchers and research groups when it comes to employment, tenure, funding etc. Your "point score" is summed up, 2 points for publication in a "level 2" journal, 1 point for "level 1".
A journal is either "level 2", "level 1" or "level 0". "level 2" is a selection of top journals from each field in science, 2000 in total (for all of science, from computational physics to the sociology of music). "level 1" means the remaining serious peer-reviewed journals. "level 0" either means "bullshit journal" or "journal that was founded just last year".
Researchers may nominate journals for a change in status, e.g. 2->1, 0->, etc. The decisions are made by a government-appointet body on a yearly basis. It's nowhere near perfect, but it's a lot better than nothing.
for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
It's more who is reviewing the material, and to what level, that matters.
A lot of high-grade peer reviewed journals, like Science and Nature, have been hoodwinked by researchers from South Korea and China, where cheating is more endemic than here.
Or, as most of us say, Wait Until The Second Journal Article.
-- Tigger warning: This post may contain tiggers! --
And if you are just writing them off and basing your reading on the "top ones", of what value are these?
While science journals are often used by researchers to find out what their colleagues are doing and can thus be vetted by the reader, they are quite often the bases for undergraduate and graduate educations, and putting deliberate crap in front of them is a Bad Thing.
In fact, over the years, Science has published numerous scientifically fraudulent papers, some of which were pretty blatant. So, in a sense, we already have a control. In addition to control experiments, it needs three more things experiments usually need: a statistically representative data set, a justification, and a clearly defined hypothesis. It lacks all of those.
Peer review isn't meant to eliminate all errors from scientific papers, it's simply intended to make life a little easier for readers by weeding out papers they are probably not interested in. So, if the hypothesis is that "lower cost journals have less stringent peer review", that doesn't require any testing: it's almost certainly true, but it doesn't matter to anybody. Publishing a bad paper in a peer reviewed journal doesn't hurt anybody, except maybe the reputation of the journal.
... the real probem is that as problem size increases, the human brain just can't deal with all the stress and energy one must expend to fact check everything. This is why we need automation in checking papers for errors and contradictions, i.e. the number of facts you need to know grows exponentially as things get more complex. What we're really seeing is that the human brain is the biggest bottleneck since human beings have limited time and energy. So no one should be surprised it's easy to 'dupe' or game a system because the resources you need to stop untrustworthy people is unrealistically expensive. Any area of human endeavor is only as good as the people themselves.
I've never heard of this person John Bohannon.
John Bohannon is a biologist, science journalist, and dancer based at Harvard University. He writes for Science Magazine, Discover Magazine, and Wired Magazine, and frequently reports on the intersections of science and war. After embedding in southern Afghanistan in 2010, he was the first journalist to convince the US military to voluntarily release civilian casualty data. He received a Reuters environmental journalism award in 2006 for his reporting on collaborative research in Gaza. He was also involved in some controversy over an article he wrote critiquing the Lancet survey of Iraq War Casualties.
At Science Magazine, Bohannon also adopts the ''Gonzo Scientist'' persona, where he ''takes a look at the intersections among science, culture, and art -- and, in true gonzo style, doesn't shrink from making himself a part of the story. The stories include original art and accompanying multimedia features.'' As the Gonzo Scientist, Bohannon's research on whether humans can tell the difference between pate and dog food led to Stephen Colbert eating cat food on the Colbert Report.
Bohannon is probably best known for creating the Dance Your PhD competition, in which scientists from all around the world interpret their doctoral dissertations in dance form. Slate Magazine ran a profile on Bohannon and the competition in 2011. He performed with the Black Label Movement dance troupe at TEDx Brussels in November 2011, where he satirized Jonathan Swift's A Modest Proposal by modestly proposing that Powerpoint software be replaced by live dancers. Bohannon then went on to perform with Black Label Movement at TED 2012 in Long Beach.
Advisory Board - John Bohannon
While visiting the Harvard University Program in Ethics and Health, he is working on two areas of research: 1) torture --- in particular the complicity of scientific and medical workers in torture, and 2) ethical problems involved with obtaining global health data, stemming from his journalistic coverage of the controversial attempts to estimate the health and mortality of the Iraqi population since the US-led invasion.
After completing a Ph.D. in molecular biology at the University of Oxford in 2002, John focused on bioethics as a Fulbright fellow (2003 --- 2004) in Berlin.
Here's the problem with doing that so systemically: it is fundamentally anticompetitive, and leads to stagnation. Nobody would bother submitting to a "level 0" journal because it won't earn them any props at all, which means that the journal can never become anything more than a "level 0" journal. This means that you don't get fresh blood with new ideas on the review boards, so progress moves at a snail's pace. There's something to be said for disruptive innovation, even in academic publishing circles.
Also, the entire notion of judging the value of your scientific contribution based on what journal agreed to publish it is as absurd as judging the value of a car based on what dealer sold it. A paper should stand or fall on its own merits. A good article that pushes science forward, even if published in a minor journal, should weigh significantly in your favor for tenure, and a lousy article, even if published in a major journal, should not. A system that does the opposite is abject stupidity, pure and simple.
Check out my sci-fi/humor trilogy at PatriotsBooks.
In all seriousness, I - as a researcher myself - understand the need of easy access to publications. However, I never supported the open access models that came into existence and are being built and pursued today. Why? Because it's all about the money and a lot of such journals absolutely do not care about quality, or about having big name editors who'd perform very thorough revision of reviews and make proper decisions about paper acceptances. Big journals have good editorial and review staff, and they simply can't allow them to be bad and irresponsible, because they actually care about their reputation and credibility. New breed open access journals on the other hand only care about revenue.
The instititue I work at has mandated open access publication as well as others did, however, they did not provide funding for us to actually publish open access versions at big name journals, so we try to play the system whenever we can, and publish in traditional journals with traditional publication schemes. I do not care about some politician-flavored scientists' (most of them not even publishing) dreams about some utopistic open access world. I care about publications appearing in credible journals, reviewed by credible people, producing quality publications - even if they are only attainable for money.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
A paper should stand or fall on its own merits.
Yep, and quality papers is what they should be competing for. The journal ranking systems used by univisities (not just in Norway) are designed to give more weight to journals that have a long track record of doing that. This is why the Nature and Science journals at at the top of the list, their long publishing history and track record of quality papers speaks for itself. A low ranked journal will stay a low ranked until it's track record is such that it can be deemed a reliable source. If it does nothing to improve it's record then it follows it will never be respected.
A good article that pushes science forward, even if published in a minor journal, should weigh significantly in your favor for tenure, and a lousy article, even if published in a major journal, should not.
The "impact factor" of individual papers is generally weighed by the number of citations, not the name of the journal.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
Science and AAAS (of which I'm presently ashamed to be a member) should be blasted for publishing this tripe. It needs to be retracted, immediately. If they want to have the slightest shred of credibility here, they should at least conduct scientifically rigorous stings.
I also doubt that they adhered to their own guidelines for Human research studies:
Informed consent must be obtained for studies on humans after the nature and possible consequences of the studies were explained. All research on humans must have IRB approval.
OS Reviews: Free and Open Source Software
Clicks are not the problem. Journals don't get any money from advertisement clicks. Real problem is :
At present, "Open Access Publishing" mostly means "Author Pays". If the author is your customer, then obviously you publish whatever they want. We must abandon the extortionate academic publishers like Elsevier all together by building an arXiv overlay filters that take over the journal's role of reviewing and declaring papers important. And these must be paid for by tax money because the customer should be society.
Just like with universities, Britain has rampant grade inflation because the students all pay 15k USD per year (9k GBP). St Andrews has a 98% graduation rate. A 98% graduation rate tells me the university did basically no "selection" on their admitted students, all selection occurred when an admissions person read their test scores from high school. In other words, the student is the customer and the product is a little piece of paper. This is why Britain sucks so bad at engineering and must create that blatantly bullshit ranking system by THES to make themselves look good.
In continental europe, almost everyone who finishes high school can attend university without paying, but the universities select students by failing out the shitty ones, well society is the customer and the students are the product. It's infinitely more fare because gaming the system in high school does nothing and people who never really hit their stride until the find challenging material do well.
The Christian religion has been and still is the principal enemy of moral progress in the world. -- Bertrand Russell