Peer Review Ring Broken - 60 Articles Retracted

blackbeak (1227080) writes The Washington Post reports that the Journal of Vibration and Control's review system was hijacked by a ring of reviewers. 60 articles have been retracted as a result. "After a 14-month investigation, JVC determined the ring involved “aliases” and fake e-mail addresses of reviewers — up to 130 of them — in an apparently successful effort to get friendly reviews of submissions and as many articles published as possible by Chen and his friends.'On at least one occasion, the author Peter Chen reviewed his own paper under one of the aliases he created,' according to the SAGE announcement."

Comments

The Good News?

[KibibyteBrain]

Peter holds a very high standard for himself, I'm sure.

Re:The Good News?

[Cryacin]

It's just the new strategy employed to increase the speed of scientific research and development. It's called the self-peer-review.

Amazingly articles can get released on the same day as submission with this method.

Re:The Good News?

[m00sh]

Peter holds a very high standard for himself, I'm sure.

The standard practice is to form a unspoken agreement between several reviewers that they will all favorably review each others papers.

Peter couldn't find his circle and created a self-circle.

Wish I could say I was surprised

[Crashmarik]

We live in a day and age where you can make a pretty decent living as a scientist without actually advancing science, or doing very much technologically related labor, only natural people would game the system. While science should be immune to this sort of thing, just how many unimportant not particularly interesting results do people actually try to reproduce ?

Re:Wish I could say I was surprised

[Anonymous Coward]

You have it backwards. the fault is not that not every scientist has a breakthrough.

the fault is that in academia its pretty much "publish or die". The incentive to publish over anything else pushes the unscrupulous to do things like this.

the system itself creates this sort of situation.

Re:Wish I could say I was surprised

[antifoidulus]

No, "publish or perish" really dis-incentivizes novel research because guess what, often times really novel research fails. All "publish or perish" really does is incentivize either cheating or the lowest risk research imaginable. There are other mechanisms for making sure a researcher is actually doing their work, punishing them for taking risks shouldn't be among them.

Re:Wish I could say I was surprised

[Karmashock]

If you pay scientists to do science and they are contracted to do it... they fraudulently do not do science yet continue to cash your checks... that is a crime.

Re:Wish I could say I was surprised

[Anonymous Coward]

Scientifically useful negative results don't merely fail peer review, they are simply unpublishable in a major journal.

Re:Wish I could say I was surprised

[alvinrod]

It's not a matter of failing peer review, it's a general disinterest in publishing negative results. If you find a cure for cancer it's a big deal, but if you just found one more thing that doesn't work any better than a sugar pill, none of the journals are going to care about publishing it even if it's the most well-run study in the history of the world.

If someone starts doing some novel research that's going to take five years to possibly produce results and nothing pans out, they aren't going to get anyone to publish the findings.

Re:Wish I could say I was surprised

[Anonymous Coward]

1. It is reasonable for scientists in the pay of the public to be required at intervals to publish the results or at least what they were currently doing over the past few months or year or whatever interval is deemed reasonable.

The basic problem here is that you seem to have at best a shaky grasp of what "publish" means in academia. No journal is going to just publish "what they were currently doing over the past few months" unless that includes a significant result. I'm honestly not sure of where to recommend that you go to get a better understanding of how academic journals work, but I suppose the Wiki article couldn't be the worst place to start.

2. Works thus published should be subjected to reasonable audits to detect fraud, laziness, waste, or incompetence.

Using what money? Peer review currently is done on a voluntary basis; no journal that I'm aware of pays its reviewers. You seem to think that just because it would be nice if all published research was reproduced that it *should* be reproduced, without concern for where the researcher-time and money will come from to accomplish this. The current reality is that any professor or research scientist who devoted significant amounts of time to reproducing already-published science would quickly find himself out on his ass, because publishing *original* research is the first and foremost factor in maintaining/advancing a career in academia. To put it in software development terms, it would be like expecting a programmer to spend a large chunk of his time on the clock refactoring code while his bosses are telling him to leave it alone and work on implementing new features.

3. The nature of audits should make it difficult or impossible for conflicts of interest to corrupt the auditing process.

Agreed, the journal from this article should absolutely have done a better job of verifying the identities of its peer reviewers.

4. The auditing process should be sufficient to determine what is and is not valid science.

In what sense? If someone publishes a paper based on years of astronomical observations, is the peer reviewer obligated to spend years making his own observations to see if he can find (more or less) the same result? If "Yes", then the simple reality is that no one will volunteer to peer review such work, and you'll end up in a situation worse than the present one. If "No", then you're back to admitting that at some point the reviewer has to trust the article's author(s).

5. Reproduction of work obviously cannot be done with all papers however, they should be done with all significant work deemed significant.

I would argue that, certainly in my own field, "significant results" intrinsically draw more attention once published, and thus any mistake or malfeasance is more likely to be caught.

6. The deeming of significant or insignificant work could be down to collective or crowd sourced choices made by other scientists to cite a given work or say they found it interesting or significant. When X number of scientists say its significant then someone in the community should be tasked with verifying it through reproduction.

And, again, whence comes the money and time to reproduce it? This is a point that most scientists, I believe, would agree with -- but no one is going to sacrifice their own career to help accomplish it. And the "publish or perish" mentality contributes to this problem, as journals do not publish articles which simply say "yes, this other article seems to be good science".

Overall, I don't see anything too objectionable in what you want -- but it is basically a list of demands without any suggestion of how they could be accomplished or any understanding of why they are *not* being accomplished (to the extent that they aren't) in the current syst

Re:Wish I could say I was surprised

[Maow]

1. I'm not interested in being brow beaten by some fool more interested in winning an argument then in addressing the argument.

If you're going to keep attempting an ad hominem then I'm going to simply not talk to you. And then what will you have accomplished? ...you're going to get asinine...

Jeez, pot meet kettle.

To top it off, he addressed your points quite well and it appears that it's you that seems intent upon winning an argument with your long-winded reply, which, of course, doesn't specifically and concretely address the issues raised by the person you're replying to.

Funding to reproduce coming from same institution? So they'll have half the money for original research then. And the suckers tasked with the reproduction won't be advancing their own careers under the Publish (original, ground breaking work) Or Perish model used today.

Like it was stated, in a fairly appropriate analogy, reproducing others' work is akin to re-writing a new software project - in software dev, it's a losing game.

In science it's important, but like in software dev, the boss isn't interested. And while the result may be beneficial, it's hard to convince people that it's a rewarding career move to play catch-up to others' work.

Having said all that, I think we all agree that reproducibility is important -- question is, how to go about it as the current system kinda disfavours it in all but the most important projects.

We need to implement specific, concrete changes -- having grad students do some of that is a good idea, but not sure if it'll completely solve the issue.

But laymen will at least understand what has and has not be verified. That is important. Science cannot be something only scientists understand any more then the law can be something only lawyers understand.

Laymen will never understand cutting edge science (unless they're quite keen on the topic at hand - a miniscule minority), and any layman that thinks they understand the law as well as lawyers generally get their arses handed to them should they attempt pro se representation.

Specialization in complex fields is natural.

Re:Wish I could say I was surprised

[Rich0]

And this is part of why all the drug development work ends up happening in private industry.

A scientist will come up with a molecule that inhibits some enzyme and get some publishable result. At that point they issue the typical "possible cure for cancer" press release and move on to the next thing. 5 years and $10M later a pharma company figures out that it causes heart valve degeneration or that inhibiting the enzyme isn't the magic bullet everybody hoped for. They don't bother publishing it, but none of their scientists get paid by the publication anyway. The companies interest is that if it eventually works out they make billions.

So, in that sense you actually have an example of a way in which industrial research is actually less risk-averse than academia, which should be shocking.

That said, when it comes to the basic research side of things pharma companies do tend to let the academics do the work for them.

Re:Wish I could say I was surprised

[Rockoon]

I was quoting someone else... please correct them instead... *yawn*

So you plagiarize too?

Re:Wish I could say I was surprised

[Anonymous Coward]

I think what the poster you quoted wanted to say is that often to make major contributions you have to do something that has never been done before, and not just follow up on previous research. Pushing on current trends is not difficult, at all, and is basically guaranteed to get you a publication in a decent journal. A lab head can do several dozens of these papers a year if he has a few handfuls of people in his group and decides to have his focus on this. Now doing this more than guarantees a comfortable living as an academic. Quite often this research can even be wrong: It's middling at best, and nobody really cares, so nobody will notice (and yes, the STAP scandal with Obokata et al. is really a special case and not what usually happens; their claims were "too" interesting to the general public and in any case outright fraudulent). Many fields are saturated by this type of managers (rather than scientists), and they have been rewarded for their noninnovative research for so long I doubt they would even recognize a basic flaw in a paper when they saw one.

This means that peer review has become useless (when your peers are managers rather than scientists) and in fact every month I spot papers from my field in some of the top journals that have zero scientific contribution: their methods are only borderline correct, and the conclusions known for decades. But they have nice pictures and peer reviewers are probably their manager-friends or manager-somebodyelses who did not have a clue what was done and well-known 50 years ago (and indeed, why care, if by ignoring old research you can accidentally redo them and get more papers!). Try publishing a paper showing that their experiment must be wrong as it violates the second law of thermodynamics and you will be shot down and now they know your name. Good luck with grants and peer reviews.

I got a bit derailed above, but no, I am not bitter nor is the above a completely accurate presentation of my personal experience. This said, it is obvious that many scientists are afraid of speaking their mind and criticizing others even when others are wrong, and that this is corrupting the entire system where one is supposed to be able to trust one's peers.

Back to the topic: Coming up with a totally new idea, trying it, and failing at it will never get written up. You say that this is the right thing to do, if you don't publish, you ought to perish. Now is failed research "wrong"? Should you have known beforehand that your idea is stupid and not even test it? Not being able to publish this failed idea and only regarding publications as a measure of your success would certainly imply this.

Hack a Day publishes fails of the week. They are not meant as articles where we laugh at someone's stupidity or bad luck, but are informative writeups about new ideas where something in the implementation went wrong, or serve as examples of how even experienced people can fail to consider some basic (or advanced) principles. Related to this, perhaps my favourite TED talk is that by Eddie Obeng. He talks about business, not research. And I remind you that the only reason university research exists is that otherwise fundamental or high risk projects would not get funded as you might not have a direct way to make money off of them, or you might lose a lot, which makes them unattractive for business. Surely Obeng will then tell you that as a business manager, do the safe projects, punish those whose ideas don't work. Well this is what he says: "You're doing something new that nobody's done before, you get it completely wrong. How should you be treated? Well, free pizzas! You should be treated better than the people who succeed. It's called smart failure. Why? Because you can't put it on your CV." Companies can treat their employees with pizzas when they fail at something new, but academia is not a structured system where you could get different kinds of rewards: it is only about publish or perish. This is why it is a horrible system.

Re:Wish I could say I was surprised

[Anonymous Coward]

I forgot to add this recent article to my post. It goes to show that the problems I am talking about are not just my personal anecdotes or limited to my field.

I remember Journal of Vibration and Control!

[RevWaldo]

That was one high class bondage mag, right up there with Bizzare and Exotique.

I don't think "peer review" means what WaPo thinks it means...

.

Web of Trust

[Dr_Barnowl]

People should cryptographically sign peer reviews (and their papers). And journals should only trust signing keys that themselves have been signed by respected experts. The more respected you get, the more signatures your keys and papers get.

Not surprising

[teakillsnoopy]

I've been proofreading engineering/medical papers for universities in Taiwan for over 7 years and this is not surprising in the least. There is almost no stigma regarding plagiarism in this region (I've done work for Malaysian, Vietnamese, Indonesian, etc. authors). When I alert an author about copy/pasted text, their reaction is one you would get if you told someone that their reference format needs to be change. "Oh, ok. I guess I'll change it.". The universities here never seriously investigate plagiarism because all the big fish at the top did it themselves to get to the top.