Peer Review Ring Broken - 60 Articles Retracted

blackbeak (1227080) writes The Washington Post reports that the Journal of Vibration and Control's review system was hijacked by a ring of reviewers. 60 articles have been retracted as a result. "After a 14-month investigation, JVC determined the ring involved “aliases” and fake e-mail addresses of reviewers — up to 130 of them — in an apparently successful effort to get friendly reviews of submissions and as many articles published as possible by Chen and his friends.'On at least one occasion, the author Peter Chen reviewed his own paper under one of the aliases he created,' according to the SAGE announcement."

Comments

The Good News?

[KibibyteBrain]

Peter holds a very high standard for himself, I'm sure.

Re:The Good News?

[Cryacin]

It's just the new strategy employed to increase the speed of scientific research and development. It's called the self-peer-review.

Amazingly articles can get released on the same day as submission with this method.

Re:The Good News?

[marcello_dl]

Besides, who's more peer with respect to the author than the author itself? I tell you, it can't get more peery than this.

Re:The Good News?

[m00sh]

Peter holds a very high standard for himself, I'm sure.

The standard practice is to form a unspoken agreement between several reviewers that they will all favorably review each others papers.

Peter couldn't find his circle and created a self-circle.

Re:The Good News?

[thunderclap]

Peter holds a very high standard for himself, I'm sure.

The standard practice is to form a unspoken agreement between several reviewers that they will all favorably review each others papers.

Peter couldn't find his circle and created a self-circle.

Otherwise known as a circle jerk.

Wish I could say I was surprised

[Crashmarik]

We live in a day and age where you can make a pretty decent living as a scientist without actually advancing science, or doing very much technologically related labor, only natural people would game the system. While science should be immune to this sort of thing, just how many unimportant not particularly interesting results do people actually try to reproduce ?

Re:Wish I could say I was surprised

[Anonymous Coward]

You have it backwards. the fault is not that not every scientist has a breakthrough.

the fault is that in academia its pretty much "publish or die". The incentive to publish over anything else pushes the unscrupulous to do things like this.

the system itself creates this sort of situation.

Re:Wish I could say I was surprised

[Karmashock]

Wrong. The issue is that publishing is considered sufficient.

It should be publish or die. How do you know they're doing anything if they don't publish? they could be watching tv all day for all you know otherwise.

But as is made clear here, simply publishing and getting it through peer review is clearly not good enough. We need to increase what they have to do to avoid this situation.

For example... maybe one scientist pays another scientist to reproduce his work.

Maybe you have big collections of graduate students that as part of their process of getting a degree get assigned some random papers submitted by scientists in their field and they have to reproduce the work.

Obviously this isn't always possible... but whenever it isn't possible that needs to be put as a giant red asterisk on the paper saying "this work has not been reproduced"...

Do that and you're not going to get as much fraud or laziness.

Re:Wish I could say I was surprised

[antifoidulus]

No, "publish or perish" really dis-incentivizes novel research because guess what, often times really novel research fails. All "publish or perish" really does is incentivize either cheating or the lowest risk research imaginable. There are other mechanisms for making sure a researcher is actually doing their work, punishing them for taking risks shouldn't be among them.

Re:Wish I could say I was surprised

[Karmashock]

If you pay scientists to do science and they are contracted to do it... they fraudulently do not do science yet continue to cash your checks... that is a crime.

Re:Wish I could say I was surprised

[Karmashock]

Okay... I am going to assume you're reasonable and not a loony... and that what we have here is a failure to communicate.

Here is what I want... please throw out your existing notions of what is currently going on or whatever talking point score card you're reading from here...

1. It is reasonable for scientists in the pay of the public to be required at intervals to publish the results or at least what they were currently doing over the past few months or year or whatever interval is deemed reasonable.

2. Works thus published should be subjected to reasonable audits to detect fraud, laziness, waste, or incompetence.

3. The nature of audits should make it difficult or impossible for conflicts of interest to corrupt the auditing process.

4. The auditing process should be sufficient to determine what is and is not valid science.

5. Reproduction of work obviously cannot be done with all papers however, they should be done with all significant work deemed significant.

6. The deeming of significant or insignificant work could be down to collective or crowd sourced choices made by other scientists to cite a given work or say they found it interesting or significant. When X number of scientists say its significant then someone in the community should be tasked with verifying it through reproduction.

Do you have a problem with any of the above?

I do not claim my notions above are perfect and am open to modification. However, the basic gist of my post I think is defensible and if challenged, I will defend it.

Re:Wish I could say I was surprised

[Anonymous Coward]

Scientifically useful negative results don't merely fail peer review, they are simply unpublishable in a major journal.

Re:Wish I could say I was surprised

[AK Marc]

If I proved that cats claws grow proportionally to zinc in their system (assuming otherwise healthy), up to zinc overdoses, it may be valid, presumably interesting, but I doubt that it would be picked up by a "major" paper. But if I proved that human IQ was proportional to the temperature of the blankets one slept under, that would be published many places and gain me much fame. Even if the results were faked and peers paid off.

The solution would seem to be the university publishing all the unpublished articles, peer reviewed by others in the same system. They'd be published, and if they can't pass a peer review, that should come out.

Re:Wish I could say I was surprised

[alvinrod]

It's not a matter of failing peer review, it's a general disinterest in publishing negative results. If you find a cure for cancer it's a big deal, but if you just found one more thing that doesn't work any better than a sugar pill, none of the journals are going to care about publishing it even if it's the most well-run study in the history of the world.

If someone starts doing some novel research that's going to take five years to possibly produce results and nothing pans out, they aren't going to get anyone to publish the findings.

Re:Wish I could say I was surprised

[Anonymous Coward]

1. It is reasonable for scientists in the pay of the public to be required at intervals to publish the results or at least what they were currently doing over the past few months or year or whatever interval is deemed reasonable.

The basic problem here is that you seem to have at best a shaky grasp of what "publish" means in academia. No journal is going to just publish "what they were currently doing over the past few months" unless that includes a significant result. I'm honestly not sure of where to recommend that you go to get a better understanding of how academic journals work, but I suppose the Wiki article couldn't be the worst place to start.

2. Works thus published should be subjected to reasonable audits to detect fraud, laziness, waste, or incompetence.

Using what money? Peer review currently is done on a voluntary basis; no journal that I'm aware of pays its reviewers. You seem to think that just because it would be nice if all published research was reproduced that it *should* be reproduced, without concern for where the researcher-time and money will come from to accomplish this. The current reality is that any professor or research scientist who devoted significant amounts of time to reproducing already-published science would quickly find himself out on his ass, because publishing *original* research is the first and foremost factor in maintaining/advancing a career in academia. To put it in software development terms, it would be like expecting a programmer to spend a large chunk of his time on the clock refactoring code while his bosses are telling him to leave it alone and work on implementing new features.

3. The nature of audits should make it difficult or impossible for conflicts of interest to corrupt the auditing process.

Agreed, the journal from this article should absolutely have done a better job of verifying the identities of its peer reviewers.

4. The auditing process should be sufficient to determine what is and is not valid science.

In what sense? If someone publishes a paper based on years of astronomical observations, is the peer reviewer obligated to spend years making his own observations to see if he can find (more or less) the same result? If "Yes", then the simple reality is that no one will volunteer to peer review such work, and you'll end up in a situation worse than the present one. If "No", then you're back to admitting that at some point the reviewer has to trust the article's author(s).

5. Reproduction of work obviously cannot be done with all papers however, they should be done with all significant work deemed significant.

I would argue that, certainly in my own field, "significant results" intrinsically draw more attention once published, and thus any mistake or malfeasance is more likely to be caught.

6. The deeming of significant or insignificant work could be down to collective or crowd sourced choices made by other scientists to cite a given work or say they found it interesting or significant. When X number of scientists say its significant then someone in the community should be tasked with verifying it through reproduction.

And, again, whence comes the money and time to reproduce it? This is a point that most scientists, I believe, would agree with -- but no one is going to sacrifice their own career to help accomplish it. And the "publish or perish" mentality contributes to this problem, as journals do not publish articles which simply say "yes, this other article seems to be good science".

Overall, I don't see anything too objectionable in what you want -- but it is basically a list of demands without any suggestion of how they could be accomplished or any understanding of why they are *not* being accomplished (to the extent that they aren't) in the current syst

Re:Wish I could say I was surprised

[Maow]

1. I'm not interested in being brow beaten by some fool more interested in winning an argument then in addressing the argument.

If you're going to keep attempting an ad hominem then I'm going to simply not talk to you. And then what will you have accomplished? ...you're going to get asinine...

Jeez, pot meet kettle.

To top it off, he addressed your points quite well and it appears that it's you that seems intent upon winning an argument with your long-winded reply, which, of course, doesn't specifically and concretely address the issues raised by the person you're replying to.

Funding to reproduce coming from same institution? So they'll have half the money for original research then. And the suckers tasked with the reproduction won't be advancing their own careers under the Publish (original, ground breaking work) Or Perish model used today.

Like it was stated, in a fairly appropriate analogy, reproducing others' work is akin to re-writing a new software project - in software dev, it's a losing game.

In science it's important, but like in software dev, the boss isn't interested. And while the result may be beneficial, it's hard to convince people that it's a rewarding career move to play catch-up to others' work.

Having said all that, I think we all agree that reproducibility is important -- question is, how to go about it as the current system kinda disfavours it in all but the most important projects.

We need to implement specific, concrete changes -- having grad students do some of that is a good idea, but not sure if it'll completely solve the issue.

But laymen will at least understand what has and has not be verified. That is important. Science cannot be something only scientists understand any more then the law can be something only lawyers understand.

Laymen will never understand cutting edge science (unless they're quite keen on the topic at hand - a miniscule minority), and any layman that thinks they understand the law as well as lawyers generally get their arses handed to them should they attempt pro se representation.

Specialization in complex fields is natural.

Re:Wish I could say I was surprised

[Rich0]

And this is part of why all the drug development work ends up happening in private industry.

A scientist will come up with a molecule that inhibits some enzyme and get some publishable result. At that point they issue the typical "possible cure for cancer" press release and move on to the next thing. 5 years and $10M later a pharma company figures out that it causes heart valve degeneration or that inhibiting the enzyme isn't the magic bullet everybody hoped for. They don't bother publishing it, but none of their scientists get paid by the publication anyway. The companies interest is that if it eventually works out they make billions.

So, in that sense you actually have an example of a way in which industrial research is actually less risk-averse than academia, which should be shocking.

That said, when it comes to the basic research side of things pharma companies do tend to let the academics do the work for them.

Re:Wish I could say I was surprised

[Rockoon]

I was quoting someone else... please correct them instead... *yawn*

So you plagiarize too?

Re:Wish I could say I was surprised

[PvtVoid]

.It is reasonable for scientists in the pay of the public to be required at intervals to publish tts or at least what they were currently doing over the past few months or year or whatever interval is deemed reasonable.

Dear NEJM: For the last six months, I mostly sat around in my office, read Slashdot, and mined Bitcoin on the cluster.

Re:Wish I could say I was surprised

[serviscope_minor]

Well, when it comes to doing browbeating, you're doing a bang up job. You are assuming that your solution is right and will not accept any criticism of it no matter how much of a bad idea it is. Your solution is unfortunately unworkable.

Is the current system perfect? No, not even slightly? Is your solution actually a solution? No, yours is a cure worse than the disease, or at least unworkable.

The point is that money goes to scientists and the people that provide that money have a right to expect something be done with it.

That's what publish or perish is! The money is largely provided by the public and not unreasonably, they want to see that the money is being used. Spending that money going in circles replicating results for the sake of it doesn't yield much of use, sadly. So, you'll have to convince people that it's a good thing that now (say) half the amount of research is getting done.

Furthermore, they must share information... you don't like the term publish or you're going to get asinine on the issue? fine... We'll start using other words. I'll speak chinese if I need to get you to stop trying to make this a semantics debate.

The point about publishing is that as soon as you have a result worth sharing, you share it. In fact if anything, publish or perish encourages that. The opposite: dumping out stuff because there's a reporting deadline does actually happen. For example any EC funded projects of which there are many hav mandatory reporting deliverables. I think a few of the RCUK bunch do as well.

Guess what? No one ever reads the damn things, mostly because any results worth sharing are published as papers. So in fact what you advocate does happen and is demonstrably useless. You can in fact go and request copies of these documents if you wish. I believe they are a matter of public record.

By all means... put out as much research as you want that no one could possibly verify or reproduce. Make my fucking day. But it gets the red asterisk.

I don't see what function that would serve. All research is already considered to have a big red asterisk by default. You might weigh the liklihood of correctness by the content of the paper, the believability of the result and the track record of the researchers, but new publications are generally taken to be unverified.

So your red asterisk would not serve any purpose.

You claim you want to make it easier for "laymen" but the red asterisk won't help, because it's already effectively there. The best thing you can do is stump up the money to educate them instead.

6. As to the money to reproduce it, that can be provided by the same institutions that hire the scientists in the first place as part of their quality control policy. Which is in large part what all of this in the first place.

And where does that money come from? Who wants to double the research costs of universities just to do what's already being done by a less formal method? Whether it's privately or publicly run, the people stumping up the hard cash are going to want to know why the output has halved or the cost has doubled.

Would that money go to the same scientist or the same type of scientists? Probably not. We might have specialists that ONLY reproduce other people's work. That might be literally all they do. And they might be paid by the scientists that produced the paper who are themselves taking the money from their grants or working budgets as a cost of publishing.

Well that would shoot a huge amount of research in the foot. I've published a fair chunk. All of my most major publications were done either on the side with no budget at all or in a small research group that had enough money for 3 PhD students (and this was way back when we were 10k per year) and almost no equipment (seriously we were on hand-me-down computers that the better funded groups junked due to obsolescence).

Come to think of it quite a lot of people I know got their big breaks by looki

Re:Wish I could say I was surprised

[TheRaven64]

In the UK, university research departments are assessed base on the Research Excellence Framework (REF, formerly the Research Assessment Exercise [RAE]). Each faculty member is required to submit 4 things demonstrating impact. These are typically top-tier conference or journal papers, but can also be artefacts or examples of successful technology transfer. The exercise happens every four years, so to get the top ranking you need to write one good paper a year. The only incentive for publishing in second-tier venues is meeting other people who might lead to interesting collaborations.

Re:Wish I could say I was surprised

[Anonymous Coward]

I think what the poster you quoted wanted to say is that often to make major contributions you have to do something that has never been done before, and not just follow up on previous research. Pushing on current trends is not difficult, at all, and is basically guaranteed to get you a publication in a decent journal. A lab head can do several dozens of these papers a year if he has a few handfuls of people in his group and decides to have his focus on this. Now doing this more than guarantees a comfortable living as an academic. Quite often this research can even be wrong: It's middling at best, and nobody really cares, so nobody will notice (and yes, the STAP scandal with Obokata et al. is really a special case and not what usually happens; their claims were "too" interesting to the general public and in any case outright fraudulent). Many fields are saturated by this type of managers (rather than scientists), and they have been rewarded for their noninnovative research for so long I doubt they would even recognize a basic flaw in a paper when they saw one.

This means that peer review has become useless (when your peers are managers rather than scientists) and in fact every month I spot papers from my field in some of the top journals that have zero scientific contribution: their methods are only borderline correct, and the conclusions known for decades. But they have nice pictures and peer reviewers are probably their manager-friends or manager-somebodyelses who did not have a clue what was done and well-known 50 years ago (and indeed, why care, if by ignoring old research you can accidentally redo them and get more papers!). Try publishing a paper showing that their experiment must be wrong as it violates the second law of thermodynamics and you will be shot down and now they know your name. Good luck with grants and peer reviews.

I got a bit derailed above, but no, I am not bitter nor is the above a completely accurate presentation of my personal experience. This said, it is obvious that many scientists are afraid of speaking their mind and criticizing others even when others are wrong, and that this is corrupting the entire system where one is supposed to be able to trust one's peers.

Back to the topic: Coming up with a totally new idea, trying it, and failing at it will never get written up. You say that this is the right thing to do, if you don't publish, you ought to perish. Now is failed research "wrong"? Should you have known beforehand that your idea is stupid and not even test it? Not being able to publish this failed idea and only regarding publications as a measure of your success would certainly imply this.

Hack a Day publishes fails of the week. They are not meant as articles where we laugh at someone's stupidity or bad luck, but are informative writeups about new ideas where something in the implementation went wrong, or serve as examples of how even experienced people can fail to consider some basic (or advanced) principles. Related to this, perhaps my favourite TED talk is that by Eddie Obeng. He talks about business, not research. And I remind you that the only reason university research exists is that otherwise fundamental or high risk projects would not get funded as you might not have a direct way to make money off of them, or you might lose a lot, which makes them unattractive for business. Surely Obeng will then tell you that as a business manager, do the safe projects, punish those whose ideas don't work. Well this is what he says: "You're doing something new that nobody's done before, you get it completely wrong. How should you be treated? Well, free pizzas! You should be treated better than the people who succeed. It's called smart failure. Why? Because you can't put it on your CV." Companies can treat their employees with pizzas when they fail at something new, but academia is not a structured system where you could get different kinds of rewards: it is only about publish or perish. This is why it is a horrible system.

Re:Wish I could say I was surprised

[Anonymous Coward]

I forgot to add this recent article to my post. It goes to show that the problems I am talking about are not just my personal anecdotes or limited to my field.

Re:Wish I could say I was surprised

[Ol Olsoc]

Is something special about science?

If you pay someone to do a job and they don't do it... you fire them. That's it, no worrying about crime and so on. If they're really contracting, then at worst it's a breach of contract and you could sue to get your money back.

Agreed. One small point to add though is that the people they are working for put a lot of pressure on them to publish.

This is just stage one of a process. It's been known that the "publish or perish" culture has produced some problems. Self plagiarism is a biggie, but they are weeding out the fraud at the same time.

So while the more politically inclined might wring their hands and moan "we have to do something!" this is all happening because we are doing something The system is working. Tools are in place now to catch the bad boys and girls. Soon they'll know they can't plagiarize themselves, come up with bogus peer review, or doctor the data. It's good, what is happening.

It's a science solution. If it were a political solution, the fraudsters would just end up winning, and would become more powerful.

Re:Wish I could say I was surprised

[golden age villain]

For example... maybe one scientist pays another scientist to reproduce his work. Maybe you have big collections of graduate students that as part of their process of getting a degree get assigned some random papers submitted by scientists in their field and they have to reproduce the work.

You don't work in science do you? Being paid for reproducing someone else's work means you are not producing anything original of your own. It doesn't advance your career. Then with respect to your second point, being a graduate student means to perform original research. If your PhD is about reproducing someone else's work, you won't be able to publish anything of significance.

The problem is the system globally: journals, which push for high impact sexy stories; promotion committees, which only look at how many high impact papers scientists have published and at how much funding they have attracted; and finally funding bodies, which only look at publications. If you are not lucky enough to get into a big lab which automatically publishes in high impact journals based on the labhead's reputation, the incentive to game the system is high. You just need to look at all the scandals that have come to light in the last years. You can even buy authorship on papers to which you have contributed nothing (http://www.sciencemag.org/content/342/6162/1035.summary).

Re:Wish I could say I was surprised

[the gnat]

Alternatively (or in addition), we could increase the penalties for those caught cheating.

FYI, cheating like this is already a guaranteed career-ender. People who do things like this aren't rationally weighing the cost of getting caught against the career advancement that comes from publishing; they simply don't expect to get caught.

Re:Wish I could say I was surprised

[interkin3tic]

That's true for large clinical trials, but clinical trials aren't all or even most research. For basic research like what this journal seems to publish on, no. It's rare that you'd have an experiment which would take months, let alone 5 years, and it would only be at the very end that you'd get a yes or a no.

For example, a study in that journal is entitled "Predicting blast-induced ground vibration using general regression neural network." The abstract is

Blasting is still an economical and viable method for rock excavation in mining and civil works projects. Ground vibration generated due to blasting is an undesirable phenomenon which is harmful for the nearby inhabitants and dwellings and should be prevented. In this study, an attempt has been made to predict the blast-induced ground vibration and frequency by incorporating rock properties, blast design and explosive parameters using the general regression neural network (GRNN) technique. To validate this methodology, the predictions obtained were compared with those obtained using the artificial neural network (ANN) model as well as by multivariate regression analysis (MVRA). Among all the methods, GRNN provides excellent predictions with a high degree of correlation.

Emphasis mine: they're testing if they can predict how the ground will shake after an explosion.

They're not going to spend five years dreaming up a model for it completely on a dry-erase board, set off a stick of dynamite, realize they were wrong, and throw out five years of work. They get preliminary results that are positive and encourage the project to go forward. They make up a model based on data, then refine it with subsequent bombs. If one were completely unable to use the method they propose to figure out how the ground shakes during explosions, they'd likely find that out before months had gone by.

I don't know what the breakdown is between clinical trials and basic research. I'd bet there's more research money spent on clinical trials than basic research (necessary, given certain realities about clinical trials) but that there are more scientists in basic research. So I'd bet that most research out there is actually not really affected a whole lot by the negative results issue: if you get a negative result, you get it before you commit yourself to it, and you move your reserarch in a different direction.

I remember Journal of Vibration and Control!

[RevWaldo]

That was one high class bondage mag, right up there with Bizzare and Exotique.

I don't think "peer review" means what WaPo thinks it means...

.

In other news,

[Zomalaja]

There actually is a Journal of Vibration and Control. Must be some thrilling stuff to read.

Dildos and bondage!

[Anonymous Coward]

Yay!

Re:Chen? Sounds Chinese

[Tablizer]

He is from Taiwan, not China.

You sure about that? Maybe he lied on that also. Hell, maybe he's not even Asian, or a "he".

Chen-Yuan Chen

[Anonymous Coward]

There's a lot of weirdness about this story. Firstly, guy's name is Chen-Yuan Chen, not "Peter". Secondly, he works at a teachers' college. Thirdly, he's supposed to be a researcher in methods for using electronics to help people learn, so why would he suddenly start writing a bunch of papers about mechanical systems? In addition to spamming 60 fraudulent papers in a few years, he also had each of the 60 papers cite all the other papers!

And the weirdest thing is that a bunch of right-wing crackpots are coming out of the woodwork to argue that this has some implication for climate change research. The fuck are these people smoking?

Re:Chen-Yuan Chen

[Anonymous Coward]

What's also weird is none of the "official" artifacts used the name "Peter" anywhere (always Chen, C-Y).
Where did "Peter" so prominently paraded by the lynch mob come from?
If anything, any official release should have used the name that's splashed all over offending articles listed.

Unrelated problem is, you can't get anything published in the trashiest tabloids without a figurative full cavity search.
JVC is suppose to be a proper academic journal, but they seem to have nobody with bullshit radar on the editorial board?
Does Ali H. Nayfeh do anything for the said journal beyond collecting a paycheck?

Re:Chen? Sounds Chinese

[backslashdot]

And what about Hendrik Schön, where was he from?

Or maybe it's not just the Chinese, your brain locks onto fake patterns.

Web of Trust

[Dr_Barnowl]

People should cryptographically sign peer reviews (and their papers). And journals should only trust signing keys that themselves have been signed by respected experts. The more respected you get, the more signatures your keys and papers get.

Not surprising

[teakillsnoopy]

I've been proofreading engineering/medical papers for universities in Taiwan for over 7 years and this is not surprising in the least. There is almost no stigma regarding plagiarism in this region (I've done work for Malaysian, Vietnamese, Indonesian, etc. authors). When I alert an author about copy/pasted text, their reaction is one you would get if you told someone that their reference format needs to be change. "Oh, ok. I guess I'll change it.". The universities here never seriously investigate plagiarism because all the big fish at the top did it themselves to get to the top.