Some Science Journals That Claim To Peer Review Papers Do Not Do So (economist.com)

A rising number of journals that claim to review submissions do not bother to do so. Not coincidentally, this seems to be leading some academics to inflate their publication lists with papers that might not pass such scrutiny. The Economist: Experts debate how many journals falsely claim to engage in peer review. Cabells, an analytics firm in Texas, has compiled a blacklist of those which it believes are guilty. According to Kathleen Berryman, who is in charge of this list, the firm employs 65 criteria to determine whether a journal should go on it -- though she is reluctant to go into details. Cabells' list now totals around 8,700 journals, up from a bit over 4,000 a year ago. Another list, which grew to around 12,000 journals, was compiled until recently by Jeffrey Beall, a librarian at the University of Colorado. Using Mr Beall's list, Bo-Christer Bjork, an information scientist at the Hanken School of Economics, in Helsinki, estimates that the number of articles published in questionable journals has ballooned from about 53,000 a year in 2010 to more than 400,000 today. He estimates that 6% of academic papers by researchers in America appear in such journals.

Comments

The result of "publish or perish"

[whitesea]

They prey on people whose career depends on the quantity of publications. My friend published a paper that became famous in his area of research overnight. Everybody and their brother cited the paper, even mainstream media mentioned it. His dept chairman said, "We are satisfied with the quality of your papers. It's the quantity that's insufficient."

Re:The result of "publish or perish"

[gl4ss]

well they gave themselves a set of criteria and didn't think further than that.

also with rise of pseudoscience in general, like publishing a paper on how michael jackson was able to do the smooth criminal 45 degree lean.. something that was revealed in making of documentary 20 years+ ago.

it's not looking good. on the upside, good science is as healthy as ever. it's just that there's shitloads of fluff science careers in effect right now and they only count those numbers.

so of course there's more journals to cash in on that. look, 12 000 journals means that they might just as well be fucking blogs. nobodys really reading them and I mean nobody.

scientists shouldn't be publishing shit papers just for numbers. if they can't find out anything new, that should be the criteria.

Re:The result of "publish or perish"

[jma05]

Peter Higgs says he would not have survived in this system.

https://www.theguardian.com/sc...

Re:The result of "publish or perish"

[Anonymous Coward]

As a tenured professor, I can say the papers are just the tip of the iceberg. Academic science, at least in the biomedical sciences, is falling apart due a variety of problems: ponzi schemes with doctoral and postdoctoral training, indirect funds of grants inflating their value to universities as profit margin, cuts from state governments and passing the buck of research funding to the federal government, and a general "widget production" model of science being demanded from administrators and conservative legislative overseers.

I was in a faculty meeting a couple of years ago. A junior faculty member was undergoing annual review, and some of the faculty expressed concern that they were publishing too much in open access journals. These weren't questionable open access journals, though: they were pretty well-established ones that just weren't traditional academic journals. More importantly, the junior faculty member's impact factors, number of citations, etc. were all fine, comparable to any other successful junior faculty at that stage. However, some of the senior faculty felt that the journals weren't prestigious enough. So they created a memo to be circulated around the department, a list of journals, saying "these are journals junior faculty should be publishing in."

The memo was justified in the interest of fairness and clarity, I and I get the intent, but on the face of it is absurd. The focus should be on the quality of the research, not the reputation of the journal. It's as if someone denigrated Lolita as a work of literature because the publisher was of poor reputation.

These lists of predatory journals that float around are useful, and the journals should be criticized. But when you get to this scope of problem, these journals aren't the problem, they're a symptom. What you have now is an oversupply of very talented researchers, an underfunding of science (above and beyond the annual federal research budget, which we shouldn't be so dependent on), a focus on celebrity over substance, superficial indicators of productivity, nepotism... I could go on and on.

Publishing in particular is sort of a house of cards. A rational outside observer would ask what the economic reasons for the current structure are, with such low costs to publishing now on the web. Why do peer-reviewed journals even exist now? Are scientists really paying attention to what they should be? Open access journals are one solution, but if you look at them closely, they probably in aggregate do more harm than good because they are pay-to-publish, which creates huge misincentives.

These sorts of predatory journals are completely predictable, and are the tip of the iceberg. Keep in mind these are journals where there are *obvious* improprieties. Things get even more problematic if you realize that there even more "legitimate" journals that leverage moral greyness or plausible deniability as a way of avoiding these lists.

Most of the time I feel like academic science is in crisis, and even more so every day. There's a huge disparity between how science actually occurs and how people are compensated and recognized.

Re:The result of "publish or perish"

[Anonymous Coward]

"conservative legislative overseers"

How hilarious. Even when you manage to worm your way into an industry that's self-admittedly 99% liberal, you still blame conservatives for your problems.

Re:The result of "publish or perish"

[thePsychologist]

The memo was justified in the interest of fairness and clarity, I and I get the intent, but on the face of it is absurd. The focus should be on the quality of the research, not the reputation of the journal. It's as if someone denigrated Lolita as a work of literature because the publisher was of poor reputation.

I'm not sure you really got the intent. The reason why science has these flaws is because of senior faculty, and they enforce these ridiculous norms to keep their own reputation. If junior faculty move away from traditional journals and methods of publishing, then senior faculty will lose prestige and their ability to direct the flow of grant money.

Academia has degraded.

[Qbertino]

As far as I can tell adademia has turned into quite a self-referential naval-gazing afair in recent decades. Point in case: I've finally gone into college and started my BsC Media-CompSci and on my first project I get a stern look for quoting details on RSA from a book from 1998. They told me they're embarrased to quote anything that's older than 5 years and I should never quote a book from 1998. ... WTF? Seriously? Even if it covers elemental basics that haven't changed a single bit since decades ago? Mind you, this is CompSci where you would expect some sort of academic credibility, unlike Sociology or "Gender Studies". Given, CompSci is still miles behind math, physics and engineering, but I still would expect it to me somewhat mature.

This paper-publishing apears to me like some intelectual masturbation academics like to indulge in and not neccesarly something that brings humanity forward.

Maybe I'm wrong, but I still can't shake the feeling that academia has degraded severly and the avantgarde has moved on.

Re:Academia has degraded.

[Cassini2]

References to papers less than 5 years old help your fellow colleagues get tenure, and help your fellow students get Masters Degrees and PhDs.

Older references are essentially sites to influential papers. Those papers were written by influential people that already have tenure. Those people would prefer you help the junior faculty, the Master's and PhD students.

To be a friend, you need to cite the newer research.

BTW: I didn't invent this system.

so?

[superwiz]

The whole theory behind peer review breaks down in a world in which areas of expertise are too narrow.

A study that no one seems to want to do is which fact finding ends up being more accurate? The criminal court system (with its trial system and adversarial review) or quality control system of scientific publications? Most advanced scientific research can only be checked by very few peers. And the only checks they perform is whether they can replicate the results. And even that is not always the standard. They often simply try to see if they can arrive at the same results with a different method. But then they know ahead of time what to expect (based on reputation of those "anonymous" results they are checking). And I put "anonymous" in quotes because if there is only 3 labs in the world which look into a certain area, then lab C knows whether they are checking results from lab A or lab B just by reading their proposals.

Again, no one is checking whether adversarial review produces better more fact-finding accurate results. And yet all the people who rely on grants for their work are against it.

Even commercial scientific research requires adversarial review. This is what drug trials are. Drug companies have to prove that their drugs perform better than placebo and that the drugs do not have a high frequencies of adverse effects. How would we like it if drug companies could put their drugs out on the market as long as they passed peer review? We definitely would think that was a scary proposition. Well, then why do we use such a weak standard for scientific exploration which is funded by public money (and which sometimes drives public policy)?

Re:so?

[godrik]

I publish Computer Science articles frequently. While I am not necessarily happy about how the peer review process works in my field, it often means something else than people expect.

In my opinion the review process verifies two things: Does the result seem correct? Is the paper interesting?

Whether the paper is "interesting" or not is a judgment call. In the context of a conference, I have asked to reject paper that were correct but I could not believe the problem and results would interest a room of 80 people for half an hour.

Whether the result is correct is a much more complicated question. If the paper is theoretical, you should be able to verify it during the peer review process. There are typically 3 reviewers, and that is usually enough to get a clear idea of whether the models and proofs are sound. If a part of the paper is still obscure to the 3 reviewers, then clearly the paper lacks clarity and should be revised before being accepted.

The real problem in CS comes from experimental papers. Because reproducing experimental results is hard and sometimes not possible. Maybe you don't have access to the code (research codes are not always made available). Maybe you don't have access to the data (some data is proprietary and can not be shared). Maybe you don't have access to the machine (I think only Chinese nationals can get access to tianhe-2 for instance; I myself wrote paper about an experimental not-yet-released system). Even if you could reproduce the result, it could take month to reproduce. So in practice, you don't attempt reproduction of most experimental CS paper.
What you do is check for consistency. Does the result make sense? Does the technique provide an output that is coherent with expectation? If it doesn't, is the discrepancy explained in the paper? Is there a clear drawback to the method that is not mentioned in the paper? Do I believe that the paper contains all the information necessary for reproducing the results if I wanted to? That is about the type of things that you check. Some are pushing for including experimental results as supplementary material to experimental papers or to make experimental results more reproducible in general. (See the work of Arnaud Legrand or of Lucas Nussbaum for instance, but many other work on that.) The SC conference has now a reproducibility initiative to help with that.

The adversarial review that you are talking about happens AFTER publication. That is where the review peer review starts. It starts when dozens of master student or PhD student will compare their method to the state of the art. And that is when you will know what will stick and what won't. Because they will make the comparisons to different frameworks, on different machines, on different datasets.

Re:The buck ALWAYS stops with YOU.

[LynnwoodRooster]

If a researcher will not share their data and their methods, it doesn't matter WHERE it was published it should be highly suspect if not outright discounted. Back when I was in university (mid 80s) you had to show your work, show your data, show your experiment setup, and show the link between all of it or you'd get zero credit. Even if your experiment failed, you'd still get credit because you showed what you did and what data you collected.

More and more it's simply "we used a process like this, and we had this result, but we cannot share the data and actual process because it's proprietary and worth money but trust us - it's good!" Nope. Science is a process and requires disclosure of data and process so that your experiment can be done, exactly, by others. Science is skeptical by nature - NO ONE should accept the result of a study or paper unless there is sufficient data and process shared that would allow you to replicate if you so choose. The scientist should, on hearing any claim, think "OK, that's interesting - now what data is there and can I replicate their results?" - if not, it's not science.

A list please.

[bluegutang]

Exactly which papers allow you to publish papers without peer review?

Not that it matters, but I will be facing a tenure committee soon...

Practical Considerations

[Roger W Moore]

The memo was justified in the interest of fairness and clarity, I and I get the intent, but on the face of it is absurd. The focus should be on the quality of the research, not the reputation of the journal.

Agreed but the problem is how do you fairly evaluate a colleagues research at a global level when you do not work in that area? Given the need to do this high quality, peer-reviewed journals make sense since, if your colleague can get his/her work published in one then clearly the rest of their field also think their research is high quality.

Academia part of Society

[Roger W Moore]

Perhaps the citation age limit is part of the field since I know that my CS colleagues rarely publish papers and instead focus on conferences because they claim it is rare to have a research topic that is still cutting edge by the time it has gone through the writing, review and publication process which can take up to a year. Certainly it does not apply in physics - I cited papers all the way back to 1932 in my thesis and 1963 for papers.

As for degrading, we are part of society and as society moves away from meritocratic ideals such as equality of opportunity in favour of equality of outcomes it has an impact on all aspects of society, including academia.

Re:Academia part of Society

[invalid_user]

In theoretical CS, (except for the results that are immediately applicable to specific field), many papers are first cited 5~6 years after their publication. Reasons:
1. Theory is hard. Few can do.
2. Breakthroughs are hard to come by.

Re: The buck ALWAYS stops with YOU.

[LynnwoodRooster]

You shouldn't publish at that point. If you cannot share your data, then you cannot - legitimately - make a scientific claim any stronger than "trust me". Not all research needs to be published, and that would be a perfect solution for data that needs to be kept private. However, if a researcher wants to publish then they should expect to share everything - otherwise how can anyone trust their research?

on the industry of indolence

[epine]

I recently read George Dyson's book on the founding of the IAS.

The intellectual and academic caliber of people fleeing Germany (and nearby regions potentially subject to German influence) was unbelievable, yet the cowboy-era administrative end-runs to secure stipends for many of these people were off the charts.

Then America had its middle-class golden era between 1950 and 1980. If I've learned anything from my recent return to the history books it's this: this is the least economically normative period of the last 400 years. Short of handing Stalin the keys to the hydrogen bomb years earlier than he should have got them, it was a pretty amazing time for an empire unlike any that came before it.

Not so long ago, 10% of the population went off to university. Now 41% of women in Canada attain a bachelor's degree or higher. This is associated with an inevitable pressure to make the filters of meritocracy ever more fine-grained. So, of necessity, academia invents a cascade of credentialism mountain ranges. But there isn't enough legitimate signal to make this work, so we're forced to invent credentialist hoops.

Zoom all the way up to the TARP bailout of 2008. There's a large group of economists who think this was too large/unnecessarily/ineffective, another large group who think it was too small/absolutely essential/effective so far as it went (with maybe a sliver of fence-sitters eating porridge at the perfect temperature).

Prospective judgement is counterfactual. Retrospective judgement is counterfactual. In none of these matters are we afforded a virtual mulligan to run the simulation again.

If you're not Einstein, you're probably facing some kind of peer-group proxy measure, with the intervals between the scythe are having contracted from leap years to stutter-step quarters (who can summon the effort to leap any more, when the relentless ankle-blades never let up?)

Goodbye Erdos number two. Hello author position number 997 on a single published paper.

Meritocracy is this weird nodable concept. Sure sounds like a good idea (especially after a long stretch where things are anything but meritocratic). But merit turns out to be a terribly, terribly hard thing to implement well in practice, with acceptable consequence.

On TARP, we never achieved a retrospective standard of merit, all we got was two lousy, tribal camps locked into an egocentric crowing competition. I mention TARP mainly because the stakes here are in the trillion dollar range. Surely if incentive porn FTW, this would be the ultimate case study concerning the collective human incentive to get the individual incentives sorted out.

———

I found these two books exceptionally interesting to read back to back:

* Twilight of the Elites: America After Meritocracy â" 2012
* Coming Apart â" 2012

Same publication year, same thematic material, anode vs. cathode political perspective, but ultimately the same message: implementing meritocracy is far harder than it looks.

Basically the problem here boils down to not enough lions. Actual survival was once a pretty good proxy measure on who had it together, and who didn't. (Until the fated day came where your—former—best friend hid your Nikes.) Problem: the objective measure of the lion cull was a just a tad morally blind.

One of the problems with incentive porn is the notion that incentive gradients should be pervasive and perpetual: don't go to university, struggle to pay the rent; don't graduate at the top of your class, struggle to pay your student loans. Etc.

The other model is that you mill around aimlessly (sort of) until something clicks, and then you go off on a mad tear, when there's clearly something special you feel that you can achieve. If you succeed, you get perks (recognition, fancy jobs, fancy peers). If not, you're simply cast back into the milling pond.

A UBI is one way to provide a giant milling pond of opportunity.

It would surely

Re:on the industry of indolence

[smallfries]

Thanks for this - the most interesting thing I've read on slashdot in over a decade. The only point that I don't think that you covered is that as society moves further into specialized roles there are many people who simply do not rise above the threshold in any particular specialty - the statistics behind this are quite simple. Yet we need to maintain these "unproductive" people specifically so that we have a broad enough pool to avoid over-fitting what we focus on.

See also: coercitive citation

[Anonymous Coward]

Posting anonymously to avoid legal problems by disclosing the following information.

Not only fake journals are a problem. Supposedly "serious" journals with a corrupt editor using a coercive citation scheme are also an important issue. Coercive citation is typically used to increase the metrics of the journal, but I have seen at least one case in which it is used to increase the metrics of the Journal Editor (!)

Specific example: Springer's Journal of Supercomputing is edited by infamous Hamid Arabnia. Yes, this is the same folk that used to run the WorldComp conference series in Vegas, which is now rebranded as CSCE after it was widespread that they accepted any crap as long as you paid for registration.

Well, I submitted some years ago a real research paper to this Journal of Supercomputing. It was serious research, not top-level, but reasonable. Reviews were reasonable, but H. Arabnia requested to add citations to FOUR of his own personal papers, completely unrelated with our submission, in order to accept the paper. We didn't add any (and the paper was eventually accepted), but we could check that he did this routinely: you can check for example this paper, in which authors cite TEN unrelated papers from the editor of this journal. I don't blame the authors: in many cases, they badly need the publication and agree to the coercive mechanism.

You can also check H. Arabnia's Google Scholar page, with a very high h-index value. However, this page also allows you to check the citations of the papers. If you check the 88 citations to this paper from 1995, you can see that it was almost unnoticed for twenty years, and suddenly it resurged in 2015... with ALL citations coming from the Journal of Supercomputing, which he edits!!

The funny fact: The journal of Supercomputing has a JCR impact factor of 1.326 in the last (2016) list, being in the second quartile (Q2) of its category. Let's see the update, coming in a few days/weeks. According to the rankings, this should be a respected journal, but it happens to be the playground of this clown, abusing it to increase his own metrics.

Re:See also: coercitive citation

[invalid_user]

Can someone please mod this up? I can confirm that this is a common practice by many today... but who can blame them? The only lesson is that citation count is no longer trustworthy. Also, Google Scholar now counts blog posts as papers, and web links as citations.

(I have mod points but can't moderate since I have coomented on this article)

People know what's reputable

[SoftwareArtist]

Scientists know which journals are reputable. In any field, there are maybe a half dozen or a dozen journals that most papers get published in. Everyone knows what they are. And then there's a hundred wannabe journals with intentionally similar names that exist just to make money. No one reads them, and no one reputable publishes in them. If you look at someone's publication list and see a lot of papers in journals you've never heard of, you immediately know to be suspicious.

Re:The buck ALWAYS stops with YOU.

[serviscope_minor]

If a researcher will not share their data and their methods,

It always makes me chuckle when someone says this. I take it you're a aoftware guy? For the point of this explanation I hope so, and a lot of us here are, so I'll assume you are.

Imagine you had a 23 year old programmer fresh out of school. Actually forget that imagine you had a 23 year old STEM grad who did something like physics or engineering. They're smart, did a little programming on the course and probably self taught to learn a bit more.

They have zero software engineering experience. Zero experience in building large systems, zero experience in building reliable software, no idea about unit tests, version control or release management. They have yet to experience the pitfalls of doing things like hardcoding paths because they've never shared their software with anyone.

Now consider that they're in a lap where their adviser has zero software experience either (what with being a scientist not a professional software person). Layer on top of that that the system positively discourages all the best practices because they need to get the result out fast as a one off then move on to the next result.

Now let that student loose on a pretty large project, say a year's worth of work.

That, the code is the "show me what you did" bit for a large part of it. I've been there on both sides. sharing tha tis not nearly as useful as you might expect.

What peer review means

[sjbe]

Agreed but the problem is how do you fairly evaluate a colleagues research at a global level when you do not work in that area?

You ask someone who does work in that area. I think that would be obvious. Merely having a paper published in a "prestigious" journal will not answer the question of whether they do good work in general or whether the specific work in question is actually valuable. Even the best journals with the highest reputations sometimes publish some shit science unintentionally. What determines the credibility of research is the quality of the work that builds from it.

If you really want to evaluate the work of a researcher you need to evaluate number of citations, and the quality of those citations. Influential and important works tend to have a lot of citations and the works based on them tend to be influential as well. Simply publishing a lot or being published in a particular journal doesn't really mean anything.

Given the need to do this high quality, peer-reviewed journals make sense since, if your colleague can get his/her work published in one then clearly the rest of their field also think their research is high quality.

Not sure how you could draw that conclusion. All it tells you is that a few "peers" felt the paper was potentially worthwhile. Maybe not even that much depending on how seriously they took their job. It doesn't mean they've done a deep dive to corroborate the research. Peer review does not mean the rest of the field respects their work.