Some Science Journals That Claim To Peer Review Papers Do Not Do So (economist.com)

A rising number of journals that claim to review submissions do not bother to do so. Not coincidentally, this seems to be leading some academics to inflate their publication lists with papers that might not pass such scrutiny. The Economist: Experts debate how many journals falsely claim to engage in peer review. Cabells, an analytics firm in Texas, has compiled a blacklist of those which it believes are guilty. According to Kathleen Berryman, who is in charge of this list, the firm employs 65 criteria to determine whether a journal should go on it -- though she is reluctant to go into details. Cabells' list now totals around 8,700 journals, up from a bit over 4,000 a year ago. Another list, which grew to around 12,000 journals, was compiled until recently by Jeffrey Beall, a librarian at the University of Colorado. Using Mr Beall's list, Bo-Christer Bjork, an information scientist at the Hanken School of Economics, in Helsinki, estimates that the number of articles published in questionable journals has ballooned from about 53,000 a year in 2010 to more than 400,000 today. He estimates that 6% of academic papers by researchers in America appear in such journals.

Comments

The result of "publish or perish"

[whitesea]

They prey on people whose career depends on the quantity of publications. My friend published a paper that became famous in his area of research overnight. Everybody and their brother cited the paper, even mainstream media mentioned it. His dept chairman said, "We are satisfied with the quality of your papers. It's the quantity that's insufficient."

Re:The result of "publish or perish"

[jma05]

Peter Higgs says he would not have survived in this system.

https://www.theguardian.com/sc...

Re:The result of "publish or perish"

[Anonymous Coward]

As a tenured professor, I can say the papers are just the tip of the iceberg. Academic science, at least in the biomedical sciences, is falling apart due a variety of problems: ponzi schemes with doctoral and postdoctoral training, indirect funds of grants inflating their value to universities as profit margin, cuts from state governments and passing the buck of research funding to the federal government, and a general "widget production" model of science being demanded from administrators and conservative legislative overseers.

I was in a faculty meeting a couple of years ago. A junior faculty member was undergoing annual review, and some of the faculty expressed concern that they were publishing too much in open access journals. These weren't questionable open access journals, though: they were pretty well-established ones that just weren't traditional academic journals. More importantly, the junior faculty member's impact factors, number of citations, etc. were all fine, comparable to any other successful junior faculty at that stage. However, some of the senior faculty felt that the journals weren't prestigious enough. So they created a memo to be circulated around the department, a list of journals, saying "these are journals junior faculty should be publishing in."

The memo was justified in the interest of fairness and clarity, I and I get the intent, but on the face of it is absurd. The focus should be on the quality of the research, not the reputation of the journal. It's as if someone denigrated Lolita as a work of literature because the publisher was of poor reputation.

These lists of predatory journals that float around are useful, and the journals should be criticized. But when you get to this scope of problem, these journals aren't the problem, they're a symptom. What you have now is an oversupply of very talented researchers, an underfunding of science (above and beyond the annual federal research budget, which we shouldn't be so dependent on), a focus on celebrity over substance, superficial indicators of productivity, nepotism... I could go on and on.

Publishing in particular is sort of a house of cards. A rational outside observer would ask what the economic reasons for the current structure are, with such low costs to publishing now on the web. Why do peer-reviewed journals even exist now? Are scientists really paying attention to what they should be? Open access journals are one solution, but if you look at them closely, they probably in aggregate do more harm than good because they are pay-to-publish, which creates huge misincentives.

These sorts of predatory journals are completely predictable, and are the tip of the iceberg. Keep in mind these are journals where there are *obvious* improprieties. Things get even more problematic if you realize that there even more "legitimate" journals that leverage moral greyness or plausible deniability as a way of avoiding these lists.

Most of the time I feel like academic science is in crisis, and even more so every day. There's a huge disparity between how science actually occurs and how people are compensated and recognized.

Academia has degraded.

[Qbertino]

As far as I can tell adademia has turned into quite a self-referential naval-gazing afair in recent decades. Point in case: I've finally gone into college and started my BsC Media-CompSci and on my first project I get a stern look for quoting details on RSA from a book from 1998. They told me they're embarrased to quote anything that's older than 5 years and I should never quote a book from 1998. ... WTF? Seriously? Even if it covers elemental basics that haven't changed a single bit since decades ago? Mind you, this is CompSci where you would expect some sort of academic credibility, unlike Sociology or "Gender Studies". Given, CompSci is still miles behind math, physics and engineering, but I still would expect it to me somewhat mature.

This paper-publishing apears to me like some intelectual masturbation academics like to indulge in and not neccesarly something that brings humanity forward.

Maybe I'm wrong, but I still can't shake the feeling that academia has degraded severly and the avantgarde has moved on.

Re:so?

[godrik]

I publish Computer Science articles frequently. While I am not necessarily happy about how the peer review process works in my field, it often means something else than people expect.

In my opinion the review process verifies two things: Does the result seem correct? Is the paper interesting?

Whether the paper is "interesting" or not is a judgment call. In the context of a conference, I have asked to reject paper that were correct but I could not believe the problem and results would interest a room of 80 people for half an hour.

Whether the result is correct is a much more complicated question. If the paper is theoretical, you should be able to verify it during the peer review process. There are typically 3 reviewers, and that is usually enough to get a clear idea of whether the models and proofs are sound. If a part of the paper is still obscure to the 3 reviewers, then clearly the paper lacks clarity and should be revised before being accepted.

The real problem in CS comes from experimental papers. Because reproducing experimental results is hard and sometimes not possible. Maybe you don't have access to the code (research codes are not always made available). Maybe you don't have access to the data (some data is proprietary and can not be shared). Maybe you don't have access to the machine (I think only Chinese nationals can get access to tianhe-2 for instance; I myself wrote paper about an experimental not-yet-released system). Even if you could reproduce the result, it could take month to reproduce. So in practice, you don't attempt reproduction of most experimental CS paper.
What you do is check for consistency. Does the result make sense? Does the technique provide an output that is coherent with expectation? If it doesn't, is the discrepancy explained in the paper? Is there a clear drawback to the method that is not mentioned in the paper? Do I believe that the paper contains all the information necessary for reproducing the results if I wanted to? That is about the type of things that you check. Some are pushing for including experimental results as supplementary material to experimental papers or to make experimental results more reproducible in general. (See the work of Arnaud Legrand or of Lucas Nussbaum for instance, but many other work on that.) The SC conference has now a reproducibility initiative to help with that.

The adversarial review that you are talking about happens AFTER publication. That is where the review peer review starts. It starts when dozens of master student or PhD student will compare their method to the state of the art. And that is when you will know what will stick and what won't. Because they will make the comparisons to different frameworks, on different machines, on different datasets.

Re:The buck ALWAYS stops with YOU.

[LynnwoodRooster]

If a researcher will not share their data and their methods, it doesn't matter WHERE it was published it should be highly suspect if not outright discounted. Back when I was in university (mid 80s) you had to show your work, show your data, show your experiment setup, and show the link between all of it or you'd get zero credit. Even if your experiment failed, you'd still get credit because you showed what you did and what data you collected.

More and more it's simply "we used a process like this, and we had this result, but we cannot share the data and actual process because it's proprietary and worth money but trust us - it's good!" Nope. Science is a process and requires disclosure of data and process so that your experiment can be done, exactly, by others. Science is skeptical by nature - NO ONE should accept the result of a study or paper unless there is sufficient data and process shared that would allow you to replicate if you so choose. The scientist should, on hearing any claim, think "OK, that's interesting - now what data is there and can I replicate their results?" - if not, it's not science.

on the industry of indolence

[epine]

I recently read George Dyson's book on the founding of the IAS.

The intellectual and academic caliber of people fleeing Germany (and nearby regions potentially subject to German influence) was unbelievable, yet the cowboy-era administrative end-runs to secure stipends for many of these people were off the charts.

Then America had its middle-class golden era between 1950 and 1980. If I've learned anything from my recent return to the history books it's this: this is the least economically normative period of the last 400 years. Short of handing Stalin the keys to the hydrogen bomb years earlier than he should have got them, it was a pretty amazing time for an empire unlike any that came before it.

Not so long ago, 10% of the population went off to university. Now 41% of women in Canada attain a bachelor's degree or higher. This is associated with an inevitable pressure to make the filters of meritocracy ever more fine-grained. So, of necessity, academia invents a cascade of credentialism mountain ranges. But there isn't enough legitimate signal to make this work, so we're forced to invent credentialist hoops.

Zoom all the way up to the TARP bailout of 2008. There's a large group of economists who think this was too large/unnecessarily/ineffective, another large group who think it was too small/absolutely essential/effective so far as it went (with maybe a sliver of fence-sitters eating porridge at the perfect temperature).

Prospective judgement is counterfactual. Retrospective judgement is counterfactual. In none of these matters are we afforded a virtual mulligan to run the simulation again.

If you're not Einstein, you're probably facing some kind of peer-group proxy measure, with the intervals between the scythe are having contracted from leap years to stutter-step quarters (who can summon the effort to leap any more, when the relentless ankle-blades never let up?)

Goodbye Erdos number two. Hello author position number 997 on a single published paper.

Meritocracy is this weird nodable concept. Sure sounds like a good idea (especially after a long stretch where things are anything but meritocratic). But merit turns out to be a terribly, terribly hard thing to implement well in practice, with acceptable consequence.

On TARP, we never achieved a retrospective standard of merit, all we got was two lousy, tribal camps locked into an egocentric crowing competition. I mention TARP mainly because the stakes here are in the trillion dollar range. Surely if incentive porn FTW, this would be the ultimate case study concerning the collective human incentive to get the individual incentives sorted out.

———

I found these two books exceptionally interesting to read back to back:

* Twilight of the Elites: America After Meritocracy â" 2012
* Coming Apart â" 2012

Same publication year, same thematic material, anode vs. cathode political perspective, but ultimately the same message: implementing meritocracy is far harder than it looks.

Basically the problem here boils down to not enough lions. Actual survival was once a pretty good proxy measure on who had it together, and who didn't. (Until the fated day came where your—former—best friend hid your Nikes.) Problem: the objective measure of the lion cull was a just a tad morally blind.

One of the problems with incentive porn is the notion that incentive gradients should be pervasive and perpetual: don't go to university, struggle to pay the rent; don't graduate at the top of your class, struggle to pay your student loans. Etc.

The other model is that you mill around aimlessly (sort of) until something clicks, and then you go off on a mad tear, when there's clearly something special you feel that you can achieve. If you succeed, you get perks (recognition, fancy jobs, fancy peers). If not, you're simply cast back into the milling pond.

A UBI is one way to provide a giant milling pond of opportunity.

It would surely

See also: coercitive citation

[Anonymous Coward]

Posting anonymously to avoid legal problems by disclosing the following information.

Not only fake journals are a problem. Supposedly "serious" journals with a corrupt editor using a coercive citation scheme are also an important issue. Coercive citation is typically used to increase the metrics of the journal, but I have seen at least one case in which it is used to increase the metrics of the Journal Editor (!)

Specific example: Springer's Journal of Supercomputing is edited by infamous Hamid Arabnia. Yes, this is the same folk that used to run the WorldComp conference series in Vegas, which is now rebranded as CSCE after it was widespread that they accepted any crap as long as you paid for registration.

Well, I submitted some years ago a real research paper to this Journal of Supercomputing. It was serious research, not top-level, but reasonable. Reviews were reasonable, but H. Arabnia requested to add citations to FOUR of his own personal papers, completely unrelated with our submission, in order to accept the paper. We didn't add any (and the paper was eventually accepted), but we could check that he did this routinely: you can check for example this paper, in which authors cite TEN unrelated papers from the editor of this journal. I don't blame the authors: in many cases, they badly need the publication and agree to the coercive mechanism.

You can also check H. Arabnia's Google Scholar page, with a very high h-index value. However, this page also allows you to check the citations of the papers. If you check the 88 citations to this paper from 1995, you can see that it was almost unnoticed for twenty years, and suddenly it resurged in 2015... with ALL citations coming from the Journal of Supercomputing, which he edits!!

The funny fact: The journal of Supercomputing has a JCR impact factor of 1.326 in the last (2016) list, being in the second quartile (Q2) of its category. Let's see the update, coming in a few days/weeks. According to the rankings, this should be a respected journal, but it happens to be the playground of this clown, abusing it to increase his own metrics.