Slashdot Mirror
Some Science Journals That Claim To Peer Review Papers Do Not Do So (economist.com)
A rising number of journals that claim to review submissions do not bother to do so. Not coincidentally, this seems to be leading some academics to inflate their publication lists with papers that might not pass such scrutiny. The Economist: Experts debate how many journals falsely claim to engage in peer review. Cabells, an analytics firm in Texas, has compiled a blacklist of those which it believes are guilty. According to Kathleen Berryman, who is in charge of this list, the firm employs 65 criteria to determine whether a journal should go on it -- though she is reluctant to go into details. Cabells' list now totals around 8,700 journals, up from a bit over 4,000 a year ago. Another list, which grew to around 12,000 journals, was compiled until recently by Jeffrey Beall, a librarian at the University of Colorado. Using Mr Beall's list, Bo-Christer Bjork, an information scientist at the Hanken School of Economics, in Helsinki, estimates that the number of articles published in questionable journals has ballooned from about 53,000 a year in 2010 to more than 400,000 today. He estimates that 6% of academic papers by researchers in America appear in such journals.
Peter Higgs says he would not have survived in this system.
https://www.theguardian.com/sc...
As a tenured professor, I can say the papers are just the tip of the iceberg. Academic science, at least in the biomedical sciences, is falling apart due a variety of problems: ponzi schemes with doctoral and postdoctoral training, indirect funds of grants inflating their value to universities as profit margin, cuts from state governments and passing the buck of research funding to the federal government, and a general "widget production" model of science being demanded from administrators and conservative legislative overseers.
I was in a faculty meeting a couple of years ago. A junior faculty member was undergoing annual review, and some of the faculty expressed concern that they were publishing too much in open access journals. These weren't questionable open access journals, though: they were pretty well-established ones that just weren't traditional academic journals. More importantly, the junior faculty member's impact factors, number of citations, etc. were all fine, comparable to any other successful junior faculty at that stage. However, some of the senior faculty felt that the journals weren't prestigious enough. So they created a memo to be circulated around the department, a list of journals, saying "these are journals junior faculty should be publishing in."
The memo was justified in the interest of fairness and clarity, I and I get the intent, but on the face of it is absurd. The focus should be on the quality of the research, not the reputation of the journal. It's as if someone denigrated Lolita as a work of literature because the publisher was of poor reputation.
These lists of predatory journals that float around are useful, and the journals should be criticized. But when you get to this scope of problem, these journals aren't the problem, they're a symptom. What you have now is an oversupply of very talented researchers, an underfunding of science (above and beyond the annual federal research budget, which we shouldn't be so dependent on), a focus on celebrity over substance, superficial indicators of productivity, nepotism... I could go on and on.
Publishing in particular is sort of a house of cards. A rational outside observer would ask what the economic reasons for the current structure are, with such low costs to publishing now on the web. Why do peer-reviewed journals even exist now? Are scientists really paying attention to what they should be? Open access journals are one solution, but if you look at them closely, they probably in aggregate do more harm than good because they are pay-to-publish, which creates huge misincentives.
These sorts of predatory journals are completely predictable, and are the tip of the iceberg. Keep in mind these are journals where there are *obvious* improprieties. Things get even more problematic if you realize that there even more "legitimate" journals that leverage moral greyness or plausible deniability as a way of avoiding these lists.
Most of the time I feel like academic science is in crisis, and even more so every day. There's a huge disparity between how science actually occurs and how people are compensated and recognized.
I publish Computer Science articles frequently. While I am not necessarily happy about how the peer review process works in my field, it often means something else than people expect.
In my opinion the review process verifies two things: Does the result seem correct? Is the paper interesting?
Whether the paper is "interesting" or not is a judgment call. In the context of a conference, I have asked to reject paper that were correct but I could not believe the problem and results would interest a room of 80 people for half an hour.
Whether the result is correct is a much more complicated question. If the paper is theoretical, you should be able to verify it during the peer review process. There are typically 3 reviewers, and that is usually enough to get a clear idea of whether the models and proofs are sound. If a part of the paper is still obscure to the 3 reviewers, then clearly the paper lacks clarity and should be revised before being accepted.
The real problem in CS comes from experimental papers. Because reproducing experimental results is hard and sometimes not possible. Maybe you don't have access to the code (research codes are not always made available). Maybe you don't have access to the data (some data is proprietary and can not be shared). Maybe you don't have access to the machine (I think only Chinese nationals can get access to tianhe-2 for instance; I myself wrote paper about an experimental not-yet-released system). Even if you could reproduce the result, it could take month to reproduce. So in practice, you don't attempt reproduction of most experimental CS paper.
What you do is check for consistency. Does the result make sense? Does the technique provide an output that is coherent with expectation? If it doesn't, is the discrepancy explained in the paper? Is there a clear drawback to the method that is not mentioned in the paper? Do I believe that the paper contains all the information necessary for reproducing the results if I wanted to? That is about the type of things that you check. Some are pushing for including experimental results as supplementary material to experimental papers or to make experimental results more reproducible in general. (See the work of Arnaud Legrand or of Lucas Nussbaum for instance, but many other work on that.) The SC conference has now a reproducibility initiative to help with that.
The adversarial review that you are talking about happens AFTER publication. That is where the review peer review starts. It starts when dozens of master student or PhD student will compare their method to the state of the art. And that is when you will know what will stick and what won't. Because they will make the comparisons to different frameworks, on different machines, on different datasets.