Slashdot Mirror
Some Science Journals That Claim To Peer Review Papers Do Not Do So (economist.com)
A rising number of journals that claim to review submissions do not bother to do so. Not coincidentally, this seems to be leading some academics to inflate their publication lists with papers that might not pass such scrutiny. The Economist: Experts debate how many journals falsely claim to engage in peer review. Cabells, an analytics firm in Texas, has compiled a blacklist of those which it believes are guilty. According to Kathleen Berryman, who is in charge of this list, the firm employs 65 criteria to determine whether a journal should go on it -- though she is reluctant to go into details. Cabells' list now totals around 8,700 journals, up from a bit over 4,000 a year ago. Another list, which grew to around 12,000 journals, was compiled until recently by Jeffrey Beall, a librarian at the University of Colorado. Using Mr Beall's list, Bo-Christer Bjork, an information scientist at the Hanken School of Economics, in Helsinki, estimates that the number of articles published in questionable journals has ballooned from about 53,000 a year in 2010 to more than 400,000 today. He estimates that 6% of academic papers by researchers in America appear in such journals.
I publish Computer Science articles frequently. While I am not necessarily happy about how the peer review process works in my field, it often means something else than people expect.
In my opinion the review process verifies two things: Does the result seem correct? Is the paper interesting?
Whether the paper is "interesting" or not is a judgment call. In the context of a conference, I have asked to reject paper that were correct but I could not believe the problem and results would interest a room of 80 people for half an hour.
Whether the result is correct is a much more complicated question. If the paper is theoretical, you should be able to verify it during the peer review process. There are typically 3 reviewers, and that is usually enough to get a clear idea of whether the models and proofs are sound. If a part of the paper is still obscure to the 3 reviewers, then clearly the paper lacks clarity and should be revised before being accepted.
The real problem in CS comes from experimental papers. Because reproducing experimental results is hard and sometimes not possible. Maybe you don't have access to the code (research codes are not always made available). Maybe you don't have access to the data (some data is proprietary and can not be shared). Maybe you don't have access to the machine (I think only Chinese nationals can get access to tianhe-2 for instance; I myself wrote paper about an experimental not-yet-released system). Even if you could reproduce the result, it could take month to reproduce. So in practice, you don't attempt reproduction of most experimental CS paper.
What you do is check for consistency. Does the result make sense? Does the technique provide an output that is coherent with expectation? If it doesn't, is the discrepancy explained in the paper? Is there a clear drawback to the method that is not mentioned in the paper? Do I believe that the paper contains all the information necessary for reproducing the results if I wanted to? That is about the type of things that you check. Some are pushing for including experimental results as supplementary material to experimental papers or to make experimental results more reproducible in general. (See the work of Arnaud Legrand or of Lucas Nussbaum for instance, but many other work on that.) The SC conference has now a reproducibility initiative to help with that.
The adversarial review that you are talking about happens AFTER publication. That is where the review peer review starts. It starts when dozens of master student or PhD student will compare their method to the state of the art. And that is when you will know what will stick and what won't. Because they will make the comparisons to different frameworks, on different machines, on different datasets.
I recently read George Dyson's book on the founding of the IAS.
The intellectual and academic caliber of people fleeing Germany (and nearby regions potentially subject to German influence) was unbelievable, yet the cowboy-era administrative end-runs to secure stipends for many of these people were off the charts.
Then America had its middle-class golden era between 1950 and 1980. If I've learned anything from my recent return to the history books it's this: this is the least economically normative period of the last 400 years. Short of handing Stalin the keys to the hydrogen bomb years earlier than he should have got them, it was a pretty amazing time for an empire unlike any that came before it.
Not so long ago, 10% of the population went off to university. Now 41% of women in Canada attain a bachelor's degree or higher. This is associated with an inevitable pressure to make the filters of meritocracy ever more fine-grained. So, of necessity, academia invents a cascade of credentialism mountain ranges. But there isn't enough legitimate signal to make this work, so we're forced to invent credentialist hoops.
Zoom all the way up to the TARP bailout of 2008. There's a large group of economists who think this was too large/unnecessarily/ineffective, another large group who think it was too small/absolutely essential/effective so far as it went (with maybe a sliver of fence-sitters eating porridge at the perfect temperature).
Prospective judgement is counterfactual. Retrospective judgement is counterfactual. In none of these matters are we afforded a virtual mulligan to run the simulation again.
If you're not Einstein, you're probably facing some kind of peer-group proxy measure, with the intervals between the scythe are having contracted from leap years to stutter-step quarters (who can summon the effort to leap any more, when the relentless ankle-blades never let up?)
Goodbye Erdos number two. Hello author position number 997 on a single published paper.
Meritocracy is this weird nodable concept. Sure sounds like a good idea (especially after a long stretch where things are anything but meritocratic). But merit turns out to be a terribly, terribly hard thing to implement well in practice, with acceptable consequence.
On TARP, we never achieved a retrospective standard of merit, all we got was two lousy, tribal camps locked into an egocentric crowing competition. I mention TARP mainly because the stakes here are in the trillion dollar range. Surely if incentive porn FTW, this would be the ultimate case study concerning the collective human incentive to get the individual incentives sorted out.
———
I found these two books exceptionally interesting to read back to back:
* Twilight of the Elites: America After Meritocracy â" 2012
* Coming Apart â" 2012
Same publication year, same thematic material, anode vs. cathode political perspective, but ultimately the same message: implementing meritocracy is far harder than it looks.
Basically the problem here boils down to not enough lions. Actual survival was once a pretty good proxy measure on who had it together, and who didn't. (Until the fated day came where your—former—best friend hid your Nikes.) Problem: the objective measure of the lion cull was a just a tad morally blind.
One of the problems with incentive porn is the notion that incentive gradients should be pervasive and perpetual: don't go to university, struggle to pay the rent; don't graduate at the top of your class, struggle to pay your student loans. Etc.
The other model is that you mill around aimlessly (sort of) until something clicks, and then you go off on a mad tear, when there's clearly something special you feel that you can achieve. If you succeed, you get perks (recognition, fancy jobs, fancy peers). If not, you're simply cast back into the milling pond.
A UBI is one way to provide a giant milling pond of opportunity.
It would surely