Slashdot Mirror
Does Journal Peer Review Miss Best and Brightest?
sciencehabit writes: A study published today indicates that the scientific peer review system does a reasonable job of predicting the eventual interest in most papers, but it may fail when it comes to identifying really game-changing research. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere (abstract). And papers that were rejected went on to receive fewer citations than papers that were approved by an editor. But there is a serious chink in the armor: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
Go science!
For those of you just joining us, peers given mod points hand them out to review comments posted here. CmdrTaco's site has seen a lot of controversies with this system in the earlier days, and M2 was invented to review the mod point decisions. Lots of discussion has been sorted by this system, and the crap found on other servers has been eliminated.
We cannot solve our problems with the same thinking we used when we created them.
-- Albert Einstein
Full disclosure: I found this at this web site.
-- I ignore anonymous replies to my comments and postings.
While I don't disagree with the conclusions, this summary equates paper "quality" with number of citations. High numbers of citations do not mean high quality, and is very field-dependent.
Quality can only be assessed by people reading the paper.
Every experiment which ends in a big bang is a good experiment.
People tend to cite papers from higher ranked journals more. In addition, said journals are higher ranked by search engines so the virtuous cycle continues. Thus this result is bogus.
The right way to do this study is to do a controlled study. Fortunately, this has been done in the recent NIPS conference (note that in CS conferences are more important than journals). See http://mrtz.org/blog/the-nips-experiment/ for details. Essentially, the noise is *huge*.
An excerpt:
"Relative to what people expected, 57% is actually closer to a purely random committee, which would only disagree on 77.5% of the accepted papers on average:"
University level:
I Just finished my PhD at Imperial College, a world leading institute, and all they care about is incremental research that industry will fund. New ideas just get thrown away until another university does the ground work and the IC jumps in with bigger wallets and then takes it on/steals it.
UK Government funding:
If you have a new or interesting approach forget about getting grant funding, you only get money in the UK if the work has already been proven to be successful. Quite literally the funding peer review of your grant can be rejected because you don't have the end answer (with a high degree of certainty) the research would give.
Original ideology of peer review is great, what is being practiced today (at least in the UK) is broken.
This is yet another statistical study that itself has flaws.
1.What was the selection process for the studies. The phrase "All 14 of the most highly cited papers in the study" implies that there were papers not in the study. Possible selection bias?
2. They do not go into why the 14 papers were cited so much and if any further research or refinement of the papers were done before they were accepted by other journals. Surface analysis of numbers can be manipulated to say anything.
3. They also say that it might be better to not have a review and just publish everything. This just means that everyone who reads the papers has to do the review. That is not practical. There are many papers that should not be published due to shoddy practices or malfeasance. Instead of trowing out the whole system how about looking at why the 14 papers were rejected and modifying the system accordingly.
4. The article does not give access to the study so we can review it. It is behind a paywall.
I would say the finding confirms that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
It's typical human politics and ideology at work. What would you expect from a large group of people, all with vested interests?
“He’s not deformed, he’s just drunk!”
I frequently had papers rejected as "not new" without citation and then accepted elsewhere where they told me on request that they checked carefully and found the content was indeed new and interesting. My take is that this may also quite often be "reviewers" that are envious or trying to compete by putting others down, not only ones that are incompetent. Fortunately, I had nothing stolen by reviewers (after they rejected it) but I know people that had this happen to them. As the peer-review system is set up at this time, the best and the brightest need longer to get their things published, need much longer to get PhDs and have significantly lower chances at an academic career as a result. Instead those that publish a lot of shallow and/or meaningless incremental research get all the professorships, while barely qualifying as scientists. The most destructive result of this is that many fields have little or no real advancement going on as new ideas are actively squashed. After all, new ideas would show how abysmally bad at science the current position-holders are. But this is not new. Cranks like Newton had research by others squashed or suppressed, because it was better than his stuff. This is also the reason most scientific discoveries take about one research-career-length to go into actual use. The thing is that the perpetrators of the status-quo have to retire before their misconceptions and mediocre approaches can be replaced. Exceptionally stupid, but all too human.
Personally, I am now in an interesting industrial job, and I still do the occasional paper as a hobby. But I would strongly advise anybody bright against trying for an academic career. Wanting to do research right reliably ensures that you will not be able to do an academic career at all. The core ingredients for a scientific career are mediocrity, absence of brilliance, hard work and a lot of political maneuvering. Oh, and you must not care about doing good science!
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Maybe they got cause and effect reversed even for those papers...
That's why when people claim breakthroughs in fusion, but don't present their results at the biggest plasma physics conference in the United States, you can pretty much dismiss them.
Where does this leave people outside the United States who claim breakthroughs in fusion but don't want to set foot on United States soil for fear of being the next Dmitry Sklyarov?
[Rejection due to reviewer envy] is also the reason most scientific discoveries take about one research-career-length to go into actual use.
Are you sure it isn't because a patent takes "about one research-career-length" to expire?
my interest isn't so high as to fork over cash for it.
If Slashdot users can't discuss the featured article meaningfully without having to buy something first, then the story should be considered a Slashvertisement, the product being copies of the featured article.
"unconventional... may be more prone to rejection"
Isn't that the definition of 'unconventional'?
For a minute, I thought the headline said that someone who is a journal peer reviews someone nicknamed "Miss Best and Brightest".
If I understand correctly, very highly cited papers were not published in peer-reviewed journals. Where were they published then? How did they became famous?
It is just journals doing their work when they bounce unconventional research, asking for further proof or clarifications. There is a lot of unconventional research out there, and while some may be the beginning of a breakthrough, most of it is not. Just look at atomic fusion.
Of course it makes it harder for really new and exciting things to get in the journals, it also keeps a lot of the crap out.
I went to grad school with a guy who was researching how to stream video over TCP. This was... 15 years ago maybe? Every where he went, his papers got rejected because "you dont stream over TCP. You are supposed to use UDP. Everyone knows that." Everyone knows that. No criticism of his work besides "everyone knows that." That's not the scientific method, yet it was nearly impossible for him to publish.
Fast-forward to 5 years ago and have a look at YouTube. Steaming video over TCP. The new hotness. He works there now thankfully.
Like many things in life, it's a popularity contest first, and a meritocracy second (at best).
"Unconventional methodology" is not.
Papers that don't use sufficiently rigorous methods should be rejected, regardless of their conclusions - even if those conclusions eventually turn out to be right. It's the only way to have any confidence about the research. If the authors are so sure of their results, they should do them more carefully, and submit again.
Far too often, rejections are taken as evidence of cronyism or groupthink (usually by those whose beliefs are contradicted by established science), when it's simply obvious flaws in methodology. When your methods are bulletproof, only then you can expect with confidence to pass review.
Why would anyone engrave "Elbereth"?
Maybe those 14 articles were cited more because they weren't buried under the paywalls imposed by the three "elite" journals? Scientists could actually get their eyes on these articles without paying a steep subscription or per unit cost?
Maybe the elite journals actually hinder the exchange of information and ideas that science needs to move forward? Nah! Can't be!
Left MS Windows for Linux Mint and never looked back!
Vote for Bernie in 2016!
The process of getting rejected from a journal may lead to improvements in a paper, or lead the paper to be submitted to a journal that's more tier-appropriate than one's first choice. Both can be very healthy.
For every problem, there is at least one solution that is simple, neat, and wrong.
Slashdot reader for 10 years and this is my first post because this encapsulates my life.
1.) work at major university ( > 25k students)
2.) learn databases
3.) write app to allow logistic regression as a service for stroke doctors to determine certain outcomes
4.) wait a year for IRB approval to get access to real data after I show POC
5.) get paper published
6.) make no more money.
7.) quit grad school, quit research become a software engineer and get twice the money.
Everything you might want to do in grad school meets the abstraction of better software. I chose to program my life to an interface and not an implementation. It has worked out well for me.
I agree with you, but will point out that TFA is discussing something else. Science is not about having a majority understand, or be "interested". This means that real science tends to be ignored. Sometimes real science goes against the grain, and that science is never peer reviewed either. Meaning that the same status quo is maintained even if it's not scientific.
Slashdot's biggest topics are not "Science" but politics. This is an area which is extremely complex and there may not be a definitive answer to a problem.
That said, I can name several topics where one side will be censored by moderation. I can do this with a science topics as well as political topics, and anyone that's been here for a while knows the same thing. Slashdot can do this because it's not a scientific paper publishing site, it's an OP/ED site primarily focused on topics geeks like.
"Science" today is in trouble, don't get me wrong. Watching bogus papers get published and treated as fact has become all too common. Some have been outright spoofs, but others are published for political agendas. The latter is extremely troubling for society..
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
The easiest way to get in is to submit mediocre, recycled work. Even AAAI has been trying to mitigate this by introducing a track for papers that were already well-received at other conferences.
The principle is simple, really. If PhD students are rating the work, and are all obsessed with last year's cool thing, then they will accept papers that are about that cool thing. It breeds mediocrity.
There are two further effects not discussed.
1. Crazy multiplies faster than normal.
2. Peer reviewers rejecting research, then publishing under their own name.
According to the Center for Responsive Politics, [David L.] Cohen expanded [Comcast's] lobbying team from 31 bodies in 2002 to 103 in 2009, when the merger [with NBC] was announced, and increased its lobbying spending more than five-fold over the same period.
This is what freedom of speech means.
> Considering the papers were eventually published anyway
That seems to be a clear cut case of biased sampling. What you heard about is all there is?
http://en.wikipedia.org/wiki/S...
This is what you would expect from sub-genius judging genius.
E Proelio Veritas.
http://en.wikipedia.org/wiki/Streaming_media
QUOTE ...
The term "streaming" was first used in the early 1990s as a better description for video on demand on IP networks; at the time such video was usually referred to as "store and forward video",[1] which was misleading nomenclature.
During the late 1990s and early 2000s, Internet users saw:
greater network bandwidth, especially in the last mile
increased access to networks, especially the Internet
use of standard protocols and formats, such as TCP/IP, HTTP, HTML
commercialization of the Internet.
UNQUOTE
Hmmm.
The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals
That is the way it should be. It is not a bug, it is a feature. Extraordinary claims demand extraordinary evidence. The signal to noise ratio is very poor when it comes to unconventional research and findings. For every deserving paper made to jump through the hoops, there are 100 papers sent to the dust heap of history very deservedly.
Think about it, Einstein was a patent office clerk. Srinivasan Ramanujam was a clerk on Madras Port Trust. Eddington destroyed the Chandrashekar on the first international presentation Chandrashekar made [*]. That paper the defined what later came to be called "Chandrashekar Limit" for black holes got Nobel Prize. But on the conference in 1935. It took 15 years before that paper was noticed and gained prominence. Science found them and made heroes out of them. If the unconventional research has any merit, it will jump through the hoops, become the accepted research and it will be highly cited too.
[*] Apparently Chandrashekar had referred to a paper by Eddington's arch rival, without being aware of the rivalry between them. That irritated Eddington enough to have a grudge against Chandrashekar. Not realizing these undercurrents, Chandrashekar, young and quite naive, freely shared all his research work with Eddington for weeks prior to the conference. All the while Eddington was gathering information silently to destroy Chandrashekar's presentation publicly in the upcoming conference. Eddington at that time has Himalayan reputation as an astrophysicist. He had confirmed the predictions of Einstein's theory of relativity by direct observation during a solar eclipse. In retrospect, today, Eddington is seen more as a competent astronomer, like Tycho Brahe. But when it comes to astrophysics the prize goes to Chandrashekar (and Kepler, not Brahe). Proving, if you have the merit, science will find it.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
That people take this article as gospel and don't question it's premise. When you are comparing the peer review process to the NBA, you have a problem.
And why would Journal Peer... review her? ;)
What they need is for reviewers that screw up and reject good stuff or accept bad stuff, to have their statistical weight reduced.
Research published in lesser read journals gets fewer citations than elite ones. If you are writing your own paper, you might want to drop a few big names rather than some obscure ones to gain credibility. The initial reason for the elite journals' editors rejection may have little to do with an in-depth analysis of the research. So it goes on to some lesser publication.
From TFA:
It's a sign that these editors making snap decisions really quickly still have a nose for what quality is and isn't
That could be bad logic. The initial rejection by the editor of an elite journal may have little bearing on the quality of the research. But it does influence its subsequent exposure to the scientific community. That factor alone may cause it to be less cited.
I would expect that the study done on this topic does make some attempt to correct for this. But I can't be bothered to follow up on it as a new paper by Bennett Haselton on the distribution of ice has come across my desk and requires my immediate attention.
Have gnu, will travel.
Yes peer review is indeed flawed.
So what?
The article even shows by the number of cites of articles rejected by top tier journals that it is self correcting.
Well, yes.
When we build distributed systems, the need to setup a distributed consensus algorithm is appearing in front of us, time and again. Leslie Lamport (of LaTeX & Time-Clocks fame) came up with a novel algorithm during early 90s about to solve this is a very competitive way (Paxos is its name). Sadly, the algorithm remained shunned for a number of years, due to rejection via the very same channel in which it was eventually published many years later. If you realise the immediate practical impact of that algorithm and what an 8 years delay means in the world of CS, and the cost putting all these together, the result is staggering and sobering at the same time.
So, yes, let's now all peer-review this statement: "peer-review systems are imperfect and provide no guarantee for any certain quality result".
Peer review is merely a compromise to increase throughput of papers, which are relatively median and more easily digestible, because this is what keeps the academia salary system in good lubrication. It provides no level of assurance that the most impactful paper gets noticed first, neither that it receives sufficient feedback in order to improve upon original concepts. In sort, human intellect won't be easily replaced via a procedural setup, yet.
...is published in a journal (in)famous for its lack of a peer review process. Makes complete sense.
If you can't beat them, embrace and extend them.
I got first post!!!! Yipppeeeee!!!!
...but you 'did' get FRIST PSOT!
Ah, right. What would you think of peer-reviewers who clap their hands for an article critical of peer-review process? Who's going to pay for this? ;-)