Slashdot Mirror
Study Attempts To Predict Scientists' Career Success
First time accepted submitter nerdyalien writes "In the academic world, it's publish or perish; getting papers accepted by the right journals can make or break a researcher's career. But beyond a cushy tenured position, it's difficult to measure success. In 2005, physicist Jorge Hurst suggested the h-index, a quantitative way to measure the success of scientists via their publication record. This score takes into account both the number and the quality of papers a researcher has published, with quality measured as the number of times each paper has been cited in peer-reviewed journals. H-indices are commonly considered in tenure decisions, making this measure an important one, especially for scientists early in their career. However, this index only measures the success a researcher achieved so far; it doesn't predict their future career trajectory. Some scientists stall out after a few big papers; others become breakthrough stars after a slow start. So how we estimate what a scientist's career will look like several years down the road? A recent article in Nature suggests that we can predict scientific success, but that we need to take into account several attributes of the researcher (such as the breadth of their research)."
Ah, nah, what was I thinking. Whether someone produces future scientists or students who know science, doesn't matter one bit. Let's continue to fetishize publication, and the system of duchies it rests on!
Not Hurst.
I am interested in how anyone would predict the successfull contributions of people who have been hiding in the patent office for several years being denied promotions for their lack of credentials.
Exceptions are exceptionally hard to predict.
Even if they start successfully predicting individuals careers, wouldn't the system eventually break down since professors would probably change based on the results of the prediction?
"Publish or Perish" is a great way to put it. In the current world one has to to go above and beyond to reach people through various social media and internet channels to get their information out. People now find the information that they want to read instead of just trusting the media to find it for them. Well said :)
Yes if your had the top thesis advisor, went to the best schools and work in a lab with good funding you do well. What a surprise! This would probably ignore patent clerks that discover Relativity however. I recall one paper that claimed to be able to predict your whereabouts by some kind of cell phone info. I can predict it without any data. %90 of the population spends %90 of the of their time within 1/4 mile of their place of residence or employment/school etc. Wow that was hard. Can I get a grant for that?
The summary is pretty inaccurate. The h-index was proposed by Jorge Hirsch, not Jorge Hurst. Rather than give a vague description, why not simply provide an exact definition? The h-index of a scientist is the largest number h, such that he/she has at least h papers each of which have received h or more papers.This is easier to understand if you look at the picture in the Wikipedia entry for h-index.
....wait a minute............ I see what the author is trying to do.......... lol
This reminds me of some research about artists which found that you could divide the most 'successful' artists into two rough categories: those who made a big splash right away and those whose classic work did not emerge until much later.
http://www.nber.org/papers/w8368
Even if they start successfully predicting individuals careers
That's a very big 'if'. For example in experimental particle physics we publish in large collaborations. My h-index score is better than Dirac's but Dirac was a far greater scientist than I! (and that is just the theory/experiment difference in the same field!) Similarly I have papers in a large variety of journals but so have the majority of us in the field. In fact to accurately assess experimental particle physicists you need to rely more on what they do inside their collaborations than the publishing record so it is hard to see how any system which relies on publication record will work.
This reminds me of a relevant comic - where once this system is worked out, scientists will be trying to game the system:
http://www.smbc-comics.com/index.php?db=comics&id=1624#comic
It should be noted that the usefulness of h-indices varies from field to field. For example, in various branches of pure mathematics, a heavily-referenced paper is one that, maybe, garners 25 to 100 citations. In applied mathematics and certain subsets of statistics, the threshold would be a factor of magnitude larger.
Also, as a preference, I tend to ignore metrics like h-indices when evaluating a researcher, as they provide very little evidence for his her her capabilities, let alone the quality of the work.
To elaborate, at least from my own experiences, in certain portions of applied mathematics that bleed over into computer vision, machine learning, and pattern recognition, I've seen papers that are relatively mathematically prosaic, but possibly easy to understand or where the code is made available, be heralded and heavily cited for a period. In contrast, I've come across papers that provide a much more sound, but complicated, framework along with better results, possibly after the topic is no longer in vogue, and go unnoticed or scarcely acknowledged.
In a different vein, there are times when a problem is so niche, but nevertheless important to someone or some funding agency, that there would be little reason for anyone else to cite it.
Touching on an almost completely opposite factor, there are times when the generality of the papers, coupled with the subject area, artificially inflates certain scores. For instance, if a researcher spends his or her career developing general tools, e.g., in the context of computer vision, things like optical flow estimators, object recognition schemes, etc., those papers will likely end up more heavily cited, as they can be applied in many ways, than those dealing with a specific, non-niche problem, e.g., car detection in traffic videos. Furthermore, the propensity for certain disciplines to favor almost-complete bibliographies can skew things too.
Finally, albeit rare, are those papers that introduce and find the "best" way to solve a problem that no other discussion is warranted.
Unless you need publishing cred for your job, I can't see why anyone would bother going that route.
It's only really useful for tenure in a teaching position, and *slightly* useful for other job prospects. If you're not pursuing either of those, why bother?
1) Your information is owned by the publisher, you can't reprint or send copies to friends.
2) You make no money from having done the work.
3) The work gets restricted to a small audience - the ones who can afford the access fees
4) It's rife with politics and petty, spiteful people
5) The standard format is cripplingly small, confining, and constrained.
6) The standard format requires jargonized cant to promote exclusion.
A website or blog serves much better as a means to disseminate the information. It allows the author to bypass all of the disadvantages, and uses the world as a referee.
Alternately, you could write a book (cf: Quantum Electrodynamics by Feynman). There's no better way to tell if your ideas are good than by writing a book and submitting it to the world for review.
Alternately, you could just not bother. For the vast majority of people, even if they discover a new process or idea publishing it makes no sense. There's perhaps some value in patenting, but otherwise there's no real value in making it public.
Today's scientific publishing is just a made-up barrier with made-up benefits. In the modern world it's been supplanted by better technology.
What if this person and their articles are cited as an example of a moron?
Yeah, I know; peer reviewed articles tend not to drag colleagues through the muck, so to speak. Citations are made to build your own case, not so much to cut others down.
Have gnu, will travel.
That sounds like exactly what Google does with its pagerank search algorithms. Though I suspect Google is much, much further along in thwarting people's attempts to game the system.
... quality is now going to be measured by popularity?
I see the little narcissists are growing up and setting policy now...
The h-index of a scientist is the largest number h, such that he/she has at least h papers each of which have received h or more papers. [nb. excluding self-citations].
And that definition shows that the results of this paper are in fact so trivial as to be meaningless.
h-index is cumulative. Their results were "Five factors contributed most to a researcher’s future h-index: their total number of published articles to date, the number of years since their first article was published, the total number of distinct journals in which they had published, the number of articles they had published in “prestigious” journals (such as Science and Nature), and their current h-index."
Are any of these factors even slightly surprising? h-index is about citations. They discover that, wow, the more articles scientists have in more journals, and more widely read journals, the greater the number of times they get cited. That's not news.
They continue "Not surprisingly, the best indicator of a scientist's h-index one year in the future was their current h-index."
"Not surprisingly" is rather an understatement, since the h-index one year in the future is their current h-index, plus a small change for the citations they get this year. It's like saying the best predictor of your bank account next year is your bank account this year (except bank accounts can go down).
It would have been slightly less trivial to predict the change in h-index. But even there, it's prettyobviously the same factors.
http://www.geoffreylandis.com
This is going to be slightly off-topic, but it something I have been mulling in my head
Rating research of people who supposed to be also teaching puts them firmly in publish or perish mode and that's not good for students. Universities are both research institutions and teaching facilities, really they should choose one. There was a time you needed all three together because of the cost and learning efficiency, however, that time has come and gone as the number of students entering all three segments has increased dramatically over the last 100 years. Many researchers who happen to be professors don't like to teach undergraduates and many undergraduates would rather have a professor who knows them and is open to them rather then learning in a lecture hall from a TA. There is no need to subject these two against each other anymore. Lifting the grunt teaching will free our researchers to explore and continue to push their craft in addition institutions focused on undergraduate learning will deliver a better, more hands-on education. Face reality, very little of research gets into undergraduate education and the longer we hold up this charade the worse this process will get.
The East Anglia incident confirmed peer-reviewed studies are squewed by political views.
In "Future impact: Predicting scientific success", Acuna, Allesina, Kording predict the future h-index of scientist using their current h-index, the square root ofnumber of articles published, years since first publication, number of distinct journals, and the number of articles in top journals. They vary the coefficients of a linear regression with the number of years in the forecast and note that, in the short term the largest coefficient is (not surprisingly) the scientist's current h-index, but in the long term, the number of articles in top journals and the number of distinct journals become more important for the 10 year h-index forecast. They achieve an $R^2$ value of 0.67 for neuro-scientists which is significantly larger than the $R^2$ using h-index alone (near 0.4).
Additionally, they provide an on-line tool you can use to make your own predictions. (Click here to see this comment rendered in Tex.)
There should be some rule on /. about posting articles that you can only see if you pay someone money...grrr.
It's too damned hard to succeed as a scientist. These men and women are already the best of the best, and they've chosen a field where you need to be the best of the best of the best to succeed. They've already gone through trials to get where they are, but it's still not enough to guarantee a permanent position or a decent wage. There's so much pressure and the competition for tenure is so tough. How is one supposed to distinguish oneself when everyone is a genius and a workaholic? It's true that competition can sometimes bring out the best in people, but at some point, people are just going to say, "I'm fed up with this game and I'm not playing anymore," and switch to a more lucrative job in the finance industry or a simpler cozy job which gives them time to spend with their family.
No, sir/mme, you are dead WRONG!
If the student takes a sincere interest in the subject matter, and came from a background where he/she knows that working hard will help avoid:
1) A low paying career
2) A meaningless job
3) A lifetime of misery
Then that student either forces his or her own self to either show a committed interest in the field, or finds one where he or she can naturally develop such an interest.
You seem to be of the impression that lectures should instill a DESIRE in students. My background is that excellent teachers challenge students, and expect them to know the practical aspects of a career instead of just the theoretical and the simple facts needed to pass a test. Real profs provide students that know more than the rest. Sadly, in North America, no child left behind means no child gets ahead.
Background info: I've been pulling in $100k+ per year since 24, but went into my undergrad on a basic scholarship from poverty. Now working with my biomed/computer B.Eng. and M.Eng. systems engineering degrees, and making more than my profs since I was IN my undergrad. So no marketing/handout/BS comments are applicable here.
...cynical about a career as a Scientist/Academic Researcher.
IMHO, there is absolutely no legitimate way of quantifying "success of a scientist". It is down to: 1) how a particular study stands the test of time; 2) extended studies that reassures the accuracy of original results, will make the original investigating scientist a true success. Best example I can provide is, Prof Higgs... even Prof Einstein.
All these 'publish-and-perish' claptrap will only do is: dilute the quality of academic research, discourage collaborations, proliferation of academic malpractices/dishonesty, and perhaps drive-away all the truly passionate scientists/researchers-alike from active research in to obscurity.
I finished my PhD last year in EE/CS. Personally, I did enjoy the pain/pleasure of doing research and the campus life in large. However, about half way through my graduate school, I increasingly felt hopeless being a researcher in academia. I went with the good intention of becoming a down-to-earth true-blue scientist/researcher. But the environment I worked was too toxic to keep to my humble wishes. I just couldn't stay there and keep doing research with a clear conscience knowing the academic dishonesty going around, and wrong-doers getting ahead in the "academic rat race" while I am getting scrutinised constantly for not being productive as them. So I did the bare minimum to defend my thesis, and got out on time with a sane mind to start a career in the industry as a software developer.
I regret about my decision in many ways. But I am happy that I do not have to sell-my-soul to cling on to my current position. Plus, I foresee a much better career path now compared to academia (promotions, ability to move to different institutions/career paths); and finally, got decent pay-cheques to enjoy life like I never did before.