Slashdot Mirror
Scientists Don't Read the Papers They Cite
WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."
As shown earlier the wright brothers were not the first in flight and had not done their research.
You can't judge a book by the way it wears its hair.
I'm almost tempted to say that this is a side-effect of all those teachers who said 'I want at least 10 references and a 5 page paper'. At least, I can't think of any serious reason why, even if someone was just publishing fluff, they'd need to bulk up the references with irrelevant ones. The only other thing I can immediately think of is that a reference becomes somewhat standard, so they use it for something they learned and forgot where they learned it from (you can't exactly say [11], 11. Professor Ragan's Astrophysics 521 class or [12], 12. Two dozen vaguely remembered textbooks). Even then, I suppose its bad form not to find some reference with the relevant information just to prove you're not making it up (yes, pi IS 3.1415....).
Another major problem with research papers is the "dissappearance" of those who actually do properly cite their sources.
/ web_citations.html)
As many of you know, the Internet is a great research tool these days. But unfortunately, it's too dynamic for the research world. "Most URL references [stand] more than a 50 percent chance of not existing after only six months." (from a Cornell study at http://www.news.cornell.edu/chronicle/00/12.14.00
I don't care as much if some researcher only reads parts and pieces of papers that they cite, but when the entier papers dissappear, that's a much bigger problem.
"The study, using term papers between 1996 and 1999, found that after four years the URL reference cited in a term paper stood an 80 percent chance of no longer existing."
Anyone who see how sme articles are written, knows perfectly that "bibliography" is usually created as a "necessary evil". Most scientific articles are done basically in the light of several "obligatory templates": abstraction, main article, citations, bibliography and notes. Frequently, real authors are not the ones you see first in the header of the article but someone in the end of it. Also, sometimes, certain people do the most flagrant plagiates out of the work of their students or co-workers.
What I call "academical science" is full of huge problems, which sometimes reach the level of flagrant falsifications and demagogic manipulation of facts. While not being a scientist per se, I have seen how these things pass the limits ethics and moral in such a thing like Mars. There is one scientist who tragically died in a very strange situation. Apart of the conditions of the tragedy, there was one big "authority" on Mars who lied with all his teeth about the work of his deceased colleague. Frankly, it was shocking to see how this guy flagrantly and demagogically "reinterpreted" the intentions of the scientific work of his colleague. One should note that both guys were highly considered in the community. However, they were adversaries. One died, the other became a big scientific authority on Mars. One of the reasons, was that he made a lot to desmise the works that went against his theories
And that's how it all begins. You have academic staff being rated more for how much they publish than how much they teach; how much time do they have, really, to *teach* their assigned students, much less grade their assignments and papers? When grading their papers, how much time do they really have to pursue all the references etc.?
:-). You did NOT want to be one who gets an email saying something along the lines of "I've never heard of this. Show me. "
.
I had one professor who would randomly check up on various references in papers submitted to him (the joys of statistical sampling
Knowing that there's some (realistic) percentage of being found out, presumably most people would be careful - but nonetheless since he cannot possibly check EVERYTHING, some people will be tempted to try their luck. At least some (most?) will get away with it. And for those who know their profs won't be hunting down everything... what's to stop them?
And like most crimes, you just keep doing it and doing it... until (if) you get caught (I don't think Winona Ryder's *only* "shoplifting experience" was the one she went to trial for). So presumably at least a percentage of those doing scientific research had had undergrad + postgrad experiences of "getting away with it". Could get to be a habit.
Heck, if even reusing *faked* graphs in multiple papers can be gotten-away-with...
When I wrote my thesis for MS in Computer Science, my advisor strongly suggested that I include several references to his previous work, the work of several of his past students and a Professor at another school that would write reviews of his books (he would review the other Prof's books). All of this occurred during the final chop and two weeks from graduation. If it was up to me, I would not have included any of these references. But I was not the one signing off the last two years of my life as complete.
The funny part is that I received the largest portion of help from a couple of Sun engineers who were able to get me through some code which my advisor could not and except for the acknowledgement, their contribution was poorly documented (at least in my mind, not the advisor's).
So, if you read my paper, you would think that I am an idiot because some of the referenced work is so basic and at other times a super genius because the code was assisted by some great programmers (after all, how many people read the acknowledgements).
Copying a reference string doesn't mean that you haven't read the paper in question. To take a personal example of what I've done:
1. Find a reference to a paper which looks interesting.
2. Walk down to the library, remembering that you're looking for Bob's paper about bars in the Journal of Foo.
3. Arrive in the library, find the paper, read it, decide it is important.
4. Walk back to computer, copy out reference string.
It's quite easy to look up a paper from a slightly-wrong reference, and as long as the reference is close to correct, it's fairly easy to not realize that the reference was wrong in the first place.
Tarsnap: Online backups for the truly paranoid
Anyway, this comic seems appropriate.
To support the view that observations got better and better, requiring more and more circles, you'll probably find most of these sources citing a book by J.L.E. Dreyer, written in the beginning of the previous century, but it exists in a few editions published later.
But Dreyer says the opposite:
Basically, if these people had actually read Dreyer, we wouldn't have had to struggle with this myth any longer. Of course, there's a lot more to this story than this, but I don't have time to write it now... :-)
Employee of Inrupt, Project Release Manager and Community Manager for Solid
The root problem is papers are a form of scientific social capital. And when people think you are well read, your paper is worth more. I worked in a research facility where grad students were literally held hostage so they could produce more papers for the professors to take credit for. One student came to use with his masters and was held *7* years for his PHD. It was getting so bad the graduate department was *forcing* the director to graduate students by saying, "So and so has to leave by the end of the year -- with or without his degree." (and after 7 years, who could blame them)
Add this to an already paper obsessed culture, and you have a serious problem.
Religion is a gateway psychosis. -- Dave Foley
Doesn't this point to a failure of the peer review process? Aren't the reviewers bothering to check whether the references are relevant, and for the ones that are, whether the paper actually interprets and builds on the prior work in a reasonable manner?
As far as twisting up evidence, yes, this does happen. But most definitely not 100% of the time. How was the solar neutrino problem ever discovered in the first place? How was a re-evaluation of the cosmological constant initiated? These (and many other ideas) were brought forth not because someone wanted their ideas to be put forth, but because their hypotheses did not match the experimental data! It most definitely is not bullshit. AFAIAC, science is still the most altruistic of professions, not to mention one of the most self-sacrificing.
I wanted to take up the point in the article that many researchers are more interested in publishing than in solving the issues they investigate. I'm going to preface this by stating that I'm a psych. major and, as such, do not have much knowledge of the specifics of other fields, but I assume their requirements are similar.
In university settings, it is all about how many papers you have published. When a professor is first accepted to the faculty of a university, he/she must "publish or perish" for the first 5(+[?]) years. If you do not publish often enough in those first years, you are not retained. Things get better after you get tenure; you are not required to publish as often. So, it should not come as too great a surprise if people are more interested in publishing than solving the issues.
I personally think the requirements of universities should change so that we are not searching through a glut of papers, all saying many of the same things (or close enough). I am more concerned with the falsification of data, which totally throws everything off, than with a tendency to publish papers that don't necessarily solve the issues, which makes finding relevant research difficult but shouldn't substantially hurt the future of the field.
"I swear I won't break you if you let me take you where the willows never weep" -- Switchblade Symphony
This is a serious problem in a lot of disciplines, though I've heard of a rather elegant solution to the problem that's now become common (if informal) practice. The solution is that when a student thinks that he's done enough to justify getting a PhD, he starts applying for jobs that require a PhD. When somebody is willing to offer him one, that's proof that an outsider views his accomplishments as being worth a degree and his advisor has to let him write up his dissertation. It serves as a very effective independant outside check on the system.
There's no point in questioning authority if you aren't going to listen to the answers.
The real problem here is inherent in the academic system. Research faculty are in a situation where they are being judged by the amount of papers they put out, and not on the quality or the potential of their work. This leads to unscrupulous individuals doing "whatever it takes" to get ahead.
What needs to be done is to reform the way merit is assigned in academia. Research funding and tenure need to be allocated based not only on the quantity of publications but on other factors which may be harder to measure, factors that would be better indicators of the value of their research.
A somewhat related issue is that more and more private sector funding is flowing into universities and along with that funding comes the expectation of a quick return on investment. This creates more pressure to pursue short-term goals with little long-term impact on the field of study.
Taken together, US scientific research is destined to fall behind and stop making new breakthroughs. Seemingly, the only apparent solution to this is to increase the amount of public funding available for basic research. It would seem, though, this is not likely to happen given the current regime in Washington. A more likely outcome will be that our scientific institutions will all be doing R&D for the big corporations in the near future.
I think people forget that the Hard Sciences are made up of people, same as the social sciences, and also have the usual problems associated with using people to try to get stuff done. (Although I'm not sure I'd put not reading all of the papers you site real high on the list - if all you're after is one point in a long and complex paper that seems like a fairly inefficient use of time. Some of these papers are HARD to understand.)
What gives the Hard Sciences the right to that title is that, eventually, someone will root out the bull that someone else has published, brand it as such, other people will check it and agree, and it dies. You can prove someone WRONG. Try that in the social sciences - has anyone ever heard of a huge scandal where someone faked results in the social sciences? They would get in trouble if they didn't do the studies and were found out, but can you prove that they cheated just by taking their conclusions, working with them, and crying foul when something doesn't work? In the Hard Sciences, you can. That's what makes them so strong and practical.
Not that Social Sciences are worthless, mind you. It's just that BS seems to be a lot easier to get away with there. Sort of like in English class, when we were supposed to get the meaning out of a book. I never get the meaning the author's trying to convey (or at least what they say later he/she was trying to convey), but I wrote down something and got a good grade. Because how could they prove my thinking about the book wrong? I think the social sciences have a little of that problem in them somewhere. Controlled experiments are really tough to do, so you run into problems.
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
>It's not profit. Academic scientists have no monetary incentive to publish papers.
...
Bullshit. The monetary incentive is TO KEEP YOUR JOB (i.e. get tenure).
>In academia, you are pretty much your own boss.
Again, bullshit. Maybe after you get tenure you are your own boss. But before that you spend considerable time answering to your department chair, the dean, the provost, the university president, funding agency program managers,
In atmospheric science everyone worships something called "baroclinic instability" and every paper on the subject dutifully references the two papers by Eady and Charney that started it all.
But for the Eady paper, I've seen it referenced as "E. T. Eady (1949)", "E. J. Eady (1949)", "E. A. Eady (1947)", etc. And since the journal's so old, it took me a while to figure out the guy's real name. I guess very few people have read the actual paper.
Yes, that is the tinfoil hat explanation.
Now try this one: authors are human beings who make typos. They cut and paste erroneous references because they don't want to waste time retyping the reference. They read articles from the online versions of journals, and sometimes the citation info provided online is incorrect or altogether absent.
One thing that does disgust me is the explosion in the number of footnotes associated with a typical academic paper these days. I recently submitted a paper with a not-particularly-important result to a not-very-important journal, and the paper had forty-one footnotes. (Most were added by my coauthor.) If you visit an mature university library, pull out a copy of an older periodical. Copies of Philosophical Transactions from the nineteenth century are a delight to read. I read a paper by Kelvin from (IIRC) 1807, and it had seven references. Seven!
The growth of massive, searchable databases of papers (eg Medline) has led to many more footnotes per paper, and many more potential typos. For the record, the paper I mentioned above contained at least three errors in the footnotes that were noted and corrected by the journal publisher. Perhaps New Scientist should be writing a scathing expose on the decline of proofreading and rise of profligate namedropping in footnotes.
~Idarubicin
This wasn't a hick lawyer either.. She was senior partner in one of the largest law firms in BC, had a reputation for never losing a case, and became a judge a year or so later (Judgeship is more of a peer-review process in Canada than it appears to be in the US).
This left me with a feeling that lawyers don't pay as much attention to their authorities as they could. Probably more so than scientists do with their citations.
OS Software is like love: The best way to make it grow is to give it away.
Sure, but I assume you're going to read it, right?
On the large experiments (500+ authors) I have worked on there has been a review committee that checked over the paper in detail of 30 or so people.
Sure, some people are going to be more involved than others in the nitty gritty of the writing. But something's really wrong if there are authors who don't even read the paper that's going out with their name on it.
Find free books.