Scientists Don't Read the Papers They Cite

WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."

Comments

Not the only problem

[SteweyGriffin]

Another major problem with research papers is the "dissappearance" of those who actually do properly cite their sources.

As many of you know, the Internet is a great research tool these days. But unfortunately, it's too dynamic for the research world. "Most URL references [stand] more than a 50 percent chance of not existing after only six months." (from a Cornell study at http://www.news.cornell.edu/chronicle/00/12.14.00/ web_citations.html)

I don't care as much if some researcher only reads parts and pieces of papers that they cite, but when the entier papers dissappear, that's a much bigger problem.

"The study, using term papers between 1996 and 1999, found that after four years the URL reference cited in a term paper stood an 80 percent chance of no longer existing."

That's usual...

[Ektanoor]

Anyone who see how sme articles are written, knows perfectly that "bibliography" is usually created as a "necessary evil". Most scientific articles are done basically in the light of several "obligatory templates": abstraction, main article, citations, bibliography and notes. Frequently, real authors are not the ones you see first in the header of the article but someone in the end of it. Also, sometimes, certain people do the most flagrant plagiates out of the work of their students or co-workers.

What I call "academical science" is full of huge problems, which sometimes reach the level of flagrant falsifications and demagogic manipulation of facts. While not being a scientist per se, I have seen how these things pass the limits ethics and moral in such a thing like Mars. There is one scientist who tragically died in a very strange situation. Apart of the conditions of the tragedy, there was one big "authority" on Mars who lied with all his teeth about the work of his deceased colleague. Frankly, it was shocking to see how this guy flagrantly and demagogically "reinterpreted" the intentions of the scientific work of his colleague. One should note that both guys were highly considered in the community. However, they were adversaries. One died, the other became a big scientific authority on Mars. One of the reasons, was that he made a lot to desmise the works that went against his theories

Idiots.

[cperciva]

Copying a reference string doesn't mean that you haven't read the paper in question. To take a personal example of what I've done:

1. Find a reference to a paper which looks interesting.
2. Walk down to the library, remembering that you're looking for Bob's paper about bars in the Journal of Foo.
3. Arrive in the library, find the paper, read it, decide it is important.
4. Walk back to computer, copy out reference string.

It's quite easy to look up a paper from a slightly-wrong reference, and as long as the reference is close to correct, it's fairly easy to not realize that the reference was wrong in the first place.

It's not a new thing

[KjetilK]

Well, you've all heard that the geocentric world system was abandoned because it was required to add more and more epicycles for the model to fit observations, right? That's what many text-books say, that's what Thomas Kuhn says, and that's what Encyclopedia Britannica said up to recently.

To support the view that observations got better and better, requiring more and more circles, you'll probably find most of these sources citing a book by J.L.E. Dreyer, written in the beginning of the previous century, but it exists in a few editions published later.

But Dreyer says the opposite:

[...] One looks in vain [in Alfonso's work] for any improvement over Ptolemy; on the contrary, the low state of astronomy in the Middle Ages is nowhere better illustrated.

Basically, if these people had actually read Dreyer, we wouldn't have had to struggle with this myth any longer. Of course, there's a lot more to this story than this, but I don't have time to write it now... :-)

Re:What about referencing one's own stuff?

[Monkelectric]

I agree, citations is the scientific equivalent of "name droping".

The root problem is papers are a form of scientific social capital. And when people think you are well read, your paper is worth more. I worked in a research facility where grad students were literally held hostage so they could produce more papers for the professors to take credit for. One student came to use with his masters and was held *7* years for his PHD. It was getting so bad the graduate department was *forcing* the director to graduate students by saying, "So and so has to leave by the end of the year -- with or without his degree." (and after 7 years, who could blame them)

Add this to an already paper obsessed culture, and you have a serious problem.

Peer review

[Eric Smith]

Doesn't this point to a failure of the peer review process? Aren't the reviewers bothering to check whether the references are relevant, and for the ones that are, whether the paper actually interprets and builds on the prior work in a reasonable manner?

Re:even if they do read other's work...

[DrLudicrous]

I have to disagree strongly. When one is doing basic science research in an academic setting, 95% of the time there is no chance for profit. If one lies, there is the risk of being caught, as evidenced by the Bell Labs fraud; perhaps this is even more likely to pass in an industrial environment where profit can be a motive behind "[making] up shit and [lying]".

As far as twisting up evidence, yes, this does happen. But most definitely not 100% of the time. How was the solar neutrino problem ever discovered in the first place? How was a re-evaluation of the cosmological constant initiated? These (and many other ideas) were brought forth not because someone wanted their ideas to be put forth, but because their hypotheses did not match the experimental data! It most definitely is not bullshit. AFAIAC, science is still the most altruistic of professions, not to mention one of the most self-sacrificing.

Re:What about referencing one's own stuff?

[rgmoore]

This is a serious problem in a lot of disciplines, though I've heard of a rather elegant solution to the problem that's now become common (if informal) practice. The solution is that when a student thinks that he's done enough to justify getting a PhD, he starts applying for jobs that require a PhD. When somebody is willing to offer him one, that's proof that an outsider views his accomplishments as being worth a degree and his advisor has to let him write up his dissertation. It serves as a very effective independant outside check on the system.

Lawyers don't either

[Black Copter Control]

I once represented myself in the BC Court of appeals. The lawyer I was up against cited some authorities to support her case. I don't think that she expected me to actually read the authorities because, when I did, I found that the authorities, taken as a whole supported my position more than they did hers.

This wasn't a hick lawyer either.. She was senior partner in one of the largest law firms in BC, had a reputation for never losing a case, and became a judge a year or so later (Judgeship is more of a peer-review process in Canada than it appears to be in the US).

This left me with a feeling that lawyers don't pay as much attention to their authorities as they could. Probably more so than scientists do with their citations.