Slashdot Mirror
Scientists Don't Read the Papers They Cite
WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."
Gee ... most scientists use a program (like Endnote) to format bibliographies, using data downloaded from a database (like PubMed). I suspect that this is more a deficiency in proofreading reference lists and assuming that databases are correct, rather than a lack of reading the original material. Whether people read articles carefully is another matter, of course.
In fact, a blatant miscitation of a given reference would often get caught during the peer review process. This happened to me once when I rewrote part of a paper and forgot to remove one of the references that no longer applied ...
I am probably one of the few people out there who has ever leafed through academic journals for fun. Still, those things are incredibly boring.
The issue here is that people expect articles to have a certain shape, form, and style, including a literature review. And a lit review can be a pain. You don't want to read an article more than is required to get the basic gist of its relevance to your work. Sometimes, that can be done by reading just the abstract.
The suggested rate of non-reading articles is also possibly overstated. That one has mis-cited a work does not necessarily mean that one has not read it. I can, for example, read an article ten years ago and remember the basic meaning I need to take out of it, and include it in my own references upon seeing it in the references of another's work without refreshing my knowledge of the work. Or I could just use another work's references as a reading checklist and not bother to correct it (or be unaware of the mistake if I sent a poor grad student or some other lackey to the library to copy the journal for me).
I assume the full article by Simkin and Roychowdhury probably states the likely sources of commonly copied errors. I'm a tad curious to se whether the authors of those progenitor articles propagated their own mistakes in future articles or if they corrected them.
While the article claims that "a billion different versions of erroneous reference are possible," in practice that may not be as true. With the errors being volume, page, or year, the most likely errors are transposition of two digits, deletion of a digit, insertion of a digit, or replacement of a digit. In the latter two, the error will most likely be the use of a neighboring number on the keyboard. A one is much less likely to be replaced by a nine than by a two. That is unlikely to lower the probably number of copied citations to below 50%, but it is still a possible source of error that may or may not be accounted for.
Take a look at the ACM or IEEE and the number of journals they support, then toss in folks like Springer Verlag. Figure out how many articles are published in these each year. Just from counting you might determine that many of these are pretty meaningless. Try reading a few at random and see if you change your mind.
Now remember that the folks on a tenure/promotion committee know nothing about what a researcher might do - they're even more ignorant of the research field of someone else than they are of their own. So, how do they determine how good a researcher might be? They're sure as hell not going to wade through yet another meaningless paper. Its simple. They count. How many publications? How many grants? How many citation from other papers to the researcher's papers?
And its an interesting feedback loop: even getting a publication or grant can depend on your publication and grant history. And if you suspect that someone might be reviewing your paper/proposal who works in the same area, you might want to make sure there are a couple of citations (always positive, naturally) of that persons work included.
So, we know someone wants publications/grants/citations and they need p./g./c. to get p./g./c.. They do some research, it depends heavily on two or three other bits of research. But two or three citations aren't enough. So they might want to use the citations they find in the work they cite. OK. This citation looks good perhaps, but the original article isn't available in the local library and inter-library-loan will take a month to get it and the deadline is next week. Oh well. Cite away - the original author isn't likely to complain (after all this is another citation to his/her work).
And so it goes.
When I was a postdoc at Argonne National Lab, my group's policy was that every group member got his name on every paper. On this paper, one of my coauthors refused to read the paper before publication. He said he was busy, and it was a long paper. I wanted to take his name off, but he insisted on having it on. This is not uncommon at all. I'm just quoting an example that I know of from personal experience.
There was a recent scandal where a group at Berkeley claimed to have discovered a new element. Later on, it turned out that the evidence had been fabricated. However, the group claimed it was only one guy who was a loose cannon who had invented the data. In other words, if it was right, they were ready to take credit. But if it turned out to be a fraud, they had plausible deniability.
There is huge pressure on young people in the sciences to establish a long list of publications, because permanent jobs are so hard to get. Another common phenomenon is where you have a senior administrator at a lab who "blesses" every paper done at the lab, and gets his name on every single one of them, even though he never actually makes any scientific contribution.
Find free books.
Excellent point. In fact, let me add to it:
Many citiations now are copied from ONLINE SOURCES. We read the papers, but we hate typing in our bibliography from scratch. I can just go to ADS (http://adsabs.harvard.edu) and have it print out the reference in BIBTeX format for me. Now, there are quite a few typos in that database. I know because we're finding them while creating a bibliography for the upcoming Juptier book.
None of this implies that we're not reading papers we cite. In fact, from experience, people in my field (astronomy) know the papers they reference pretty well, as a rule. There are times when we don't read an entire paper because we're just taking a few numbers or an equation, but as a whole, it pays to know the papers. If you don't, you'll usually find yourself under fire from your collegue who wrote the mis-quoted paper. Quite often, that collegue will be the journal referee who is reviewing the paper before publication. This is *not* a position you want to be in, so people generally work to avoid it.
Yup! I've had the same happen to me. A bunch of people attend meetings, and get added to the paper. Perhaps a few of them helped proofread, but that's nothing like making an original scientific contribution. On the one hand, I was added to a number of papers based largely on my a) meeting attendance and b) technical expertise. This helps me immeasurably in furthering my career, except for the difficulty of explaining what I did on some projects (ranging from "absolutely zilch" to "constructed the web page"). However, when it was my turn to write a paper, about five people ended up as co-authors with minimal contributions.
I'm now in a position where I don't let that happen any more; I've asked to be taken off of papers because my contribution was minimal, and the last time I wrote something I stated up front who would be on the author list. I do not forsee that this will be a permanant solution, however; I'm still too junior a scientist to have much sway. I've seen other cases where someone asked to be added to a project because he "needed more publications", or where a senior investigator did not realize he was a co-author until well after the paper had been published.