Slashdot Mirror
Scientists Don't Read the Papers They Cite
WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."
I've also seen the case where scientists will constantly refer to their own, or their coleagues' papers. This is an easy way to increase the "cited" count of the refered paper, making one's work look more usefull, even when the citation has little or no relevance to the current topic.
(cough) read the article (cough)
They cover that.
Is Slashdot written to the maxim "no news is new news"?
Charles Darwin is known to have cited other people's work that he hadn't read (I forget the name of the author involved - not being in the field myself). Then there was the entire field of molecular biology in the 1990s, which suffered more scandals than a dyslexic shoe factory.
Slightly more relevant (though still stretching back decades) is that some authors don't read the papers they co-author - look at all the people who co-authored papers with Jan Schoen, the team who, with Ninov, "discovered" Ununoctium, etc.
Next you'll be telling us that (shock! horror!) some scientist pass off other peoples' work as their own, with a fascinating NEW revelation about Rosalind Franklin's work in the discovery of the DNA structure.
Well, it may be a troll, but the fact remains that some very prominent scientists claim that the HIV -> AIS theory wasn't proven before it was adopted.
Their claim is that this exact thing happened in the early 80's, and that instead of actually reading the research that said that HIV may cause AIDS (which was inconclusive) they simply took the ball and ran with it, causing years of research to be based on the same incorrectly cited source.
Who knows what the answer is, but it's a fascinating subject to read up on.
This doesn't come as news for me..
As a student starting my PhD studies, I once asked a researcher at the department about a paper. He told me he hadn't read it.
The next day, I saw that he had indeed quoted that paper in one of his.
However, it usually isn't such a big problem,
when papers are cited without being read, since it usually only happens with papers periferial to the subject.
(For example to justify a certain method or procedure that is common practice)
Also, sometimes the relevant portion of an paper can be summed up in one sentence, or in the abstract.
Gee ... most scientists use a program (like Endnote) to format bibliographies, using data downloaded from a database (like PubMed). I suspect that this is more a deficiency in proofreading reference lists and assuming that databases are correct, rather than a lack of reading the original material. Whether people read articles carefully is another matter, of course.
In fact, a blatant miscitation of a given reference would often get caught during the peer review process. This happened to me once when I rewrote part of a paper and forgot to remove one of the references that no longer applied ...
Unfortunately, your characterization of science is flawed. Rarely does a scientist go into a lab and perform an experiement that is 100 percent original. Generally, the origins of the experiment can be traced back to earlier work, that he/she learned about thru publications, conferences, etc. Furthermore, scientists try to be somewhat original. Therefore, considerable effort is spent researching the published literature to make sure you're not repeating something someone did 5 years ago. If you repeat it, you want to put your own "spin" on it. E.g., look at new aspects of the problem.
I am probably one of the few people out there who has ever leafed through academic journals for fun. Still, those things are incredibly boring.
The issue here is that people expect articles to have a certain shape, form, and style, including a literature review. And a lit review can be a pain. You don't want to read an article more than is required to get the basic gist of its relevance to your work. Sometimes, that can be done by reading just the abstract.
The suggested rate of non-reading articles is also possibly overstated. That one has mis-cited a work does not necessarily mean that one has not read it. I can, for example, read an article ten years ago and remember the basic meaning I need to take out of it, and include it in my own references upon seeing it in the references of another's work without refreshing my knowledge of the work. Or I could just use another work's references as a reading checklist and not bother to correct it (or be unaware of the mistake if I sent a poor grad student or some other lackey to the library to copy the journal for me).
I assume the full article by Simkin and Roychowdhury probably states the likely sources of commonly copied errors. I'm a tad curious to se whether the authors of those progenitor articles propagated their own mistakes in future articles or if they corrected them.
While the article claims that "a billion different versions of erroneous reference are possible," in practice that may not be as true. With the errors being volume, page, or year, the most likely errors are transposition of two digits, deletion of a digit, insertion of a digit, or replacement of a digit. In the latter two, the error will most likely be the use of a neighboring number on the keyboard. A one is much less likely to be replaced by a nine than by a two. That is unlikely to lower the probably number of copied citations to below 50%, but it is still a possible source of error that may or may not be accounted for.
Take a look at the ACM or IEEE and the number of journals they support, then toss in folks like Springer Verlag. Figure out how many articles are published in these each year. Just from counting you might determine that many of these are pretty meaningless. Try reading a few at random and see if you change your mind.
Now remember that the folks on a tenure/promotion committee know nothing about what a researcher might do - they're even more ignorant of the research field of someone else than they are of their own. So, how do they determine how good a researcher might be? They're sure as hell not going to wade through yet another meaningless paper. Its simple. They count. How many publications? How many grants? How many citation from other papers to the researcher's papers?
And its an interesting feedback loop: even getting a publication or grant can depend on your publication and grant history. And if you suspect that someone might be reviewing your paper/proposal who works in the same area, you might want to make sure there are a couple of citations (always positive, naturally) of that persons work included.
So, we know someone wants publications/grants/citations and they need p./g./c. to get p./g./c.. They do some research, it depends heavily on two or three other bits of research. But two or three citations aren't enough. So they might want to use the citations they find in the work they cite. OK. This citation looks good perhaps, but the original article isn't available in the local library and inter-library-loan will take a month to get it and the deadline is next week. Oh well. Cite away - the original author isn't likely to complain (after all this is another citation to his/her work).
And so it goes.
When I was a postdoc at Argonne National Lab, my group's policy was that every group member got his name on every paper. On this paper, one of my coauthors refused to read the paper before publication. He said he was busy, and it was a long paper. I wanted to take his name off, but he insisted on having it on. This is not uncommon at all. I'm just quoting an example that I know of from personal experience.
There was a recent scandal where a group at Berkeley claimed to have discovered a new element. Later on, it turned out that the evidence had been fabricated. However, the group claimed it was only one guy who was a loose cannon who had invented the data. In other words, if it was right, they were ready to take credit. But if it turned out to be a fraud, they had plausible deniability.
There is huge pressure on young people in the sciences to establish a long list of publications, because permanent jobs are so hard to get. Another common phenomenon is where you have a senior administrator at a lab who "blesses" every paper done at the lab, and gets his name on every single one of them, even though he never actually makes any scientific contribution.
Find free books.
Excellent point. In fact, let me add to it:
Many citiations now are copied from ONLINE SOURCES. We read the papers, but we hate typing in our bibliography from scratch. I can just go to ADS (http://adsabs.harvard.edu) and have it print out the reference in BIBTeX format for me. Now, there are quite a few typos in that database. I know because we're finding them while creating a bibliography for the upcoming Juptier book.
None of this implies that we're not reading papers we cite. In fact, from experience, people in my field (astronomy) know the papers they reference pretty well, as a rule. There are times when we don't read an entire paper because we're just taking a few numbers or an equation, but as a whole, it pays to know the papers. If you don't, you'll usually find yourself under fire from your collegue who wrote the mis-quoted paper. Quite often, that collegue will be the journal referee who is reviewing the paper before publication. This is *not* a position you want to be in, so people generally work to avoid it.
Yup! I've had the same happen to me. A bunch of people attend meetings, and get added to the paper. Perhaps a few of them helped proofread, but that's nothing like making an original scientific contribution. On the one hand, I was added to a number of papers based largely on my a) meeting attendance and b) technical expertise. This helps me immeasurably in furthering my career, except for the difficulty of explaining what I did on some projects (ranging from "absolutely zilch" to "constructed the web page"). However, when it was my turn to write a paper, about five people ended up as co-authors with minimal contributions.
I'm now in a position where I don't let that happen any more; I've asked to be taken off of papers because my contribution was minimal, and the last time I wrote something I stated up front who would be on the author list. I do not forsee that this will be a permanant solution, however; I'm still too junior a scientist to have much sway. I've seen other cases where someone asked to be added to a project because he "needed more publications", or where a senior investigator did not realize he was a co-author until well after the paper had been published.
The logic benind the articel is very, very weak. The basis of the article is that misquotes in citations (wrong volume, page number etc.) propagate from one paper to another. Whech shows that the authors cut-and-pasted citations from earlier papers. Sure. But the researchers quoted claim that this means that the researchers didn't read the papers concerned. Rubbish.
During the reserch shage of a project, you read the papers. Error in th citation - no sweat; you know authers and title, and a search engine will give it to you in nothing flat.
Weeks or months later, it is writeup time. Open the first paper to cite it. And there are all the other references you followed (a little trouble in the lookup is long forgotten) and dutifully read. And - get this - it is easier to cut-and-past the citation than to go back to the paper and assemble - separately - the publication, title, authors and page numbers.
Then only thing the research quoted proves is that papers are overwhelmingly circulated electronically ans the dead tree format is, for scientific papers, obsolete.
Consciousness is an illusion caused by an excess of self consciousness.
True. But I'm old enough to remember the days before word processors. Back then it was not uncommon to literally cut-and-paste from earlier drafts, photocopies of other documents, etc. into a new document. Using the copy held together with glue and tape, the secretary would create a clean copy. So assuming a good secretary, its not surprising that transcription errors are propagated.
The people who insist on such models tend to have a very parochial view of science.
If I spend 5 years designing an experimental apparatus and gather data then a collegue (or more likely his grad students) takes that data and produces an analysis I have the right to have my name on the paper, the analysis is only a part of the work.
Science is based on trust. Consider the case where a physicist wants to measure some effect but does not know how to build the apparatus to test the theory. I might well design the apparatus to test his theory even though I don't fully understand the theory he is testing, ultimately I have to trust him on that point. Equally if I provide him with a bunch of experimental data he trusts me not to have fabricated it.
On the large experiments (500+ authors) I have worked on there has been a review committee that checked over the paper in detail of 30 or so people.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/