Scientists Don't Read the Papers They Cite

WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."

Comments

um...

[jeffy124]

This means that if one paper misreads a work the misreading propagates.

You're assuming the paper which mis-cites another gets read when it gets cited.

even if they do read other's work...

[garcia]

what's to say that even if other people write something about the topic that it's right? Plenty of poorly researched ideas move through circles and even end up in other's research.

Most people twist previous research to fit what they are trying to say anyway (that's the nature of it). AFAIAC it's all bullshit anyway.

Unless they are showing HARD evidence (which in recent months they have been making up as well) and others have reproduced the same results, it's all about money/greed/profit.

Yes.

1. Research
2. Make up shit.
3. Lie.
4. ???
5. Profit.

Re:even if they do read other's work...

[BistroMath]

Um, OK. I'll try it:

1. Read /. headline
2. Form angry, uninformed opinion.
3. Post
4. ????
5. Karma!

Doing science for the money is like having sex for
the exercise. There are many other ways to make considerably more money that require
far less work. The raison d'etre of science is the joy
of discovery; no one spends 6-8 years in higher education
getting a PhD just for the paycheck. People do it
because they love it.

As far as scientists faking results, yes, it happens.
However, the beauty of the scientific method is that
it is self-policing. Anyone can read the journals;
anyone can write the editors of said journals and
report anything that's not above board. As for papers
not being read in the first place, well, let's hop on
the Magic School Bus and take a quick tour of the
scientific publishing process.

First, write the paper. Then, submit it to either a
journal or a conference. In either case, the pool
of available papers will be divided over the number
of people on the review board of the respective
journal/conference, so a bunch of people read a few
papers. Once here, the aforementioned paper is either
rejected or accepted. If accepted, it is published.

After the paper is published, other scientists read
the paper. If it is useful for their work, they may
incorporate some of the ideas into their own work,
at which point, they'll test the idea that they're
borrowing to see if it makes sense.
If it does make sense, they'll use it. If not, they'll
tell the whole world, discrediting the work and
embarassing the original author. Thus there is plenty
of pressure to do good science. The people doing legitimate
work far outnumber the charlatans just submitting
gibberish.

Matt

Not necessarily...

[Pentagram]

The study seemed to be checking for typos in citations. Just because a scientist has copied the text of a (wrongly typed) citation does not mean s/he has not read the paper. There is no law that says someone writing a paper has to type up every citation they make from scratch.

Re:Not necessarily...

[doublegauss]

Exactly. Scientific research is my job, so believe me, I know what I am talking about. Sometimes you mis-cite papers you know extremely well just because you copy/paste an incorrect citation. That doesn't imply at all that you have not read the article.

Proof: in one occasion, I misquoted a paper that I had written myself, just because I copied its title from the preliminary version, forgetting that eventually the title had (slightly) changed in the published version.

IMHO, this is cheap sensationalism. On the other hand, it is true that the academic profession is too loaded with the "publish or perish" thing, which leads researchers (and eventually publishing) sloppy research.

Re:Not necessarily...

[daoine]

Exactly.

I think that the article itself is making a huge leap here, and it's not one I'm about to believe.

They noticed in a citation database that misprints in references are fairly common, and that a lot of the mistakes are identical. This suggests that many scientists take short cuts, simply copying a reference from someone else's paper rather than reading the original source.

They go even further...

The pattern suggests that 45 scientists, who might well have read the paper, made an error when they cited it. Then 151 others copied their misprints without reading the original. So for at least 77 per cent of the 196 misprinted citations, no one read the paper.

Copying the reference format from a paper does not mean that the scientist has not read the original. When writing my papers [research conferences, not just assignments] I'd often grab citations out of papers that were in the proper conference format. I had the paper in my hands -- but sometimes the citation information gets separated from the paper, and you need to rely on someone else's citation. That doesn't mean I didn't read the paper, nor does it mean that I was using another author's interpretation of the original work. It's an absurd leap.

While I do believe that authors do skimp on what they've read and what they just pretend to have read, I'm not sure that using typos in references is the best way to determine the degree to which this occurs. What it *does* show is that people are clearly lazy when it comes to references, and many will copy a reference without checking to see if it's correct. I'd be interested to see if they've looked into lesser known papers -- a popular paper is more likely to withstand an error, since everyone knows what it is anyway...

Not reading != misreading

[AlphaHelix]

This paper takes some very simple statistical models and turns them into what seem to be totally unfounded generalizations about the way science is done. Taking their statistical conclusions at face value, we find that 77% of the people who cited the paper didn't read it in its original form. But, they go on to conclude that a) the only source of information about the paper could have come from a single other paper (namely, the paper with the original citation), and b) misunderstandings about the conclusions drawn by a paper will spread "like wildfire." They do not actually demonstrate this latter conclusion, and don't show that any of the papers actually did misconstrue the science in the original paper.

This is because heavily cited papers become very widely known and understood. Not everybody who's ever cited "The Origin of the Species" has read the whole thing, but it certainly then does not follow that they took their understandings of its conclusions from a single other citing paper.

They end their article with a smug admonition to "read before you cite." These guys sound like the guy with a clean desk who never gets anything done complaining about all the clutter on your desk. Smug social scientists criticizing physicists for their lack of citation rigor does not impress me. There are plenty of better reasons to criticize physicists this year (e.g., Ninov and Schoen). This one seems a bit silly.

Did not prove causality

[MyNameIsFred]

This suggests that many scientists take short cuts, simply copying a reference from someone else's paper rather than reading the original source.

So they copy and paste, that doesn't imply that they didn't read it. I copy and paste references from old reports routinely, its called saving time. That doesn't mean I didn't read the reference.

Idealism vs. Reality

[DirtyJ]

• Ideally the point both in science and in academics in general is to understand the ideas.

True, and most scientists go into the profession with this in mind. I think that most hold to this ideal as much as they can throughout their careers, but they also have to face the reality that their job security (achieving tenure, for an academic) and funding are based in part on their publication count. That's counter to the ideal situation. We'd all like to think of scientists locked away in their labs, very nobly trying to understand the world and explain it to their fellow citizens. Scientists would like to think that what they do is that romantic, too. But then they forge through grad school, get a post-doc somewhere, and realize that they damn well better publish a bunch of papers, or their career is going nowhere. Then maybe they get lucky and find a tenure-track job somewhere, and suddenly they have a teaching load to worry about, plus ever-increasing committee work within their department and university, plus smaller, but still significant tasks, like refereeing their peers' papers. It's a lot of work. More hours of work in a week than the majority of the working force has to put in. Research gets squeezed thinner and thinner, and your time becomes more precious. Yet you're still expected to remain productive and publish a bunch. So the romantic ideal of the lone scientist exploring the mysteries of the universe with a complete focus only on the nobility of the pursuit of knowledge for the sake of knowledge is a little far-fetched.

This, of course, does not justify in any way the falsification of work (which, I think, is extremely uncommon - it's just that we've recently heard about a particularly egregious case of this), nor does it justify propagating misinformation as a result of improper literature citation. I'm just pointing out that the ideal mentioned by the submitter is just that - an ideal.

Duh!

[Starky]

Of course most scientists don't read the entirety of the papers they cite. Is this news?

Sometimes all someone wants is a certain result from a paper. Reading and understanding the full reasoning behind a result rather than the result itself may mean the difference between an afternoon of work and 3 weeks of work. Multiply that by the number of citations a paper has, and a hapless but well-meaning scientist would spend all their time digesting their citations rather than publishing papers and would soon be relieved of their position.

Understanding the details behind cited results is certainly very important, but in the real world there are real tradeoffs that researchers constantly have to evaluate professionally regarding how much time they spend understanding and in how much detail they understand any given result.

This posting is interesting, certainly, but it is not news.

Huh?

[Pedrito]

...most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates.

If they're not reading the papers, why would it propagate?

How research is done in nutrition 'science'

[hlh_nospam]

I have lost over 100 lbs on a high-fat, low-carb, moderate-protein diet, while at the same time achieving good blood-sugar control, lowering my cholesterol, improving my HDL/LDL, dropping my triglycerides from the high 300's to the teens, and numerous other health benefits.

I've grown tired of hearing members of the so-called 'medical' profession lecture me on how 'risky' my 'high-protein' diet is (seems most doctors are functionally deaf and/or immune to learning anything at all from a non-doctor). I gotta wonder how much more 'risky' my MODERATE protein is than being more than 100 lbs overweight. Seems doctors only read the conclusions of studies, and not the actual studies. I have come to the conclusion (based on my personal experience, and comparing notes with several dozen others in the same situation) that the typical 'research' paper follows these steps:

1: Write down a conclusion

2: Write a paper supporting that conclusion

3: Do some 'research', carefully structured to support that conclusion

4: Discount or discard any data that doesn't support that conclusion

5: Get the paper reviewed by a group of associates that agree with your conclusion

6: Publish the paper in some mutual-admiration society journal

My favorite along these lines is one entitled "Type 2 Diabetics Benefit From Reducing Intake Of Animal Protein". If you read the summary very carefully, you will see that the 'researchers' removed the SUGAR from the diet, and then concluded, from the resulting health improvements, that animal protein causes type II diabetes. (!!) This is, unfortunately, typical of what passes for 'science' in the study of diet.

In college...

[shylock0]

So, honestly... How many of you, in college, fudged footnotes and works cited every once in a while? That totally doesn't make it right, and I'm certainly not advocating doing so, but generally speaking my professors never checked up on stuff like that -- and who can blame them, in a class of 100 or more they certainly don't have the time. But it does foster the same thing down the line, which might be what we're seeing here...

this proves little

[ksteddom]

The article in question was cited 4300 times. That is a lot. This would suggest this must be a fundamental paper for that particular field. How many times has the paper been discussed in classes, discussions, literature clubs, etc.? If so, the scientists are probably very well acquainted with the work, with out having the paper on their desk while typing in the reference. You can easily grab another paper that cites the original, with out digging through your file cabinet full of papers. Managing references can be a huge task. I have a small collection of papers, but even this is over 250. I know others with over 1,000. Any yes, we have read every one of them. That doesn't guaranty I will pull the paper out of my drawer every time I cite it. Many others in this discussion have also mentioned inaccuracies in the databases. It does happen, if you don't agree talk to your local inter-library loan. Contrary to the media's perspective, you should never believe what you read until you have tested it yourself or others have confirmed the work. Didn't Einstein say "Believe Nothing"?

Cost of journals, pride of reviewers, contribute

[Jonathan]

As someone who has written a number of scientific papers (and yes, sometimes, but not often, cited articles that I haven't read), I think there are a couple of reason contributing to the problem:

1) Cost of journals -- often there is an article that ought to be cited in your work (because it was published before yours, and is related), but is in a journal unavailable at your university's library. There are thousands of journals, and their high costs (often thousands of dollars a year each) means that no library can have them all. But why not simply ignore an article you haven't read? Read on.

2) Pride of Reviewers -- When a scientific article is sent to a journal, it is passed on to several researchers who are doing similar work for peer review. While it would nice to think that reviewers are not so petty, the fact is, if you haven't cited their work, they might get angry and reject the paper. So, authors feel that it is better safe than sorry and cite freely.

Publication number.

[LothDaddy]

Being a recent Ph.D., a current Post Doc., and a future Prof in Plant Pathology I understand this comment like few others:

A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate.

The problem is that the higher-ups in the university system essentially mandate a certain number of peer reviewed publications for promotions, hell even to keep your job if you're not tenured. This, I feel, is part of the problem in that we're pushed so hard to get X number of publications per year. In a sense it's necessary to weed out the smucks (anyone can get a Ph.D. nowadays), but it also can cause the quality of the research to decline. The whole quality vs. quantity argument.

Just my $0.02.

Re:Not the only problem

[zook]

This highlights a potential problem for traditional publications as well.

A research library used to serve a double role, both providing access to resources and in some sense backing up them up, but with many libraries moving their journal subscriptions from paper to web-based electronic ones, should the journal go away for some reason these resources have a much grater chance of simply disappearing.

Electronic papers are great---they allow for better searches, easier distribution, and let me avoid peeling my butt out of my chair to go to the library. However, libraries really must endeavor to keep local copies of as much of their inventory as possible.

Re:Not really a big problem.

[the gnat]

Also, sometimes the relevant portion of an paper can be summed up in one sentence, or in the abstract.

Of course, which is why the article sort of misses the point. For instance, if I were to mention offhand in an introduction that protein synthesis by the ribosome is done by catalytic RNA, there is an obvious reference to cite [Nissen et al. (2000) Science etc.]. I know this is correct, it's been extensively covered, and I have a copy lying around somewhere, but I've never actually read it all the way through. You can just look at the abstract and that's plenty for these purposes- if I were extensively discussing the mechanism I'd need to thoroughly read the paper, but for an introduction I just need to mention the proper source.

Now, I could be making an error- what if they just pulled something out of their ass, or used sloppy methodology? Usually, people will just say "if it's good enough for the editors (and peer reviewers) of Science, who am I to argue?"

Re:first post?

[quintessent]

What it all boils down to is partial dishonesty. I wish poeple would take credit for what they actually do. Perhaps a list of authors should be annotated indictating very honestly the degree and type of participation. Then you might have more people choosing not to be named.

Watched presentation; corrected spelling in three places.

Re:first post?

[Anonymous Coward]

My father worked at Univerisity of Maryland at College Park's physical lab as a foreign scientist doing experimental research, he often griped about the quality of the papers he reads or writes due to grant pressure and a simple need to just publish. In the theory-related papers, the topics are too obscure to be understood; in experiment-based papers, it's usually a survey of some experiemental procedures + the data collected, with no insight to what has happened (other than some mindless statistical tests and pretty graphs).

the Humanities

[ferrous oxide]

I'm a PhD student in Literature (I know...) and although there's definitely a bit of a problem in the Humanities with people not responding to others in a useful dialogue at times, and there is certainly the same "publish or perish" imperative, it is really a *huge* faux pas to not have read the entirity of the paper/book you cite. In my field, you can easily be discredited for your entire academic carrer for that sort of thing.

Incidentally, it seems to me that the peer review process that exists in both the humanities and the sciences ought to catch these people who are completely misreading their source material. If neither the people writing the papers nor the reviewers are familiar with secondary materials, a real problem exists.

The article is total bullshit

[orange7]

Look, as someone who's written scientific papers, the claims in the article are not only false, but indicative of poor science themselves. They're making the classic experimental stats mistake. Namely, copying and pasting citations from other sources is *absolutely uncorrelated* with whether those papers have been read by the author.

Formatting citations is fussy, tedious, and annoying. You have to look up the page numbers in the journal (which you may not even have in these days of online papers), figure out who the publisher was, the issue or journal number.

I read every single one of the papers I've ever cited. But it was rare that I ever typed in a citation from scratch. Usually you get them either from an on-line citation database, from the bibtex entry helpfully supplied on the cited author's web page (scientists like being cited!) or, yes, by typing out a citation from a printed paper.

In any given field, usually some kind-hearted soul starts collecting a database of citations for others to use. For instance, here's one here:

http://www.helios32.com/resources.htm#Bibliographi es

Have a look; you'll soon twig to why people don't type these in from scratch.

Creating the citation all over from scratch when it's right there in front of you is about as pointless as adding a link to a web page by retyping some monstrous 200-character URL. Just because you copy & pasted a link doesn't mean you didn't read the article did you? (I guess slashdot is the wrong place for that particular piece of rhetoric.)

I'm disappointed in New Scientist. The pissy little diatribe about science in the story submission is par for the course. Please, leave the pontificating to people who have a clue.

In fact, how about a retraction? (Ha ha ha ha!)

A.

Re:The article is total bullshit

[tgibbs]

I agree. I have never cited a paper without having read it. But I have shared my citations file with many students who have passed through my lab. And I have also, when I couldn't remember the exact citation of a paper that I'd read, looked it up from another published paper that cited the same reference.

Another problem is that they seem to be estimating the frequency with which people read papers they cite from the frequency of repeating errors. The underlying assumption of this analysis is that scientists who read all the papers they cite and those who don't are equally likely to make errors. But it seems quite likely to me that scientists who are careful not to cite papers they have not read are also extra careful to get their citations correct. So sloppy scientists will tend to be overrepresented in the authors' statistics.

Re:The article is total bullshit

[Anonymous Coward]

The man says it like it is. I can think of a situation where you might not read a paper though. It's when the paper is really old and really famous. Take this situation: You develop a new widget for processing DNA samples for sequencing or PCR or whatever, fields with thousands of workers and billions of dollars of turnover. You write a paper for a cheerful process oriented journal like "Biotechniques". Your opening line is. Since the discovery of DNA[1] high throughput process tools have become increasingly important...Blah Blah [1] refers to the original watson and crick paper. Now everyone should have read that classic one pager, but if you haven't it doesn't really call into question the existance of DNA. The same could happen quoting Binning ahd Rohrer's 1986 paper on the scanning tunnelling microscope In fact I haven't read it and I built a working STM which I published.

So it's not the famous papers you have to worry about, it's the low profile papers covering recent developments or still poorly understoon areas of nature. I'd like to see those crackers at the new scientist research that., becuase everyone I know reads those ones very carefully.

Uh... You Haven't Worked With Medical Academics

[occamboy]

There is ungodly profit potential in medical "science", and research physicians and similar types will do almost anything to make a buck, or many hundreds of thousands of bucks, from big pharmaceutical companies and so forth. My perspective comes from years in the medical industry.

To paraphrase James Carville, it's amazing what happens when you drag a $100 bill through a research hospital. These folks will make stuff up for money. Some of them deal in half truths -- they'll play clever games with the truth to support things that they know to be nonsense. Others just make stuff up out of whole cloth.

Some people in the field are good -- but there are many others for who you'd be well advised to count your fingers after shaking their hand to make sure they gave 'em all back.

Citation Typos != Not Reading the Cited Paper

[Anonymous Coward]

I always read the papers I cite, but I don't always look at the paper for the title, authors, journal, volume, etc., but instead pull it from another paper or from our group's citation database or one from the web because:

1. Poor reproduction caused such things to be lost off the edges of the pages or to be illegible.
2. The micromanagers require it.
3. It's easier.

--
There's no conspiracy, they just want you to think there's one.

Re:first post?

[Ed Avis]

I am not a scientist, I am just a humble student (or rather ex-student, I graduated this year). When I wrote my project report I was asked by my supervisor to get in references to papers X, Y and Z. So I ended up putting in a few fairly meaningless or irrelevant sentences just to cite the correct paper. Of course here the aim is to get marks, not to get kudos or whatever else real researchers write for, so it's not really a problem.

There's a tradition...

[Farang]

...that tolerates sloppy writing and even outright fraud in science, and it's not new.

In the 1960's, I was informed that a biology prof at a California state university was telling his undergrad classes that a female gorilla had been articifically inseminated with human semen and had subsequently given birth to a live baby--half ape, half human. I was astounded that anyone would perpetrate such a hoax, and eventually not only had an interview with the hoaxer, but corresponded with many biologists, zoologists and institutes in an effort to prove that his (shifting) sources were sheer inventions.

My point: as I tried to bring up the issue of academic ethics and scientific responsibility, I was shocked to discover that no one wanted to deal with this hoax. The prof stuck to his story, and his colleagues avoided comment and involvement. In fact I was warned to shut up and stop making waves. The scientific community was upset with ME.

There is no happy ending to this story of hoax and lies and disgraceful cover-up. The scientific community is, IMHO, unwilling to police itself adequately; it is sloppy, lacking in stringent ethical guidelines, and lethargic. That's a generalization that doubtless has exceptions.

I was able to establish that the hoax was not a teaching technique. The lies seemed to have been part of an effort by the prof to discredit political figures: he evidently wanted to show that because of the nature of the "research," creationists had used political pressure to cut funding for the program.

The hoaxer has a Ph. D. from the University of Southern California, which means...nothing. Back in the 1960's, I thought that an advanced degree was not only an academic accomplishment, but also a kind of certification as to the ethics of a researcher. How naive I was.

So this story about sloppy writing comes as no surprise at all. I believe that if more research were done into the ethics of researchers, a lot of nasty things would be exposed. Scientists are not demi-gods--some are rascals, and many of those who are not have a distressing tendency to tolerate sloppy research and even blatant hoaxes.

Umm - can anyone say "Duh"?

[Ignorant Aardvark]

I'm not surprised many scientists don't read all the papers they've cited. To be published in a reputable peer review journal, you have to cite other literature. And you have to cite lots of other literature. It's almost like back in high school when the English teacher says, "You must have ten sources." So the scientists do a quick search for vaguely relevant material, and judging by the abstract alone, choose whether to add it in to their cited sources list. That's just the way things are: some journals are hard to come by, and to get some articles you have you pay $20 online per 24 hour period. Now the abstracts are always free. Frankly, I don't blame the scientists. It just reinforces the importance of the abstract. As long as such an emphasis is placed on citing, I think a large emphasis should be placed on abstracts as well.