Slashdot Mirror
Scientists Don't Read the Papers They Cite
WatertonMan writes "Very interesting and sure to be controversial study that suggests most scientists don't read the papers they cite. This means that if one paper misreads a work the misreading propagates. It's a very interesting study and has big implications for science, in my opinion. New Scientist has a good overview of the work. Given that most attention to work has been in sloppy work on the experimental side (poor methadology or outright fraud) this suggests a whole other problem. A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate. Ideally the point both in science and in academics in general is to understand the ideas. Yet those of you who've looked up footnotes realize that actually engaging the ideas of other researchers typically falls by the wayside. Often footnotes are there simply because references are needed. Engaging others works is secondary. I've always thought that the hard sciences were more immune to that effect than the humanities. I guess not."
I wouldn't either -- those things are boring! ;-)
Most of my classmates don't read the papers they write. Do we hold others to a higher standard?
You can't judge a book by the way it wears its hair.
...where no one reads the articles they cite. We are in good company!
I want to drag this out as long as possible. Bring me my protractor.
The study seemed to be checking for typos in citations. Just because a scientist has copied the text of a (wrongly typed) citation does not mean s/he has not read the paper. There is no law that says someone writing a paper has to type up every citation they make from scratch.
Another major problem with research papers is the "dissappearance" of those who actually do properly cite their sources.
/ web_citations.html)
As many of you know, the Internet is a great research tool these days. But unfortunately, it's too dynamic for the research world. "Most URL references [stand] more than a 50 percent chance of not existing after only six months." (from a Cornell study at http://www.news.cornell.edu/chronicle/00/12.14.00
I don't care as much if some researcher only reads parts and pieces of papers that they cite, but when the entier papers dissappear, that's a much bigger problem.
"The study, using term papers between 1996 and 1999, found that after four years the URL reference cited in a term paper stood an 80 percent chance of no longer existing."
Anyone who see how sme articles are written, knows perfectly that "bibliography" is usually created as a "necessary evil". Most scientific articles are done basically in the light of several "obligatory templates": abstraction, main article, citations, bibliography and notes. Frequently, real authors are not the ones you see first in the header of the article but someone in the end of it. Also, sometimes, certain people do the most flagrant plagiates out of the work of their students or co-workers.
What I call "academical science" is full of huge problems, which sometimes reach the level of flagrant falsifications and demagogic manipulation of facts. While not being a scientist per se, I have seen how these things pass the limits ethics and moral in such a thing like Mars. There is one scientist who tragically died in a very strange situation. Apart of the conditions of the tragedy, there was one big "authority" on Mars who lied with all his teeth about the work of his deceased colleague. Frankly, it was shocking to see how this guy flagrantly and demagogically "reinterpreted" the intentions of the scientific work of his colleague. One should note that both guys were highly considered in the community. However, they were adversaries. One died, the other became a big scientific authority on Mars. One of the reasons, was that he made a lot to desmise the works that went against his theories
This paper takes some very simple statistical models and turns them into what seem to be totally unfounded generalizations about the way science is done. Taking their statistical conclusions at face value, we find that 77% of the people who cited the paper didn't read it in its original form. But, they go on to conclude that a) the only source of information about the paper could have come from a single other paper (namely, the paper with the original citation), and b) misunderstandings about the conclusions drawn by a paper will spread "like wildfire." They do not actually demonstrate this latter conclusion, and don't show that any of the papers actually did misconstrue the science in the original paper.
This is because heavily cited papers become very widely known and understood. Not everybody who's ever cited "The Origin of the Species" has read the whole thing, but it certainly then does not follow that they took their understandings of its conclusions from a single other citing paper.
They end their article with a smug admonition to "read before you cite." These guys sound like the guy with a clean desk who never gets anything done complaining about all the clutter on your desk. Smug social scientists criticizing physicists for their lack of citation rigor does not impress me. There are plenty of better reasons to criticize physicists this year (e.g., Ninov and Schoen). This one seems a bit silly.
* mild mannered physics grad student by day *
* daring code hacker by night *
http://www.silent-tristero.com
Copying a reference string doesn't mean that you haven't read the paper in question. To take a personal example of what I've done:
1. Find a reference to a paper which looks interesting.
2. Walk down to the library, remembering that you're looking for Bob's paper about bars in the Journal of Foo.
3. Arrive in the library, find the paper, read it, decide it is important.
4. Walk back to computer, copy out reference string.
It's quite easy to look up a paper from a slightly-wrong reference, and as long as the reference is close to correct, it's fairly easy to not realize that the reference was wrong in the first place.
Tarsnap: Online backups for the truly paranoid
To support the view that observations got better and better, requiring more and more circles, you'll probably find most of these sources citing a book by J.L.E. Dreyer, written in the beginning of the previous century, but it exists in a few editions published later.
But Dreyer says the opposite:
Basically, if these people had actually read Dreyer, we wouldn't have had to struggle with this myth any longer. Of course, there's a lot more to this story than this, but I don't have time to write it now... :-)
Employee of Inrupt, Project Release Manager and Community Manager for Solid
The root problem is papers are a form of scientific social capital. And when people think you are well read, your paper is worth more. I worked in a research facility where grad students were literally held hostage so they could produce more papers for the professors to take credit for. One student came to use with his masters and was held *7* years for his PHD. It was getting so bad the graduate department was *forcing* the director to graduate students by saying, "So and so has to leave by the end of the year -- with or without his degree." (and after 7 years, who could blame them)
Add this to an already paper obsessed culture, and you have a serious problem.
Religion is a gateway psychosis. -- Dave Foley
As far as twisting up evidence, yes, this does happen. But most definitely not 100% of the time. How was the solar neutrino problem ever discovered in the first place? How was a re-evaluation of the cosmological constant initiated? These (and many other ideas) were brought forth not because someone wanted their ideas to be put forth, but because their hypotheses did not match the experimental data! It most definitely is not bullshit. AFAIAC, science is still the most altruistic of professions, not to mention one of the most self-sacrificing.
Um, OK. I'll try it:
/. headline
1. Read
2. Form angry, uninformed opinion.
3. Post
4. ????
5. Karma!
Doing science for the money is like having sex for
the exercise. There are many other ways to make considerably more money that require
far less work. The raison d'etre of science is the joy
of discovery; no one spends 6-8 years in higher education
getting a PhD just for the paycheck. People do it
because they love it.
As far as scientists faking results, yes, it happens.
However, the beauty of the scientific method is that
it is self-policing. Anyone can read the journals;
anyone can write the editors of said journals and
report anything that's not above board. As for papers
not being read in the first place, well, let's hop on
the Magic School Bus and take a quick tour of the
scientific publishing process.
First, write the paper. Then, submit it to either a
journal or a conference. In either case, the pool
of available papers will be divided over the number
of people on the review board of the respective
journal/conference, so a bunch of people read a few
papers. Once here, the aforementioned paper is either
rejected or accepted. If accepted, it is published.
After the paper is published, other scientists read
the paper. If it is useful for their work, they may
incorporate some of the ideas into their own work,
at which point, they'll test the idea that they're
borrowing to see if it makes sense.
If it does make sense, they'll use it. If not, they'll
tell the whole world, discrediting the work and
embarassing the original author. Thus there is plenty
of pressure to do good science. The people doing legitimate
work far outnumber the charlatans just submitting
gibberish.
Matt
This is a serious problem in a lot of disciplines, though I've heard of a rather elegant solution to the problem that's now become common (if informal) practice. The solution is that when a student thinks that he's done enough to justify getting a PhD, he starts applying for jobs that require a PhD. When somebody is willing to offer him one, that's proof that an outsider views his accomplishments as being worth a degree and his advisor has to let him write up his dissertation. It serves as a very effective independant outside check on the system.
There's no point in questioning authority if you aren't going to listen to the answers.
As someone who has written a number of scientific papers (and yes, sometimes, but not often, cited articles that I haven't read), I think there are a couple of reason contributing to the problem:
1) Cost of journals -- often there is an article that ought to be cited in your work (because it was published before yours, and is related), but is in a journal unavailable at your university's library. There are thousands of journals, and their high costs (often thousands of dollars a year each) means that no library can have them all. But why not simply ignore an article you haven't read? Read on.
2) Pride of Reviewers -- When a scientific article is sent to a journal, it is passed on to several researchers who are doing similar work for peer review. While it would nice to think that reviewers are not so petty, the fact is, if you haven't cited their work, they might get angry and reject the paper. So, authors feel that it is better safe than sorry and cite freely.
A lot of the ultimate problem is that many in research are concerned more about publishing than in solving the issues they investigate.
The problem is that the higher-ups in the university system essentially mandate a certain number of peer reviewed publications for promotions, hell even to keep your job if you're not tenured. This, I feel, is part of the problem in that we're pushed so hard to get X number of publications per year. In a sense it's necessary to weed out the smucks (anyone can get a Ph.D. nowadays), but it also can cause the quality of the research to decline. The whole quality vs. quantity argument.
Just my $0.02.
Look, as someone who's written scientific papers, the claims in the article are not only false, but indicative of poor science themselves. They're making the classic experimental stats mistake. Namely, copying and pasting citations from other sources is *absolutely uncorrelated* with whether those papers have been read by the author.
i es
Formatting citations is fussy, tedious, and annoying. You have to look up the page numbers in the journal (which you may not even have in these days of online papers), figure out who the publisher was, the issue or journal number.
I read every single one of the papers I've ever cited. But it was rare that I ever typed in a citation from scratch. Usually you get them either from an on-line citation database, from the bibtex entry helpfully supplied on the cited author's web page (scientists like being cited!) or, yes, by typing out a citation from a printed paper.
In any given field, usually some kind-hearted soul starts collecting a database of citations for others to use. For instance, here's one here:
http://www.helios32.com/resources.htm#Bibliograph
Have a look; you'll soon twig to why people don't type these in from scratch.
Creating the citation all over from scratch when it's right there in front of you is about as pointless as adding a link to a web page by retyping some monstrous 200-character URL. Just because you copy & pasted a link doesn't mean you didn't read the article did you? (I guess slashdot is the wrong place for that particular piece of rhetoric.)
I'm disappointed in New Scientist. The pissy little diatribe about science in the story submission is par for the course. Please, leave the pontificating to people who have a clue.
In fact, how about a retraction? (Ha ha ha ha!)
A.