Slashdot Mirror
The Scientific Paper Is Obsolete (theatlantic.com)
James Somers, writing for The Atlantic: The scientific paper -- the actual form of it -- was one of the enabling inventions of modernity. Before it was developed in the 1600s, results were communicated privately in letters, ephemerally in lectures, or all at once in books. There was no public forum for incremental advances. By making room for reports of single experiments or minor technical advances, journals made the chaos of science accretive. Scientists from that point forward became like the social insects: They made their progress steadily, as a buzzing mass.
The earliest papers were in some ways more readable than papers are today. They were less specialized, more direct, shorter, and far less formal. Calculus had only just been invented. Entire data sets could fit in a table on a single page. What little "computation" contributed to the results was done by hand and could be verified in the same way.
The more sophisticated science becomes, the harder it is to communicate results. Papers today are longer than ever and full of jargon and symbols. They depend on chains of computer programs that generate data, and clean up data, and plot data, and run statistical models on data. These programs tend to be both so sloppily written and so central to the results that it's contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you've actually discovered, clearly enough that someone else can discover it for themselves.
The earliest papers were in some ways more readable than papers are today. They were less specialized, more direct, shorter, and far less formal. Calculus had only just been invented. Entire data sets could fit in a table on a single page. What little "computation" contributed to the results was done by hand and could be verified in the same way.
The more sophisticated science becomes, the harder it is to communicate results. Papers today are longer than ever and full of jargon and symbols. They depend on chains of computer programs that generate data, and clean up data, and plot data, and run statistical models on data. These programs tend to be both so sloppily written and so central to the results that it's contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you've actually discovered, clearly enough that someone else can discover it for themselves.
As a scientist who has published papers in peer-reviewed journals, I strongly disagree. I don't care for many aspects of the publication process or how academia works, but the scientific paper isn't obsolete.
Regarding the use of software in creating results, it is definitely a problem when that software is difficult to use or isn't available at all. The same goes for data sets, many of which aren't released publicly for a variety of reasons. These are issues that need to be addressed.
In my own work, I try to release the software used to perform my analysis under the GPLv3. I also try to include adequate documentation to allow the work to be reproducible. I also support making data publicly available, but sometimes the volume of data sets makes it prohibitive to redistribute the work. The best option in that case is to provide detailed instructions on how to generate the data set. But sometimes it's even worse, such as when the research requires using a closed source program with a license that prohibits redistributing the output. The license makes it relatively difficult to obtain the software for anyone outside of the US government or academia, unless they pay for a commercial license. There isn't a comparable piece of software, but the university that licenses the software uses restrictive licensing requirements to increase their revenue.
To the extent that it's possible, scientists receiving grant funding ought to release their software as free and open source software. The data sets should be released or detailed instructions should be included to allow others to generate the same data set.
Papers can be difficult to read, but there are at least some steps in the right direction. One of the encouraging changes is the use of more first person in scientific papers and less passive voice. The formal tone is being being phased out in favor of readability. I wholeheartedly support this because the subject matter is difficult enough to understand without awkward sentence structures. Peer-reviewed papers also provide some level of quality control for research, though not to the extent of reproducibility.
Scientific papers still have a place, because they're still the best opportunity for scientists to present the key points of their results and describe their methods in detail. Simply releasing software and data sets is not enough. Those also aren't peer-reviewed. The bigger issue is that there just isn't a lot of funding to ensure that results are reproducible. Due to funding limits, there just isn't a lot of effort to reproduce all but the most surprising results. It would be great if funding agencies like NSF would allocate more funds for ensuring the reproducibility of existing results. Unfortunately, funding is so competitive that researchers have to sell their work as being very novel rather than verifying existing research.