Slashdot Mirror
Current Scientific Publishing Methods Problematic
A recent examination of current scientific publishing methods shows that they are problematic at best, treating the entire process like an economic system, with publishers as bidders at an auction, authors as sellers, and the community at large as consumers. "The authors then go on to discuss a variety of economic terms that they think apply to publishing, but the quality of the analogies varies quite a bit. It's easy to accept that the limited number of high-profile publishers act as an oligarchy and that they add value through branding. Some of the other links are significantly more tenuous. The authors argue that scientific research suffers from an uncertain valuation, but this would require that the consumers — the scientists — can't accurately judge what's significant. "
What is wrong with the free market? When has it ever failed us?
A recent examination of current scientific publishing methods show that it is problematic at best. Treating the entire process like an economic system, with publishers as bidders at an auction, authors as sellers, and the community at large as consumers.
Agreed. I think we need to switch this whole process to Vickrey Auctions. Then you can explain to the authors of the papers that they will receive $75 for their paper instead of $100 because whoever bid $100 was gaming the system. Why is it suddenly so popular to turn everything possible into an auction system with 75 different flavors of said auction system?
My work here is dung.
I have this fantasy of writing a program which makes some big combinatorial breakthrough and from time to time to motivate myself I imagine what I might do with such a thing should I actually bumble into creating it.
I looked at scientific journals, and I honestly can't see much of an incentive to appear there. I mean sure, you might get published and that's got some merit, but it seems to me that these journals don't really make a lot of money for the scientists who write the articles. Does Science or Nature pay its writers? It can't be that much, even if they did. So what's the point?
From the scientist's perspective, if they have pure research, then, they can put it on a web site, such as the university web site or even their own, and just skip the b.s. Or, they can sell it. Either option is better to achieve the altruistic or commercial ends of the scientist than being in a magazine.
This is my sig.
publishers as bidders at an auction, authors as sellers, and the community at large as consumers
Hmmm, there's some term I'm thinking of that deals with people in the middle of the source and the destination that take money for acting as men in the middle when they're not doing anything or providing any service except being in the middle of the transaction. Also, it's beneficial to the sellers & the consumers to eliminate these people. I think they're called 'rich greedy bastards.'
Seriously, hosting a document for me to view doesn't cost $100/mo. so why are you trying to charge me that? I know it's primarily physics but if any other field wanted to pull their heads out of their asses, they would leave the journals to the professors and start up something like arxiv for the rest of humanity that can't afford an outrageous premium!
My work here is dung.
No, but seriously. They have an auction site with nothing to sell... what's not to like?
There are things that you can run like a business, and there are things that you cannot. Without meaning to be political about it, look at what 8 years of running the country like a business with an MBA at the top has got us.
I have not read the article. If the summary is accurate reflection of the authors' point about this, then it is at once misguided, and foolhardy. The purpose of business in a modern capitalist economy is to produce goods at low prices that the consumers can afford, generate enough profit to please the shareholders and to set aside enough money to do research to develop the goods and services to increase these profits and consumer good down the road. Sure, businesses cannot be left alone to do what they wish and government regulations limit unchecked profit-mongering, but the primary purpose of businesses is to establish a market share and earn profit for the shareholders.
Contrast this with the purpose of scientific research. The purpose varies from gaining a more accurate understanding of physical, chemical and biological phenomena to leveraging these phenomena into processes and contraptions that improve the quality of human life (where you lie on this spectrum depends on how pure/fundamental or applied your area of research is). The only shareholders in this process are the authors of scientific work, and their reward varies from just scientific renown to funding for future research or even commercialization of the fruits of their research. However, to achieve the most progress, scientific research tends to be 'open source', in the sense that anyone capable of understanding, and with financial resources to buy access to the journals (if the work is not presented in the growing number of free journals online) can read not only what was done, but also how it was done (something commercial concerns never reveal).
Of course, scientific journals are often run like a business (at least successful and well-renowned ones), but to extend these ideas to the actual business of carrying out research is utterly misguided. The goals of business (from a businessman's pov) and science (from a scientist's pov) are very different. The authors might as well apply these ideas to conduct of a military for all the relevance it has.
There are a host of other objections to such treatment as well, but I will pause here as people know what they are.
Why don't they just start their own wiki? I would find a great deal of value in a wiki moderated by a team of reputable scientists that published their findings to the great peer review workflow.
They could do more to help. At $15-25 a read most scientific papers are beyond the means of ordinary folks. Even if you are at a university or institute that subscribes to some journals the chances of getting what you want if you're casually browsing across disciplines (from which many great creative insights come) is slim.
I think the search engines contribute to this problem because the search algorithms do not take account of availability. Search for any specialist subject and you will see the first 10 pages dominated by links to the gatekeepers of knowledge who offer shallow abstracts and for-pay access. Somewhere after page 10 you will find an obscure link to the actual authors website where you can get the paper.
Google has the technology to rank availability above kissing the publishers asses. I can only assume they choose not to act in the best interests of the community.
As a scientist who has published work in a few journals, I know that the process is arcane and fraught with peril. There are publishers who have axes to grind and it sometimes keeps good information out of the scientific discourse. Of course, I can't offer a real solution because all peer-reviewed journals involve humans with all of our attendant weaknesses.
Having a high publication helps a graduate student land a good post-doctoral opportunity. As a post-doc, you'll need a good publication record (Nature, Science, Cell) if you want to land a good faculty position at a top university (tenure track). A scientist that can semi-regularly publish in the top journals will have an easier time earning grants (without such, they wouldn't be able to run a lab). Without a good publication record, a junior faculty won't get tenure (the review is typically 5-7 years for the biological sciences post hire). Publications - no, make that publications in good journals - is everything.
From the scientist's perspective, if they have pure research, then, they can put it on a web site, such as the university web site or even their own, and just skip the b.s.
Any yahoo can post on a website. The reasoning behind scientific journals is that the science is peer reviewed before being accepted. While not everything published on Nature, Science or Cell is top quality work (politics does play a role), the signal to noise ratio is much higher than say, International Immunology. The science presented in the top journals usually has a much higher impact factor than the 'lower' journals; i.e A paper published in Nature Immunology or Nature Medicine typically has a much broader impact on the field than, say, a paper published in Journal of Immunology. That's not to say that the JI paper is worse than the Nat. Imm. or Nat. Med. paper - it's not. Just that the JI paper will likely be much more narrow in scope.
The authors then go on to discuss a variety of economic terms that they think apply to publishing
Which is probably the most problematic point.
While even in the current crisis "market models" are still hip, they don't give the answers to all questions. A scientific conclusion that starts with "if X were a market" must question, among all the other validations, the "if" part as well.
And while economy provides interesting theories that are helpful in many cases - just like evolution, it does not fit everywhere. So the very first thing that would've to be established is that the model fits.
In this case, I've not seen enough of that, so any conclusions drawn are meaningless until then.
Assorted stuff I do sometimes: Lemuria.org
Current Scientific Method Problematic? For like 2 seconds or so my jaw fell off my face.
Is this serious, or just push back from economists who are upset that a number of papers and editorials have recently appeared in high profile scientific journals questioning the description of economics as science? Allegories, for example, are not scientific.
I think there are a few websites geared towards solving problems with scientific publications (specifically in the life sciences). I think Labmeeting (labmeeting.com) is one of the earliest websites in this field and already has some good functionality for researchers. As a researcher myself, I know that one of the biggest problems is getting scientists to use new tools. Hopefully, when the right tools comes along we will see some big changes in scientific research methods...
One issue brought up in the article is scientists don't know the value of the information they produce. The pieces of information created by scientists can't be divided up into portions of equal value because what is valuable to one scientist is going to be based upon their field of interest and research. The problem is that your "consumer audience" isn't a single market of half a million scientists, it's half a million markets that happen to be made up of scientists.
Instead of claiming that the whole system is broken, just fix the breakdowns in the current system.
The authors argue that scientific research suffers from an uncertain valuation, but this would require that the consumers â" the scientists â" can't accurately judge what's significant.
Any given paper does suffer from 'uncertain valuation', but uncertain doesn't mean the consumers have no clue.
Consider the impact factor of the journal to which the paper was submitted, the reputation of the author, the actual evidence that has been presented in the paper, the fact that the paper has undergone peer review, and what impact the paper would have its claims were to have merit.
In combination, these factors allow better papers to tend to float to the surface. This is fairly typical of a market.
Most fields have decent market regulation built-in, in the form of peer review and independent verification of results.
Politically charged fields (global warming) tend toward unregulated market behavior. Papers are no longer selected based on scientific merit, but instead on hype/scare factor. This makes the value of a paper much more uncertain, and leads to a nasty failure mode (see the current world economy).
There are models that do not have such nasty failure modes (or at least have very different failure modes). Usually, these also fail to produce such good results in the common case.
I dislike the current scientific publishing system because the publishers tend to be paid by both the author and the consumer, and can generally force the author to relinquish copyright. However, the quality of the system seems to me to be far better than this article supposes.
-V-
Who can decide a priori? Nobody.
-Sartre
I'm a grad student in the natural sciences. Some other friends of mine and I started Labmeeting.com because we are so eager to help change the way science gets published.
The current system of peer review is inefficient, arbitrary, and hidden from public view. We definitely need something new, but, as we said in our talk at BioBarCamp a while back, change needs to be gradual enough to preserve consensus.
That's why we're starting by just trying to make research tools that are useful to scientists in their everyday professional lives.
This publishing method is problematic. Damn I need to rewatch Firefly.
Getting publishes is just a further extension of the bullshit that is rampant in "academia".
Your value is based not on what you know or contribute, but on who you rub elbows with and which side of the political fence your "research" can be construed to support.
I long for the days where scientists actually produced something. I'm sick of hearing about study A which shows that X is true, and study B which shows that X is not necessarily true. I'm sick of hearing about a new groundbreaking proof in mathematics, only to later find out that it's for a small, specific set of cases, or that it's just plain wrong. I'm sick of people supporting "peer reviewed" hobnobbing bullshit work, and denouncing anything that isn't. I'm sick of hearing about the scientific "consensus" on global warming, when in fact there is NO consensus, and the vast majority of scientists who are counted in the consensus specifically state their research, data, and conclusions can not be used to verify OR refute global warming. I'm sick of "research" that is politically motivated bullshit that exists only to support a particular ideology, group, or class. I'm sick of scientists in the west being restrained to the point of obsolescence by laws and "ethics" that are merely political. I'm sick of useful research and inventions and ideas being held back due to legal issues, corporate greed, and political bullshit. I'm sick of "science" that has no regard for the scientific method. I'm sick of "scientists" merely massaging inconsistent data from tests sloppily performed by some grad students.
ETC.
Due to the publish or perish mindset at universities, scientific authors must be prolific. Putt's Law (good book) lampoons this quite well, roughly akin to an Amway style ponzi scheme. Sharing new knowledge with the larger community is no longer among any of the major motives for cranking out papers. Frankly the system punishes those who would compile and distill the huge number of obtuse and often stupid articles into a useable form for us rank and file engineers. Such useful efforts are not "new and novel", despite what would be a great service for those of us doomed to wade through the stacks and stacks of crap papers written in acadamia-ese.
In my job, a hardware design engineer, I find that most of the modern papers are indecipherable and irrelevant at best. Only occasional gems make it through and actually apply to my day job (of designing state of the art T&M hardware). By contrast, the old journals from the 70's and 80's easily have a 10:1 better signal to noise ratio, despite their dated nature.
That this article got published in PLOS Medicine is a data point saying that the publication model for PLOS Medicine is flawed. That's about it.
The authors don't bother to back up any of their assertions. If there is a winner effect, for example, the most prestigious journals should have the highest rate of publication of junk results, whereas lower-ranked journals should be more accurate. So, is this true? Did the authors bother to look, or even to think about and discuss it?
Also, does "overpayment" correspond to "poor quality science" or to "only slightly more cool than the rejected paper, on second thought"?
Now, it is more true in the medical sciences that positive results are published that claim to show p0.05, but are one of a dozen similar studies 11 of which have not shown an effect (i.e. overall there was no significant finding). But this recognition has nothing to do with bidding per se; it's not that the journals are picking the high tail of a distribution of value so much as that they're seeking statistical significance without controlling for the number of times that the study was done.
And as the summary says (which is actually better than the research article itself, IMO), there are a number of other problems.
There are certainly ways that one might seek to improve scientific publishing. But this seems almost entirely off target and/or ill-supported to me.
I love Thomas Jefferson as much as the next American, but there are certain things you listen to him on and some things you don't. Civil liberties, the scope of government, certainly. The economy.... not so much.
Jefferson wanted us to farm our way to victory. Here's some primary source stuff on the subject for your edification/amusement.
Just because he's a founding father doesn't make him a visionary on everything. See also: slavery.
Ceci n'est pas une signature.
One of the major problems with scientific journals and the peer review process is that we have a positive bias for publication, in that you are much more likely to be published if your study has positive results than if it has - equally valid so as not to have everybody else keep doing the same thing and failing - negative results.
Half of getting into Science and Nature is politics, not science.
And just TRY to get something published about improved methodology in statistics for genetics studies ... hah! You have to publish in obscure journals or start your own self-publishing annual or biannual workshop and then attach it to a positive study to get it out there.
-- Tigger warning: This post may contain tiggers! --
If its a reputable journal, the evaluators assume that the articles have been thoroughly peer reviewed , and the quality can be taken as read.
The big assessment exercises (such as the 5-yearly RAE in the UK which determines the research ranking of universities) have to "assess" a metric shedload of papers - so they're not going to spend too much time on each one!
Of course, the reliability of this assumption is legendary.
In a survey of 100 programmers, 111111 thought that duck-typing was a good idea.
Did they try saying how it is like a car?
A softball question. One simple example of the failure of the market is the apparent inability of science publishers, particularly in the pharma area, to publish so-called negative results or to spin negative results as if they are postive. In epidemiology and in pharmacology, negative results are at least as important as postive ones ("first, do no harm"). Yet, the greater economic forces of pharmaceutical sales (and nutricutical sales, and outright woo sales) incent the supression, or simple failure to publish, of such findings in pernicious ways. Check out Ben Goldacre's site (and buy his book while you are there). Tucked away among various rants against, among other things, media coverage of medicine, you will find several discussions about this very phenomonon, and why it is so incredibly bad.
A paper that was published in the open access journal PLoS Medicine has now examined scientific publishing using economic concepts and concluded that the way things are done now is inevitably problematic.
Seems a bit of a conflict of interest here. Of course the open-access journals are going to suggest that "the way things are done now" --i.e., traditional journals-- are "oligarchic" and "distort science".
http://www.geoffreylandis.com
... to know which way the wind blows.
You do, however, need a working and publishing scientist, myself among many here, to be able to tell you TFA is wrong in that it's not right enough, and confusing in that it's a ridiculous metaphor being used where facts would suffice. Furthermore, there are more problems in the process than they don't even touch on. One that comes to mind, and certainly should have to an economics viewpoint on the subject, is the effect of research grants on the production of and bias created in science, the effect of prior publications on getting those grants, and the effect of possible future grants to be gained by producing and publishing certain things in certain ways and places.
Another point that fails due to faulty assumption is that science shouldn't be incorrect and should always be correct, this assumption being used to specify in which cases they feel this most likely to occur. We know full well it's not correct. It never is. If it were we'd have had the Blinding Glimpse of the Obvious Truths. Science runs on statistics, with chances of false positives and negatives calculated and presented. Independent replications are required to narrow those margins but can never close them. We produce the best supported educated guesses and then argue over them until we come up with better ideas and tests, ad infinitum. Economics as considered in TFA would assume the same mistaken model that results in the inevitable end point of monopoly when the best possible product is created. No scientist who understands the process would ever try to present themselves as having done so. We operate under the model as presented by John Nash* wherein the best collection of choices are all sub-optimal because this produces the best aggregate result. We do not do so in order to achieve that result, we achieve that result because we know science to be imperfect and thus sub-optimal.
Finally, this appears in a medical journal, where economics is much more a driving force than most other fields. The model presented comes from bias within that field and so is not very generalizable. Other fields would produce equally valid and equally invalid/biased results based on their own models.
Taken out of the economic context, the points raised are well known to the philosophy of science (where from Ph.D. is supposed to be derived), and have been explicated far more deeply and with better generalization throughout the history of the scientific method. These errors are merely compounded and made more relevant to economics when the present publication models are given their present level of import.
* John Nash's writings and conclusions are fairly astounding in their revelations, but are fairly dense. The presentation of his major discovery regarding best aggregate results coming from individually sub-optimal results are very well presented in "A Beautiful Mind" where he tells his friends that the best way for them to all get laid is for none of them to go for the blond, and all go for the brunettes. This is one of the most insightful translations of science to popular media that I've ever seen.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
"... treating the entire process like an economic system, with publishers as bidders at an auction, authors as sellers, and the community at large as consumers."
Not really, not at least in biomed papers.
With those the scientist actually *pays* the publisher to print the article. A paper, especially if it contains colour images (microscopic slides, colourful graphs etc) can set you back by several thousand dollars. That's why there's a little disclaimer under each article that states that since the author paid for the publication, the article legally is paid advertisement.
So no, the authors are not sellers. Only the publishers are the sellers, selling article space to the scientists, advertisement space to corporations and the end result back to scientists. The scientists who publish don't even get the complimentary free copy, although as an author, you can ask them to send you the PDF that you sent them in the first place; now this service is free. On the other hand, if you want the whole magazine on your bookshelf, fork out $200 for a single issue.
It is a wonderful business model:
- Get the article from scientist (free)
- Send it to other scientists for peer review (free)
- Accept it and charge the author (income)
- Sell the advertisement space, at least 50% of the mag (income)
- Print the mag (expense)
- Sell the mag *way* above production/distribution cost (income)
- Keep the copyright to every article so that the author can't republish it without paying you (possible income)
- Profit!!!
The Underpants Gnomes had no clue about business...
Plus there are further tricks - if you manage to bribe the execs of some research association so that membership in the assocation also means a compulsory subscription to your periodical, then the customer base is guaranteed. Of course it is usually sold as "we had to increase our membership fee, but now membership also includes a complimentary subscription to magazine X". This of course increases your readership, making your ranking higher among the sci mag list.
Ah, yes. Chances are that the money they take from the scientist is public money from some research grant. Yet the copyright to the article belongs to the publisher, a private organisation that actually has very little to do with scientific research and everything to do with making profit. That's clever! Even better, that when research grants are awarded, they check your publication record. If you published your articles on the Web, that is worth exactly nothing. Publishing a handful of articles in a couple of those high-ranking expensive publications significantly increases your chance to get a grant, so that you can publish some more...
Imagine if the **AA could find a way to implement this! Every musician and actor paying them to be included on a CD or in a film, plus selling ads even smack in the middle of a CD (music stops, someone screams about the advantages of washing powder X, music continues) and making it compulsory that members of any civil organisation buy a certain number of CDs or DVDs every month. Those Hollywood dudes know diddly squat about making profit.
The authors argue that this situation makes the publishers, as they try to attract the hottest research to their pages, in a position analogous to bidders at an auction, and the authors analogous to sellers.
That's just not how it works. There are two classes of papers:
1. Papers that everyone knows will be valuable because they are made by the big names in the field. The author can choose where to publish, but in reality nothing like an auction happens. Those are uncommon.
2. Papers by new and relatively unknown people. Those make up the majority of papers. In this case it's the authors who try to publish in the best journal possible. High IF journal = more citations = more recognition.
If I were to make a comparison, it's more like show business than an auction. The stars play leading roles in major pictures while the rest has to take episodic roles and obscure productions. Once in a while there's a production that's popular based on star power rather than an interesting story. One major difference here is that the filmmakers rip off both the actors and the viewers. I find it outrageous that all those intelligent people are putting up with the "scientific publishing" scam.
Those who would give up liberty to obtain working drivers, deserve neither liberty nor working drivers.
The current system is antiquated, costly and slow. It constricts rather than enhances the flow of information between scientists. Journals of publicly-financed research should be peer reviewed, supported by the universities, and be maintained on the web. Their contents should be freely available for the taxpaying public that finances them. The publishers are trolls under the bridges of understanding, exacting high tolls and impeding the spread and limiting the availability of knowledge.
Go read various articles at The Last Psychiatrist for a different (and to me, more reasonable) perspective on the problem.
Media that can be recorded and distributed can be recorded and distributed.
-kfg
Nevermind the results that emerge once one accepts the model as true. The real trick is finding the right model.
The model that's used in the article starts with a lot of assumptions.
Starting with the one that scientific articles are a commodity. Now one aspect of commodities is that they have a price, and that the value of X grams of gold is the same as that of Y kilograms of lead, simply because the price is the same. This for example does not hold for scientific articles. Many mediocre (or even weak) articles do not increase the citation factor of a publication. You need good to very good articles, and the odd seminal article to do that. There is something quantitative to the value of scientific articles, or at least some extremely non-linear relationship between "quality" and price. Therefore publications aren't "commodities" in the usual sense of the word.
Another assumption is that the consumers (scientists) can't assess the value of a publication. Well, they can assess the value of an article at the time they read it. What they have much more difficulty with is to estimate the value of an article at some point in the future. Which may be very different from its value today. Take e.g. the pure mathematics involved in coding theory. A completely academic (and therefore not so often cited) subject until the advent of the music CD, DVD's, and the need for error-correcting codes. Suddenly this transformed the "value" of theretofore obscure mathematical publications in ways non-one had foreseen.
Of course this doesn't mean that a model that assumes they are will be totally devoid of value. On the contrary. One can study the properties of a hypothetical market in which they are, and then compare the properties of that hypothetical market with what one sees in reality.
There are lots of discrepancies, and then one can adjust one's modelling assumptions, change the model, and study the changed conclusions. This is one of the ways to advance science.
The only thing one should *not* do is treat the outcomes of this little modeling exercise as anything other than a particularly systematic and disciplined form of speculation.
Perhaps going a little OT (!= Outer Thetan) here but hey this is /.
Worked for a science institute but not being a scientist i can take a laid back view and offer an independent viewpoint.
Firstly, Scientists need money. To live as well as perform their actual science. For a long time now there has been a shift in funding from individuals and organizations who are interested in promoting science to organizations that are simply looking to create patents so they can live off the back of the researcher work.
The scientists are totally caught up in this. They need to get the money, so they tend to do 'trendy' science for which they can get funding (this is a generalization of course) and the most important, their research MUST work. In other words they have a theory, and then will perform the same experiment 100 times until they get results that fit their theories (really, ive seen this happen and even had it admitted to me over a pint). Then they publish (the more well read journal the better) and everyone is happy.... until the next person comes along and tries to duplicate the earlier work or build upon it because they have a theory that us built upon the previous theory .... that is only correct 1/100 times.
They are all so desperate for money to recognition, that real science is becoming a thing of the past.
Here's a braindump of a few of my thoughts from the graduate student viewpoint:
The typical way my supervisors determine how good a particular journal is is based on its Impact Factor, basically an indication of how many citations the average article gets within a particular time period (I think its 3 years). This suggests that the value of a published article is the number of citations you get in other papers.
Since about 4 years ago, pretty much the only papers I've been reading have been those available to me for free -- This includes those publications for which my university has a subscription. It's only when a paper is absolutely critical to my research, and no other free paper exists with similar research, that I fork out the couple of dollars required (and ~1 month wait) to get a copy of an article via the library's Interloan service. Researchers are expected to read the papers that they cite, so more generally, I would expect papers published in freely-accessible journals to have more citations (and therefore have more value) than those with a pay-per-view model.
This value gets diluted by people who cite second-hand without reading the article (e.g. "X said Y said Z", so I'll say that "Y said Z"). Researchers can then get articles cited by simply getting a highly trusted researcher to cite their own paper, even if it is in a pay-per-view journal. This is the quicker way to get a paper done, but ends up with a chinese-whispers distortion of the original research.
If people respect your past research (which can be indicated by how many citations of your papers have been made), it is easier to get another paper published, with less checking carried out by the reviewers of the paper. And, with each new paper comes more citations, so a greater perceived respect for your research.
I believe that both open access and peer review are important. Doing this in such a way that the reviewers can't avoid a good critique of the article is difficult.
As an aside, funding is so driven by new/novel results that funding bodies often won't provide money for repeating experiments done by others. Richard Feynman has made some nice points on why this is a bad idea.
Ask me about repetitive DNA
Scientific research should be published anonymously, with a code to later identify the scientist(s) involved. That would take the 'branding' out of science, and make each paper stand or fall on its own merits.
Serving your airship needs since 1995.