Slashdot Mirror
Can Bad Scientific Practice Be Fixed?
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
The case against journalism is straightforward: much of the news articles, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, journalists has taken a turn towards darkness. The apparent endemicity of bad journalist behavior is alarming. In their quest for telling a compelling story, journalists too often sculpt facts to fit their preferred narrative of the world or retrofit hypotheses to fit their data.
Unlike journalists, however, science will always have to bow to reality. So, yeah, bad science practice will eventually run aground when reality hits, no matter how many epicycles one add to the model. But bad journalism will persists as long as it attracts eyeballs.
Oliver.
The MMR vaccine fiasco is of course the classic example of this; there are still people acting on the assumption that the lies were true, and that's getting people killed.
...when we replaced the scientific method with scientific consensus?
That 99 out of 100 scientists agree one thing is true doesn't make it true - it may be, it may not be, but the number of people that believe doesn't make it so.
When the scientific community is caught 'correcting' raw data and ostracizing 'non-believers' that challenge their beliefs they undermine the public trust in 'science'.
I was taught that the scientific method welcomed challenges to accepted beliefs - a return to that position would go a long way towards reforming belief in science.
Ken
There will always be shitty studies out there. With the proliferation of these pseudo-journals there will be even more bad science out there. This science is a waste of time of money but I don't think it poses much of a direct threat to progress. The bulk of the wrong studies are likely also the obviously bad and unintersting studies. These are the studies that nobody reads. The quantity of genuinely significant work (stuff that pushes forward a field) is tiny. When something that looks like this comes out it is immediately mobbed: people rush to reproduce the results and/or use the new techniques. If it's wrong we'll know very soon. In practice there is always an attempt to replicate the important stuff, even though the publish or perish nature of science means that pure replication studies are rarely carried out and instead are dressed up as a minor extension of preceeding work. The lesson is that it's dangerous to treat a single study as definitive. Wait for the field to catch up and, where appropriate, wait for the meta-studies.
soylentnews.org
Much of the problem comes from studies being published whose data is not robust because the sample size is too small to be meaningfully significant. This needs to be headlined in the abstract if it is published at all; the best magazines should refuse anything without a decent sample size, whilst the ones further down the food chain should have statisticans on hand to ask hard questions.
Discovering an apparent effect should result in more research - not a rush to believe...
Too many "scientists" are more concerned with the next big grant than with doing quality research. And getting grants is often a lot more about politicking and ass-kissing than making a case for why you actually deserve it.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
Science has always been full of bad science. The people involved have always had agendas. The problem is that we are creating so much data, that it is hard to process and identify what was created in a rigorous process and what is just a pile of crap. And, it is not easy to tell them apart. Then you have people involved. Newton tried his best to discredit Hooke. Hooke was lacking in some areas, but a genius in others. Some scientists just create large quantities of data, and don't know what to do with it. Others have a specific idea, and ignore anything which proves them wrong. Science has just gotten so big, it is hard to find the good amongst the, not really bad but, useless. Scientists must publish or be ignored, so they create anything they can to keep going.
not as much as billions of $$$$ of some new drug that may or may not work and by the time the lawsuits come you are rich and retired
Feynman's take:
We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Two more examples from Ignition! by John Clark.
James Dewar (later Sir James, and the inventor of the Dewar flask and hence of the thermos botde), of the Royal Institute in London, in 1897 liquefied fluorine, which had been isolated by Moisson only eleven years before, and reported that the density of the liquid was 1.108. This wildly (and inexplicably) erroneous value (the actual density is 1.50) was duly embalmed in the literature, and remained there, unquestioned, for almost sixty years, to the confusion of practically everybody.
Bill Doyle, at North American, had also fired a small fluorine motor in 1947, but in spite of these successes, the work wasn't immediately followed up. The performance was good, but the density of liquid fluorine (believed to be 1.108 at the boiling point) was well below that of oxygen, and the military (JPL was working for the Army at that time) didn't want any part of it.
This situation was soon to change. Some of the people at Aerojet simply didn't believe Dewar's 54-year-old figure on the density of liquid fluorine, and Scott Kilner of that organization set out to measure it himself. (The Office of Naval Research put up the money.) The experimental difficulties were formidable, but he kept at it, and in July, 1951, established that the density of liquid fluorine at the boiling point was not 1.108, but rather a little more than 1.54. There was something of a sensation in the propellant community, and several agencies set out to confirm his results. Kilner was right, and the position of fluorine had to be re-examined. (ONR, a paragon among sponsors, and the most sophisticated —by a margin of several parsecs — funding agency in the business, let Kilner publish his results in the open literature in 1952, but a lot of texts and references still list the old figure. And many engineers, unfortunately, tend to believe anything that is in print.)
For years people had noted that a standing drum of acid slowly built up pressure, and had to be vented periodically. But they assumed that this pressure was a by-product of drum corrosion, and didn't think much about it. But then, around the beginning of 1950, they began to get suspicious. They put WFNA in glass containers and in the dark (to prevent any photochemical reaction from complicating the results) and found, to their dismay, that the pressure buildup was even faster than in an aluminum drum. Nitric acid, or WFNA at least, was inherently unstable, and would decompose spontaneously, all by itself. This was a revolting situation.
All of this goes to show that even well-respected scientists and engineers are not immune to bad science.
"Publish or Perish", Degrees that require new original ideas, Strict hierarchy structure...
Academic institutions are culturally stuck in victorian times. So if you want to work up, get the choice projects and research, you need to publish. The more your publish, the higher the chances you will move up. Because there is so much published material, people don't read it much, so they found that they can get credit for half ass work.
Your name becomes your brand, so when you try to get a grant your name+institution you will work for will get you the grant money.
There isn't any reason why Say State University of New York Buffalo can't get a grant to study seismology, but chances are it will go to University of California Berkeley not because they will do a better job, but because of the name.
Finally institutions haven't learned how to deal with today's political climate with the attempt for breaking news. Every Hypothesis is sold to the public as a new Theory... Then if that Hypothesis is shown false (as it is common in science) then the media who may have a political slant will go and say see Science is Wrong again, just like our political stance has predicted!
Science for the most part is quite work, collaborating with like minded people, with checks and balances to try to filter out strong egos. But it has gone commercial so these checks and balances are weaken as strong egos will win out.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Can bad scientific practices be fixed?
I whipped together a quick study that shows that it is completely impossible. I'm sorry, it can't be fixed.
On individual days and in individual studies the science can be protected, but you will never completely remove even unintentional bias.
Willful misrepresentation of the facts to satisfy an agenda will continue as long as humans are involved in the experimentation or in the compilation of the results.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
When 1000's of studies, all done differently, with different data, in different places all come to the same conclusion....you ask for 1001 because, hey, you never know, right? Here's three we can put to bed: the world is not flat, vaccines work well, and smoking causes lung cancer.
No, they were illiterate in the truest sense of the word.
I had students who were unaware that books have page numbers. That's how frequently they cracked a book during twelve years of compulsory education:
I.e., never.
They couldn't read the textbooks. They couldn't read my PowerPoint presentations. They were incapable of following lab manuals -- a complete killer if you're in a systems or network administration class. They detested typing and would not accept my assertion that it's a key skill, one that they'll use continuously in the field.
No, sadly, they are simply illiterate,
Microsoft leads to Bluescreen; Bluescreen leads to downtime; downtime leads to suffering.
I have witnessed way too many brilliant, and I mean off the scale brilliant graduate students who are forced to pretty much credit their work to some 60+ year old very tenured professor because he is the only one who can get access to the money. But worse than that I see the same off the scale brilliant students being told that they are wrong wrong wrong. Not because they are wrong but because when they are shown to be correct it will upend the research and conclusions that entire careers were built upon.
I find that many senior professors/scientists never really accomplished anything and simply became experts in an established field further establishing that field. They are threatened by anyone who comes along and shakes the tree which might cause a few of their most rotten fruit to fall. But they are also threatened that if recognized that a truly great young scientist will come along and "steal" all the grant money that is rightfully theirs because of their seniority.
There are the rare senior scientists who encourage new and radical thinking along with making sure that credit is properly assigned (first name) but pretty much without exception these are scientists who accomplished something in their day.
I find a very common song sung by these terrible scientists is that all science is now to be done by groups. Yes groups are often required to conclusively put something new to bed but almost without exception great science had some key crack opened by some one person(or two) thinking way outside the box; not merely going through a checklist.
I have long thought that one of the reasons that so many great scientists are a bit autistic is that only this way can they ignore the continuous social pressure to conform to the groupthink that the lesser scientist would prefer they would. Whereas the more social but less capable scientists are the ones who can rise to the top on little or no accomplishments and cajole and structure the system so as to provide them with a huge cut of the grant money.
Some of you may be visiting that Amazon link and wondering how the hell weilawei ended up spending $1,000 on a book ("Ignition!"). The book is too wonderful to be limited to the big spenders. Search for it online and you will find a PDF easily enough. It's an awesome read.
I think part of the problem is that nobody wants to publish a paper where the experiment failed--but they should.
Failures are useful; they're not wasted time. You've almost certainly learned something from a failed experiment. Maybe you learned that the setup wasn't rigorous enough, or maybe you just learned that a certain avenue of research wasn't viable for one reason or another. I get that journals are looking for breakthroughs, but it would be so useful to read a paper in your field and find out that someone already tried the thing you're attempting, and now you don't have to fail in exactly the same way.
But that requires a much more collaborative system, and one where the community is interested in finding answers, not glory.
There is no vaccine for the common cold, not even for the "unusually vulnerable." There are over 200 different cold viruses. As a kid, you get lots of different colds, as you get older, you get fewer because you've already been exposed to a large cross-section of them. The next generation is going to have much bigger problems because they won't have been exposed to many of them when they were young - kids with colds are not allowed in day cares so nobody else gets exposed, nobody lets their kids play in the mud any more, everything has to be sanitized (like good old soap and water isn't good enough - you have to have an antibacterial soap).
"Transparent" is a shit show that trades on every stereotype going. A man in drag is NOT a transsexual.
I agree it's not a problem. As can be seen at Retraction Watch, lots of bad science if found out and retracted. That's a good thing not a bad thing. One could ask how much of published science is made up and undetected but a better question would be how many results are simply crappy in the data or crappy in the analysis. It surely dwarfs the latter. But who cares. If the result is important it will be replicated. if it's not important then no one will cite it.
ultimately it's the well cited articles that also get vetted by reproduction. Those constitute the body of science moving forward. the rest goes into the gutter of history.
In skiing the saying is, if you fall and your fall isn't forward your not being aggressive enough. It's the same in science. People will make errors. If they weren't then then were not paying for aggressive enough research.
Some drink at the fountain of knowledge. Others just gargle.
Actually there is not a problem with science there is a problem with biomedical research which the author of the article keeps confusing with all science despite actually referring to fields such as particle physics which does not have this problem. That's not to say that we do not have mistakes but these tend to get caught quickly and retracted e.g. faster than light neutrinos.
Except for medical research, I'd say most of science is the same way as particle physics: the odd mistakes which tend to get caught quickly. I don't hear of frequent retractions or contradictions by chemists, mathematicians, computer scientists, geologists or even non-medical biologists like you do frequently for medical studies. In fact it is incredibly ironic that an article written by a medical researcher criticizing the poor practices in his field is so inaccurately and carelessly written. This aptly illustrates at least part of their problem.
The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue.
That has ALWAYS been true. In fact just about the only way to make a name for yourself in science is to show that someone else is wrong about something. Einstein is famous because he showed how Newton was wrong. We put forward hypothesis, test them and (in what should be a surprise to no one) most of them ultimately turn out to be wrong or defective in some way. As a general rule that is both acceptable (to a point) and expected.
Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.
Again, why the notion that any of this is somehow new?
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative.
Bullshit they aren't incentivized to be right. Being right is hugely incentivized. The problem is that it is hard to be right about something that is actually complicated and meaningful. So we have to break big problems up into little problems and most of those aren't consequential and many are going to turn out to be wrong or dead ends. Not every bit of science is going to be of world altering importance. Some people are doing some shady things to earn a paycheck and stay in the game but they tend to get found out in due time. Science is remarkably effective in weeding out bad data over time.
"Publish or Perish", Degrees that require new original ideas, Strict hierarchy structure...
Academic institutions are culturally stuck in victorian times. So if you want to work up, get the choice projects and research, you need to publish. The more your publish, the higher the chances you will move up. Because there is so much published material, people don't read it much, so they found that they can get credit for half ass work.
Your name becomes your brand, so when you try to get a grant your name+institution you will work for will get you the grant money.
There isn't any reason why Say State University of New York Buffalo can't get a grant to study seismology, but chances are it will go to University of California Berkeley not because they will do a better job, but because of the name.
Finally institutions haven't learned how to deal with today's political climate with the attempt for breaking news. Every Hypothesis is sold to the public as a new Theory... Then if that Hypothesis is shown false (as it is common in science) then the media who may have a political slant will go and say see Science is Wrong again, just like our political stance has predicted!
Science for the most part is quite work, collaborating with like minded people, with checks and balances to try to filter out strong egos. But it has gone commercial so these checks and balances are weaken as strong egos will win out.
This reminded me of two things:
1- One of my favorite Roy Scheider lines from 2010: "Look, just because our governments are behaving like asses doesn't mean we have to! We're supposed to be scientists, not politicians!"
and
2- Dr. Jeff Hawkins, the inventor of the palm pilot and handspring lines of devices, who is an avid researcher in the field of artificial intelligence, pointed out in his book, On Intelligence, the following about his approach to his interests and career path:
"Frequently hypotheses in the academic environments don't pan out into ground breaking research and as a result can be career enders." This is why he approached his study of neuro-biology to the end of designing and building intelligent machines, to the corporate research and development environment which tends to take more of a "Back to the drawing board" approach to engineering and science programs that don't pan out into discoveries or innovation. This is a much better approach for many obvious reasons, but part of the problem is that academic research is too quick to blame the researcher and not the questions or the actual research or research approach and black list the people involved, which is very much like throwing the baby out with the bath water. It is no surprise that academia has serious problems with the integrity of it's publications (which is the root of the actual problem pointed out here) because they have created an environment where it is profitable or expedient to be less than honest, at least in the short term, if there is one constant in life, it is that nothing remains a secret forever. Academia would do well to reward the actual merits of research that does not pan out into something groundbreaking, because like Edison, it adds to the body of research that can hep to define later research that does pan out into something novel. (like the 1000 tries at finding the appropriate material to use as a filament in the first light bulb and the famous quote "I just found 999 ways not to make a light bulb" before he settled on tungsten.)
There are so many talented scientists and engineers that are unable to find places to apply their talents due to the system that is in place in academia making the process work against itself in this manner. I would say this is why (coming full circle here) we did not actually end up exploring the outer solar system in the last decade. (2000 - 2010)
If I remember correctly, the scientists were saying that they MEASURED faster than light neutrinos, and were soliciting community aid in figuring out what was going on. They weren't confident at all of their results.
It's arguable that if they hadn't published their measurements, it would have taken a lot longer for them to have got the help which resolved the issue.
Did I just make a case for knowingly publishing results which are very likely wrong? Does it in fact boil down to simple honesty from the scientist about the likely validity of his claims/observations?
Even in the biomed area: "The subject study is admittedly small, however, if the results can be firmly established in a larger study, then significant medical benefits will accrue...."
--PM
Measles vaccine effectiveness is one that is specifically in doubt.
Having looked at this problem, I note that before and after the measles vaccine was introduced, we saw a three order of magnitude drop in US measles cases with similar declines in other countries, correlating with the introduction of measles vaccines in those countries. There's just too much of an effect to hand wave away with the assertion that the world no longer practices measles parties as much as it used to or with the other assertions you make.
Also, lab tests were developed and began being introduced at the same time as the vaccines that only verify 100/25,0000 of suspected cases. A suspect case of measles is not a case of measles. It is not even a diagnosis of measles. It is a case where doctor is covering their ass for a measles-like illness by ordering the test. There is no reason today to expect a "suspected case of measles" in the developed world to have a high likelihood of being a case of measles, especially with the extremely rare incidence of measles. There is no actual evidence here that doctors have a high likelihood of misdiagnosing measles.
You know, this stuff has been explained to you before and yet you continue with your erroneous assertions. When are you going to listen to reason?
Papers get 'impact scored'. Based on the number of times they are cited by other papers, especially other high impact papers. Basically Google page rank for papers. If Google ever tried to patent 'page rank', scientific papers 'impact scores' are prior art. It's even done 'on the internet'.
Not surprisingly, this is also gamed.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
First, the gold standard of scientific proof is experimentation.
Uh... there's a lot more to science than that. But even if we take your word for it, the climatologists create statistical models based on observable variables and fit those models to collected data. The better the fit, the more accurate the predictions.
One issue with academia is that all research must be the stated hypothesis is confirmed, i.e. a negative hypothesis result is not considered valuable. Even though the elimination of a degree of freedom from consideration for further study is one of the cornerstones of science. Instead, everyone must make something new and groundbreaking.
"Who are you?" "No one of consequence." "I must know." "Get used to disappointment."
Right, I'm in the humanities and there is this running joke that you only need to publish one really bad and obviously flawed paper on a really popular topic, and your career is certain. It's true, one bad paper, a followup book that is even worse published at 'prestigious' publisher like Oxford UP*, and you will get cited everywhere and get full tenure within about 3 years after the book has been published. I swear I'm not kidding, I've seen this more than once.
So much for impact scores and citation indices ...
-----
* I mention this publisher because he's well respected and nevertheless publishes many bad or at least dubious books without a proper peer review. I should know, because they once contacted me, a lowly postdoc from an unknown university, to review the latest book project by one of the most famous researchers in my area. It's obvious that they just googled me, as I'm easier to find on the net than some of my more established colleagues.
As to corporate labs, I'm sorry I made the qualification because you're focusing on it beyond its relevance.
We're talking about scientific fraud. Your comments about the corporate labs is not on topic.
Saying "well they're too focused"... so the fuck what? What does that have to do with fraud? Am I saying "do away with government labs and only have corporate labs"?... Nope.
So kindly shut the fuck up about that because you're embarrassing yourself.
Just forget I said anything about corporate labs at all. No no. Enough. Done. Over.
We're talking about fraud and bullshit science.
Spending 2000 years testing polycarbonate limbs or whatever is not related to either FRAUD or faulty science. Is it a waste of time? Perhaps. But it isn't fraud.
is it fraud?
Answer that question for me... please. Tell me now.
Is it fraud?
Yes or no?
If you say no... and you will because that's a rhetorical question, then you're off fucking topic.
Okay? So... enough.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
My position is that fraud is not ubiquitous due to the checks and balances you appear to have failed to notice (otherwise you wouldn't be pushing such a radical view). Thus your cloak and dagger fantasy would be better used in a situation where it is easier to fake things, such as in finance.
There's a very wide void between the few frauds that are difficult to discover and a situation where your odd espionage ideas would make a difference if they worked at all. Your second point for instance makes zero sense since peer reviewed papers are "audited" with more rigor than you suggest as a matter of course. The first is a nice movie plot but isn't going to work in reality - people cite stuff they read whether their own research is any good or not, so your stuff that doesn't exist is never going to pop up in the "barium meal test", ever. Fraudsters are going to cite the most solid stuff they can find to try to appear legit.
With respect, while you may be for instance a master fisherman this is like attempting to use fishing equipment for mountain climbing - doesn't matter how good you are at using it, it's the wrong tool for the job and there not really a problem since there is a chairlift to the top anyway. You are trying to solve a non-problem with tools that are not as good as are already in use.
No it is not.
You are being very simplistic and also getting things backwards. It's not a case of either close to 100% fraud or 0% fraud. Reality bites when people try to prove physical things that are not real so fraud in most sciences cannot be sustained for long since when experiments are repeated reality asserts itself.
I'm not worried, it's just a demonstration of something in one field that makes zero sense in another. Copying other people's work is not the problem and that's the only situation where your suggestion is going to work.
Sorry, you've been very poorly informed or are just guessing. Peer review is why it takes time to publish. Peer review is why it takes so long to submit a thesis.
Only when there are idiots intent on discrediting entire fields of science for political purposes.