Slashdot Mirror
Most Science Studies Tainted by Sloppy Analysis
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
We all know that a good portion of all studies are based on bogus facts or on other studies based on other studies based on nothing.
:(){
They can study my taint anytime they want. /Karma
Living With a Nerd
And they are routinely reported sensationalistically in the media, and most of you people who are reading this right now swallow it all hook and sinker.
Are you adequate?
Quo usque tandem abutere, Nimbus, patientia nostra?
all i can say is "duh!". Everybody, being under pressure of "you have to publish", publish whatever they can. Sad but true
It is way off the base to say that "most published research findings are wrong". It is often the case that data analysis and interpretation for particular aspects of a research project (like 1-2 figures in a 7 figure paper) are up for vigorous debate. The scientific community can, in the long run, converge on very robust ideas, and drop those that are flimsy. To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.
It should be noted that "medical research" (epidemiology, clinical studies etc...) is very different from basic research (mechanisms, pathways, etc...) and the threshold for acceptance in journals that cover basic research is much higher than that for medical journals. i.e. There is significantly higher oversight and peer review criticism over basic research than there is medical research and the two fields should not be confused.
Visit Jonesblog and say hello.
With our current education system and a public that's not willing to scrutinize things carefully enough, I don't find it shocking at all that science journals are publishing bogus studies. It is a shame, certainly, but its inevitable until the scientific community as a whole moves in a different direction.
How do we know the study that shows that most studies are tainted isn't tainted?
According to my research, most studies involve about 84% error rate due to flawed statistical analysis caused by people pulling statistics out of their arse. The other 16% are flawed due to NOT actually pulling statistics out of their arse.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
This was certainly true in my experience. When I did a review of mathematical methods in my area a while back, most papers had basic calculation errors, missing information that made reproducing the work difficult or impossible, and they all used carefully selected examples to show their work in the best light.
And one of the first rules is, "Never take a single study as proof of anything! Wait till the results are replicated before you even think of moving to a conclusion."
The major problem is really poor reporting on science research. The news media routinely blazon some **NEW * Scientific * Discovery!!!**. Then you read the story and somewhere around the 10th paragraph you might see that this is based on only one study - and oftentimes even before peer review.
Every scientists knows this. It's a shame the public doesn't. They wouldn't worry so much.
The secret is out! Investigators are looking into whether or not millions of scientists have been using modified versions of SCIgen for their work. The FBI and Department of Termpaper Security have acknowledged the investigation but declined to speculate on the alleged ties between SCIgen and grammar terrorists citing a new law just passed by pResident Bush which allows warrantless underwear tapping.
Authorities are also investigating the connections between Malda, Bush Laden, Bill Gates, Dvorak and Borat SCIgen is a program that generates random Computer Science research papers, including graphs, figures, and citations. It uses a hand-written context-free grammar to form all elements of the papers. Our aim here is to maximize amusement, rather than coherence.
Infiltrated dot Net
/. commentors commenting on sloppy submission about sloppy analysis
pot, meet kettle
His work seems to focus on population genetics and epidemiology, which is notorious for having unreproducible claims due to a combination of uncorrected multiple testing, publication bias and statistical incompetence. This "gender and genes" is a perfect example: someone does a study, finds nothing, slices and dices the data until he gets p = 0.04 for females or Asians or smokers and publishes his breakthrough finding. I'd have been surprised if he hadn't found almost all of those to be wrong.
If you look at more in-vitro molecular biology and biochemistry work, I doubt if nearly as high a percentage of it is clearly "wrong", although quite a bit of it is worthless.
What I'm listening to now on Pandora...
I just read the article and checked his statistics. He did his numbers wrong.
The sample sizes are often small in medical/health studies and human beings have a lot of extra variables that are hard to control. The best thing to do is to wait for the meta-study, where someone analyzes all the studies that relate to an effect. After many studies have been done, do they agree? Do they appear to have been well done? Adding the studies together creates a larger sample size and hopefully averages out some of the variation due to flaws in the method.
I am not a scientist.
That being said, it's my understanding that most scientists work off of grants, and those grants fund novel research. Replicating results is of obvious importance in validating those results, but doing so seems at odds with the funding mechanisms that are the reality for what I would believe to be most researchers.
Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?
(As rhetorical as that might have sounded, I actually welcome those with first-hand experience to respond to it)
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
My favorite scenario is when the author makes claims that are technically true but which a naive reader will misconstrue as meaning something more. The naive population typically consists of the vast majority of readers, but in some cases, it may consist of everyone except two people in the world, one of which is the author. A retraction is irrelevant, because the claims are "true", and clarifications just don't happen in the scientific literature.
And I wasn't disappointed. You also managed to please me with your inclusion of the bogus "The Great Global Warming Swindle", "Global Cooling", and conspiracy theories. Excellent!
As an interesting aside, I thought that this argument had been dropped because it was a little too easy to shoot down: The interesting thing is that, despite warming temperatures, the oceans are holding more CO2 than before (which lowers their pH level as CO2 + H20 = C2H03, carbonic acid). This is possible because increasing the partial pressure of CO2 in the atmosphere (as we've done significantly) more than counteracts the decreased solubility due to temperature rises. It's possible that in the past this was a factor (although you should read up on those time courses and realize that your 800 year figure is also bogus), but it's clearly not true today. Global warming theories aren't based on correlations, they're based on fundamental principles of science.Ben Hocking
Need a professional organizer?
At least an interested group of amateurs are working to replicate some of the more controversial research results (e.g. the Britta reconstruction, MBH hockey stick, and the GISS temperature record). See http://www.climateaudit.org/ and http://www.surfacestations.org/ for more.
TFA
Since the criterion is that the claim is published, someone had to find the study new and interesting. Most new ideas are going to be wrong, especially true the more significant it is. After all, how many crackpot theories were postulated between Newtonian and Relativistic physics? On the other hand, most things easily verifiable, etc, are too obvious to me considered new and interesting. Note, while I find this interesting, I did not come up with this idea. Some economists published a similar study over a year ago postulating this as a reason. Of course, it's probably wrong.
Your ad here. Ask me how!
So, is what you're basically saying is that this study was tainted by sloppy analysis?
Ben Hocking
Need a professional organizer?
When it becomes too difficult for a small mind to sort through all the data out there, there is a place where one can turn: the mindless belief of the religious.
Otherwise known as the ol' "If it isn't obvious to me, a god did it!" mis-thought.
Blar.
The secret shame of the scientific community is that statistical analysis is the foundation of all good research but few Ph.D programs offer more than a single semester worth of train in the subject. Truth is, training in statistical analysis should start in grade school but I doubt that will happen any time soon. One solution is dropping high school and college requirements of calculus and replace them with a year of statistics, which would be useful to more students.....
It's psychosomatic. You need a lobotomy. I'll get a saw.
It is fairly common knowledge that 3 things factor into tenure (in this order): (1) being published (2) bringing funding into the university and (3) teaching.
...
1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.
2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.
3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).
The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.
SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know
The study does not refer to science in general, only to medicine. Medicine is hardly a typical example of standard science.
1) Physicians are mostly trained to be healers (practitioners) not scientists.
2) Medical data is very inaccurate compared to the data in standard science (physics, chemistry, geology, etc)
3) Dealing with inaccurate data requies advanced knowledge of matematical statistics and most medical doctors do not have a basic grasp of this field.
4) Many MAJOR ERRORS in the medical literarure are due to the ignorance of basic principles of statistics.
5) Sure ignorance in the field of statistics is only one cause of poor medical research; however it cannot be ignored.
[The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.]
[No one knows how much shoddy research is out there.]
I'm not sure what the point of this article is besides fear mongering. The goal of most scientific research is to prove a set of assertions - and sure this set may not be fully encompassing or comprehensive - but you've got a model and you try see what fits - and its not always exact.
Take for example the recent debacle
So now I see a new wave of WSJ luddites missing the point.
(Oh please tag this as a troll, I've bashed WSJ again.)
/\/\icro/\/\uncher
Hmmm... what fraction of news published and edited in reputable journals contains factual errors? That would be another interesting story.
But come on... 90% of everything is crap. This is no more or less true in medical research, which is a fraction of the $50 billion total spent on research. OMG that's like "One MILLLLIIION DOLLLLARS". That's like 0.5% of GDP so don't be surpised when it's bunk, it's drop in the bucket compared to development costs. If you haven't figure this out you're still a little naive.
So what do real scientists and engineers (that includes doctors) do? They build something that works, and it's a long hard slog, and most of the problems you encounter aren't the cool sexy ones you thought they would be at the start. They don't make big headlines, and they've mostly been discovered (and solved) by a zillion people before you. The key is solving a problem so well that nobody has to solve it ever again! Then the other bad solutions slowly go away, and people can work on other problems (yes QWERTY is good enough).
Now with medicine, things were made a bit better because of the FDA and the clinical trials model, before the more significant political interference of the last decade. Unfortunately, there's the issue that people really want to believe in magical cures that will save uncle Milt, or their cat so they'll pressure doctors and believe con-men for the rest of time. This is fed by the journalistic-industrial lie complex.
Sure scientists are wrong all the time, or certainly less accurate than they could be, but not usually about big things. More often than not they miss the real interesting results making things come out the way they think they should. It takes someone who really believes in their ability to do something right to discover something really new... and if it's truely astounding they better be prepared to spend the next decade proving it and developing some cool new tools with it. If those things work as they predicted and not the guys before... then hey they must be on to something!
The problem is that there are all sorts of quick-fix "cranberries cure cancer" quacks out there. They're in sales where it doesn't matter if it works. That's why your computer doesn't run any faster with the new whizzy RAM than the old stuff, but it doesn't mean Moore's Law doesn't progress. The key is the guys doing the development and implementing the new ideas, and proving they work. They keep making better faster cooler stuff !THAT WORKS!
The BS you read in most hype journals (EETimes are you listening?) is mostly unproven tripe. It occasionally has a grain of truth, along with a lot of interpretation bias, and if you bothered to read the original article (did you?) and some of their other articles (hah that's harder) you can often tell what the nuance is, how careful the methods were, and whether it's worth trying to replicate yourself. Guess what if you're not in the field that paper really wasn't meant for you (it was meant for the tenure comittee or the guys who gave you a grant etc). The stock market guys may go crazy, but that's because they don't give a damn about science or the truth. Same thing with most journalists these days, frankly. They care about money, and figure their reputation won't be much worse than anyone elses.
Go read the actual article:
http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124&ct=1
It's not nearly as interesting as reading a sensationalistic WJ article, but I give them props for linking to it.
last weeks article about Liberals being smarter than Conservatives. I kid!! I'm a moderate so what do I care! ;)
But once the study is out there, it's taken as gospel unless some other study comes along to discredit it. The vetting should come before the printing.
That might be a better title for this article. Journalists love to take findings from scientific literature and mangle, misconstrue, and misinterpret them. Let's see... a journalist finds a biologist who looks at a particular subfield of genetics, and somehow manages to blow that up into "most published research findings are wrong."
Never mind that there are thousands of journals in enormously diverse fields - biology, chemistry, physics, all the branches of engineering. We can take a sample of 432 articles on a single subject, and extrapolate that to state that ALL scientific research is 'probably' wrong.
Way to further undermine the value of science in the eyes of the layman, jackass.
To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.
Irony police, your analysis?
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
But it's not as funny when you say it that way...
Ben Hocking
Need a professional organizer?
I'll give you points for making an argument I hadn't heard before, even if it's wrong. The basic science is only a starting point, but detailed analysis allow you to make meaningful predictions about what happens when the concentration of CO2 increases. In fact, these predictions have borne out fairly well.
Ben Hocking
Need a professional organizer?
People need to realise that a lot of those calling themselves scientists are not really scientists at all. They don't apply the scientific method. They massage data regularly. They misapply statistics constantly. They don't subject their theories to falisfiability. They waffle, hand wave, engage in rhetoric, and generally do just about everything except an honest to goodness, old fashioned solid, scientific experiment.
Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists. They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.
It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.
Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.
There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.
May the Maths Be with you!
Universities are pumping out PhD's at a prodigious rate. As a manager of R&D, I've interviewed and hired more than my share. Virtually all say they want to do research.
Here's my problem. Only a fraction (I'm guessing 1 out of 5) are actually capable of doing good research. The rest are competent employees for developing other people's research into useful products, but aren't terribly original thinkers, nor show a lot of initiative, nor show the rigour and clarity of thought one wants to see in a researcher.
Frankly, when I "unleash" employees on open-ended problems without much guidance, the majority soon begin to flounder.
There is nothing wrong with getting advanced degrees, but many then feel they are obliged to do original research when in fact they really aren't up to it. This may be one reason why the quality of papers isn't where it should be.
Ya know, you guys just write these papers that are all wrong and riddled with errors. Lucky for civilization, there are plenty of engineers that turn this slop into useful products for truth, justice, and the American way!
This is my sig.
I'll talk about my master's thesis work. I was designing a guided bullet to be shot out of a 40mm cannon with a linearized guidance system and a pack of squibs. The combination of a linearized guidance system and a controller hooked up to the squibs would cancel out any pointing errors and initial guidance errors of the gun+RADAR system as the bullet was in flight, and then hopefully hit the target. I wrote a 6DOF simulation to model all this and that was the basis for my thesis.
Now, to answer your question, I am not personally aware of people getting funded to replicate results, although it would not surprise me. However, there is plenty of research that rubs shoulders. For example, in designing my guidance system, I came across tens of papers of people who had done it already. And there are many hundreds of people who have written 6DOF simulations at some point in time.
When papers go to be submitted, many times they are rejected (by the good journals, anyways) because 'they are too similar to something already submitted' or 'they do not really add anything unique to the body of science'. In other words, more or less, they replicate what has already been done. Whether intentional or not, engineers and scientists do from time to time re-invent the wheel, or come up with a solution very similar to something already done. Which isn't all bad, because then you can research what was done before and compare the existing body of results. (I did, in my case. I ran my numbers through a fellow researcher's sim [with his permission of course] to test one portion of the guidance model, and I ran my numbers through a commercial package to validate my core 6DOF model)
Number 3 is something akin to citizenship: participating in meetings, bringing in guest speakers, etc.
Ben Hocking
Need a professional organizer?
... is available for free: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1182327
Is this Wall Street Journal article on the credibility of (medical) scientific research an invitation to scientists to publish in Nature that modern economic theory is grounded in the flawed assumptions that individuals will make choices based on the most desirable outcome in every instance? Or their insistence on ignoring that an economic system is inherently finite, and thus unlikely to settle into an optimal state? Or how about the time scale of settling, which in many cases would be longer than human lifespans and thus unrealistic in the light of basic human psychology? Oscillatory systems?
They are inviting physicists to take aim at the flaws in modern economic theory. This is sorely needed.
[Speaking to self...] .... this...
A scientific study published that most scientific studies are wrong... therefore there is a good change of it being wrong.... Which means that most scientific studies are right... But if most studies are right then this one is also right... which means...
c.. an.. t take
[Head explodes]
How is your comment in any way relevant? The story is about a tenured scientist complaining about the quality of verification of science results; there is no mention of religion anywhere in this thread, except in your post, where you insert some gratuitous flamebait (as if there aren't enough religious arguments on Slashdot yet).
MOD PARENT DOWN.
> In specific, in the ice core samples used to create the data CO2 levels rose about 800 years AFTER the temperature rose. [etc]
This is a *very* highly disputed interpretation of the data. As I recall, the original study put an upper limit of around 800 years on the temperature rise, but later studies have narrowed that upper limit considerably. There are several that suggest it occured much more rapidly - within 2-3 years. And yes, ice core studies can be that accurate since the cores have layers of ice in them that accumulate every winter and are detectable like tree growth rings.
The particular "documentary" you mention has been widely criticised and soundly discredited now in many countries - the BBC has edited it several times to remove inaccuracies since its original airing and it is now substantially shorter (about 1/3 or 20 minutes) as a result.
Science is ALWAYS right. Its fact. Especially if the scientists I agree with say what I want to hear.
Even if scientists were perfectly honest and disinterested, new results in a "hot" field should be suspect.
Most publications require that results happen at the p <= 0.05 level: that is, that the chance that the result happened by chance is less than one in twenty. For every twenty experiments, one will show the result. If a field is "hot" and one hundred groups of honest and disinterested scientists work on a problem, five would have publishable papers, no matter whether the result is real.
A better way to do science would be to publish a proposed experiment before any data is collected, then publish the results -- no matter whether the experiment succeeds or fails.
"Torture statistics enough, and it will admit to anything."
I'd rather have someone respond than be modded up.
This guy's main beef appears to be with medical studies and other sciences which rely heavily on statistics (sociology, psychology and the other wannabe-sciences). This is not surprising, to be honest. Statistical analysis isn't difficult, but I've known many social science students. They consider statistics to be extremely advanced and have no other mathematical background. As a result, they don't have a very deep understanding of how to mathematically model a system. Naturally, this will lead to bogus conclusions and incompetent analysis work. Medicine has a similar problem, albeit on a smaller scale. Most of the time, statistical analysis will yield correlations, but they won't tell you anything about the mechanism behind what you're seeing, which is what's important in science.
I'd expect the rate of error for physics experiments to be much lower than that of, say, sociology.
http://www.msnbc.msn.com/id/20113753/site/newsweek/
http://www.pbs.org/wgbh/pages/frontline/hotpolitics/etc/cron.html
You'll note this bit from 1979 (nearly 30 years ago): As early as the late 50s some scientists were already discussing how increased CO2 would lead to higher temperatures. This issue is not 10 years old.
Ben Hocking
Need a professional organizer?
an "enviro" and could quite give a shit less in general about how you live your life, but yes, I will care what you do if it involves dumping dioxin into my local river.
It's not your river. It's mine.
signed, Jacks Nuclear Dumping.
PS. If we catch you on our river, we will shoot your for trespassing!
This is my sig.
Soon enough all the believers will arrive to say something similar to my post, except that they actually mean it.
Blar.
"A leading US climate scientist is considering legal action after he says he was duped into appearing in a Channel 4 documentary that claimed man-made global warming is a myth. Carl Wunsch, professor of physical oceanography at the Massachusetts Institute of Technology, said the film, The Great Global Warming Swindle, was 'grossly distorted' and 'as close to pure propaganda as anything since World War Two'.
He says his comments in the film were taken out of context and that he would not have agreed to take part if he had known it would argue that man-made global warming was not a serious threat. 'I thought they were trying to educate the public about the complexities of climate change,' he said. 'This seems like a deliberate attempt to exploit someone who is on the other side of the issue.' He is considering a complaint to Ofcom, the broadcast regulator."
What would the environmental scientist gain by claiming that global warming is real even though it is not? Don't you think it would _hurt_ their credibility and funding? And what do you think there is to gain for making such bold assumptions if they're likely to be false?
Said the WSJ editorial board...
It must have been something you assimilated. . . .
I won't analyze again.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
So - what do these findings have to do with Global Warming? Especially GW pushed by CO2?
If we do have GW its probably pushed by changes in Water Vapour - but of course we can't measure that.
The Wall Street Journal headline is a tautology. (Note that he's not talking about scientific misconduct, only honest mistakes, incorrect analysis or experimental design which could be improved.)
There are almost no areas of science we're "done" with. The most recent paper on a subject almost always points out where previous papers have gone wrong. Thus, the previous papers have some mistake such as a miscalculation, poor design or incomplete analysis. If you pick any paper published in a peer reviewed journal this month, there's a very high probability that at some point in the future it will be amended or improved by some other paper.
What Ioannidis *has* shown in his recent reports is that in genetics, not enough people are publishing on the same subjects. There are not enough "other papers" out there to check on the previous ones. The result is that papers which in other fields would be recognized as needing improvement are instead treated as the final word.
from science citation index, that aprox half of all journal articles have 1 or 0 citations - this suggests that a large fracton of scienc studies are not worth much (altho you could have a long /. discussion on citation anaylsis)
>According to the scientists
According to the scientists, the program misrepresented what they said.
>the concept of CO2 warming was a fairly small area of research that wasn't taken very seriously
On the contrary, it goes back to Arrhenius and is generally agreed to be the reason the oceans aren't frozen over. The existence of a "greenhouse" effect was in science textbooks decades ago.
>CO2 levels rose about 800 years AFTER the temperature rose.
After the temperature BEGAN to rise. Temperature and CO2 feed on each other in a positive feedback cycle. The Milankovitch cycles, by themselves, aren't enough to account for the temperature swings in the geological record. There needs to be some mechanism that amplifies the temperature swings, and CO2 accounts for it.
That positive feedback implies some important things for making policy. In particular, it means warming will go further than you'd expect -- CO2 production leads to more CO2 production, rising temperatures cause temperature to go up further.
and
Coder's Stone: The programming language quick ref for iPad
As soon as anybody discovers anything wrong with scientific studies, everybody from creationists and holocaust deniers to global warming opponents comes out of the woodwork and uses it as an excuse.
While it is probably true that many scientific papers are wrong or poorly supported, what remains still allows science to make progress.
If you haven't seen the documentary, I highly recommend it. One of the key issues they point out was that Gore's graph [...]
I see: so you prefer, instead of peer reviewed journals, to advance human knowledge through the medium of "published documentaries". Science may have its flaws, but it's still better than that.
Your example illustrates another flaw: it really doesn't matter what Gore's graphs show; Gore is a politician, not a scientist. He may be lying through his teeth and it doesn't affect science or the scientific consensus on the question of global warming one iota. The only facts that matter in science are those that are published in peer reviewed journals. That process isn't perfect, but it's still better at getting at truth than any other process we have.
Ben Hocking
Need a professional organizer?
by promoting crap like this -- selfish, self-interested parties often promote their own agenda, even in the science community -- we're actually helping bush, inc. wage war against science. all of the scientists and science professors I've ever met were more interested in finding the truth than in furthering a single point of view. it's that basic curiosity about how things work which leads to discoveries great and small. by focusing on little people who lack such vision or commitment, it's easy to promote the far-right's attacks on human knowledge.
Remember kids, if you're not paying for the service, YOU ARE THE PRODUCT THAT IS BEING SOLD.
We don't have tenure here as such, and the publication requirements for the closest equivalents aren't that harsh (approximately five papers, no time limit).
However most of the university researchers are paid by external research grants, and the unofficial "first cut" for the main source (our NSF equivalent) is purely based on publications. A new Ph.D. should have five papers, after that, it is three papers per year. If the applicants fulfill these requirements, the review committee will look at the actual application.
Three papers per year is less mad than it sounds, as you don't have to be the primary author.
It no longer matters if the research was done properly or not. It doesn't even matter if the conclusions are correct. All that matters now is how neatly the conclusions fit into one's world view.
MOD PARENT UP!!! Excellent contribution to the discussion!
:: cough ::" You know something
is wrong when even first posters complain about accuracy.
The media often contributes by being dishonest and over-interpreting results.
Most "scientific" papers aren't really scientific. The first clue is that they are poorly written, suggesting that the writers want to hide their poor contribution behind bad expression.
Slashdot editors often are fooled by "junk science", I notice. For example, this article was fraudulent in my opinion: Imaging Breakthrough "Sees" Lung Disease.
The Slashdot article The Car That Makes Its Own Fuel [slashdot.org] has a +5 moderated First Post that expresses the consensus of the comments on that story: "first post to call bullshit!
The Slashdot article Hydrogen Won't Save Our Economy said that water was rare. That's a stretch considering most of the surface of the planet is covered with deep water. Maybe Slashdot editors had never heard of the Pacific ocean. Then there's that small pond called the North and South Atlantic ocean.
He implies or states that there is very little to no proper randomized, double blind, placebo controlled studies in this area that valid conclusions can be drawn from. This is false.
Just off the topic of my head, the most compelling recent population-based, double-blind, randomized placebo-controlled trial showing a statistically significant effect was for using vitamin D to prevent cancer. The effect in this 4 year study was so large and statistically significant that researchers say that nearly _all_ previous studies of cancer will need to be revised to control for the effect of latitude (because latitude effects vitamin D levels).
Also, just looking at one hit from a single quick google search brought up randomized, double blind, placebo controlled studies for omega-3 fatty acids and the following health effects: joint function, cognitive/emotional health, respiratory function, and gastrointestinal health. These results are further supported by our detailed mechanistic understanding of the function of specific types of fats in membrane functioning, and as precursors to important hormones in specific synthesis pathways, etc. It can be shown in atomic detail exactly why a membrane doesn't function as well when constructed with the wrong fats. We also know that certain of these required fats can not be synthesized by the human body, and therefore must be consumed in the diet.
The problems of healthy user effects, compliance effects, and prescriber/eager patient effects that he mentioned are factored out in the randomized, double blind, placebo controlled studies I refer to above.
Just because some of the science is junk, it doesn't mean there isn't also good science being done.
Putting aside for a moment the question of whether Genetic Association Studies - the focus of the research paper - are representative of "Most Science", the article does not say the analysis is invariably sloppy, it says it is often mistaken. For genetic association studies, this is not surprising, since it is very difficult to publish a negative result. So, small studies that show a statistically significant relationship are published, but small studies showing no relationship are not. Then, when larger studies are done, the small studies that had the "significant" relationship because of a fortunate or unfortunate set of samples is not confirmed. Indeed, this is what the research article points out; if your threshold for statistical significance is 0.05, then you will report that a chance relationship is significant once in 20 experiments. But, if you can't publish the 19 negative experiments, then lots of chance results get published.
But Dr. Ioannidis has a very narrow definition of science - he only includes statistical studies that use p 0.05 as a threshold for significance. There are, of course, lots of papers that do not show p-values - the purification of a protein, the determination of a genome sequence, the identification of a new fundamental particle. In many cases, p-values are not provided because they are not considered informative - something that happens when the p-value is much much much less than 0.05 (I like my p-values less than Avagadro's constant. With that p-valuep, I think most of my results are correct.)
And, of course, the WSJ misses all of this. The point of the research paper is that you can do everything right, and still be mislead with marginal p-values (0.05). Not sloppy, just not significant enough. We could, of course, require more stringent values, but then we would miss the genuinely rare, but important results.
As the research article points out, results that are reproducible are, in fact, quite likely to be correct. It is perhaps useful to distinguish between science as a paper and science as a process. Most results that stand up to scientific scrutiny over a period of years (that any one cares enough about to validate), are (probably) correct. In some disciplines, which rely heavily on modest thresholds for statistical significance, many results cannot be confirmed.
I downplay exclaimations about "scientific charlatans," but mention two factors that may help to explain momentum in this general direction: first, the proliferation of MA and PhD's, and, second, increasing specialization. As more people enter the university, more of them end up seeking advanced degrees. University and advanced degrees begin to establish themselves as entitlements of the middle class. Today people take advanced degrees without basic prerequisites like talent, or drive. Consequently, the MA and PhD no longer mean as much as they used to. Stupid, incompetent people can earn these degrees, and go on to be stupid, incompetent researchers and practitioners. Not retarded, mind you, just stupid. To my second point, those in graduate studies must make their mark. They must "contribute" something "novel" to the subject they are studying. In some fields this results in a high degree of specialization, because the most general topics have been written to death. You gain funding and your degree certification by picking the most obscure topic. This topic may very well be outside of the competency of one's supervisor, who is encouraged to take on as many theses as possible for the rewards of sabbaticals, and other departamental perks. In the end, one becomes an expert in an area for which there are no experts, and perhaps few peers, the hope being that, generally speaking, one's peers can detect on the broad issues whether you are a fraud or not. But can they detect this? Do they really care enough?
I have worked in a biotech / pharmaceutical environment for over five years now, and I don't trust the average medical researcher or biologist to accurately calculate the weight of a kilogram of stuff. I would say that their data analysis is habitually poor, if I were not convinced that it is actually habitually awful.
I have been trying to change this for five years. My success in this has been such that it contributed strongly to my recent decision to start searching for another job. The reality is that biomedical researchers simply do not believe in doing mathematical analysis of data properly. They consider it an eccentric habit, forgivable but socially objectionable, like smoking. By common consensus, it is considered much too complicated to expect that any of them can be expected to understand. Your average biologist is innumerate to the nth degree, and proud of it.
I blame their education, which seems to stress naive and antediluvian (excuse the word) analysis practices, if at all. I have seen course materials which in their expression of basic mathematical formulas, betrayed that they had been left unchanged since the days when people used slide rules and logarithmic tables for calculations. Most of their other training is strictly qualitatively, not quantitatively, and focussed more on memorizing that on understanding.
If necessary, they will find a crutch to help themselves to stumble along: Find a paper that defines a formula that looks relevant, and then fill in the numbers. They would not bother doing their own analysis, or trying to understand how the calculation works or whether it is relevant at all. The notion that a good statistical analysis of mathematical modeling can actually contribute to the scientific understanding of an issue, is well beyond most of them.
I am frankly, sick and tired of their attitude, and I still have to work with these people every day. And in my experience, my colleagues are actually better than most. I strongly suspect the WSJ is correct on this one.
Before there is consensus on an issue, there is contention. Before contention there is no theory at all. Only at the the consensus phase can a majority of 'scientists' be correct. During the contention phase, there is the old theory that didn't include the new theory or which may in fact preclude the new theory. In both cases, if the new theory is correct, then the old theory was not. At least part of the pile of peer reviewed papers for the old theory can now be viewed as incorrect.
What's interesting is when an old theory that was passed over can sometimes be resurrected and proven correct, often times to the consternation of the consensus.
Exactly. Incidentally, this is also the way that the no-global-warming myth started, too.
Are you adequate?
I recall a comment given by one of our university lecturers. He was quite well-known in his field, and a Nobel Prize winner to boot, so he was asked to review various government-funded scientific research projects to see if they were on time and within budget.
"I said I would agree," he explained to us, "with one condition: if it's on time and it's within budget, then we have to shut down the project --because, whatever that project is doing, it couldn't possibly be research."
He never did get to review the projects.
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
Actually, it was this one http://environment.guardian.co.uk/climatechange/story/0,,2119695,00.html.
It is important to understand what you've read, also. To me it seems that your understanding of the environmental issues is as deep as your understanding of that article.
Also, hyperlinking is the act of linking two or more documents together. Following hyperlinks is more like reading the document. Especially if they are quoted in the context.
So stop being such a pompous asshole, you are dumber than you realize.
But it means the the correlation between CO2 and global temperature on Gore's graph is caused by the absorption of CO2 into the ocean. That means that you can't use the correlation to estimate the temperature rise that would be caused by increased atmospheric CO2 levels.
"Global warming theories aren't based on correlations, they're based on fundamental principles of science."
Yes, but global warming predictions are based on correlations. The fundamental principles of science tell us that since CO2 effectively insulates the planet, increasing the concentration of CO2 will increase the surface temperature of the earth. But the earth is such a complicated and vast system that global temperatures can not be modeled based on fundamental principles alone. So we know it will go up, but we don't know how much.
If the current temperatures are a result of CO2 concentration (and they don't really correlate very well) then we don't have much to worry about. Ocean levels may rise 3 feet over the next 100 years, but people will be able to move out of the way, or adapt in other ways. Rivers under the influence of glaciers will experience more flooding, but those rives can be damed and mediated (doing so would increase agricultural output as well). The temperature increase itself will do little to make the earth less habitable to humans, and there's always air conditioning if we need it. On the other hand, devastating the world's economy in order to reduce (not eliminate) growth in CO2 production will cause millions to die, and everyone else to live at a significantly reduced quality of life. And it will do little or nothing to stem the continued build up of CO2 in the atmosphere.
... where are you getting the time to read /. ?
Some Epidemiological Claims of Sex Differences for Genetic Effects Not Replicated.
This is a *very* small number of claims from a subsection of a single field of one small bit of science. Tarring all of science based on some potentially dubious epidemiology is badly out of line. It would be like claiming that since some spinach has made people sick, all food is unsafe to eat. Absurd.
Epidemiology itself has a bit of a reputation of having a hard time finding really solid effects, partly because the effects that are measured are frequently multi-variate with lots on confounding effects, partly because you need huge numbers to have very much analysis power, partly because such studies are generally more observational then experimental. This guy has published a bunch of papers in the past arguing (and presenting models for) exactly this kind of problem. He comes up with the logical (if rather obvious) suggestions that amongst others: 1. Smaller studies are less likely to be true. 2. Smaller observed effects are less likely to be true. 3. The greater the financial interests there are in the study, the less likely it is to be true. 4. The "hotter" a topic is, the less likely a study is to be true. Largely these are no shit, sherlock kinds of things.
So, to sum up, there are lots of epidemiological claims in published articles out there that might not be right. This represents neither a new idea, nor a meaningful comment on anything but epidemiology.
-Ted
-=-=- Quantum physics - the dreams stuff are made of.
There are aspects of discussion sections that are likely wrong, or at least a little incorrect, in almost all papers. But that's becuase you're stating your next HYPOTHESIS, which is not fact. Unfortuantely, discussions are more easily read than methods/results and are likely what the popular press, and perhpas this guy, latch onto.
That said, an examination of how science gets done may be in order. He faults the community for not reproducing other people's work enough. This can often require specialized materials or equipment that few other labs might have. Taking on the task of your own research combined with playing "results police" for the community could place a burden. In grad school we did our best to provide our unique materials to everyone who asked, but it did get really really expensive to make all that material and a less well funded lab simply couldn't have done it.
Also, how does it help my career to spend months to years (yes, some science takes years to do a single experiment properly) to reproduce someone elses' work. I can't write a paper that says, "Yeah, Smith et. al., got the right answer." The REALITY of it is sometime you rely on a previous result to do the next step, and if their work was flat out wrong you get stopped and can't proceed. Then you clean it up.
I also have to fault this study because the information gets cleaned up in other ways besides outright retractions. The results can appear in other papers, or as part of other papers without specifcially calling attention to it. In grad school I used someone elses' results as a launchign point for an experiment that was part of a much larger study. When it didn't work we looked into it and the original work was messed up by seriously flawed design, controls, and materials. I redid that study and found the correct answer, and used that to carry on with the experiment I wanted to do. All of that work appeared in a single figure in a 7 figure paper, and we didn't say, "So and so et. al., made up their paper and we've corrected it in figure 4, panel A." We just simply made those new controls and kept going. Those steeped in the field know what happened, but this situation wouldn't be noticed if you were only couting retraction.
It's all too complicated for these reductionist studies. That said, we can do better. Paper reviews are often too cursory. I've reviewed papers with serious issues, either rejected or asked for revisions, and then seen the paper appear in the journal a couple months later anyway without those revisions. Scientists are part of the equation, the journals and their staff are another!
But I think you have to bounce that one down to the construction teams... Obviously, a superior engineer would not design a superstructure that would failed, unless, of course, using the shoddy science provided to him or her by the Phd.... obviously, there would never be a misapplication!
tongue_in_cheek!
This is my sig.
Ben Hocking
Need a professional organizer?
from what I've seen, most scientists care far more about self than science
getting a paper published is the very first step in peer review, not the final word.
So yes, this might be a problem, but when other peers review it the problems are likely to get pointed out.
A peer review paper isn't a paper that HAS been peer reviewed, it one that is being peer reviewed.
Yes, I know that war redundant, but people for get to all to often.
Another reminder - Scientist live to disprove hypothesis and theories.
The Kruger Dunning explains most post on
Hack, cough, choke, choke, String Theory! Ahem.
This problem would be solved most readily by increasing the standards of transparency in science. The community is still stuck inside the journal paper format, which is really only a highly synthesized final report. There is no space in hardcopy journals to publish things like base datasets, detailed statistical analyses, or simulation source code. Without these details even the most qualified peer reviewer can only do a partial job. It's a little bit like a programmer trying to do a code review with access to program output but not source code.
The online journals and repositories like arXiv could improve this situation by making more supplementary material publishable. Ideally one should be able to start from base data and follow the analytical chain to the synthesized numbers and charts appearing in the paper (and where simulations are involved, download source code that generates the results). It would be nice to see standard formats for observational data, simulation source code, etc. emerge -- and have these be published online with the paper itself. The standard in science should be as it is in mathematics, where the entire logical chain is presented for review.
I supplied this story to /., and I'd like to note that the title of the WSJ article is somewhat misleading since the researcher was concerned with medical studies, not science in general. (That is not to say, however, that some of the same problems of non-replication and unintentional data manipulation don't exist in other disciplines. However, one cannot draw conclusions on those fields from this work.) Also the global warming tag, which has now been removed, was not added by me and was inappropriate for the same reason.
For anyone who's taken (and passed) a statistics course, one should understand confidence intervals.
Nowadays, the bar for confidence on studies is so low, you'd swear they accept stuff in crayon.
Thirty years ago none of these studies would have made it to publication, But standards have fallen so low now, that pretty much ANYTHING that's nicely written (regarless of how large a piece of tripe it is) will get printed.
Chas - The one, the only.
THANK GOD!!!
That was the title of a NOVA film in 1998.
`Abstract: This video examines the troubling question of scientific fraud: How prevalent is it? Who
commits it? And what happens when the perpetrators are caught? Factors contributing to "bad science"
include sloppy research, personal bias, lack of objectivity, "cooking and trimming", "publish or perish"
pressure, and outright fraud. The limits of peer review and other quality control systems are discussed.'
The results of the study determined that 48% of all published data was fraudulent. The data was trimmed, cooked or outright falsified. Some cases made famous by public exposure were analyzed.
While recieving a lot of lip service from the establishment science, the two government researchers who made the report were reassigned to worthless tasks in isolated areas. One was sent to shuffle papers in Alaska, IIRC. So much for whistle blowers, even government whistle blowers.
In the last 19 years it seems nothing has changed. Besides this latest report how can I tell? Simple. The news is filled with stories of drugs being recalled because they are more dangerous that the problems they are supposed to treat. How would they ever have gotten on the market in the first place if their FDA "studies" weren't rigged? And you don't wonder about the revolving door policy between Pharmaceutical employees and FDA employees? Corporate influence in research is as corrupting as Microsoft influence in ISO standards voting.
What really burns me is that MUCH of our basic research is done at academic institutions by professors funded by government grants, i.e., tax payers. But, thanks to the best congress that money can buy (because most of them have been bought off) OUR research is "monetized" (sold to special interests) for pennies on the dollar. These interests then reap HUGE license profits for decades. To make matters worse, many of the "special interests" are the very academic researchers who were paid to do their work. Having discovered key facts, without reporting them, they resign academia and begin a corporation to capitalize on what we paid them to learn.
IF we had a congress worth what they are paid there would be a law which prohibits recipients of gov grants, or their families relatives, or former business associates to personally benefit from what they learned using that grant money for a period of 15 years. Secondly, the ONLY corporations which should be allowed to receive IP licenses from the gov should be NON-PROFITS, whose board, management or employees cannot include the professor or his family or relatives.
Another thing that this recent study shows is what the NOVA film revealed: Peer-review is worthless for vetting research. Replication is worthless for vetting research. Obviously, personal integrity is also a worthless indicator of research quality.
Running with Linux for over 20 years!
Interestingly enough, a couple years ago my Biostats teacher mentioned that about 50% of scientific articles in biological journals contain at least one statistical error. Personally, I think this can be attributed to the general lack of math skills of most Biology majors (not all of course). Biology is a memorization-based science, and most classes require virtually no math skills. In the courses that do have some math, it's never above the algebra level (with the exception of Biostats), and even that is considered quite difficult for a lot of people. Most of my classmates have an amazingly hard time with non-Calculus based physics. My university is even lowering the math required to get a Biology degree (from Calculus II to Calculus I). Even if students do learn it, if you don't use it you tend to forget it. Therefore it doesn't surprise me that many scientists and doctors (which many of my classmates may become) make mathematical errors in journal articles.
It sounds like you're of the opinion that once we run out of fossil fuel, we're hosed. (I'm assuming that you're not a YEC who believes that fossil fuels are renewable.) Don't you think we have the technological know-how to develop other means of generating a sufficient amount of electricity?
Ben Hocking
Need a professional organizer?
No company is going to want to pay someone to repeat someone else's experiment, just to see if it was right. Unless they really don't agree with the study and are hoping it will fail. (think Tobacco). But in general they won't bother.
If you adminster a block of research funds, and have a choice between someone who's doing new research, and somone who says he just wants to check Newton's math, who's going to get the funding?
If your a professor, and some distant aquaintence asks you to review his paper, you'll probably say yes, because you figure then you can ask him to review yours next year. But when it gets there, and its 120 pages, how much time are you going to dedicate to really hammering on it to see if they made a mistake?
There's no glory in checking someone else's work, and there's no funding to do it. So why be suprised that mistakes are not noticed?
You're equivocating over the use of "new" in order to talk about something that nobody is suggesting.
1) It's one thing to "stop increasing how much we emit per year".
2) It's another thing to "stop emitting anything beyond what we've already omitted".
All the serious debate is #1. #2 is the "devastated economy" model.
Murdoch now owns the WST oops that should be a J. Soon it will say global warming studies are flawed. /ignore wsj
MISSED THE POINT:
The point of the grandparent post, and of this Slashdot story is that many "scientific" studies are fraudulent.
If you are seriously interested in this topic, I recommend reading the comments (for and against,... and at least one response from the author) found at
http://medicine.plosjournals.org/perlserv/?SESSID=ffa6bd3b3cff3235d54a3cbcc6f295a1&request=read-response&doi=10.1371/journal.pmed.0020124#r1812
-=-=-
The obvious sometimes takes a while,... and, then, it too disappears.
-=-=-
The truth is what remains after all the lies are removed,... even then, it is overwhelming.
-=-=-
EAoBA [even afraid of being anonymous]
Saying it's based on the fundamental principles of science is not the same as saying that correlations and observations are worthless, and I'm not sure where you get that out of what I'm saying. I think your particle accelerator example is excellent. We have fundamental principles of science there that allow us to make predictions. The accelerators allow us to make observations that allow us to test those predictions. This sometimes results in changes to those fundamental principles. However, we are not talking about post-hoc analyses here or in the case of global warming. Do you understand the distinction?
Ben Hocking
Need a professional organizer?
The study referred to in the article is only analysing data from one field of medical science; to tarnish all of the sciences with the same brush, as the summary title suggests, is misleading.
SJW: Someone who has run out of real oppression, and has to fake it.