Meta-Research Debunks Medical Study Findings

jenningsthecat writes "From The Atlantic comes the story of John Ioannidis and his team of meta-researchers, who have studied the overall state of medical research and found it dangerously and widely lacking in trustworthiness. Even after filtering out the journalistic frippery and hyperbole, the story is pretty disturbing. Some points made in the article: even the most respected, widely accepted, peer-reviewed medical studies are all-too-often deeply flawed or outright wrong; when an error is brought to light and the conclusions publicly refuted, the erroneous conclusions often persist and are cited as valid for years, or even decades; scientists and researchers themselves regard peer review as providing 'only a minimal assurance of quality'; and these shortcomings apply to medical research across the board, not just to blatantly self-serving pharmaceutical industry studies. The article concludes by saying, 'Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.' I've always been somewhat suspicious of research findings, but before this article I had no idea just how prevalent untrustworthy results were."

Comments

For example

[Anonymous Coward]

That fat in your diet is bad for you.

Re:For example

[erroneus]

That's one of my favorites... or the presumption that eating fat makes you fat.

People seriously do not understand nutrition or how diet and exercise work. Lately, I have been doing an kind of experiment for the people around me. First, for about a month or more, I started riding my bicycle to work. I was working it hard. Then, after it was established that I had been riding my bike for at least a month, I started on a low-carb diet. Within two weeks people started to notice the weight loss. Some still wanted to believe it was the bicycle riding. I had to lay it out to them what the deal was. Exercise burns carbs and then fat. Trouble is, the carbs we take in our daily diet still outnumbers that which I burn from riding 10 miles each day. It is only after I limited the intake of carbs that a difference could be made and observed.

Here's why I did it like this:

People don't listen for more than a few sentences and are especially resistant when the information conflicts with what they think they know. Eating fatty meats is contrary to their beliefs about what a weight loss program should contain so they simply refused to accept it. Hell, even many doctors don't yet fully acknowledge that making your body burn fat will reduce cholesterol. (Hello? cholesterol is fat floating in the blood!)

Having lost almost 30 lbs in a 6 week period has been noticed by all and the evidence is right in front of their faces. I lost the weight, and this is what I have been eating.

Re:For example

[CosmeticLobotamy]

For a year and a half I tracked every ounce of food I ate, recorded it in software I wrote, meticulously recorded calories burned on exercise machines (this is the largest source of error, but I did my best to compensate) and found the following:

Graphing my actual weight and my projected weight using (start weight - ((2660 - calories eaten + calories exercised) / 3500) will never be more than 3 pounds apart, even over a weight change of over 100 lbs, regardless of whether my caloric intake was from a tub of Crisco or from cucumbers dipped in virgin blood. Conclusion: eat whatever the hell kind of food you want, just keep the calories low, your long-term weight change will be the same. But it's easier to eat fewer calories, in my experience, if you cut out carbs. They make me hungrier.

The +/- 3 pounds fluctuates based on when you last peed, how much you drank and when, and how much meat is sitting in your colon pending expulsion.

Re:For example

[c6gunner]

Not healthy, dude, and Ketosis is not your friend. In the short term, ok, you're probably not going to kill yourself. In the long term, you're going to burn lean muscle, including some important ones like, oh, I don't know ... the heart. And by "long term" I mean "you crossed the line 20 days ago". I suggest you stop. Being overweight may be bad for you, but starvation diets are far worse.

Re:For example

[turbidostato]

"Reduce the primary source of fuel so that it will use the secondary source of fuel. It's really that simple."

No, it isn't.

The (almost) only direct source of "fuel" for the organism is sugar (glucose), full stop*1.

Anything else the body ingests or stores has to be reduced to sugar (usually by means of the krebs cycle) prior to be "burned".

While this is a very basic simplification, this, and the fact that the blood can carry a limited level of sugar at a time, is what explains, at a whole body level, everything else.

Like...:
* Since you can only burn sugar, sugar-equivalent contents is all that counts for weight imballance (of course, within limits: you can't just stop your ingestion of, say, oligoelements). That's what we really talk about when we talk about food calories.
* If you directly eat sugar (glucose), the sugar will be immediatly burn, but since your blood has limited sugar carriage capacity, you should be continously eating like a hummingbird to sustain that, so you usually just can't eat sugar in excess.
* If you eat carbs, they'll be transformed into sugar and burned. Any carb in excess will be stored as glucogen in your liver. If there's still carbs in excess once your liver can't hold any more glucogen, it will be transformed into fat and stored under your skin.
* If you eat fat, it will be transformed into sugar and burned. Any fat in excess will be stored under your skin unless you are so low in glucogen (which usually won't happen) that part of the fat is transformed into glucogen and stored in the liver.
* If you eat proteins, they'll be used for structural development (like muscle mass). Usually, anything in excess will be trashed away, unless you are very low in sugar, carbs and fat intake (it usually doesn't happen) in which case, it will be uneffitiently transformed into sugar and then, burned.
* To explain for long term weigth, all that's needed is accounting for your ballance between ingested calories and burned calories: if you eat more calories than you burn, your weight increases; if you eat less, you lose weight.

For a practical example:
If you eat less carbs and more fat to the point that daily calories stay the same, You Will Not Lose Weight (but in the long run you will develop cardiovascular illness).
If you eat less carbs and more proteins to the point that daily calories stay the same, You Will Not Lose Weight (but in the long run you will destroy your liver).

Given the ballance between ingress and burn, you can obviously go two (complimentary) routes:

1) Eat less calories. Sustaining a varied and ballanced diet, only eating less, is the way any sensible nutritionist will suggest since it's the easiest to do properly long term and the easiest to lead to you changing your habits. But as long as you stay to the First Principle "eat less calories", and within sensible limits, you will get it right.

2) Burn more calories. That's where exercise and rising your basal metabolism come into account. Aerobic exercise is an obvious recomendation, but other less obvious things like lowering your home thermostat 3~4 degrees in winter will have it's effect too. Again, it's not what you do, but what you achieve with regards of burned calories.

Everything else about diets is about making acceptable for you to eat less calories/burn more calories (like, unless you are a kind of iron-man you won't have the will for strengh training like weight lifting unless you go heavy on sugars; the same with aerobics, like long distance running or bycicling unless you go heavy on carbohidrates, or you'll probably break your diet if you are just told "eat exactly the same as you did, only on third the quantities", so you are offered a diet with much less calories but about the same or even more volume so you feel satiated, or you'll probably will abandon a diet if you don't see fast results at least at the beginning, so you are offered a diet very low in calories for the first weeks so you fastly see your efforts are

Reality check

[koreaman]

If medical research were really as close-to-useless as The Fine Summary claims, we'd be hardly better off with modern Western medicine than with homeopathy and prayer. Clearly, we are, refuting the idea that medical research doesn't do a huge amount of good. I'm not saying it isn't flawed, but give it some credit.

Re:Reality check

[Notquitecajun]

I was thinking along the same lines - stuff like this only gives the anti-immunozation people more ammunition.

Re:Reality check

[oldspewey]

Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.

TFS seems to be suggesting there is no value in incremental knowledge. No, no you don't discover penicillin every day you go to the lab. But you usually achieve something that advances the state-of-the-art and the things that you learned from your predecessor(s).

Re:Reality check

[UnknownSoldier]

You ARE aware of the placebo effect, right? It is a BIG problem for big pharma ...

http://www.newscientist.com/article/mg18524911.600-13-things-that-do-not-make-sense.html

--
"The Inner Space (of Mind), not Outer Space is the FINAL frontier."

Re:Reality check

[Anonymous Coward]

Over time I became immune to placebos, I now take Extra-Strength Placebos (liqui-gels). Twice the inert ingredients!

Re:Reality check

[instagib]

I skimmed TFA, and it's important to note that scientific base research (for new pills, procedures, etc.) is not the issue here. This is about studies, i.e. field testing of large numbers of patients, and the (wrong, causation != correlation, etc.) interpretations that are made public afterwards. Funny enough, until recently, criticising the official results of medical studies was seen as conspiracy theory by those in power in medical circles.

Re:Reality check

[zrbyte]

If medical research were really as close-to-useless as The Fine Summary claims, we'd be hardly better off with modern Western medicine than with homeopathy and prayer.

True.

Top notch research is what makes all the medical breakthroughs, but this is only the top few percent of ALL medical research. IMHO one of the main reasons there are so much bogus papers out there is because of the publish or perish attitude in academia, which requires researchers to have a set number of papers published to be eligible for research funding, tenure, other career advancements. I know from experience (although not in medical research, but natural sciences) that sometimes you have to publish a paper even if you know that the results aren't meaningful, or of value to anyone. Then there are people who publish things that were not subjected to rigorous testing, double checking of data, etc. which can easily turn out to be wrong. Lastly there are the cheats. All I'm trying to say is that it's more of a science policy problem than a problem with the integrity of researchers. If the number of publications has to go up, then their quality will surely decrease. Very few research groups (the ones which have good funding) have the luxury of publishing only every now and then. But when they do it's usually a Science or Nature paper. This problem os quality VS quantity is most serious in China. However, not even journals such as Nature are immune to this.

Re:Reality check

[TubeSteak]

Over time I became immune to placebos, I now take Extra-Strength Placebos (liqui-gels). Twice the inert ingredients!

I only use homeopathic remedies.
Now with 1000x less inert ingredients!

Re:Reality check

[dogmatixpsych]

That's because most M.D.s do not have strong statistical and scientific research backgrounds. They get clinical training, not theoretical training, so when most of those who do research try to do their studies, they don't always understand the statistics behind what they are doing (and the same goes for those editing and reviewing the articles).

For example, I'm proposing a model of cognitive dysfunction in patients with Parkinson's disease (for my dissertation) that flies in the face of what most people who deal with Parkinson's disease believe is the case. My dissertation is not complete so I cannot say if my findings match my hypotheses but preliminary data are encouraging.

My point is that we have certain set beliefs about the way things are and some people (myself included) have a hard time challenging those beliefs. What I've seen is that many researchers don't actually follow the scientific method and so at the core, their research has methodological weaknesses.

Study shows

[Drakkenmensch]

... that most people will believe anything, as long as it starts with 'study shows'.

Quack Attack

[Anonymous Coward]

The problem I know because I am (retired) RN. I went to my MD for a severe sinus infection and chest infection some years ago. During the visit I asked for Abuterol Inhailer to assist with clearing my chest. This is standard Respiratory Therapy stuff. They gave me a puffer which I took home, used 10 or 15 puffs out of and threw it in the drawer. Ever after that BCBC has me as Diagonsis Asthma. I am not asthmatic. This will screw up my healthcare for the rest of my life! Makeing all of these errored stupid databases cross link will do far worse than this. My daughter (age 23) was emergency taken to the local hospital with what appeared to be an Epileptic Seizure. Consequences included she couldn't drive for 6 months! I will skip the details, her seizure was a cardiac seizure. It took her actually taking her case to the local Fire Department to get a heart monitor strip to make this undeniable. She is now treated well but forever she will be DX Epileptic even though it is completely wrong. Does any sane person want this sort of a system where you cannot go to another doctor and have him/her look at you rather than some record first? Who wants in that trap? If you are an MD in that trap even if you see that the record is wrong, you can go to jail, lose your job etc all if you go against this insane record that is completely in error. Please wake up people this is a prison without walls! You cannot escape! You will have to leave the country to get away from a bad diagnosis or a stupid keystroke error. Remember the computers have a forever memory and no intelligence.

Re:We've known this for years

[Profane MuthaFucka]

I'd replace item C with regular fucking. Don't see the quacks, just fuck someone. Chiropractors are worthless and evil. Fucking is fucking AWESOME!

These are well known problems

[Grond]

I don't want to discount the value of the study itself. Clearly it's important to quantify how bad the problem is and try to develop solutions. But at the same time, the article and summary might give one the impression that the errors and biases involved were newly discovered by the researchers. A few examples:

The secondary marker problem (e.g. tracking cholesterol levels instead of real outcomes like deaths)
Comparing new drugs only to placebo or only to drugs that aren't best-in-class or using an intentionally weak dose of the comparison drug
Using meta-analysis of other studies instead of doing new research (and often doing it badly)
Doing retrospective analyses like chart reviews instead of prospective studies (and often doing those badly)

To expand upon that last example: common problems with the methodology of chart review studies were investigated thoroughly by Gilbert and Lowenstein in 1996. Despite their findings and recommendations for how to do a chart review properly, things haven't improved much since.

Many doctors and researchers have been critiquing studies and warning about these problems for years. In the emergency medicine context, for example, Jerome Hoffman, a UCLA medical professor and emergency department physician, is well known as a critic of poorly designed studies in the emergency medicine literature. He has critically reviewed studies since 1977 as part of a continuing medical education program called Emergency Medical Abstracts.

So the problems are well known. The bigger issue is how to fix them.

Re:We've known this for years

[turing_m]

Ask any Chiropractor, Naturopath or Homoeopath and they will all tell you the same thing: MODERN MEDICINE IS POISON

I'm a firm believer in Homoeopathy. I don't believe in all of it though, just a homoeopathic proportion of its principles.

I see your problem right there: "Science"

[wonkavader]

Doctors are notoriously bad at doing scientific investigations, and since peer review journals are reviewed by peers, they don't know bad science when they see it.

I've been arguing with my father about this for... what, fifteen years?!? He subscribes to the New England Journal of Medicine and is forever telling me about the latest study telling you to that standing on one leg reduces breast cancer in nuns. These studies are, without fail, trivial to eviscerate as their control groups are either non-existent or very poorly picked.

Doctors just don't know much about science. That shouldn't be surprising. They have terrible educations. After all, they studied premed in college and then went to a vocational school. MD/PhDs really ought to do better, but I suspect they do their doctoral work with faculty who come from the same under-educated group.

The problem is one of prestige. You cannot walk into a group of doctors and tell them why they're dumbasses and what to change. They know, and society confirms for them, that they are the best educated people on the planet and smarter than anyone else. "Heck, just look at our paychecks," they might respond. "That proves it right there, doesn't it? Figure it out from that data point only -- you don't need a control group."

Spinal Fusions and Hormone Replacement Therapy

[assertation]

Don't do either without doing your own research.

I recently read a book called "How Doctors Think" by a Hematologist associated with Harvard. He went around interviewing the top doctors in the country about their cognitive fuck-ups and triumphs.

Serendipitously, along the way, some doctors were more than honest with him about medical dogma, being lazy in keeping up with research and how drug company money/gifts influence medical opinions.

Spinal fusions have no proven clinical benefit for reducing pain. At all. It has been known for a long time, but financial interests keep that procedure being done.

Hormone replacement therapy has been proven to be potentially dangerous ( cancer ,etc ) but with negligible benefit. Aside from middle aged women & estrogen, I'm sure you noticed the new commercials trying to convince aging men to supplement with testosterone.

It is natural for testosterone levels to gradually decrease as men age. Testosterone levels fluctuate often and hard. If a doctor tells you that you have below normal testosterone asked to be tested again. Get off of the things that lower testosterone ( ie alcohol ) and do the natural things that boost like hard exercise ( lift weights ) for a few weeks. Then get tested again.