Slashdot Mirror
Meta-Research Debunks Medical Study Findings
jenningsthecat writes "From The Atlantic comes the story of John Ioannidis and his team of meta-researchers, who have studied the overall state of medical research and found it dangerously and widely lacking in trustworthiness. Even after filtering out the journalistic frippery and hyperbole, the story is pretty disturbing. Some points made in the article: even the most respected, widely accepted, peer-reviewed medical studies are all-too-often deeply flawed or outright wrong; when an error is brought to light and the conclusions publicly refuted, the erroneous conclusions often persist and are cited as valid for years, or even decades; scientists and researchers themselves regard peer review as providing 'only a minimal assurance of quality'; and these shortcomings apply to medical research across the board, not just to blatantly self-serving pharmaceutical industry studies. The article concludes by saying, 'Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.' I've always been somewhat suspicious of research findings, but before this article I had no idea just how prevalent untrustworthy results were."
That fat in your diet is bad for you.
If medical research were really as close-to-useless as The Fine Summary claims, we'd be hardly better off with modern Western medicine than with homeopathy and prayer. Clearly, we are, refuting the idea that medical research doesn't do a huge amount of good. I'm not saying it isn't flawed, but give it some credit.
Le français vous intéresse?
... that most people will believe anything, as long as it starts with 'study shows'.
... it's the most useless way to progress, except for all others.
Be a skeptic, but don't confuse skepticism with truthiness. Unfortunately, I expect a rise in the use of truthiness over science when people will investigate reality.
Those who can, do. Those who can't, sue.
In 2001, rumors were circulating in Greek hospitals that surgery residents, eager to rack up scalpel time, were falsely diagnosing hapless Albanian immigrants with appendicitis. At the University of Ioannina medical school’s teaching hospital, a newly minted doctor named Athina Tatsioni was discussing the rumors with colleagues when a professor who had overheard asked her if she’d like to try to prove whether they were true—he seemed to be almost daring her. She accepted the challenge and, with the professor’s and other colleagues’ help, eventually produced a formal study showing that, for whatever reason, the appendices removed from patients with Albanian names in six Greek hospitals were more than three times as likely to be perfectly healthy as those removed from patients with Greek names.
Okay - so I only bothered getting this far into TFA.
Now - I'm no medical junkie, I didnt' even take Bio in high school, but I have occaisonally watched Scrubs and House and ER and a bunch of other medical dramas from time to time.
One thing that always seems to surface in these TV shows is the patients history, like their religion, nationality, where they work, etc. This leads me to believe that maybe - JUST MAYBE - there is actually some correlation between something in the Albanian culture and society that has an increased chance of appendicitis, and that its entirely possible that this pushes doctors towards diagnosing that when some of the symptoms appear. (Not that this is particularily the best course of action, but what else would you do? Run every test?)
But at the same time I know how incredibly innaccurate a lot of television can be about portraying a subject. However, the IT Crowd has basically mimicked my life, but thats another story for another time. Anyways, so if I'm absolutely and completely wrong, feel free to mod me down - but I just thought I'd interject.
The problem I know because I am (retired) RN. I went to my MD for a severe sinus infection and chest infection some years ago. During the visit I asked for Abuterol Inhailer to assist with clearing my chest. This is standard Respiratory Therapy stuff. They gave me a puffer which I took home, used 10 or 15 puffs out of and threw it in the drawer. Ever after that BCBC has me as Diagonsis Asthma. I am not asthmatic. This will screw up my healthcare for the rest of my life! Makeing all of these errored stupid databases cross link will do far worse than this. My daughter (age 23) was emergency taken to the local hospital with what appeared to be an Epileptic Seizure. Consequences included she couldn't drive for 6 months! I will skip the details, her seizure was a cardiac seizure. It took her actually taking her case to the local Fire Department to get a heart monitor strip to make this undeniable. She is now treated well but forever she will be DX Epileptic even though it is completely wrong. Does any sane person want this sort of a system where you cannot go to another doctor and have him/her look at you rather than some record first? Who wants in that trap? If you are an MD in that trap even if you see that the record is wrong, you can go to jail, lose your job etc all if you go against this insane record that is completely in error. Please wake up people this is a prison without walls! You cannot escape! You will have to leave the country to get away from a bad diagnosis or a stupid keystroke error. Remember the computers have a forever memory and no intelligence.
I was going to suggest just the opposite. Medical studies aren't like doing physics experiments in the lab: you can't control the minutae of the experiment to anywhere near an ideal degree. You need to have control groups, you need to factor out all possible other causes (and even then, you can be sure you won't catch them all), you need to have long-term observations and follow-up studies. Sometimes you'll see a trend and it turns out it was pure chance. Everyone is different, both in terms of genetic makeup and environment, and that's going to mean that everyone has different reactions to just about anything.
There's clearly room for a great deal of improvement here, but people might need to accept that results of medical studies are never going to be as clear-cut as math papers. I think, and now I'm moving into personal opinion, the most important step-- and this applies to both the media and to scientists who might want to cite or cross-reference something-- is that nothing is ever, ever proved after the first paper. Ever. The first paper on some hypothetical connection really should just be ignored, except for prompting further investigation Like I said above, it needs follow-ups, and experiments with different variables controlled for.
Dislike the Electoral College? Lobby your state to join the National Popular Vote Interstate Compact.
I'd replace item C with regular fucking. Don't see the quacks, just fuck someone. Chiropractors are worthless and evil. Fucking is fucking AWESOME!
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
I don't want to discount the value of the study itself. Clearly it's important to quantify how bad the problem is and try to develop solutions. But at the same time, the article and summary might give one the impression that the errors and biases involved were newly discovered by the researchers. A few examples:
The secondary marker problem (e.g. tracking cholesterol levels instead of real outcomes like deaths)
Comparing new drugs only to placebo or only to drugs that aren't best-in-class or using an intentionally weak dose of the comparison drug
Using meta-analysis of other studies instead of doing new research (and often doing it badly)
Doing retrospective analyses like chart reviews instead of prospective studies (and often doing those badly)
To expand upon that last example: common problems with the methodology of chart review studies were investigated thoroughly by Gilbert and Lowenstein in 1996. Despite their findings and recommendations for how to do a chart review properly, things haven't improved much since.
Many doctors and researchers have been critiquing studies and warning about these problems for years. In the emergency medicine context, for example, Jerome Hoffman, a UCLA medical professor and emergency department physician, is well known as a critic of poorly designed studies in the emergency medicine literature. He has critically reviewed studies since 1977 as part of a continuing medical education program called Emergency Medical Abstracts.
So the problems are well known. The bigger issue is how to fix them.
I'm a firm believer in Homoeopathy. I don't believe in all of it though, just a homoeopathic proportion of its principles.
If I have seen further it is by stealing the Intellectual Property of giants.
Why I don't doubt that some good critical thinking, and legitimate questioning come from these meta-research studies, I fear that the process is ripe for abuse, as basically being so awash in data (information overload) that given enough data you can pick and choose to fit your a priori or posteriori hypothesis.
I applaud the increased scrutiny of statistical analysis, which is truly difficult to administer on even the best designed and controlled biological and medical studies, where you have very little "total control" of the experimental subjects - damn ethics committees on testing human subjects, and using double blind testing is the best you can do to eliminate bias, yet may mask discovery of experimental flaws during the testing phase. Things go "wonky" in strange ways, for example testing a heart medication, and a freak snowstorm skews all the results because of the rise of heart attacks from the increase in shovelling. We can't legally put 1000s of humans in vats for 10-20 years to test everything, and computational models are primitive and only address what the model is designed to look for, while most medical testing focuses on the unexpected results and effects that may only appear in a small fraction of society, yet if the consequences are dire, it can kill an entire potentially life-saving product.
I fear that the "undergrad social science" approach of meta-study research will make the approach stained with a reputation of people who want to "do science", but without the messy get-your-hands-dirty that costs money (an increasingly mythical subatomic particle in most fields of science and labs around the world) and just do a PR-style re-spin (think: re-branding) of the results of multiple similar but different experiments to reach a conclusion that was not considered by the original experimenters, so whom may not include appropriate experimental controls to minimize draw incorrect conclusions from this re-interpretation of the experimental data. Of if they are really lazy (like social science elective takers), draw conclusions from a compilation of results, and not even bother looking at the original (raw or filtered) data at all. I guess I'm trying to say that there is limited latitude for re-interpreting data for anything beyond what the experiment was designed to test. It can be very useful for detecting and thwarting bad or biased experiments, but as far as I know, it cannot produce trustworthy results from bad experiments.
It is rare for any medical journal to publish negative results, and yet that is precisely what we need to advance science.
Both positive results - and the failures that prove they were flawed. But journals only publish the former, not the latter.
-- Tigger warning: This post may contain tiggers! --
Because he has documented statistical analysis backing his points up and if you want to verify them you just have to read his studies, look up all the studies he cites and then run statistical analysis on them yourself. That's the beauty of the scientific method (when it's implemented correctly at-least...).
I've heard a lot of disgusted (perhaps accurate, perhaps not) complaints about how politicians (usually this complaint is aimed at Republicans, as in the book title "The Republican War on Science") have "cut money for basic, fundamental research -- the kind that benefits all of us." It's a slight paraphrase, and you might consider it a straw man, but I don't think these gripes are hard to find.
Meta studies like this are worth thinking about when someone says we need more "basic research" in any given area: it might be sincere, well-intentioned (or who knows, even perfectly accurate, in some platonic universe where "needs" could be accurately, objectively assessed and compared), but it might also mean that well-intentioned money would be spent on stuff that is very sciency, but not very helpful.
timothy
jrnl: http://tinyurl.com/c2l8yr / foes: http://tinyurl.com/ckjno5
Doctors are notoriously bad at doing scientific investigations, and since peer review journals are reviewed by peers, they don't know bad science when they see it.
I've been arguing with my father about this for... what, fifteen years?!? He subscribes to the New England Journal of Medicine and is forever telling me about the latest study telling you to that standing on one leg reduces breast cancer in nuns. These studies are, without fail, trivial to eviscerate as their control groups are either non-existent or very poorly picked.
Doctors just don't know much about science. That shouldn't be surprising. They have terrible educations. After all, they studied premed in college and then went to a vocational school. MD/PhDs really ought to do better, but I suspect they do their doctoral work with faculty who come from the same under-educated group.
The problem is one of prestige. You cannot walk into a group of doctors and tell them why they're dumbasses and what to change. They know, and society confirms for them, that they are the best educated people on the planet and smarter than anyone else. "Heck, just look at our paychecks," they might respond. "That proves it right there, doesn't it? Figure it out from that data point only -- you don't need a control group."
What? Doctors have to be convinced to follow evidence-based medicine? What were they practicing before? And why are they against it?
You know, there's never been a randomized control trial of the effectiveness of parachutes versus placebo when jumping out of planes at high altitude. Would you care to volunteer?
Some things are just obvious or can be deduced from retrospective analysis or theoretical modeling. And some times it would be unethical or impractical to conduct a randomized control trial of a treatment (e.g., testing an AIDS drug versus placebo).
The idea that doctors are ALLOWED to prescribe drugs for off-label use horrifies me (it's off-label because there's no proof it works for the off-label condition).
No, there is often tons of proof. Off-label just means that the manufacturer hasn't gone through the (expensive, time-consuming) process of proving its effectiveness to the FDA. If the patent has expired there's often no financial incentive to go through the trouble.
Don't do either without doing your own research.
I recently read a book called "How Doctors Think" by a Hematologist associated with Harvard. He went around interviewing the top doctors in the country about their cognitive fuck-ups and triumphs.
Serendipitously, along the way, some doctors were more than honest with him about medical dogma, being lazy in keeping up with research and how drug company money/gifts influence medical opinions.
Spinal fusions have no proven clinical benefit for reducing pain. At all. It has been known for a long time, but financial interests keep that procedure being done.
Hormone replacement therapy has been proven to be potentially dangerous ( cancer ,etc ) but with negligible benefit. Aside from middle aged women & estrogen, I'm sure you noticed the new commercials trying to convince aging men to supplement with testosterone.
It is natural for testosterone levels to gradually decrease as men age. Testosterone levels fluctuate often and hard. If a doctor tells you that you have below normal testosterone asked to be tested again. Get off of the things that lower testosterone ( ie alcohol ) and do the natural things that boost like hard exercise ( lift weights ) for a few weeks. Then get tested again.