Another Dementia Test Oversold

An anonymous reader writes: Many prominent news organizations, including the BBC, are reporting on a study (PDF) that claims a new blood test is 87% accurate in predicting which patients will develop cognitive impairment. It's hailed as a major step forward in efforts to fight dementia and Alzheimer's disease. Unfortunately, reality isn't quite so impressive. An article at MedPage Today explains all of the statistical facts that the mainstream press glosses over: "Only about 10% of patients of patients with MCI convert to clinical dementia per year. With nearly 30% of positive results false (remember, the specificity was 71%) as well as 15% of negative results false, most of the positive results in such a group will be false. Yes, it's time once again for a tutorial in positive predictive values. If we have 100 MCI patients and a 10% conversion rate, then 10 of them will develop dementia. These are the true positives. There will be 90 true negatives — the ones who don't convert. But with a specificity of 71%, the test will falsely identify 29% of the 90 true negatives, or 26, as positive. Meanwhile, with a false negative rate of 15%, only nine (rounding up from 8.5) of the 10 true positives will be correctly identified. ... It's easy to get a high negative predictive value when the annual event rate is 10%. If I simply predict that no one will convert, I'll be right 90% of the time."

Comments

Basic statistics

[Anonymous Coward]

THis is basic statistics learned by every doctor in medical school. specificity and sensitivity, prevalence, pretest and post test probability and false positive/true negative, false negative/true positive. They all factor in to deciding to use a medical test. Every person who comes in and demands a test ussually gets a lecture on this (at least from me) (at work have to post as AC)

Re:Basic statistics

[Incadenza]

This is basic statistics learned by every doctor in medical school.

Or that they have learned, passed the test, and forgotten. There was quite an interesting read on exactly this subject on the BBC website this week, Do doctors understand test results?:

In one session, almost half the group of 160 gynaecologists responded that the woman's chance of having cancer was nine in 10. Only 21% said that the figure was one in 10 - which is the correct answer. That's a worse result than if the doctors had been answering at random.

Re:Basic statistics

[oodaloop]

And yet, I just read a study that most doctors and clinicians involved with breast cancer can not correctly identify the liklihood of having breast cancer if someone tests positive for it. Answers ranged from 10-90%. Correct answer is 10%. Being taught it in medical school is one thing, but applying it on the job is another apparently.

Re:Basic statistics

[ceoyoyo]

Learned is such a strong word. I teach evidence based medicine, including test evaluation to med students and residents. I also have to try and explain it to practicing physicians.

Re:Basic statistics

[Anonymous Coward]

Sounds almost like they read the question wrong.

Re:Basic statistics

[Anonymous Coward]

Just add Peanut Butter and we can fix improve the results:
http://news.ufl.edu/2013/10/08/alzheimers-test/

Re:Basic statistics

[jeffmeden]

THis is basic statistics learned by every doctor in medical school. specificity and sensitivity, prevalence, pretest and post test probability and false positive/true negative, false negative/true positive. They all factor in to deciding to use a medical test. Every person who comes in and demands a test ussually gets a lecture on this (at least from me) (at work have to post as AC)

Statistics aside there are two more pressing questions: 1, are patients with MCI more likely to convert (10% or greater) in subsequent years? 2, is there a preventative process that can be used by those that test positive, that is safe for the general population? In other words, do we get to narrow our treatment focus with these results, even a little bit? We can piss and moan about statistics but at the end of the day, who really gives a fuck if you were right about the onset of someone's dementia, unless you can do something about it?

I keep reading that report...and

[Anonymous Coward]

I keep reading that report...and then I reread the report...what is the report about? I then read the report...and then reread the report...

How much closer

[Anonymous Coward]

All the mainstream articles I've read, whilst as usual glossing over the stats, are presenting this as a vast improvement, not as something that should be rolled out clinically. Didn't read it as hype.

Is it a major step towards a viable test, or is that just not true?

Re:How much closer

[sjames]

The test looks for a set of 10 proteins in the blood. I suspect that part is fairly reliable which means the theory that those proteins are markers for Alzheimer's is probably incorrect or incomplete.

That leans towards just not true.

Re:How much closer

[next_ghost]

The test looks for a set of 10 proteins in the blood. I suspect that part is fairly reliable which means the theory that those proteins are markers for Alzheimer's is probably incorrect or incomplete.

Whether or not those 10 proteins are markers for Alzheimer's is not the issue at all. When you look for markers, the search goes this way: disease => chemicals. But when you start testing patients for markers, you're actually using the inverse of that relationship: chemicals => disease. If the disease is rare, false positives will vastly outnumber the number of patients really afflicted by the disease even though you test for the right markers.

There are 3 important percentages for disease tests:
- How many people are afflicted by the disease in the general population?
- How many afflicted people get positive result?
- How many healthy people get positive result (false positive)?

I'll give you an example. Let's say we have some test with perfect accuracy (100% of people afflicted by the disease get positive test) but the test has 10% rate of false positives. We'll test 1000 people.

If the disease afflicts 50% of the population, about 550 people will get positive result but about 50 of them are healthy.

If the disease afflicts 10% of the population, about 190 people will get positive result but about 90 of them are healthy.

If the disease afflicts 1% of the population, about 109 people will get positive result but about 99 of them are healthy.

If the disease afflicts 0,1% of the population, about 101 people will get positive result but about 100 of them are healthy.

Is the problem clear now?

Re:How much closer

[next_ghost]

Just to make it clear, by "looking for markers" I meant developing a new test for the disease.

Yup

[the eric conspiracy]

This is a generic problem with media reports of medical results. No understanding of the statistics.

Add in the lack of repeatability of many studies (for example just about anything claiming bad effects from low doses of bisphenol-A has proven to not be repeatable) and outright fraud (Wakefield) or attempts to push the data using questionable statistics (Séralini) and you have a giant problem with media and medical research results.

"Thank God!"

[smittyoneeach]

Said Thad Cochran.

Surprise

[ceoyoyo]

Hey, the condescending article manages to not know what positive predictive values are!

Geeez

[Tablizer]

These sales people are demented, and I have the tests to prove it.

Somehow I lost my mind

[Anonymous Coward]

By the end of that paragraph... WHAT?!?

Useless, expensive drugs

[Anonymous Coward]

The available meds for dementia are of little practical use. No long-term double blind studies have been done with a sufficient number of patients.
There is Nothing in the literature to suggest that any of these very expensive drugs do _anything_ except run up the insurance rates for the general population.
It is naive to expect to reverse brain damage (formation of Lewy bodies). At best, the existing drugs might help the family feel better as their loved one spirals into acute phases of the disease.

faster, easier, more reliable

[Anonymous Coward]

Just ask them if Leno is funny.

Doctors suck

[Curunir_wolf]

A while back, a friend of mine went to the doctor complaining of fatigue. He was just constantly tired and unsure why. They checked him into the hospital for tests, and shortly declared he had Leukemia and needed to start chemotherapy right away. After several weeks, including a long stay in the ICU, the doctors declared the treatment a "success". He remained in the ICU and died yesterday.

Fucking quacks "practicing" medicine and performing "treatments" that have a proven low survival rate - but, hey, everybody including the pharmaceutical companies are raking in the money, so it's all good. Except for my friend, who would have lived longer and had less pain than he did under their "treatment".

Fuck them. Stop it with the myopic single diagnosis / single treatment crap. People are NOT petri dishes.

Sign of the times

[Anonymous Coward]

I suspect their claims is from the science PR department, which has a long history of overselling science. Too bad we take dementia so seriously that more than one person has looked into their claims to see that they are false. Too bad this doesnt happen with the climate sciences.

that's not what "true positive/negative" means

[Anonymous Coward]

29% of the true negatives are false positives? Wrong. Next time look up the terms you use. They have accepted meanings.

many kind of lies

[LocutusOfBorg1]

There are three kind of lies: lies, damn lies and statistics.