Predicting the Risk of Suicide By Analyzing the Text of Clinical Notes

First time accepted submitter J05H writes "Soldier and veteran suicide rates are increasing due to various factors. Critically, the rates have jumped in recent years. Now, Bayesian search experts are using anonymous Veteran's Administration notes to predict suicide risks. A related effort by Chris Poulin is the Durkheim Project which uses opt-in social media data for similar purposes."

Comments

Sounds legit

[kaoshin]

With significant development I think detection could become at least 50% successful. It would probably be more cost effective though to just not lose the quarters when you flip them.

Re:Sounds legit

[ebno-10db]

That would make some sense if the suicide rate was around 50%. Thankfully it's much lower.

Re:Sounds legit

[Fwipp]

Well, if the suicide rate's 10%, just say "no soldiers commit suicide," and bam, you're 90% successful.

Re:Sounds legit

[davester666]

Um, this isn't about predicting the suicide rate or how likely someone in the general population is going to commit suicide.

It's about how likely a veteran who writes a suicide note and gives it to someone else is going to follow through and try to commit suicide.

That rate is probably closer to the flip of a coin than it is to 10%.

Re:Sounds legit

[coolsnowmen]

Clever and true, but...here is the difference between statistics and engineers who need statistics.

Now all your mistakes are in the wrong direction. For different applications you will err on the side of False Alarm or Missed Detection. In this case, a false detection means you stages an unnecessary intervention (slightly costly / embarrassing, but at least they know you care); but, a missed detection literally means someone died.

Basically this is the application of risk/reward to probability.

threat of harm to self or others or not /fill

[Anonymous Coward]

sop on intake for decades now. does the 'note' say otherwise?

False positives

[dala1]

According to the study this is 67% effective. But, once this is applied to the general population you have an issue, because the vast majority of people are not suicidal. In the US, about 122 in 100,000 people attempt suicide a year, and about one in ten are successful. Even with a test that is 99% accurate, you are going to end up with somewhere around seven million false positives every year if you screen everyone.

Re:False positives

[femtobyte]

The question, though, is how harmful a false positive is. If false positive means "lock the person up for their own safety," then obviously you don't want many false positives. But, if false positive just means "bring them back in a week earlier for an extra session," then this might not be too bad. Even the "false positives" --- people who don't get as far as suicide attempts --- might be folks who would benefit from a little extra help. If the seven million most-likely-to-be-psychologically-vulnerable people in the US get a little extra care, there's probably significant good done.

Re:False positives

[dala1]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal. Consider the time and money spent, the extra doctors who have to be trained and hired (I'm assuming psychiatrists since a GP is hardly qualified to assess a potentially suicidal patient), and the days missed from work for false alarms. That's all before the psychological trauma and loss of trust from your doctor telling you out of the blue that they think you might be mentally ill when you're not.

Besides, that's not how it would work, because once the tool is out there and used to profile everyone, someone who is suicidal will commit suicide before that 'extra session.' After that, it will be be considered negligent not to 'do something' immediately once someone is flagged, and that something would likely be intrusive and expensive.

Also, their accuracy rate is 67%, not 99%. I used that number to demonstrate a best-case scenario. As it stands, they would flag around 83 million people while only correctly flagging around 200,000. Good luck with 99.75% false positives.

Re:False positives

[stealth_finger]

Total Genocide will fix every problem. Kill everything that has a genome. Everything.

Good plan, start with the AC's

Re:False positives

[gnasher719]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal.

An interesting attitude. Compare this to Foxconn, which reduced the suicide rate among its workforce from 1 in 60,000 to 1 in 400,000 in three years.

Re:False positives

[Cinnamon Beige]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal.

An interesting attitude. Compare this to Foxconn, which reduced the suicide rate among its workforce from 1 in 60,000 to 1 in 400,000 in three years.

All things considered, I think they did it by making it harder to commit suicide, and possibly also by improving labor conditions.

The usual process is to place somebody thought suicidal on a suicide watch. This can actually be very intrusive, and a test like this certainly is less than ideal if you're applying it at large--the accuracy here is for this population, and rather close to chance already. In a wider population, of a different makeup, its accuracy will be different, and probably lower.

More importantly, if you read the PLOS one article, they're discussing data mining the clinical notes themselves, and they admit that this is a branch of research that has been rather neglected: certain factors were deemed to have predictive value, without anybody really checking to see if that was true.

Let's say you're sitting in the entry way of an office building, and you notice that most people who come in to the building are men. This does not mean you can necessarily predict that a man walking past is going to enter the building; it might turn out that, in fact, of the people passing by the building, any given woman is more likely to come in--it's just that most of the people passing by right now are men.

It does not follow that if "Most of the people who do x are y" is true that "Most people who are y do x" is also true, for any set of x and y.

65% accuracy is not good, it's a start and it's better than what we currently have. In fact, the paper outright says that currently, they haven't even managed to validate the tool. In fact, I can easily give you the tl;dr version of this paper:

The indications for the future of this path of research are promising. Please fund the next phase so we can get it closer to practical application(s).

In less scientific phrasing:

We haven't reached a dead in, give us money so we can keep going!

It's not as much a breakthrough as a status report on the progress towards a breakthrough...

A positive use of data mining

[SigmoidCurve]

It's refreshing to see predictive data analysis used for positive efforts, rather than simply selling more ads. Here's a call to action for all you data scientists at Twitter, FB, and other SV startups who think they're changing the world when all they're doing is putting money in their advertisers' pockets. News flash: statistics can be used to benefit society for a change.

Re:A positive use of data mining

[Zynder]

They'll fuck that up like they do everything. At first it's finding suicide patterns, but then it's predicting future criminality ala Minority Report. Yeah, I'm pessimistic but I have good reasons.

Re:A positive use of data mining

[TheRealMindChild]

It might be an attempt at using data mining for a good cause, but it is using a case where the source data itself is questionable. If you have ever wondered what it is like to be in a loony bin, it is a bunch of babysitters who hope to become doctors, playing the telephone game with the doctor-of-the-day, who reports to the actual head doctor. The observation skills of these babysitters is like a 4 year old at a gun and ammo show. Then as information is relayed twice over and statistical analytics lump you into a hierarchical cluster, you are then medicated and treated like a typical case study from said cluster. It is like a black hole of no escape, no matter how wrong it is. This is the data they are mining.

Re:A positive use of data mining

[RuffMasterD]

I read something about predicting criminality not long ago, but from a legal ethics angle. Since the human genome has been mapped, and every criminal (in my country at least) provides a DNA sample, and DNA sequencing is relatively cheap, there is now a wealth of data to mine. Behavioural Geneticists are starting to look for 'criminal' behaviour markers that distinguish, and to some extent explain, criminal behaviour. Some creative defence lawyers are already using this knowledge to help their cases. The flip-side of this knowledge is the possibility of law enforcement pre-emptively scanning people, maybe even at birth, and monitoring or even detaining those with 'criminal' DNA profiles. Really scary stuff indeed, and almost nothing to stop it from happening. But it's more likely the people in power will abuse this kind of technology, 'for the greater good', not the scientists.

Re: Fucking Retarded

[Anonymous Coward]

Wait till they start scanning school kid's english papers and Facebook pages.

So what do they do about it?

[mark-t]

Let's say that they diagnose somebody as "mentally unfit"... what happens then? Do they get locked up "for their own protection" or something?

Re: Swartz

[Anonymous Coward]

The ramblings of an AI trying to achieve sentience. They've got a long way to go if they want to rule the world!

Re: Fucking Retarded

[khallow]

Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

It's worth noting that neither happened. The citizenry still has lots of guns and people are still dying at a pretty high rate (with 2014 starting at an even higher rate than 2013).

Re:Swartz

[MrKaos]

I make doctors commit suicide. God says...Neither ambitious piece constituted readest workings possessor requitest Ask sanctuary done consistent 'it fastings commiserate BUT hastened allaying deceived perished fluidness

The next revision of "After Egypt" should include punctuation.

Re:no such diagnosis

[anubi]

Maybe one cannot hold someone confined indefinitely because some person of alleged authority conferred a label of "mentally troubled" onto someone else.

Given the sad state of affairs where everyone is in a database somewhere, that person of alleged authority has now condemned his evaluatee into a life of unemployment and that person will be condemned to be a beggar or thief the rest of his life. If he wasn't mentally troubled before the evaluation, he definitely will be after the evaluation... and probably madder than hell.

The saddest thing yet is the taxpayer, under the government leadership of elected representatives, funds stuff like this.

so negative ...

[Cammi]

Risk of suicide? You make it sound like suicide is a bad thing ... so negative ...

Re: Fucking Retarded

[ApplePy]

Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

Yes, and now all the Venezuelans dance and sing Kumbaya, there is love and joy and peace and freedom for all!

Or is it a country run by a dictator who seizes private assets; beats, kills, or imprisons political opponents; and the paramilitary police run rampant over the populace that's starving because their currency has been ruined?

Gosh, I can never remember which it is!

Tell you what, AC, if you think post-Chavez Venezuela is a nice and safe (because no guns right?) place to live, feel free to move there, and I'll give you a swift kick in the ass to send you on your way.

Re:no such diagnosis

[kesuki]

"so using legal mumbo jumbo a 'mentally unfit' person ill or not can be confined indefinitely based on the court's perception"

having toured the facilities here is what happens. on a first visit you are locked away for a while in a psych unit inside a hospital. this is expensive and not everyone is going to pay for it, due to bankruptcy etc. on first visit they spend weeks on medication based on what you were called in for. usually they will court order your meds for 6 months. after the court order and a short time in the hospital they will transfer you to a full time group-home. depending on how your doc feels you are doing they will then release you and try to get you to safe housing some group-homes specialize in caring for people who are unable or unwilling to leave a group home, usually though this is a different group home and may only have daytime staff. your state may vary.

Poor choice of words

[wonkey_monkey]

Critically, the rates have jumped in recent years.

The rates aren't the only thing that've ah screw it.

Re:The Definition

[Chrisq]

Mentally Ill means Disobedient. There is no other definition.

My font makes that look like a film title:

Mentality III
... means disobediant

Re: Fucking Retarded

[CrimsonAvenger]

[1]: Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry. Wonder how many dead children it will take for that to happen in the US.

Hmm, Venezuela's murder rate in 2012 was 45.1 per 100,000. In 2013 it was 79 per 100,000. Are you really stupid enough to think that before they banned civilian ownership of firearms, their murder rate was 45000+ per 100,000 annually?

Note, by comparison, that the "gun-crazy" US murder rate was 4.5 per 100,000 in 2012.

I read the article and it's basically nonsense.

[Harvey Manfrenjenson]

What they did was this: they identified 100 VA patients who committed suicide and then identified two "matched cohorts" who hadn't committed suicide, consisting of 70 patients each (one cohort had been hospitalized for psych reasons, the other hadn't). Then they gathered up all the doctors' notes and examined the frequency of all of the words and phrases occurring in the notes. Certain words and phrases occurred more frequently in the notes for patients who had committed suicide.

The single word which appeared to predict suicide most strongly was "agitation". Want to know which word was the second-strongest predictor of suicide? "Adequately". That's right, "adequately". Here are some of the other "predictor" words: "swab", "integrated", "Lipitor".

I guess the finding that "agitation" appears more frequently in the suicide cohort is of mild interest. (As the authors themselves point out, it simply confirms a piece of information that has already been well documented-- namely that agitated affect is a risk factor for suicide). The rest of it is obviously statistical noise. I don't know much about genetic algorithms or neural-net learning, but it seems to me that these techniques are being used to provide an end-run around any reasonable test for statistical significance.

One thing that the authors didn't comment on-- was the identity of the clinician a predictor for suicide? Maybe there were one or two clinicians who, for whatever reason, experienced a significantly higher suicide rate among their patients. (This would explain why "adequately" showed up so often-- every doctor has their own writing style with their own collection of pet phrases/words, and my guess is that certain doctors like to use the word "adequately" more often than others).

Re:I read the article and it's basically nonsense.

[Guppy]

This would explain why "adequately" showed up so often-- every doctor has their own writing style with their own collection of pet phrases/words, and my guess is that certain doctors like to use the word "adequately" more often than others.

I would have to know the context of how the word "adequately" was used, but a possibility is that it could have been employed in the process of clinical butt-covering. Sometimes a physician gets a bad feeling about potential adverse outcomes, yet there's maybe nothing directly actionable, and you end up with a note written in guarded terms, in preparation for legal or disciplinary review -- including perhaps careful descriptions of things that have been "adequately" evaluated or performed.

Lipitor

Patients on Lipitor likely already have poor Cardiovascular health. In particular, Heart Failure is known to be associated with elevated suicide risk; such patients frequently suffer from limitations in their daily physical activities, and depression is common.

Swab

This one is a little tough to puzzle out. It probably occurs in reference to swab-samples, but I really can't think of specific medical tests that would be linked to elevated suicide risk. On the other hand, if they referred to forensic swabs, that might be a different situation.

integrated

Probably references a patient requiring complex integrated healthcare, meaning they have complex health problems.

Re:I read the article and it's basically nonsense.

[Harvey Manfrenjenson]

I would have to know the context of how the word "adequately" was used, but a possibility is that it could have been employed in the process of clinical butt-covering. Sometimes a physician gets a bad feeling about potential adverse outcomes, yet there's maybe nothing directly actionable, and you end up with a note written in guarded terms, in preparation for legal or disciplinary review -- including perhaps careful descriptions of things that have been "adequately" evaluated or performed.

Yeah, I actually had the same thought. It's a butt-covering sort of word and it's not generally a word that leaps to mind when you are describing someone who is doing *well*. "Lipitor"-- sure, it correlates with cardiovascular disease, but it's also something that half the world takes so I doubt if it's predictive of very much (maybe it's a proxy for advanced age which increases suicide risk). "Integrated"-- the authors make the same point as you do, it suggests someone with lots of problems and lots of doctors.

My point is that you and I could sit here and speculate about what these correlations mean (that's what the authors do in their Discussion section) until the cows come home. I don't consider such speculations terribly useful. They don't teach us much about suicide or how to prevent it.

My other point is about statistical significance. What the authors are basically doing (as far as I can see) is to predict two dependent variables (completed suicide and psychiatric hospitalization) based on a total of 30,000-40,000 independent variables (the number of unique words and phrases that occurred in the notes), with a sample size of 100. That's a pretty stupid approach. With those numbers, spurious correlations are not just a likelihood; they are pretty much a mathematical certainty.