Predicting the Risk of Suicide By Analyzing the Text of Clinical Notes

First time accepted submitter J05H writes "Soldier and veteran suicide rates are increasing due to various factors. Critically, the rates have jumped in recent years. Now, Bayesian search experts are using anonymous Veteran's Administration notes to predict suicide risks. A related effort by Chris Poulin is the Durkheim Project which uses opt-in social media data for similar purposes."

Comments

Re:Sounds legit

[ebno-10db]

That would make some sense if the suicide rate was around 50%. Thankfully it's much lower.

False positives

[dala1]

According to the study this is 67% effective. But, once this is applied to the general population you have an issue, because the vast majority of people are not suicidal. In the US, about 122 in 100,000 people attempt suicide a year, and about one in ten are successful. Even with a test that is 99% accurate, you are going to end up with somewhere around seven million false positives every year if you screen everyone.

Re:False positives

[dala1]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal. Consider the time and money spent, the extra doctors who have to be trained and hired (I'm assuming psychiatrists since a GP is hardly qualified to assess a potentially suicidal patient), and the days missed from work for false alarms. That's all before the psychological trauma and loss of trust from your doctor telling you out of the blue that they think you might be mentally ill when you're not.

Besides, that's not how it would work, because once the tool is out there and used to profile everyone, someone who is suicidal will commit suicide before that 'extra session.' After that, it will be be considered negligent not to 'do something' immediately once someone is flagged, and that something would likely be intrusive and expensive.

Also, their accuracy rate is 67%, not 99%. I used that number to demonstrate a best-case scenario. As it stands, they would flag around 83 million people while only correctly flagging around 200,000. Good luck with 99.75% false positives.

Re:False positives

[Cinnamon Beige]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal.

An interesting attitude. Compare this to Foxconn, which reduced the suicide rate among its workforce from 1 in 60,000 to 1 in 400,000 in three years.

All things considered, I think they did it by making it harder to commit suicide, and possibly also by improving labor conditions.

The usual process is to place somebody thought suicidal on a suicide watch. This can actually be very intrusive, and a test like this certainly is less than ideal if you're applying it at large--the accuracy here is for this population, and rather close to chance already. In a wider population, of a different makeup, its accuracy will be different, and probably lower.

More importantly, if you read the PLOS one article, they're discussing data mining the clinical notes themselves, and they admit that this is a branch of research that has been rather neglected: certain factors were deemed to have predictive value, without anybody really checking to see if that was true.

Let's say you're sitting in the entry way of an office building, and you notice that most people who come in to the building are men. This does not mean you can necessarily predict that a man walking past is going to enter the building; it might turn out that, in fact, of the people passing by the building, any given woman is more likely to come in--it's just that most of the people passing by right now are men.

It does not follow that if "Most of the people who do x are y" is true that "Most people who are y do x" is also true, for any set of x and y.

65% accuracy is not good, it's a start and it's better than what we currently have. In fact, the paper outright says that currently, they haven't even managed to validate the tool. In fact, I can easily give you the tl;dr version of this paper:

The indications for the future of this path of research are promising. Please fund the next phase so we can get it closer to practical application(s).

In less scientific phrasing:

We haven't reached a dead in, give us money so we can keep going!

It's not as much a breakthrough as a status report on the progress towards a breakthrough...

A positive use of data mining

[SigmoidCurve]

It's refreshing to see predictive data analysis used for positive efforts, rather than simply selling more ads. Here's a call to action for all you data scientists at Twitter, FB, and other SV startups who think they're changing the world when all they're doing is putting money in their advertisers' pockets. News flash: statistics can be used to benefit society for a change.

So what do they do about it?

[mark-t]

Let's say that they diagnose somebody as "mentally unfit"... what happens then? Do they get locked up "for their own protection" or something?

Re: Fucking Retarded

[khallow]

Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

It's worth noting that neither happened. The citizenry still has lots of guns and people are still dying at a pretty high rate (with 2014 starting at an even higher rate than 2013).

Re:Sounds legit

[Fwipp]

Well, if the suicide rate's 10%, just say "no soldiers commit suicide," and bam, you're 90% successful.

Re:Sounds legit

[davester666]

Um, this isn't about predicting the suicide rate or how likely someone in the general population is going to commit suicide.

It's about how likely a veteran who writes a suicide note and gives it to someone else is going to follow through and try to commit suicide.

That rate is probably closer to the flip of a coin than it is to 10%.

Poor choice of words

[wonkey_monkey]

Critically, the rates have jumped in recent years.

The rates aren't the only thing that've ah screw it.

I read the article and it's basically nonsense.

[Harvey Manfrenjenson]

What they did was this: they identified 100 VA patients who committed suicide and then identified two "matched cohorts" who hadn't committed suicide, consisting of 70 patients each (one cohort had been hospitalized for psych reasons, the other hadn't). Then they gathered up all the doctors' notes and examined the frequency of all of the words and phrases occurring in the notes. Certain words and phrases occurred more frequently in the notes for patients who had committed suicide.

The single word which appeared to predict suicide most strongly was "agitation". Want to know which word was the second-strongest predictor of suicide? "Adequately". That's right, "adequately". Here are some of the other "predictor" words: "swab", "integrated", "Lipitor".

I guess the finding that "agitation" appears more frequently in the suicide cohort is of mild interest. (As the authors themselves point out, it simply confirms a piece of information that has already been well documented-- namely that agitated affect is a risk factor for suicide). The rest of it is obviously statistical noise. I don't know much about genetic algorithms or neural-net learning, but it seems to me that these techniques are being used to provide an end-run around any reasonable test for statistical significance.

One thing that the authors didn't comment on-- was the identity of the clinician a predictor for suicide? Maybe there were one or two clinicians who, for whatever reason, experienced a significantly higher suicide rate among their patients. (This would explain why "adequately" showed up so often-- every doctor has their own writing style with their own collection of pet phrases/words, and my guess is that certain doctors like to use the word "adequately" more often than others).