Predicting the Risk of Suicide By Analyzing the Text of Clinical Notes

First time accepted submitter J05H writes "Soldier and veteran suicide rates are increasing due to various factors. Critically, the rates have jumped in recent years. Now, Bayesian search experts are using anonymous Veteran's Administration notes to predict suicide risks. A related effort by Chris Poulin is the Durkheim Project which uses opt-in social media data for similar purposes."

Comments

Sounds legit

[kaoshin]

With significant development I think detection could become at least 50% successful. It would probably be more cost effective though to just not lose the quarters when you flip them.

Re:Sounds legit

[ebno-10db]

That would make some sense if the suicide rate was around 50%. Thankfully it's much lower.

Re:Sounds legit

[Fwipp]

Well, if the suicide rate's 10%, just say "no soldiers commit suicide," and bam, you're 90% successful.

Re:Sounds legit

[davester666]

Um, this isn't about predicting the suicide rate or how likely someone in the general population is going to commit suicide.

It's about how likely a veteran who writes a suicide note and gives it to someone else is going to follow through and try to commit suicide.

That rate is probably closer to the flip of a coin than it is to 10%.

Re:Sounds legit

[Anonymous Coward]

No it isn't. It's about predicting whether a veteran is likely to commit suicide from his medical records. Clinical notes are notes written by the doctor, not suicide notes.

Re:Sounds legit

[coolsnowmen]

Clever and true, but...here is the difference between statistics and engineers who need statistics.

Now all your mistakes are in the wrong direction. For different applications you will err on the side of False Alarm or Missed Detection. In this case, a false detection means you stages an unnecessary intervention (slightly costly / embarrassing, but at least they know you care); but, a missed detection literally means someone died.

Basically this is the application of risk/reward to probability.

if there's any standing ovations left... typo

[Anonymous Coward]

this guy deserves much better treatment from US perfect balancing actors http://www.youtube.com/results...

it's late, cold, the cat is stuck in the microwave who wouldn't make some typos?

threat of harm to self or others or not /fill

[Anonymous Coward]

sop on intake for decades now. does the 'note' say otherwise?

False positives

[dala1]

According to the study this is 67% effective. But, once this is applied to the general population you have an issue, because the vast majority of people are not suicidal. In the US, about 122 in 100,000 people attempt suicide a year, and about one in ten are successful. Even with a test that is 99% accurate, you are going to end up with somewhere around seven million false positives every year if you screen everyone.

Re:False positives

[femtobyte]

The question, though, is how harmful a false positive is. If false positive means "lock the person up for their own safety," then obviously you don't want many false positives. But, if false positive just means "bring them back in a week earlier for an extra session," then this might not be too bad. Even the "false positives" --- people who don't get as far as suicide attempts --- might be folks who would benefit from a little extra help. If the seven million most-likely-to-be-psychologically-vulnerable people in the US get a little extra care, there's probably significant good done.

Re:False positives

[Anonymous Coward]

once this is applied to the general population you have an issue, because the vast majority of people are not suicidal.

I'm sure that can be fixed.

Re:False positives

[dala1]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal. Consider the time and money spent, the extra doctors who have to be trained and hired (I'm assuming psychiatrists since a GP is hardly qualified to assess a potentially suicidal patient), and the days missed from work for false alarms. That's all before the psychological trauma and loss of trust from your doctor telling you out of the blue that they think you might be mentally ill when you're not.

Besides, that's not how it would work, because once the tool is out there and used to profile everyone, someone who is suicidal will commit suicide before that 'extra session.' After that, it will be be considered negligent not to 'do something' immediately once someone is flagged, and that something would likely be intrusive and expensive.

Also, their accuracy rate is 67%, not 99%. I used that number to demonstrate a best-case scenario. As it stands, they would flag around 83 million people while only correctly flagging around 200,000. Good luck with 99.75% false positives.

Re:False positives

[Anonymous Coward]

Total Genocide will fix every problem. Kill everything that has a genome. Everything.

Re:False positives

[stealth_finger]

Total Genocide will fix every problem. Kill everything that has a genome. Everything.

Good plan, start with the AC's

Re:False positives

[gnasher719]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal.

An interesting attitude. Compare this to Foxconn, which reduced the suicide rate among its workforce from 1 in 60,000 to 1 in 400,000 in three years.

Re:False positives

[Cinnamon Beige]

Seven million extra doctors' visits are hardly inconsequential, especially considering that only about 1 in 175 would actually be suicidal.

An interesting attitude. Compare this to Foxconn, which reduced the suicide rate among its workforce from 1 in 60,000 to 1 in 400,000 in three years.

All things considered, I think they did it by making it harder to commit suicide, and possibly also by improving labor conditions.

The usual process is to place somebody thought suicidal on a suicide watch. This can actually be very intrusive, and a test like this certainly is less than ideal if you're applying it at large--the accuracy here is for this population, and rather close to chance already. In a wider population, of a different makeup, its accuracy will be different, and probably lower.

More importantly, if you read the PLOS one article, they're discussing data mining the clinical notes themselves, and they admit that this is a branch of research that has been rather neglected: certain factors were deemed to have predictive value, without anybody really checking to see if that was true.

Let's say you're sitting in the entry way of an office building, and you notice that most people who come in to the building are men. This does not mean you can necessarily predict that a man walking past is going to enter the building; it might turn out that, in fact, of the people passing by the building, any given woman is more likely to come in--it's just that most of the people passing by right now are men.

It does not follow that if "Most of the people who do x are y" is true that "Most people who are y do x" is also true, for any set of x and y.

65% accuracy is not good, it's a start and it's better than what we currently have. In fact, the paper outright says that currently, they haven't even managed to validate the tool. In fact, I can easily give you the tl;dr version of this paper:

The indications for the future of this path of research are promising. Please fund the next phase so we can get it closer to practical application(s).

In less scientific phrasing:

We haven't reached a dead in, give us money so we can keep going!

It's not as much a breakthrough as a status report on the progress towards a breakthrough...

Re:False positives

[Anonymous Coward]

Hello,
Far be it from me to tell others what to do, but in my honest opinion, not only is this not fit for the general population, but to be quite honest, I think the costs far outweigh the benefits, especially when used with individuals who have not contributed the way that Veterans have. In other words, I would be willing to spend the money this would cost to help Veterans, but absolutely NOT non-Veterans. If someone in the general population commits suicide, it's not as big a deal, and should not be considered as big a deal, then if a Veteran does so. I'm very sorry if this upsets some, but those who are upset obviously refuse to see the reality of this

Idiots flipping quarters.

[Anonymous Coward]

This is how I get my laundry money.

You don't want to know what I do with the dice from rolls of the dice. You couldn't sleep after learning, let me assure you.

Re:Idiots flipping quarters.

[Anonymous Coward]

Let me assure you, you're full of shit and probably old mens semen too.

Rest assured, no matter what your fucked up mind can conceive, I will still sleep soundly at night.

A positive use of data mining

[SigmoidCurve]

It's refreshing to see predictive data analysis used for positive efforts, rather than simply selling more ads. Here's a call to action for all you data scientists at Twitter, FB, and other SV startups who think they're changing the world when all they're doing is putting money in their advertisers' pockets. News flash: statistics can be used to benefit society for a change.

Re:A positive use of data mining

[Zynder]

They'll fuck that up like they do everything. At first it's finding suicide patterns, but then it's predicting future criminality ala Minority Report. Yeah, I'm pessimistic but I have good reasons.

Re:A positive use of data mining

[TheRealMindChild]

It might be an attempt at using data mining for a good cause, but it is using a case where the source data itself is questionable. If you have ever wondered what it is like to be in a loony bin, it is a bunch of babysitters who hope to become doctors, playing the telephone game with the doctor-of-the-day, who reports to the actual head doctor. The observation skills of these babysitters is like a 4 year old at a gun and ammo show. Then as information is relayed twice over and statistical analytics lump you into a hierarchical cluster, you are then medicated and treated like a typical case study from said cluster. It is like a black hole of no escape, no matter how wrong it is. This is the data they are mining.

Re:A positive use of data mining

[RuffMasterD]

I read something about predicting criminality not long ago, but from a legal ethics angle. Since the human genome has been mapped, and every criminal (in my country at least) provides a DNA sample, and DNA sequencing is relatively cheap, there is now a wealth of data to mine. Behavioural Geneticists are starting to look for 'criminal' behaviour markers that distinguish, and to some extent explain, criminal behaviour. Some creative defence lawyers are already using this knowledge to help their cases. The flip-side of this knowledge is the possibility of law enforcement pre-emptively scanning people, maybe even at birth, and monitoring or even detaining those with 'criminal' DNA profiles. Really scary stuff indeed, and almost nothing to stop it from happening. But it's more likely the people in power will abuse this kind of technology, 'for the greater good', not the scientists.

Re: Fucking Retarded

[Anonymous Coward]

Wait till they start scanning school kid's english papers and Facebook pages.

corepirate nazis operate on perceived threat

[Anonymous Coward]

so the sop for them is to be a threat to/destroy their own perception of danger which is us. that we may rise up & eat them is their greatest fear? (it's happened before). kind of psychosis with a hysterical history base. history of abuse predominate cause of violence towards everything self loathing. fathers may i & mopery had nothing to do with it?

Kuro5hin user?

[Anonymous Coward]

NT

Re: Swartz

[Anonymous Coward]

What the fuck did I just read?

So what do they do about it?

[mark-t]

Let's say that they diagnose somebody as "mentally unfit"... what happens then? Do they get locked up "for their own protection" or something?

Re:So what do they do about it?

[Anonymous Coward]

What happens is exactly the same as with any other medical diagnostic that you might think of: you get advised to follow a therapy of some kind. Whether you have been diagnosed with flu or high suicide probability does not mean you get immediately locked-up.

Re:So what do they do about it?

[Anonymous Coward]

What's frightening is that corporations and universities are increasingly using this kind of Russian Psychiatry to prevent ex-employees from pursuing employment/qui tam (whistleblowing)/discrimination/etc. suits. (See, e.g., http://abovethelaw.com/2013/11...)

Also, they can hold you indefinitely because the ALJs are essentially a rubber stamp -- they agree with the psychologists and psychiatrists over 95% of the time. And, increasingly, physicians are doing this because they're -- and their employers -- are mistakenly afraid of lawsuits. And I don't mean frivolous ones: I mean litigation that holds corporations accountable for their actions, like the one where a man was released because he was uninsured, killed himself and his family, and the hospital lost 25+ million in the subsequent litigation.

But instead of being honest about it, they just claim "terrorism" and buff the charts to make it look that way.

TL;DR: Don't see a psychiatrist in the US. Or the West.

Re:So what do they do about it?

[Anonymous Coward]

Exactly how I feel about it. I've seen numerous occasions online where people said they are concerned [about a stranger] and consider reporting them. I find it frightening that this line of thinking is socially acceptable, normal even. It's very similar to a communist dictatorship, where the one thing you mustn't do is leave, where your fellow man will report you to the authorities for such thought crime, where you'll get locked up "for your own safety".

What worldview inspires people to build such a cage?

Re: Fucking Retarded

[Anonymous Coward]

I'm sure this is will come to pass, especially when the next school shooting [1] happens... FB and other sites will be required to run a real time filter that notifies local police 24/7 if some emo kid writes a post that passes a threshold, they get picked up by the white coats and force fed Prozac until their synapses collapse.

[1]: Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry. Wonder how many dead children it will take for that to happen in the US.

Re: Swartz

[Anonymous Coward]

The ramblings of an AI trying to achieve sentience. They've got a long way to go if they want to rule the world!

no such diagnosis

[Anonymous Coward]

so using legal mumbo jumbo a 'mentally unfit' person ill or not can be confined indefinitely based on the court's perception

Re:no such diagnosis

[anubi]

Maybe one cannot hold someone confined indefinitely because some person of alleged authority conferred a label of "mentally troubled" onto someone else.

Given the sad state of affairs where everyone is in a database somewhere, that person of alleged authority has now condemned his evaluatee into a life of unemployment and that person will be condemned to be a beggar or thief the rest of his life. If he wasn't mentally troubled before the evaluation, he definitely will be after the evaluation... and probably madder than hell.

The saddest thing yet is the taxpayer, under the government leadership of elected representatives, funds stuff like this.

Re:no such diagnosis

[kesuki]

"so using legal mumbo jumbo a 'mentally unfit' person ill or not can be confined indefinitely based on the court's perception"

having toured the facilities here is what happens. on a first visit you are locked away for a while in a psych unit inside a hospital. this is expensive and not everyone is going to pay for it, due to bankruptcy etc. on first visit they spend weeks on medication based on what you were called in for. usually they will court order your meds for 6 months. after the court order and a short time in the hospital they will transfer you to a full time group-home. depending on how your doc feels you are doing they will then release you and try to get you to safe housing some group-homes specialize in caring for people who are unable or unwilling to leave a group home, usually though this is a different group home and may only have daytime staff. your state may vary.

Bad wars

[Anonymous Coward]

How about: Don't fight pointless/immoral wars?

Re:Bad wars

[Anonymous Coward]

Answer the Call Of Duty, citizen! Serve Obama for the glory of Obama!!

Re:Bad wars

[Anonymous Coward]

Is there a queue to suck Obama's thick meaty dick? I want to be next!

true psychopath both homicidal & suicidal

[Anonymous Coward]

the most glamorous of whackos,, history of abuse as a child + ptsd from helping out in 'civil' wars (harming others back) etc.. leaves us nearly spiritless unfunctioning in any positive manner poor prognosis

when evaluating one of these... individuals

[Anonymous Coward]

for best results be sure to check the gene pool & other dark matters

Re: Fucking Retarded

[khallow]

Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

It's worth noting that neither happened. The citizenry still has lots of guns and people are still dying at a pretty high rate (with 2014 starting at an even higher rate than 2013).

Re: Fucking Retarded

[BlueStrat]

[1]: Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

Chavez forbid citizens from owning/possessing firearms, meanwhile putting most of the former criminals and criminal gangs to work for him as enforcers and arming them. Citizens aren't shooting each other, they're too busy just trying to survive and avoid being shot (or much worse, especially for females) by Chavez' goons.

People are still dying, it's simply not reported as such (if noted/recorded at all) when it's Chavez' own thugs doing the killing. Do you also believe the various Utopian stats the N. Korean and Chinese governments release?

Your typical peacetime violent civilian-on-civilian crime generally trends downward among a population when there's an extremely authoritarian, oppressive, violent, and cruel regime in charge. The more extreme, the lower the numbers down to a point. It's interesting to note, however, when smaller groups are isolated and forced together, the opposite happens (see US and many other nations' prisons).

Strat

Re:Swartz

[MrKaos]

I make doctors commit suicide. God says...Neither ambitious piece constituted readest workings possessor requitest Ask sanctuary done consistent 'it fastings commiserate BUT hastened allaying deceived perished fluidness

The next revision of "After Egypt" should include punctuation.

so negative ...

[Cammi]

Risk of suicide? You make it sound like suicide is a bad thing ... so negative ...

Re:so negative ...

[Anonymous Coward]

If you like suicide so much why don't you fucking kill yourself?

Re:so negative ...

[Anonymous Coward]

Risk of suicide? You make it sound like suicide is a bad thing ... so negative ...

I cant believe I need to scroll this far for such a good comment. Whatever happened to freedom? Some people need help and turn to suicide because nobody is there for them. Some people just commit suicide. If it is the persons free choice then what is wrong with it?

Good comment Cammi

Re: Fucking Retarded

[ApplePy]

Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry.

Yes, and now all the Venezuelans dance and sing Kumbaya, there is love and joy and peace and freedom for all!

Or is it a country run by a dictator who seizes private assets; beats, kills, or imprisons political opponents; and the paramilitary police run rampant over the populace that's starving because their currency has been ruined?

Gosh, I can never remember which it is!

Tell you what, AC, if you think post-Chavez Venezuela is a nice and safe (because no guns right?) place to live, feel free to move there, and I'll give you a swift kick in the ass to send you on your way.

if i murdered and tortured tens of brown people

[Anonymous Coward]

I'd want to kill myself too.

Re:if i murdered and tortured tens of brown people

[Anonymous Coward]

what about white people. would you still want to kill yourself if you murdered millions of white people.

predict this

[Anonymous Coward]

fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you am i suicidal yet you stupid fuck

suicide is a perfectly rational response

[Anonymous Coward]

Soldiers choose to kill themselves when they realize they serve the ultimate evil Obama.

Re:Fucking Retarded

[Anonymous Coward]

the label "mentally ill" label is now apparently being used to strip people of their rights

Now being used? Where the fuck have you been, man? "Mentally ill" has been used as a label to strip people of their rights since forever. What, you think it's only recently been used to jail political dissidents? Again, where the fuck have you been, man?? Ever read any human history, ever???

The Definition

[Anonymous Coward]

Mentally Ill means Disobedient. There is no other definition.

Re:The Definition

[Chrisq]

Mentally Ill means Disobedient. There is no other definition.

My font makes that look like a film title:

Mentality III
... means disobediant

suicides should be convicted

[Anonymous Coward]

of murder

Poor choice of words

[wonkey_monkey]

Critically, the rates have jumped in recent years.

The rates aren't the only thing that've ah screw it.

Re: Fucking Retarded

[CrimsonAvenger]

[1]: Ironic that Venezuela had their violent crime rate drop by a factor of a thousand by removing guns from the citizenry. Wonder how many dead children it will take for that to happen in the US.

Hmm, Venezuela's murder rate in 2012 was 45.1 per 100,000. In 2013 it was 79 per 100,000. Are you really stupid enough to think that before they banned civilian ownership of firearms, their murder rate was 45000+ per 100,000 annually?

Note, by comparison, that the "gun-crazy" US murder rate was 4.5 per 100,000 in 2012.

Re:Swartz

[Anonymous Coward]

Dude, step away from the crack pipe...

How about just Listening?

[Anonymous Coward]

Good quality counselors and therapists have used this ground breaking technique for years...
It's called listening.

I read the article and it's basically nonsense.

[Harvey Manfrenjenson]

What they did was this: they identified 100 VA patients who committed suicide and then identified two "matched cohorts" who hadn't committed suicide, consisting of 70 patients each (one cohort had been hospitalized for psych reasons, the other hadn't). Then they gathered up all the doctors' notes and examined the frequency of all of the words and phrases occurring in the notes. Certain words and phrases occurred more frequently in the notes for patients who had committed suicide.

The single word which appeared to predict suicide most strongly was "agitation". Want to know which word was the second-strongest predictor of suicide? "Adequately". That's right, "adequately". Here are some of the other "predictor" words: "swab", "integrated", "Lipitor".

I guess the finding that "agitation" appears more frequently in the suicide cohort is of mild interest. (As the authors themselves point out, it simply confirms a piece of information that has already been well documented-- namely that agitated affect is a risk factor for suicide). The rest of it is obviously statistical noise. I don't know much about genetic algorithms or neural-net learning, but it seems to me that these techniques are being used to provide an end-run around any reasonable test for statistical significance.

One thing that the authors didn't comment on-- was the identity of the clinician a predictor for suicide? Maybe there were one or two clinicians who, for whatever reason, experienced a significantly higher suicide rate among their patients. (This would explain why "adequately" showed up so often-- every doctor has their own writing style with their own collection of pet phrases/words, and my guess is that certain doctors like to use the word "adequately" more often than others).

Re:I read the article and it's basically nonsense.

[Guppy]

This would explain why "adequately" showed up so often-- every doctor has their own writing style with their own collection of pet phrases/words, and my guess is that certain doctors like to use the word "adequately" more often than others.

I would have to know the context of how the word "adequately" was used, but a possibility is that it could have been employed in the process of clinical butt-covering. Sometimes a physician gets a bad feeling about potential adverse outcomes, yet there's maybe nothing directly actionable, and you end up with a note written in guarded terms, in preparation for legal or disciplinary review -- including perhaps careful descriptions of things that have been "adequately" evaluated or performed.

Lipitor

Patients on Lipitor likely already have poor Cardiovascular health. In particular, Heart Failure is known to be associated with elevated suicide risk; such patients frequently suffer from limitations in their daily physical activities, and depression is common.

Swab

This one is a little tough to puzzle out. It probably occurs in reference to swab-samples, but I really can't think of specific medical tests that would be linked to elevated suicide risk. On the other hand, if they referred to forensic swabs, that might be a different situation.

integrated

Probably references a patient requiring complex integrated healthcare, meaning they have complex health problems.

Re:I read the article and it's basically nonsense.

[Harvey Manfrenjenson]

I would have to know the context of how the word "adequately" was used, but a possibility is that it could have been employed in the process of clinical butt-covering. Sometimes a physician gets a bad feeling about potential adverse outcomes, yet there's maybe nothing directly actionable, and you end up with a note written in guarded terms, in preparation for legal or disciplinary review -- including perhaps careful descriptions of things that have been "adequately" evaluated or performed.

Yeah, I actually had the same thought. It's a butt-covering sort of word and it's not generally a word that leaps to mind when you are describing someone who is doing *well*. "Lipitor"-- sure, it correlates with cardiovascular disease, but it's also something that half the world takes so I doubt if it's predictive of very much (maybe it's a proxy for advanced age which increases suicide risk). "Integrated"-- the authors make the same point as you do, it suggests someone with lots of problems and lots of doctors.

My point is that you and I could sit here and speculate about what these correlations mean (that's what the authors do in their Discussion section) until the cows come home. I don't consider such speculations terribly useful. They don't teach us much about suicide or how to prevent it.

My other point is about statistical significance. What the authors are basically doing (as far as I can see) is to predict two dependent variables (completed suicide and psychiatric hospitalization) based on a total of 30,000-40,000 independent variables (the number of unique words and phrases that occurred in the notes), with a sample size of 100. That's a pretty stupid approach. With those numbers, spurious correlations are not just a likelihood; they are pretty much a mathematical certainty.

V.A.

[Anonymous Coward]

The V.A does not really do Mental Health, from my personal experience.
The V.A. does Medication.
They think that if you medicate the veterans enough that they no longer have a problem. I have been dealing with P.T.S.D. and a whole host of issues from my time in service. They issued me a whole host of drugs to make me better. These drugs mostly just made me suicidal and VERY angry. I stopped going there because of what they were doing to me, basically turning me into a Very Angry Zombie that didn't care if i lived or died or anyone else for that matter. The counseling consisted of "here take these and I will see you in six weeks" because we do not have anyone to counsel you, maybe next time. When they did get a counselor, it was usually some one that lasted a couple of sessions and left. Rinse and repeat, over and over and over. Out of all the 'help" they gave me, I had an intern suggest something to me that actually made a difference, an intern, not a doctor. I ended up throwing away all the medications (after YEARS of trying it their way). If not for my wife and family that stuck by my side through the worst of this, I most likely would have been dead.

I have found my own answers. I still have the hyper-vigilance, nightmares and even flashbacks triggered by certain sounds or smells but I am able to understand why and recognize that when these occur, I am able to deal with it in my own way. I cannot go anywhere with large crowds for any length of time, I have a very big avoidance problem. I really have no personal friends, just family, and I am OK with that. My quality of life is not what you would expect for a relatively healthy man in his mid-forties, and I am OK with that as well.

I am not advocating any veteran to toss their meds or forgo counseling, but my experience with the VA was LESS than satisfactory. So, when I hear anything from the V.A. I just think to myself "How sad for anyone in that system".