AI Can Predict Heart Attacks More Accurately Than Doctors (digitaltrends.com)

An anonymous reader quotes a report from Digital Trends: Scientists from the University of Nottingham in the United Kingdom have managed to develop an algorithm that outperforms medical doctors when it comes to predicting heart attacks. As it stands, around 20 million people fall victim to cardiovascular disease, which includes heart attacks, strokes, and blocked arteries. Today, doctors depend on guidelines similar to those of the American College of Cardiology/American Heart Association (ACC/AHA) in order to predict individuals' risks. These guidelines include factors like age, cholesterol level, and blood pressure. In employing computer science, Stephen Weng, an epidemiologist at the University of Nottingham, took the ACC/AHA guidelines and compared them to four machine-learning algorithms: random forest, logistic regression, gradient boosting, and neural networks. The artificially intelligent algorithms began to train themselves using existing data to look for patterns and create their own "rules." Then, they began testing these guidelines against other records. And as it turns out, all four of these methods "performed significantly better than the ACC/AHA guidelines," Science reports. The most successful algorithm, the neural network, actually was correct 7.6 percent more often than the ACC/AHA method, and resulted in 1.6 percent fewer false positives. That means that in a sample size of around 83,000 patient records, 355 additional lives could have been saved.

Comments

I work in this field

[Anonymous Coward]

The biggest problem with these guidelines (the paper calls it an algorithm) is that humans optimize their predictive power using simple statistical analysis based on a training dataset. Then medical professionals are told the guidelines can be reasonably used anywhere. Throw it some data from a vastly different hospital and it will degrade in performance very quickly. For example, create rules using data from academic hospital, but test it on a rural hospital and the results won't be as good. There are different label distributions and often times even feature distributions.

If you use machine learning to train on the same hospital that you test, of course it will be better than a model created by humans on a different hospital.

I see this happen all the time with readmission prediction. There's a very popular system called LACE that was optimized for a specific hospital. It's simple to implement and test so there's a lot of reproduction. LACE has wild swings in performance (from AUC=0.55 to AUC=0.75 depending the dataset). Researchers will often compare LACE to a machine learning method like Random Forest, training and testing on same hospital, and shockingly the results are better.

Re:Cannot use the same set of data

[Anonymous Coward]

This exactly!

The baseline method was a diagnostic algorithm created by humans on a training dataset. Using machine learning, these researchers split train/test using a different dataset and compared models using the same test, but different train. YOU CANNOT DO THIS. But many researchers do and it bothers me because it's not sound methodology. The classification label and feature distributions will be different between the original train and new train datasets.

Also, I've noticed a huge increase in AUC as a primary metric and I'm not really convinced it's a good metric in this field. It's difficult to analyze misclassification cost using AUC and in the medical world FP and FN often have vastly different costs. Very few researchers are currently acknowledging this and I see very little cost sensitive classification in this domain. I've also seen several arguments in journals recently arguing that comparing classifiers using AUC is comparing apples to oranges.

And how many training-test cycles did they do?

[WoOS]

And one can "meta-train" for the test data group. Like
* Train
* Compare to test set
* Worse than guideline result => Change training parameters
* Train
* Compare to test set
* Still worse than guideline result => Change training parameters more
* Train
* Compare to test set
* Better than guideline result => Publish

I will be impressed, if it is better than a human doctor on new cases.

Misleading headline

[GeekWithAKnife]

The AI does not outperform a doctor. Does not outperform any doctor period.

It has managed to perform better than guidelines.

Now let's consider that the AI is basing this on data gathered by medical science. What it's outperforming in actuality is thus old prediction models.

Overall, quite happy about it but let's not pretend that AI had 5 hours of sleep in the last 48, had a shower, drove to work, checked on the kids and yet manage to perform all work duties admirably. -because then it might actually outperform a doctor.