Slashdot Mirror


Google AI Claims 99 Percent Accuracy In Metastatic Breast Cancer Detection

Researchers at the Naval Medical Center San Diego and Google AI, a division within Google dedicated to artificial intelligence research, are using cancer-detecting algorithms to detect metastatic tumors by autonomously evaluating lymph node biopsies. VentureBeat reports: Their AI system -- dubbed Lymph Node Assistant, or LYNA -- is described in a paper titled "Artificial Intelligence-Based Breast Cancer Nodal Metastasis Detection," published in The American Journal of Surgical Pathology. In tests, it achieved an area under the receiver operating characteristic (AUC) -- a measure of detection accuracy -- of 99 percent. That's superior to human pathologists, who according to one recent assessment miss small metastases on individual slides as much as 62 percent of the time when under time constraints. LYNA is based on Inception-v3, an open source image recognition deep learning model that's been shown to achieve greater than 78.1 percent accuracy on Stanford's ImageNet dataset. As the researchers explained, it takes as input a 299-pixel image (Inception-v3's default input size), outlines tumors at the pixel level, and, in the course of training, extracts labels -- i.e., predictions -- of the tissue patch ("benign" or "tumor") and adjusts the model's algorithmic weights to reduce error.

In tests, LYNA achieved 99.3 percent slide-level accuracy. When the model's sensitivity threshold was adjusted to detect all tumors on every slide, it exhibited 69 percent sensitivity, accurately identifying all 40 metastases in the evaluation dataset without any false positives. Moreover, it was unaffected by artifacts in the slides such as air bubbles, poor processing, hemorrhage, and overstaining. LYNA wasn't perfect -- it occasionally misidentified giant cells, germinal cancers, and bone marrow-derived white blood cells known as histiocytes -- but managed to perform better than a practicing pathologist tasked with evaluating the same slides. And in a second paper published by Google AI and Verily, Google parent company Alphabet's life sciences subsidiary, the model halved the amount of time it took for a six-person team of board-certified pathologists to detect metastases in lymph nodes.

34 comments

  1. it takes as input a 299-pixel image by Anonymous Coward · · Score: 0

    That's somewhere between 17x17 and 17x18, which makes it even more impressive.

    1. Re:it takes as input a 299-pixel image by ShanghaiBill · · Score: 1

      That's somewhere between 17x17 and 17x18, which makes it even more impressive.

      The default image for Inception-V3 is 299x299 RGB = 89401 pixels.

      The journalist is a moron. All he had to do was cut-and-paste, and he screwed it up.

    2. Re:it takes as input a 299-pixel image by Anonymous Coward · · Score: 0

      Shanghai "moron often proven wrong" Bill, calling people names because someone made a mistake which his punk ass sure never ever did right kids? Lol. Arrogant asshole.

  2. False Positive Rate is Equally High by Anonymous Coward · · Score: 0

    Distrubingly so.

  3. Why focus on breasts? by Anonymous Coward · · Score: 0

    Because they are breasts? Admittedly there is a greater publicity potential with breasts than colons, and having to explain to junior what a colon is and how to test for colon cancer at the dinner table would be awkward, whereas he'd get the idea readily about breast cancer since it's like being on the playground.

    1. Re: Why focus on breasts? by Anonymous Coward · · Score: 0

      Because google incels want excuses to see boobs

    2. Re: Why focus on breasts? by Anonymous Coward · · Score: 0

      Honest Trailer Guy says:

      Beeeeuuubbbbzzz

    3. Re:Why focus on breasts? by PPH · · Score: 2

      Very large training data set available on line.

      --
      Have gnu, will travel.
    4. Re:Why focus on breasts? by hcs_$reboot · · Score: 1

      Because the AI was made by men.

      --
      Slashdot, fix the reply notifications... You won't get away with it...
    5. Re:Why focus on breasts? by Megol · · Score: 1

      If you'd even attempt to make a real argument I could provided some (layman level) reasons however...

  4. Endgame by Kohath · · Score: 1

    This will allow Google to properly target ads to breast cancer patients.

    1. Re:Endgame by Anonymous Coward · · Score: 0

      Does 99 percent AUC mean that it's near perfect, or just that you've overfit on your dataset? LOL. First mistake of data science, Google! Thou shalt not get overly excited at high AUC numbers unless you've undergone extensive post-model testing.

    2. Re: Endgame by Anonymous Coward · · Score: 0

      yes I'm sure the people working at "Google AI, a division within Google dedicated to artificial intelligence research" have never heard of overfitting before.

  5. I'm better. by sunking2 · · Score: 1

    I just tell everyone I meet they have cancer. I haven't missed someone with cancer yet.

    1. Re:I'm better. by Anonymous Coward · · Score: 0

      The inverse works quite well. Tell everyone you meet that you have a perfect machine learning black box and it says they don't have cancer. The 0.0001% of the time you'll be wrong will be so low, the noise level of people complaining will be taken care of by malpractice insurance or death.

    2. Re: I'm better. by Anonymous Coward · · Score: 0

      They did give comparison value to human professionals but I don't understand why you even doubt it. Even I could build an AI that would perform in this task better than humans if I am given teaining images. This is trivial with modern tech. This is where AI is at its best.

    3. Re:I'm better. by Anonymous Coward · · Score: 0

      TUBE - In doctors lingo Totally Unnecessay Breast Examination.
      Actually my GP corrected me by saying breast examinations are never unnecessary, and now annotates his notes RBEPC Refused breast examination, prefers cancer.

  6. Of how many patients? by Anonymous Coward · · Score: 0

    This is no different from the Waymo claims. If it is contextually irrelevant it means precisely jack shit, and human physiology is far more individuated than roads are. They are just trying to placate investors, and it's painfully obvious. Will Google be the next Soylent or Theranos? It seems incredibly likely with each passing year.

  7. Who cares by Anonymous Coward · · Score: 0

    Women are cancer so maybe thats why God puts it in their boobs

  8. It's amazing. by Anonymous Coward · · Score: 0

    It is simply amazing. The operator hits the "yes" or "no" key and voila. Analysis complete. As artificial an intelligence as a wife-beating Sherrod Brown Democrap.

  9. I never met ... by CaptainDork · · Score: 1

    ... a static I didn't like.

    --
    It little behooves the best of us to comment on the rest of us.
  10. TLDR takeaway by SuperKendall · · Score: 1

    If Google is starting to feel everyone up "looking for cancer" I'd say it's more than time to go use DuckDuckGo!

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  11. Nothing surprising here by Anonymous Coward · · Score: 0

    Classifier NN are particularly very good at this task, classifying images among predetermined image set. We already know that the given images are images of cells, possibly affected by cancer. The network that was trained specifically to classify such images will do very well because it will pick up the most subtle clues that are often missed by humans. It can pick up a single digit difference in pixel brightness level, which human can't generally do.

    The same network however can't tell that the given picture is something other than cancer, like an elephant or a couch. It will just say - x% probability of cancer, which would be totally wrong.
    I expect that we would need so much fewer trained radiologists, pathologists and in general people who look at pictures and tell us things. We can train inception like networks to do this with much more accuracy and less mistakes. So all these radiologist jobs that were outsourced to India and other places soon will be done by AI.

  12. statistics by bigtreeman · · Score: 2, Informative

    1 in 99 is really bad
    1000 women, about 120 will get breast cancer, if we miss-diagnose 10 cases, that could be as bad as 8% failure
    fuck statistics

    --
    Go well
    1. Re:statistics by religionofpeas · · Score: 4, Insightful

      99% is pretty good for a notoriously difficult problem.

      Yeah, sucks if you're part of the 1%, but you'd be part of the 100% if there wasn't any test.

    2. Re:statistics by Anonymous Coward · · Score: 0

      It is stated in a convoluted way. It could also be read as 99% of cases where cancer is present is caught.

      I hope that's the case, as you say 8% failure is not something to hail as a great stride.

    3. Re:statistics by Anonymous Coward · · Score: 1

      Yeah. Fuck facts. #MAGA.

    4. Re:statistics by Megol · · Score: 1

      My understanding is that this is about the screening detecting metastatic spread of the breast cancer. Detect tumors, treat tumors (operation, radiation, chemotherapy), take biopsy samples of lymph nodes, analyze samples to detect cancer cells - if there are any further treatment is needed (chemo). Something like that.

      So the missed case is one of the people (men can get breast cancer too) that have cancer spreading through the lymphatic system.

    5. Re:statistics by Anonymous Coward · · Score: 2, Insightful

      I wanted to comment on a few things about medical statistics that are easy to misunderstand. Unfortunately, the summary of the article misuses some terminology which further obfuscates the issue.

      Some basic measures of a test are its sensitivity and specificity.
      1. Sensitivity is a measure of false negatives. It means that if you have 100 people with the disease, the test catches this percentage. So a test with a sensitivity of 99% would be positive on 99/100 patients with the disease.
      2. Specificity is a measure of how many false positives there are. So if you are using a test with 99% specificity on 100 people without the disease then you will get 1/100 false positive.
      3. An ROC curve graphs sensitivity at different test-cutoff thresholds (because most tests give a continuous, not binary result) and 1-specificty at different thresholds on the other axis. It is a general measure of the accuracy of a test but not the be-all-end-all. A perfect test has a area-under-the-curve (AUC) of 1, a useless test has and AUC of 0.5 or less.

      Usually if you increase the sensitivity (to lower false negatives) you must choose a threshold that corresponds to more false positives, and vice versa. Where you put your thresholds depends on what you want to use the test for. This is because while us doctors look at sensitivity and specificity to have a general idea of how a test works, what we really care about is the positive and negitive predictive values. A positive predictive value is what percentage of positive tests mean the patient has the disease. To calculate this you need to know what percentage of the tested population has the disease (the disease prevelence) - which is hard to know exactly but we try to eyeball to choose the right test. While the sensitivity and specificity are test characteristics at a specific threshold, the PPV and NPV apply to a specific patient population, for example 40-50yo women with a painless breast lump. 120/1000 is NOT the prevalence in the population tested, to refer to a previous poster (aside: in the article they talk about PPV and NPV but they are of course using the prevalence of cancer in the sample slide set used to calculate this).

      An example - let's say you have two tests for kryptonite sensitivity, a rare disease seen only in comic book heroes. Lets say you are using a test with 99% sensitivity and 99% specificity, and you are testing 1,000,000 people, only one of whom has the disease. You will get 1000 positives (99% specificity), but you'll have a 99% chance (99% sensitivity) that Superman will be among those 1000. Your positive predictive value is quite low, 1/1000 or 0.1%, so you're not that confident that a given person with a positive test has kryptonite sensitivity. Your negative predictive value is quite good, approaching 100%, so you're fairly confident that a negative test means that person is an earthling. This is just an example that shows that sensitivity and specificity must be interpreted in context of the population being tested. What we usually end up doing for problems like this is starting with a very sensitive test and then following it up with a more specific test.

      The article (not the summary) says they were able to get 100% sensitivity with the algorithm but they had only 84% specificity.
      When they reoptimized the algorithm for specificity of 100% it had a sensitivity of 69%.
      Human pathologists resemble the specific but less sensitive algorithm.

      Thus, the likely utility of this would be in the sensitive mode, flagging the slides for manual review. Sawa?

    6. Re:statistics by Anonymous Coward · · Score: 0

      Please calculate also the number for patients of the human doctors. I understand that people want perfect, but can't we do a huge improvement before going to perfect?

    7. Re:statistics by Anonymous Coward · · Score: 0

      What is the accuracy rate for humans?

  13. Google Physical Cancer Screen... by Anonymous Coward · · Score: 0

    The end result of this line of research will clearly be really really accurate inappropriate touching by robots...

  14. Fart-Officals-in-smelly-gents by Anonymous Coward · · Score: 0

    G-tards

    wont stand up in the real word, google is not capable, 'hint' hint if you get MI drift a 'ton' of mistakes in their thinking.

    They have corporate hand cuffs and poor collaboration skills, if you share MI point, they cant even write papers that don't loose the plot rather quickly and you can clearly see when the authors are changing thru paragraphs when reading multi author papers .

    Are not 'good' enough OR 'fellows' of the correct academic branches OR collaborate correctly OR know the 'difference' and

    All their so called "training" is NOT able to stand up in the real world and their crap Chinese atheist ethics don't have any intelligence OR know the 'difference' take 'knaw-pig' for example a 'temporal' turd

    Only good for games and toys . they are not JEDI material

    also spatial challenged and full of poop

    these guys will now get a pack of suppository when they go in their smelly gents, they have little helpers to help insert then they SQu33L

    1. Re:Fart-Officals-in-smelly-gents by Anonymous Coward · · Score: 0

      go and gargle some dioxin google