Slashdot Mirror

← Back to Stories (view on slashdot.org)

IBM Pitched Its Watson Supercomputer as a Revolution in Cancer Care. It's Nowhere Close (statnews.com)

Posted by msmash on from the reality-check dept.

IBM began selling Watson to recommend the best cancer treatments to doctors around the world three years ago. But is it really doing its job? Not so much. An investigation by Stat found that the supercomputer isn't living up to the lofty expectations IBM created for it. It is still struggling with the basic step of learning about different forms of cancer. Only a few dozen hospitals have adopted the system, which is a long way from IBM's goal of establishing dominance in a multibillion-dollar market. And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care. From the report: The interviews suggest that IBM, in its rush to bolster flagging revenue, unleashed a product without fully assessing the challenges of deploying it in hospitals globally. While it has emphatically marketed Watson for cancer care, IBM hasn't published any scientific papers demonstrating how the technology affects physicians and patients. As a result, its flaws are getting exposed on the front lines of care by doctors and researchers who say that the system, while promising in some respects, remains undeveloped. [...] Perhaps the most stunning overreach is in the company's claim that Watson for Oncology, through artificial intelligence, can sift through reams of data to generate new insights and identify, as an IBM sales rep put it, "even new approaches" to cancer care. STAT found that the system doesn't create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.

4 of 108 comments (clear)

  1. I've seen better from high school students by TimothyHollins · · Score: 4, Interesting

    And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care.

    Are you seriously telling me that they sold a multi-million dollar machine and didn't even include a goddamn machine learning step to adapt to local variations? Aren't the IBM guys supposed to be experts? Or at least guys that know how to pick up a fucking phone and dial an expert?

    This is the kind of rookie mistake I see in my undergrads...
    I sure hope I'm reading this wrong, because it sounds like people might die from maltreatment over this.

    At its heart, Watson for Oncology uses the cloud-based supercomputer to digest massive amounts of data — from doctor’s notes to medical studies to clinical guidelines. But its treatment recommendations are not based on its own insights from these data. Instead, they are based exclusively on training by human overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.

    Ahh I guess I was wrong. There is no machine learning at all yet.

    In the case of Watson for Oncology, those human operators are a couple dozen physicians at a single, though highly respected, U.S. hospital: Memorial Sloan Kettering Cancer Center in New York. Doctors there are empowered to input their own recommendations into Watson, even when the evidence supporting those recommendations is thin.

    But hey, looks like the dying part could be correct. I only hope those doctors know what the variations across the world requires, because they will be giving recommendations both for japanese highschool girls and African village elders without even knowing it, and I don't think those groups have the same contextual issues.

    1. Re:I've seen better from high school students by CraigDavidIsTheShit · · Score: 3, Interesting

      Watson is literally a big decision tree from the bits i've seen/worked on

    2. Re:I've seen better from high school students by Anonymous Coward · · Score: 4, Interesting

      I took an IBM Watson course at my University (Partnered with IBM), and their current head researcher was our Prof for the 6 week course. From what I remember it was essentially ~34 modules attempting various methods of Brute Forcing a solution, with several modules grading their outputs. But the actual modules from what I remember were infinitely basic and it was just their multitude that actually lead to improvements in results. I remember walking away from that course having learned little about how Machine Learning is done at the upper echelons of computing, but having a 1 year license to actually use the BlueMix platform. We even had to create a project using the BlueMix API, but it only allowed us to use a pre-imported corpus submitted by the TAs, so we didn't even get to take a whack at classifying training sets. Anyway, I was very unimpressed, and I was the only First-year Bachelor student in that masters course. Even I felt it was all a bit basic.

  2. I've worked with Watson by bangular · · Score: 5, Interesting

    First of all, Jeopardy Watson has almost nothing to do with the Watson product IBM is selling. That system was largely NLP based and the team disbanded afterwards. From what I can tell, the only thing still alive from that project is Apache UIMA. Watson isn't even a single product. It's a collection of about a dozen disjoint products with the word "Watson" in front of their name.

    The current iteration of "Watson" is not interesting at all. Their machine learning portion is just SPSS. Their next-gen machine learning is just Apache Spark. Their UI to setup hardware and submit spark jobs is very unreliable. When you get an error, it's generally a "An error has occurred" or something just as useless.

    Jeopardy Watson was interesting, but the big players are doing much more interesting things these days. Google and Microsoft have very good public machine learning and AI platforms. Amazon's is OK, nothing special. If you want to work at a lower level, Stanford maintains a set of libraries with implementations of their cutting edge algorithms. Especially their NLP group. Theirs are actually user-friendly compared to many other research level projects.