Crowdsourcing Game Helps Diagnose Infectious Diseases

Lucas123 writes "Researchers at UCLA have created an online crowdsourcing game designed to let players help doctors in key areas of the world speed the lengthy process of distinguishing malaria-infected red blood cells from healthy ones. So far, those playing the game have collectively been able to accurately diagnose malaria-infected blood cells within 1.25% of the accuracy of a pathologist performing the same task (PDF). The researchers hope that users of the game can help eliminate the high cost and sometimes poor accuracy of diagnosis in areas like sub-Saharan Africa, where malaria accounts for some 20% of all childhood deaths."

Comments

Didn't know a DDX could have the word balls in it.

[RyuuzakiTetsuya]

on one hand, gamers have been known to be gracious and kind enough to donate and keep CHild's play running.

On the other hand, gamers are dicks.

I hope this ends well.

applying machine learning?

[tomlue]

Whenever I see these games I wonder if there is some effort to solve the task with ml as well. It seems like if you are getting a large number of players then you should have data on how those players are classifying the red blood cells. Given that data we can at least attempt a couple different techniques at a classifier.

The benefit could be two fold. On one hand a decent classifier could 'help eliminate the high cost and sometimes poor accuracy of diagnosis.' On the other hand if a diagnostician is looking at a blood sample having a classifier give some probability of classification (ie. saying 'this sample is 90% probable to be malaria infected') could help the doctor in their diagnosis.

I'd love to work on that problem. A very quick search of google scholar and citeseer doesn't pop up anything on the subject though, and I don't see any api on the linked site.

Re:applying machine learning?

[Anonymous Coward]

There would be severe liability issues with this. It's the same reason you'll never see cars that drive themselves become mainstream.

Re:applying machine learning?

[Iniamyen]

What's interesting though is that since you are effectively taking an average (not a mathematical mean necessarily, but broadly speaking an average), any individual answer (even the smart people's answers) could still be worse than the crowdsourced answer. I heard a story on NPR about a crowdsourcing project where this exact thing happened. I am sorry, I can't remember the details. In this case though, you can't necessarily build an algorithm, because it will still give an inferior answer to the crowdsourced answer. Or at least, you have to totally change how you frame the problem and try again all over.

Re:applying machine learning?

[fuzzyfuzzyfungus]

I suspect that you could dodge some of the liability issues in this case.

Yeah, if Joe American goes in to MGH for a CT, pays north of 5k(after insurance), and learns that some unfeeling robot, rather than a tired radiologist, misdiagnosed him, it'll be malpractice lawyer time.

However, in areas where the current standard of care often doesn't include pathologist inspection of cells; because there aren't any qualified pathologists, or they are too expensive for the majority of patients, I suspect you'll find a much greater willingness to embrace the idea that you can perform the diagnosis with a glorified webcam(wasn't there some story on slashdot a little while back about some research group hacking microscope optics onto cellphone cameras?) and a nickel worth of CPU time...

It sounds crass when you say it in so many words; but what you can get away with in medicine is very much a product of what the alternative would be. If the current standard is sufficiently dire, even mediocrity counts as lifesaving. If it just so happens that machines are actually really good at this classification problem, all the better.

Re:applying machine learning?

[Defenestrar]

A step up from that would be to embed a well trained machine in a simple lab on a chip. Print a few million and package them with a finger lance. If the LED turns red, then you foot your kid to a WHO clinic with proof in your pocket that you need to be at the front of the line. Not to mention the usefulness something like this would have at the same WHO clinic.

Re:applying machine learning?

[chooks]

From the referenced PDF:

we also developed an automated machine learning algorithm to detect the presence of malaria parasites,

Re:applying machine learning?

[Anonymous Coward]

There is a hybrid CV-ML algorithm described in the paper.

Re:applying machine learning?

[houghi]

or they are too expensive for the majority of patients

Like in America?

Re:applying machine learning?

[Anonymous Coward]

I'm running a small research lab in Uganda applying machine learning to problems including this one. A few people have indeed tried such automated classification before with varying degrees of success. It's not all that difficult as vision problems go, though one of the main obstacles so far has been the difficulty of obtaining good quality training images. The appearance of malaria parasites can vary quite widely depending on species and stage in the life cycle, so many examples are needed for a classifier to generalise well.

Given the training data, though, it definitely seems feasible to get usable accuracy with a nearly completely automated system. There are a few tricks which can be used to squeeze out better performance too - for example if diagnosis is carried out on a location-aware device, we can look up that position on a malaria "risk map" based on climatic remote sensing data to give us prior information that improves the reliability of the diagnosis.

If you'd really like to play with this problem, in a few weeks time we're planning to upload the annotated image data we have so far (~2000 microscope fields of view in which ~50,000 parasites have been identified) and some baseline detection code here.

Re:Didn't know a DDX could have the word balls in

[Anonymous Coward]

Usually it's not an issue with crowdsourcing. You rely on maybe 100 or more people classifying the same thing correctly, the key point is collective classification. See Citizen Sky for more examples.

Slashdotted-ish.

[game kid]

Signed up, website still works but game is stuck at "Retrieving Authentication" on Firefox 12 and IE8 (my machine with IE9 is getting an overhaul), so guess I'll just have to wait a while to find those buggy blood cells.

Re:Slashdotted-ish.

[Anonymous Coward]

Likewise.
 
Way to go, /. You just let tens of thousands of people with malaria die, you insensitive clods.

Re:Slashdotted-ish.

[Anonymous Coward]

Signed up, website still works but game is stuck at "Retrieving Authentication" on Firefox 12 and IE8 (my machine with IE9 is getting an overhaul), so guess I'll just have to wait a while to find those buggy blood cells.

Have you tried updating? It works for me in Firefox 19.

Re:Didn't know a DDX could have the word balls in

[Anonymous Coward]

Once the troll population reaches sufficent levels:

"I have a sore throat and a slight fever."
*evaluating*
*You have AIDS, leprosy, and lupus.*

Re:Didn't know a DDX could have the word balls in

[Anonymous Coward]

This isn't about gamers at all, this is about a group of untrained people being able to do the same job as a trained individual.

Machine learning

[kbg]

Would it not be simpler to have the pathologist train a machine learning algorithm and then use that instead of the game?

Re:Machine learning

[degeneratemonkey]

The point of this approach is that machines can make pretty good decisions ~90% of the time, but when a more refined judgment call needs to be made, a human can be provided with an abstract rendition of the data and any relevant context, and can inadvertently perform a useful classification for the solver.

Its purpose is to solve precisely the kinds of problems that arise when you simply train an ML algorithm with known inputs and outputs, and then encounter an input that appears at least partially ambiguous to said trained machine.

Use as CAPTCHA-like system?

[Guppy]

I wonder if these sort of games could be implemented as a sort of Captcha challenge for niche situations. A simple yes/no diagnosis has insufficient complexity to serve as Captcha of course; you'd have to increase the difficulty by presenting a panel of several samples, multiple diseases, or require more detailed responses ("click on the abnormal Red Blood Cells").

This wouldn't be suitable for use as general purpose Captchas of course; any user seeing it for the first time won't have a clue what they're looking at. Rather, it could be used in settings where you have a body of dedicated users who repeatedly use a service over a long period of time, yet require anonymity without permanent user profiles. Forcing the users to go through initial training could be seen as a bonus, creating an additional obstacle for Mechanical-Turk workers. As an example of such a situation that pairs a persistent user pool with anonymity, consider something like 4chan.

On the downside, because it's 4chan, half your diagnosis results will be "You have AIDS (Pool's Closed)".

Re:Didn't know a DDX could have the word balls in

[Anonymous Coward]

Oh, so it's a new form of outsourcing then.

Eve

[XCDBFPL]

I hope the player base of Eve Online does not discover this game.

Wait

[Anonymous Coward]

You're telling me humans are still allowed to licensed as doctors? What year is this? Oh, I'm sorry...