Slashdot Mirror
Software Takes On School Science Tests In Search For Common Sense
holy_calamity writes: Making software take school tests designed for human kids can help the quest for machines with common sense, says researchers at the Allen Institute for Artificial Intelligence. They've made software called Aristo that scores 75 percent on the multiple choice questions that make up most of New York State's 4th grade science exam. The researchers are urging other researchers to pit their best software against school tests, too, to provide a way to benchmark progress and spur competition.
75%? Everybody knows that you get 25% when you just guess randomly... So being able to add another 50% isn't all that amazing.
Understand how they do this though... They have taken the existing study guides and have constructed an algorithm that does basic word association. Multiple choice tests are written to have one right answer, one plausible answer and two answers which are distracters, designed to trick you. So the trick to multiple choice when you don't know the answer is to identify the distracters and pick from the remaining answers. So armed with their word association, they eliminate the distractors by finding the answers that have the more closely associated words with the question seen in the study guides and throwing out the rest. this will get you easily to a pretty good solution, and where it isn't conclusive it will easily get you to a 50/50 choice.
Problem is, this isn't how people do this. They've not invented a "common sense" way to do this that works for humans, but a way that is more suited to machine "learning". It's about pattern matching and possible associations, not knowledge of 4th grade science. They've not taught the computer 4th grade science, far from it, they've only figured out a way to winnow down the more likely answers in a multiple choice test and the computer then just guesses based on probabilities. While this is interesting, it has nothing to do with human "common sense" and is basically pointless.
Now, if they only would teach KIDS how to take multiple choice tests using similar techniques, THAT would be something worthwhile....
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
It would be easy to get ~75% correct just by looking for keywords and simple pattern matching, without actually understanding the question,
Pattern matching, without any understanding, is state-of-the-art AI.
Required reading for internet skeptics