Slashdot Mirror
EFF Launches New AI Progress Measurement Project (eff.org)
Reader Peter Eckersley writes: There's a lot of real progress happening in the field of machine learning and artificial intelligence, and also a lot of hype. These technologies already have serious policy implications, and may have more in the future. But what's the ratio of hype to real progress? At EFF, we decided to find out.
Today we are launching a pilot project to measure the progress of AI research. It breaks the field into a taxonomy of subproblems like game playing, reading comprehension, computer vision, and asking neural networks to write computer programs, and tracks progress on metrics across these fields. We're hoping to get feedback and contributions from the machine learning community, with the aim of using this data to improve the conversations around the social implications, transparency, safety, and security of AI.
Today we are launching a pilot project to measure the progress of AI research. It breaks the field into a taxonomy of subproblems like game playing, reading comprehension, computer vision, and asking neural networks to write computer programs, and tracks progress on metrics across these fields. We're hoping to get feedback and contributions from the machine learning community, with the aim of using this data to improve the conversations around the social implications, transparency, safety, and security of AI.
I am sure that Professor Jefferson [a critic of AI] does not wish to adopt the extreme and solipsist point of view. Probably he would be quite willing to accept the imitation game as a test. The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has "learnt it parrot fashion." Let us listen in to a part of such a viva voce:
Interrogator: In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?
Witness: It wouldn't scan.
Interrogator: How about "a winter's day," That would scan all right.
Witness: Yes, but nobody wants to be compared to a winter's day.
Interrogator: Would you say Mr. Pickwick reminded you of Christmas?
Witness: In a way.
Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.
Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.
And so on, What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance."
That's an example of what Alan Turing expected of the "Turing Test." And the issue isn't knowledge of sonnets or English lit here or whatever -- it's being able to parse and understand and respond reasonably to demonstrate such understanding. That was Turing's definition of AI. The kind of AI that he predicted by the year 2000 would be able to fool a skilled "interrogator" specifically trying to trip up the AI and identify the computer when an AI would be put up against a human in the "imitation game" test.
When a chatbot can do this, call me. Otherwise, all of this talk about "artificial intelligence," "deep learning," "neural networks," etc. is just fancy words for slightly more powerful statistical tools and adaptive algorithms. Maybe chaining billions of such things together could eventually lead to something that could carry on a conversation like Turing's example, but I've never encountered a chatbot with anything close to that. Most chatbots can't understand a pronoun reference to the previous sentence, let alone make abstract connections as shown in the above quotation.