EFF Launches New AI Progress Measurement Project (eff.org)

Reader Peter Eckersley writes: There's a lot of real progress happening in the field of machine learning and artificial intelligence, and also a lot of hype. These technologies already have serious policy implications, and may have more in the future. But what's the ratio of hype to real progress? At EFF, we decided to find out.

Today we are launching a pilot project to measure the progress of AI research. It breaks the field into a taxonomy of subproblems like game playing, reading comprehension, computer vision, and asking neural networks to write computer programs, and tracks progress on metrics across these fields. We're hoping to get feedback and contributions from the machine learning community, with the aim of using this data to improve the conversations around the social implications, transparency, safety, and security of AI.

Comments

One dimensional analysis

[Anonymous Coward]

Based on the amount of time it takes me to solve ReCaptchas lately, I think the # of ReCaptchas/second that can be solved by a human is probably a good cross-domain proxy. When it takes me 1 hour to solve a ReCaptcha, I'm pretty sure SkyNet is already real at that point and just biding its time until Judgement Day.

I can summarize

[110010001000]

You can just mark 0% for progress now. Playing Go or Chess or any game is NOT AI. Neither is Siri or facial or voice recognition or autonomous driving. They are just programs. Computers are good at Go and Chess because they have strict rules to follow. Computers love rules. Computers are less good at autonomous driving because the rules aren't as clearly defined.

Re:I can summarize

[Peter Eckersley]

That was true in the past, but it just isn't true of the recent progress in machine learning. Take a look at the data we've collected on problems like visual question answering, reading comprehension or learning to play Atari just by watching the screen, and you'll see that progress is happening in domains that either lack rigid rules, or where the rigid rules are non-trivial to discover.

Re:I can summarize

[ShanghaiBill]

" It is computers solving complex problems"
If THAT is what you call "AI", then the term is meaningless.

It is not AI because it is solving "complex" problems, but because it is using machine learning to figure out for itself how to solve the problem. Machine learning is a (very important) part of AI.

Look, I understand that you have seen some Will Smith movies on Netflix about robots and AI and stuff, and you think that is "AI". But this is a technical forum for nerds, not a movie discussion board. When actual researchers are discussing "AI", they are almost never talking about human level "strong AI", which is still science fiction. They are talking about "weak AI", which means exactly the type of research referred to in TFA.

Every time there is any article about AI, someone (often you) has to stick their nose in and start saying "This isn't AI", "This isn't AI". That adds nothing to the discussion. So either learn the terminology, and participate like an adult, or go back to Netflix.

Also, AI researchers did not "steal" the term "Artificial Intelligence" from Hollywood. It was the other way around.

Can it interpret a sonnet?

[AthanasiusKircher]

What Alan Turing wrote in 1950 about the "imitation game":

I am sure that Professor Jefferson [a critic of AI] does not wish to adopt the extreme and solipsist point of view. Probably he would be quite willing to accept the imitation game as a test. The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has "learnt it parrot fashion." Let us listen in to a part of such a viva voce:

Interrogator: In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?

Witness: It wouldn't scan.

Interrogator: How about "a winter's day," That would scan all right.

Witness: Yes, but nobody wants to be compared to a winter's day.

Interrogator: Would you say Mr. Pickwick reminded you of Christmas?

Witness: In a way.

Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.

Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.

And so on, What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance."

That's an example of what Alan Turing expected of the "Turing Test." And the issue isn't knowledge of sonnets or English lit here or whatever -- it's being able to parse and understand and respond reasonably to demonstrate such understanding. That was Turing's definition of AI. The kind of AI that he predicted by the year 2000 would be able to fool a skilled "interrogator" specifically trying to trip up the AI and identify the computer when an AI would be put up against a human in the "imitation game" test.

When a chatbot can do this, call me. Otherwise, all of this talk about "artificial intelligence," "deep learning," "neural networks," etc. is just fancy words for slightly more powerful statistical tools and adaptive algorithms. Maybe chaining billions of such things together could eventually lead to something that could carry on a conversation like Turing's example, but I've never encountered a chatbot with anything close to that. Most chatbots can't understand a pronoun reference to the previous sentence, let alone make abstract connections as shown in the above quotation.