Upgrading the Turing Test: Lovelace 2.0

mrspoonsi tips news of further research into updating the Turing test. As computer scientists have expanded their knowledge about the true domain of artificial intelligence, it has become clear that the Turing test is somewhat lacking. A replacement, the Lovelace test, was proposed in 2001 to strike a clearer line between true AI and an abundance of if-statements. Now, professor Mark Reidl of Georgia Tech has updated the test further (PDF). He said, "For the test, the artificial agent passes if it develops a creative artifact from a subset of artistic genres deemed to require human-level intelligence and the artifact meets certain creative constraints given by a human evaluator. Creativity is not unique to human intelligence, but it is one of the hallmarks of human intelligence."

Comments

Turing test is fine

[itzly]

There's nothing wrong with the Turing test, but it needs to have some thought put into the set up and execution, plus competent judges.

Re:Turing test is fine

[Megol]

Maybe you should inform yourself what a Turing test actually is? Eliza didn't pass it nor would a normal chatbot - unless there is true intelligence behind it.

The problem with the Turing test is that is hard and most ordinary people would probably fail it.

Re:Turing test is fine

[itzly]

So you wouldn't consider an alien from another planet intelligent unless he shared our bodily functions ?

Re:Turing test is fine

[ultranova]

Why should an AI have to think about all the things us meatbags have to think about that aren't relevant to it?

Because if it can't model a meatbag, why would it be able to model an electron (so can't do physics), an industrial robot (so can't program them), a car (can't control vehicles), abstract entities (can't do logic or math) or anything else for that matter?

Imagination is not optional for intelligence. Intelligence is the ability to build mental models and manipulate them.

AIs don't have parents (well, not in the traditional sense anyway) and so won't have a human-like childhood experience to reflect upon,

Any entity that comes to a new setting will require a period of acclimatization. Whether you call this "childhood" or not is irrelevant.

nor should they have to worry about whether that lump is cancerous, or whether they have to go into work tomorrow, or if that dish had too much salt in it.

Computers break down and require resources - more than human bodies, in fact - thus work enters the picture.

Re:Turing test is fine

[narcc]

Did you miss the last 64 years of research and philosophy? The last hold-outs, save the most delusional, we're knocked out by Searle in 1980.

It's only controversial for those who haven't read Turing's paper, or have completely failed to understand it.

Eliza, for example, highlights the massive failure in Turing's reasoning -- The question "can machines think" is not equivalent to the question "Are there imaginable digital computers which would do well in the imitation game?"

Weisenbaum found the response to his program from non-technical staff disturbing.

Secretaries and nontechnical administrative staff thought the machine was a "real" therapist, and spent hours revealing their personal problems to the program. When Weizenbaum informed his secretary that he, of course, had access to the logs of all the conversations, she reacted with outrage at this invasion of her privacy. Weizenbaum was shocked by this and similar incidents to find that such a simple program could so easily deceive a naive user into revealing personal information.

( From Eliza to A.L.I.C.E. )

Further, the so-called "Turing test" hasn't held still. Not even in his 1950 paper! (Turing proposed multiple variations on the test, if you'll recall.) Since then, a number of different versions of the "Turing test" have appeared, none of which are (like Turing's variations) are equivalent to one another!

If you need a *really* simple argument: The results of any variation of the "Turing test" are completely subjective. Consider a program that fools 100% of one set of interrogators may completely fail to fool even 10% of another set.

moving target

[Anonymous Coward]

This is just making the "Turing test" into a moving target. The Turing test makes sense, and if you have a long enough test you can eventually rule out the "abundance of if statements."

How many meat-people would pass the Lovelace test?

[Anonymous Coward]

There's a Forest Service joke that the problem with designing trash cans is that the smartest bear is smarter than the dumbest tourist.

Inference is Hard

[infogulch]

'In the following sentence: "Ann gave Sue a scarf. She was very happy to receive it." Does "she" refer to Ann? (yes/no)'
A series of similar and increasingly difficult inference questions like this one can usually knock over an AI pretty easily, while not being too difficult for humans.

Human Intelligence

[Elledan]

All I can think of while reading up on the Turing and related tests is how many humans would fail such a test.

With the many assumptions made about what constitutes 'true' intelligence, how sure are we of the assumption that a human being of at least average intelligence would pass it? What's the research telling us there so far?

Or are human and artificial intelligence somehow considered to be mutually exclusive?

We will never have "real" AI

[msobkow]

We will never have "real" AI because every time we approach it, someone moves the bar as to what is required. It's been happening since the mid-late '80s. We *have* what would have qualified as AI according to the rules of '86-'87.

The assumptions, they make a whoosh out of you

[Donwulff]

So yet another article on Turing test which completely misses the point... First of all computer scientists never considered Turing test valid test of "artificial intelligence". In fact, there's practically no conceivable reason for a computer scientist to test their artificial intelligence by any other way than making it face problems of its own domain.
Perhaps there will come a day where we really have to ask "is this entertainment droid genuinely intelligent, or is it only pretending", possibly for determining whether it should have rights, but this kind of problem still doesn't lie in the foreseeable future.
On the Other hand, as Turing himself put it in the paper where he introduced his thought-experiment, from Wikipedias phrasing: "I propose to consider the question, 'Can machines think?'" Because "thinking" is difficult to define, Turing chooses to "replace the question by another, which is closely related to it and is expressed in relatively unambiguous words." Turing's new question is: "Are there imaginable digital computers which would do well in the imitation game?"
In other words, the Turing test does not seek to answer the question of whether machines can think, because Turing considered the question meaningless, and noted that if a machines thinking was outwardly indistinguishable from human thinking, then the whole question would become irrelevant.
There is a further erroneous assumption at least in the summary - as of present times, even the most advanced computers and software are basically simply an abundance of if-statements, or for the low-level programmers among us, cmp and jmp mnemonics. If, on the other hand, we expand our definition of a "machine" to encompass every conceivable kind, for the materialistic pragmatic it becomes easy to answer whether machines can ever think - yes of course, the brain is a machine that can think.