Was Turing Test Legitimately Beaten, Or Just Cleverly Tricked?

beaker_72 (1845996) writes "On Sunday we saw a story that the Turing Test had finally been passed. The same story was picked up by most of the mainstream media and reported all over the place over the weekend and yesterday. However, today we see an article in TechDirt telling us that in fact the original press release was just a load of hype. So who's right? Have researchers at a well established university managed to beat this test for the first time, or should we believe TechDirt who have pointed out some aspects of the story which, if true, are pretty damning?" Kevin Warwick gives the bot a thumbs up, but the TechDirt piece takes heavy issue with Warwick himself on this front.

Comments

but that's the problem with the turing test...

[Anonymous Coward]

It has nothing to do with actual artificial intelligence and everything to do with writing deceptive scripts. It's not just this incident, it's a problem with the goal of the Turing test itself. I always found the Turing test a kind of stupid exercise due to this.

Re:but that's the problem with the turing test...

[flaming error]

They got 30% of the people to think they were texting with a child with limited language skills. I don't think that's what Alan Turing had in mind.

Re:but that's the problem with the turing test...

[TheCarp]

I always thought of it as more a philosophical question or thought experiment. How do you know that anything has an internal consciousness when you can't actually observe it? I can't even observe your process, I just assume that you and I are similarly in so many other ways (well I assume, you could be a chatbot, whreas I know I am definitely not)....and I have it, so you must too, aferall, we can talk.

So.... if a machine can talk like we can, if it can communicate well enough that we suspect it also has an internal cosciousness, then isn't our evidence for it every bit as strong as the real evidence that anyone else does?

Re:but that's the problem with the turing test...

[Jane Q. Public]

It has nothing to do with actual artificial intelligence and everything to do with writing deceptive scripts. It's not just this incident, it's a problem with the goal of the Turing test itself. I always found the Turing test a kind of stupid exercise due to this.

Yes. TechDirt's points 3 and 6 are basically the same thing I wrote here the other day:

First, that the "natural language" requirement was gamed. It deliberately simulated someone for whom English is not their first language, in order to cover its inability to actually hold a good English conversation. Fail.

Second, that we have learned over time that the Turing test doesn't really mean much of anything. We are capable of creating a machine that holds its own in limited conversation, but in the process we have learned that it has little to do with "AI".

I think some of TechDirt's other points are also valid. In point 4, for example, they explain that this wasn't even the real Turing test.

Re:but that's the problem with the turing test...

[Anonymous Coward]

So according to you I could make a machine that simulates texting with a baby. Every now and then it would randomly pound out gibberish as if a baby was walking on the keyboard.

Re:but that's the problem with the turing test...

[BasilBrush]

The problem is that priming the judges with excuses about why the candidate may make incorrect, irrational, or poor language answers is not part of the test.

If the unprimed judges themselves came to the conclusion they were speaking to a 13 year old from the Ukraine, then that would not be a problem. But that's not what happened.

Re:but that's the problem with the turing test...

[TheCarp]

Please tell me more about like something a chatbot would say.

Re:but that's the problem with the turing test...

[BasilBrush]

What should the program have claimed to have been?

I don't care. What I care about is what the organisers of the "test" told the judges. I was under the impression they had told the judges it was a 13 years old boy from the Ukraine. Now I look again, it's not clear who told them that. Which brings another problem: we don't know what the judges were told. Given the effort to invite a celebrity to take part as one of the judges, you'd have thought there would be video of the contest. But no.

If you've been around tech for a while, you will have come across some of Kevin Warwick's bullshit claims to the press before. He's a charlatan. So therefore we need more than his say so that he conducted the test in a reasonable way.

We also need independent reproduction of the result. You know, the scientific method and all that.

Re:but that's the problem with the turing test...

[Spy Handler]

The test as specified by Alan Turing involves a human judge sitting in front of two terminals. One is a computer and the other is human-operated. The judge asks both terminals questions and tries to figure out which one is computer and which is human. It's quite specific.

It does not involve unsuspecting normal people in everyday situations who are duped into thinking they're interacting with a human... that would be quite easy. For instance if somebody asked the TigerDirect customer service chat window questions they have about a product and receive a good answer, they might not suspect it's a bot. Doesn't mean the TigerDirect bot passed the Turing test.

Turing also didn't say anything about crippling the test by making it a child who doesn't speak fluent English.

Stupidly tricked, not clever

[gurps_npc]

Turnign test is NOT supposed to be limited to 15 minutes, nor is it supposed to be conducted by someone that does not understand the main language claimed to be used by the computer.

Similarly, the computer must convince the judge it is a human with it's full mental capacity, not child, nor a mentally defective person, nor someone in a coma.

The test is whether a computer can, in an extended conversation, fool a competent human into thinking it is a competent human being speaking the same language,at least 50% of the time.

Program pretends to be foreign child, not adult

[dunkindave]

For those who haven't read the article (I read one yesterday and assume the details are the the same): The program claimed to be a Ukrainian boy of 13 years old, a non-native English speaker, writing in English to English speakers. This allowed the program to avoid the problem of people using language to make judgements about whether the responses were from a person or a program. Also, since the program was claiming to be a boy instead of an adult, it also greatly reduced what could be expected of the responses, again greatly simplifying the programs parameters and reducing what the testers could use to test. So basically, the Turing Test is supposed to be a test if a person can tell if the program acts like a person, but here the test was rewritten to see if the program acted like a child from a different culture and who was supposed not to be speaking in his native language. Many are apparently crying foul.

I personally agree.

The Turing test

[KramberryKoncerto]

... was not actually performed in the research. End of story.

Re:I see.

[RDW]

But seriously, yes, it was 'legitimately beaten', just like it's been 'legitimately beaten' in times past, going back to ELIZA in the 60s.

How does that make you feel?

I don't care

[mbone]

The first time I saw ELIZA in action, I realized that the Turing test is basically meaningless, as it fails on two fronts. We are not good judges for it, as we are hard-wired to assume intelligence behind communications, and Turing's assumption that the ability to carry on a reasonable conversation was a proof of intelligence was wrong.

This is not to fault Turing's work, as you have to start somewhere, but, really, after all of these years we should have a better test for intelligence.

Re:I don't care

[AthanasiusKircher]

The first time I saw ELIZA in action, I realized that the Turing test is basically meaningless, as it fails on two fronts. We are not good judges for it, as we are hard-wired to assume intelligence behind communications, and Turing's assumption that the ability to carry on a reasonable conversation was a proof of intelligence was wrong.

But that wasn't Turing's assumption, nor was it the standard for the Turing test.

Turing assumed that a computer would be tested against a real person who was just having a normal intelligent conversation. Not a mentally retarded person, or a person who only spoke a different language, or a person trying to "trick" the interrogator into thinking he/she is a bot.

Note that Turing referred to an "interrogator" -- this was an intensive test, where the "interrogator" is familiar with the test and how it works, and is deliberately trying to ask questions to determine which is the machine and which is the person.

ELIZA only works if you respond to its stupid questions. If you actually try to get it to actually TALK about ANYTHING, you will quickly realize there's nothing there -- or perhaps that you're talking to a mentally retarded unresponsive human.

The "assumption" is NOT "the ability to carry on a reasonable conversation," but rather the ability to carry on a reasonable conversation with someone specifically trying to probe the "intelligence" while simultaneously comparing responses with a real human.

I've tried a number of chatbots over the years when these stories come out, and within 30 seconds I generally manage to get the thing to either say something ridiculous that no intelligent human would utter in response to anything I said (breaking conversational or social conventions), or the responses become so repetitive or unresponsive (e.g., just saying random things) that it's clear the "AI" is not engaging with anything I'm saying.

You're absolutely right that people can and have had meaningful "conversations" with chatbots for decades. That's NOT the standard. The standard is whether I can come up with deliberate conversational tests determined to figure out whether I'm talking to a human or computer, and then have the computer be indistinguishable from an actual intelligent human.

I've never seen any chatbot that could last 30 seconds with my questions and still seem like (even a fairly stupid) human to me -- assuming the comparison human in the test is willingly participating and just trying to answer questions normally (as Turing assumed). If somebody walked up to me in a social situation and started talking like any of the chatbots do, I'd end up walking away in frustration within a minute or two, having concluded the person is either unwilling to actually have a conversation or is mentally ill. That's obviously not what Turing meant in his "test."

Re:I see.

[NatasRevol]

I can't answer that right now.