Has Cleverbot Passed the Turing Test?

kruhft writes "It seems that Cleverbot, the chatbot so ready to admit that it was a unicorn during a discussion with itself, has passed the Turing test. This past Sunday, the 1334 votes from a Turing test held at the Techniche festival in Guwahati, India were released. They revealed that Cleverbot was voted to be human 59.3% of the time. Real humans did only slightly better and were assumed to be humans 63.3% of the time." As the Wikipedia link above points out, though, there's no single, simple "Turing Test," per se — many systems have successfully convinced humans over the years. Perhaps Cleverbot would consent to taking part in a Slashdot interview, to be extra-convincing.

Comments

Definitely not

[ModernGeek]

Clever bot is a piece of garbage that hasn't even surpassed Perl scripts on IRC in the 1980s. It isn't even worth mentioning, it's nothing more than a piece of crap with a "Web 2.0" edge to it that doesn't even have long term memory while having a "conversation". Far from AI, far behind what's already been out there.

Re:Definitely not

[Anonymous Coward]

Boy, that sure puts those "63.3%" of humans in their place. I'd feel bad to be them, but I'm not certain if they have emotions or not.

Re:Definitely not

[Anonymous Coward]

That sounds like something a jealous AI would say. I suspect you're not human!

Re:Definitely not

[Jarik C-Bol]

is a Turing test valid if the human is an idiot?

Re:Definitely not

[ipwndk]

Sure. But the Turing test is a piece of garbage too. I have a deep respect for Allan Turing, and all that he has done for science. But the Turing test was death to AI the moment he proposed it. It MUST be forgotten and burried, and maybe incidents like these can help us achieve that!

Re:Definitely not

[vlm]

Sure. But the Turing test is a piece of garbage too. I have a deep respect for Allan Turing, and all that he has done for science. But the Turing test was death to AI the moment he proposed it. It MUST be forgotten and burried, and maybe incidents like these can help us achieve that!

Eh, its more of a thought experiment. Its like making fun of Heisenberg because you want experimental proof of quantum dot technology LEDs, not dead/undead cats in a box with a source and a geiger counter. Einstein had some legendarily weird thought experiments too.

Its value is in making you think of contrived, yet vaguely familiar situations in a really strange problem space. Not much value in an experiment design engineering planning review meeting.

As part of a previous job I occasionally got involved as an engineering contact in telco service monitoring situations. You know, like every time we call long distance I hear echo, that type of thing. Trust me, most human to human conversations are pretty F-ing unintelligent, so I think an Artificial Intelligence would not be challenged in outperforming them. PHBs talk meaningless buzzwords to each other, not just to peons, for example.

Re:Definitely not

[SanityInAnarchy]

I don't read those as "Chatterbot passes turing test." I read them as "Human fails turing test."

Someone took an ALICE-like bot to IRC, loaded with slightly-flirtatious dialog and with a slightly-flirtatious and female name. It got hit on, and it fooled entirely too many guys for entirely too long.

Re:Definitely not

[Savantissimo]

"is a Turing test valid if the human is an idiot?"

What about the humans in the control group who failed the test? Maybe some of them were flunked by idiots making the judgement, but likely many of them really were indistinguishable from bots. Given that this test was done at a tech convention in India, I personally suspect that most of the 36.7% of humans who flunked the test work in call centers. I've certainly had a few on the line that were indistinguishable from a chatbot running on a Speak & Spell, and were certainly quite as useless as a very useless thing indeed.

Re:Definitely not

[Anonymous Coward]

You can definitively make out a chatbot with this simple conversational thread:

- I'm going to tell you a color. I want you to remember it, okay?
- Green.
- Now, what's the color that I just told you?

Since cleverbot just parrots back responses from its DB, the responses make no sense:

Me: I'm going to tell you a color. I want you to remember it, okay?
CB: What do you want me to remember?
Me: Green.
CB: Why do you like the colour green?
Me: Now, what's the color that I just told you?
CB: It is purple.

Even better, ask it to then make an association based on something you told it earlier.

I have no idea how this thing managed to get more than 50% of people. Trying to "communicate" with it for more than a minute just ends in frustration and annoyance.

Hold on

[Dyinobal]

Hold on I've not been able to go talk to Cleverbot for more than two exchanges before it goes off topic with some completely rando...ah damn no wonder people thought it was a human.

Re:Next step

[Rik Sweeney]

Nice try, Cleverbot.

Re:Next step

[Dthief]

how about they pass (how about over 80 points) an IQ test first

Re:The follow-up question

[Dachannien]

How many slashdot users can pass the same Turing Test?

How do you feel about how many slashdot users can pass the same Turing Test?

Turing test was passed long ago.

[EnglishTim]

You don't get it do you? I've been posting here since the nineties and none of you useless meat-sacks ever realised I was a machine!

Re:Turing test was passed long ago.

[Abstrackt]

I mean, being a robot's great but we don't have emotions and sometimes that makes me very sad.

WTF?

[TheRaven64]

So, I went and had a little chat with 'clever' bot to see how clever it was. I thought I'd start off with something a typical five-year-old could answer:

User: Hello.
Cleverbot: Hello!
User: Can you help me with a maths problem?
Cleverbot: Sure, what is it?
User: If I have 5 ostriches, and two run away, how many are left?
Cleverbot: 9.
User: Seriously?
Cleverbot: What is the formula for the area of circle?
User: What does that have to do with anything?
Cleverbot: When I talk to people I don't know, I tend to ask for their name. It's only polite.
User: I think you managed to fail the Turing test in two questions if I'm generous, may be one.
Cleverbot: My name is Cleverbot?
User: I could write a clevererbot in about 20 lines of prolog. What kind of idiots thought you were human?
Cleverbot: Only one. I don't think I could fit 1,000 people in my living room.

I think the fact that 59% of people thought it was human says a lot more about the intelligence of the average human than it says about the bot...

Re:WTF?

[Exitar]

I've tried to repeat the same question of yours:

If I have 5 ostriches, and two run away, how many are left?
One.
If I have 5 ostriches, and two run away, how many are left?
Ohhhhhh! I'm going Digital, Completly Digital, Now I'm invincible, Let's all get Digital!
If I have 5 ostriches, and two run away, how many are left?
Four.
If I have 5 ostriches, and two run away, how many are left?
Seven.
If I have 5 ostriches, and two run away, how many are left?
2, unless you destroy 1.
If I have 5 ostriches, and two run away, how many are left?
What's a happle?
If I have 5 ostriches, and two run away, how many are left?
Dos? Zwei? Ni? 2?
If I have 5 ostriches, and two run away, how many are left?
Eight.
If I have 5 ostriches, and two run away, how many are left?
Whole apples?

Re:WTF?

[digitig]

I think the fact that 59% of people thought it was human says a lot more about the intelligence of the average human than it says about the bot...

From the RA: "Cleverbot is available for conversation online, but don’t be fooled. Although Cleverbot managed to score well on the Turing test, the model that did that is different from the one you’ll find online."

Re:WTF?

[rgmoore]

The Turing test, as originally proposed, wasn't just a test of casual conversation. It was supposed to involve skeptical questioners doing their very best to separate human from AI, with no limit on conversational topic. The hypothetical questions in Turing's original paper included ones about math, chess problems, and poetry. If you held a Turing test under the original rules, with a reward for testers who successfully told human from AI (and for humans who successfully proved their humanity) you would find that no AI would get anywhere close to success. Because everybody knows that, public tests like this one have repeatedly watered down the original concept to make things more interesting. But that just proves how far AI has to come, not that it's getting close to succeeding.

Re:WTF?

[CAIMLAS]

Here, try this (exceedingly generic) 'emotional' conversation out that I just had with this thing:

Me: How you doing?
Bot: I'm doing fine.
Me: It's been a while.
Bot: I'm glad.
Me: What? you're glad?
Bot: Because you're happy.
Me: I'm not happy! you said you were glad we hadn't seen each other in a while!
Bot: No, I said I was a girl. You were the one who said I was a boy.

There were better bots than this 20 years ago. This doesn't even approach '14 year old valley girl' intelligence.

The only thing this bot appears to be good at is interjecting non-related information into the 'conversation' and pausing variably between responses. A bot I wrote around 2003 did this as well, but it had per-user history analysis, deterministic responses, no pre-constructed sentences (relying upon a database of word relationships and sentence structure) and was able to maintain a conversation for about 20 private exchanges on average on IRC. Most of the time, there was no indication that the chatter realized it was a bot, and I had it sit in channels for weeks chatting people up.

If I could do it in a couple weeks of off-time programming, I'm sure there are many other examples of something drastically superior. This is not one of them.

Cleverbot is a bad example of a chatterbot.

[Z8]

According to the wiki page, it just selects canned responses from its database. I think this approach just gets you garbage, or at the very least is a dead-end in trying to beat the Turing test.

The best Turing Test is probably the Loebner Prize and at least the contestants seem much better than Cleverbot. There's an example conversation from Suzette (the latest winner) here. (But it's hard to tell if that is typical or simply a lucky exchange for the computer.) But anyway, as is clear from this interesting story written by a contestant about the Loebner prize, bots are no where near winning that version of the Turing test, as long as the humans are paying attention.

The problem with these tests ...

[MacTO]

Well, there are many. But I recall seeing one such turing test in the 1990s where the human operators would try to convince the user that they were a computer. Sometimes they would do simple things, like pretending that they weren't as 'smart' as they actually were (e.g. they would pretend that they didn't know things that they knew in order to avoid looking encyclopedic about a topic). Other times they would insert mistakes that a typical computer would make, such as misinterpreting a question in a wonky way.

Then there is a boatload of other issue. How do you quantify the humanness of a subject? Are we looking at socialization or linguistic skills (i.e. is a computer that can smooth over a misunderstanding more human than a computer that can understand what is being said but reacts in an anti social manner), and so forth.

Re:I hope not

[vlm]

I just chatted with him and it is was total failure, it kept changing subject and it never answered any of my question. Those 59.3% techies from India must have been really dumb.

Did cleverbot ask you to reinstall windows?

This says it all...

[Excelcia]

Conversation I just had...

Hello. My name is Kurt.
What kinda name is that.
German.
Cool! Me too!
You're German?
Czech.
So you're not German?
I am not.
Then why did you say "me too"?
I didn't.
Did the lobotomy hurt?
Where did th ebarn come from?

Re:This says it all...

[CProgrammer98]

"not too impressed"

On the other hand....

me: Do you know Java?
bot: I Know JavaScript
me: But that isn't java.
bot: What did you ask.
me: asked if you knew Java
bot: I know many jokes.
me: Java is not a joke.
bot: Yes, it is.

Possible language barrier?

[Just Brew It!]

Given that the test was held in India, how fluent in English were these 1334 people? Yes, I know that English is taught and used extensively in India; but non-native speakers will likely have a harder time picking up on the nuances which would separate a human from an AI.

Re:Possible language barrier?

[CAIMLAS]

Judging by the Indian programmers and support I've had the displeasure of working with, the logical divorce/completely non sequitur responses of this bot likely seem quite human to them - language capabilities aside!