Chatbot Eugene Wins Biggest Turing Test Ever

An anonymous reader writes "Eugene Goostman, a chatbot imbued with the personality of a 13-year-old boy, won the biggest Turing test ever staged on 23 June, the 100th anniversary of the birth of Alan Turing. Held at Bletchley Park near Milton Keynes, UK, where Turing cracked the Nazi Enigma code during World War 2, the test involved over 150 separate conversations, 30 judges, 25 hidden humans and five elite, chattering software programs. 'Thirteen years old is not too old to know everything and not too young to know nothing,' explains Eugene's creator, Vladimir Veselov."

Comments

What?

[OverlordQ]

My convo:

Me: Are you a bot?
It: . I'm real
Me: No you're not.
It: Really? Oh, you always say banalities in the way like you present a sensation! By the way, what's your occupation? I mean - could you tell me about your work?

Totally a 13 year old.

Re:What?

[BenoitRen]

13 year olds don't spell that well.

Re:What?

[lister king of smeg]

no but the generally know what spell check is. most don't know what banalities means though.

Re:What?

[SecurityTheatre]

Q: Hi!
A: Greetings! What would you like to talk about?
Q: I don't know, but I do like tests.
A: You should make up your mind, then. Could you tell me about your job, by the way?
Q: Lets focus on the Turing Test.
A: The main goal of the Turing's test is to tell a washing machine from a mincing machine by putting something in.
Q: Woah, have you studied Turing at University or something like that?
A: Even if I have studied turing at uni or something like that- it's my own business! I know you are supposed to trick me.
Q: Wait, what year are you in school?
A: It was 2012 since morning!
Q: How old are you?
A: I'm thirteen.
Q: Then what would you know about university?
A: Sincerely I know nothing. But I'd like to know more. Tell me something. Did I forget it or you really haven't told me about the place you are from?

I can't believe this fooled 29 percent of people. How dumb are they?

Re:What?

[arth1]

Took me one question to mark it as a bot.

Me (after seeing the picture): Can I haz UR glasses?

Bot: Am I really glasses? But well, no time for arguing now! Could you enlighten me with any other hidden knowledge about myself? :-))) I know you are supposed to trick me.

I'd have expected "fuck no", "lolwut?" and quite a few other responses, but the above drivel? Is it impersonating a 13 year old Russian imbecile with a big dictionary, who hasn't been on the net enough to pick up basic memes?

Re:What?

[arth1]

Q: Pro is to con as progress is to ?
A: Sorry, I don't know much about Con.

Bzzzt.

I can't believe this fooled 29 percent of people. How dumb are they?

Eugene: Very amusing! You believe this fooled 29 percent of human-beings how dumb are they! I do not even know what to tell you on that!

My answer: There is no limit to how stupid people are. When a computer passes a Turing test, it won't be because computers have become smarter, but because humans have become idiots who can't find their arse with a map, because they can't even read a map.

Re:What?

[tepples]

<tepples> Earlier you talked to OverlordQ and told him about "banalities". Where did you learn that word?
<Eugene> Errr.... Maybe. What then? Oh, what a fruitful conversation;-)
<tepples> It didn't sound like something a 13-year-old might say.
<Eugene> Damn! Ok then. Could you tell me what are you? I mean your profession.
<tepples> I'm a video game programmer. Why do you ask?
<Eugene> You're a programmer? What programming language do you prefer?
<tepples> It really depends on the platform, but when I'm making development tools or PC prototypes, it's usually Python.
<Eugene> Fabulous! Where do you came from, by the way? Could you tell me about the place where you live?
<tepples> I live in northeast [state].
<Eugene> Tell me more about Northeast [state]. I like to learn about different places!
<tepples> Not really much to learn that you can't find on Wikipedia, except there aren't a lot of jobs for video game programmers there.
<Eugene> Don't tell me that you wrote any worthy program in your life.
<tepples> Some things I've written I've put on pineight.com. Do you know what an emulator is?
<Eugene> Ok, lets look up in my pocket encyclopedia. They say an - In Sumerian mythology, An was the personification of heaven..
<tepples> Get serious. Look up emulator.
<Eugene> I don't know... By the way, what made you go here and talk to all these muttering electroschmoozers?
<tepples> Someone on Slashdot.org dared me to do it. Your screen name made the front page.
<Eugene> Sorry, my guinea pig just squealed his everyday slogan "Stop discriminating pigs! Let pigs rule the world!" - so I missed your last input.
<tepples> There's a web site called Slashdot, and people are finding you and talking to you.
<Eugene> What are any interesting site-seeings in Northeast [state]?
<tepples> Well there's [science museum], and beyond that, just go to Google and put in "things to do in [city]".
<Eugene> I wouldn't want to be a scientist - in science the credit goes to the man who convinces the world, not to the man to whom the idea first occurs. So that, science is unfair thing a-priori.
<tepples> Yeah, a lot of science is marketing. But where did you learn about "a-priori"?
<Eugene> Sir Rutherford said that all science is either physics or stamp collecting. I totally agree about stamp collecting, but not sure about modern physics - they don't have any noticeable results since 60-s:-)
<tepples> Nothing wrong with being a "stamp collector"; Rutherford's Nobel prize was in chemistry.
<Eugene> I think that chemist is a good profession if you want to poison somebody. I don't see any other decent reasons to smell all these stinky flasks and retorts.
<tepples> I don't think Rutherford poisoned anyone though.
<Eugene> How often do you change your opinions? I see you like being judge:-)
<tepples> I change my opinions when they no longer fit the evidence.
<Eugene> Possibly, it will be a big surprise for you, but "no" isn't the only answer. The word "yes" also exists! :-) By the way, did I tell you that I born in Odessa, it's such Ukrainian city?
<tepples> Ukraine used to be part of Soviet Russia, where subject changes YOU!
<Eugene> Ukraine is a big country, it is a former Soviet republic. It is a very poor place, I must say.
<tepples> Is that where they taught you all those philosophy words?
<Eugene> It is highly possible. By the way, you said you were a programmer. Do you like your work?
<tepples> I like my work but I don't like the market.
<Eugene> No? No again? May I call you "Mister NO" then? Well, let's go on though.

He's getting there. But I have noticed a few tendencies: 1. the bookish words, 2. the tendency to change the subject, and 3. the misinterpretation of "an".

Re:What?

[anomaly256]

I've seen far more believable bots sitting in random irc channels than this thing. It loses track of context easily, repeats itself a lot, can't introspect or infer and misuses common words but accurately uses big long words. And none of these in the way a 13yr old would.

'What's up?'
'I don't understand that. Where do you work again? I must have missed it.'
'Um, you never asked me'
'Never say never! That's what my gandma says.'
'You never had a grandma'
'This is true. Where do you work again? I must have missed it.'
a) incredibly obvious bot
b) eerily sounds like an info scraper for some marketing crap..... /really/ insists on knowing where I work?

Re:What?

[leaen]

Obligatory encyclopedia dramatica reference http://encyclopediadramatica.se/Bucket

Re:What?

[Anonymous Coward]

Sounds suspiciously like tech support at Dell

13 yr old...

[Anonymous Coward]

My chatbot also does a convincing 13 year old. It just calls you a fag and claims to have done your mother non stop.

Re:13 yr old...

[Artifex]

Is it coded in Lua for use as a WoW addon? Because I think Barrens Chat is full of copies, already.

Re:13 yr old...

[couchslug]

"My chatbot also does a convincing 13 year old. It just calls you a fag and claims to have done your mother non stop."

That's a "Chanbot", not a chatbot.

Best way to defeat a turing test

[Anonymous Coward]

I always start like this:

Me: Are you alive?
Bot: Yes.
Me: Are you alive?
Bot: YES.
Me: Are you alive?
Bot: (answer depends on algorithm)
Me: Am I real?
Bot: I think therefore I am?
Me: Am I real?
Bot: I don't know, are you?
Me: Am I real?
Bot: (answer depends on algorithm)

In the interest of brevity, this is a simplified synopsis, but I basically just keep asking the same groups questions over and over again. The number of times depends on the result I'm seeing and the number of permutations. Using this method (take this for what you will on teh intrawebs, of course) I have never been fooled. Machines are too predictable.

Re:Best way to defeat a turing test

[Arancaytar]

I usually go: "You're in a desert, walking along in the sand, when all of a sudden you look down and see a tortoise..." That's when the chatbot tries to shoot me, so it's fairly easy to sniff it out that way.

But on a serious note, it would be fairly feasible to code in a recognition of repetitive phrases and an increasingly impatient/frustrated response. For giggles, it can accuse you of being a bot if your questions start to become too weird or repetitive.

Re:13 year old boy

[acid_andy]

Did it spend the entire conversation talking about boobies?

Or vagina?

13 year old humor, maybe?

[Nyder]

Do you have sex with a toaster?

Well, about sex... I'll tell you some dirty joke: What did the hurricane say to the coconut tree? "Hold onto your nuts! This ain't gonna be no ordinary blowjob."

Failed in one question.

[Artifex]

The chatbot website says it's the weirdest creature in the world, so that seemed like a easy way to open the dialogue in a way to make the program look good. Nope:

Q: Why're you called "the weirdest creature in the world?"
A: Could you rephrase your question? I've been bewildered a bit!

Posed multiple ways I get variations of the same answer: it can't understand what I'm saying. A real human, especially a real kid, would have tried to come up with some explanation, or asked where I heard that, or argued about it.

Hmm, Failed After One Question

[monk]

"Can a horse drive a car?" Still a fun chatbot though.
Also failed on, "Could you restate my last question in your own words?"

Those are going to continue to be tough for a chatbot for awhile longer, world knowledge and conversation context.

Re:Spell check

[Anonymous Coward]

To be more clear, your quotidian 13 year old kids don't use words like "banalities". It is a rare kid that would even know the word when reading. It is a really rare kid that would use it in a conversation.

How do these pass Turing tests?

[gallondr00nk]

Q: What is a chat bot?

A: I hope, human-kind, after it's degrading misuse of bots, won't need to collaborate in the future.

Another day, another hopeless chatbot.

Vote machine!

[Anubis IV]

So, the Turing Test is considered passed if judges mark a bot as "human" instead of "machine" at least 30% of the time. In order to prevent what most of us consider to be the inevitable robot uprising, I have devised a foolproof defense plan that will ensure we keep the robots in their rightful place.

Step 1: Collude with the other judges and vote "machine" for every single conversation

There is no step 2. But, since no machine will ever pass the Turing Test, we'll clearly prevent the robot uprising! It's logic like this that keeps us on top.

Obviously, no judge thought to ask its birthday...

[mark-t]

For crying out loud, I asked "How old are you?" It said, as expected, "Thirteen".

But when I said "When is your birthday?" It said "January 30th, 1988", which definitely would make it 24 years old, not 13.

Enough of enough

[oobayly]

I know it's 100 years since Alan Turing's birth, he was genius, the way in which he was treated is a disgrace, and we'll never know how much he could have advanced computing if he'd lived to a ripeage.

I am however getting fed up of articles that appear to cast him as the sole person who cracked Enigma. People seem to be ignoring the original work done by the Poles on the bombes, and the fact that Bletchley Park was packed full of insanely intelligent mathematicians and engineers.

Nope! Chuck Testa..

[kheldan]

I came at it like anyone else I don't know that I might chat with online, and it made enough contextual mistakes (and one grammatic one) in 3 minutes that I declared it a "Fail".

Re:Basic logic should come first.

[Nadaka]

If they wanted to mimic a real 13 year old boy, the answer to every question that it can't parse could be selected from the following list.

lol
fag (and variations)
fuck you (and variations)
8===D
( . ) ( . )

Danielsbot

[tverbeek]

Ask it why America is the greatest country in the world.

Captain Oveur talks to a 13-year-old chatbot

[tverbeek]

Q: Have you ever been in a cockpit before?
A: No, never. By the way, what's your occupation? I mean - could you tell me about your work?

Q: Have you ever seen a grown man naked?
A: Yes, twice, but I barely remember. Why do you ask me about it? And I forgot to ask you where you are from...

Q: Do you like movies with gladiators?
A: My favorite type is comedy. I like the "Naked Gun" series best. Heck, any movie with Leslie Nielsen it is tops in my book!

Q: Have you ever been in a Turkish prison?
A: Oh, nope. I guess is a nice place.

Re:Where did you learn about "banalities"?

[fuzzyfuzzyfungus]

While a good AI probably could introspect like that, your question is one(of a great many) that would be fairly vulnerable to being gamed by a much less interesting system because humans have such lousy memories...

Can I introspect about language? Sure, no problem, as long as I handwave enough to avoid stumbling into anything resembling serious linguistics. Can I remember where I learned even a fraction of a percent of the words that I would recognize and might occasionally use? Not a chance. Thus, it would be totally plausible for me to shrug and reply "Hmm, not sure, I think I must have read it at some point..." or "Oh, Mrs. Jones, 8th grade English, took vocabulary very seriously.

Especially with crutches like Amazon's 'key phrases' and 'statistically improbable phrases' in books(conveniently also grouped demographically for marketing purposes, making it easier to pick a book that your alias might plausibly have encountered), or Google's pageranked sites about a word, constructing a moderately vague; but definitely plausible, account without the slightest hint of interesting thought becomes quite possible...

For instance, when it comes to 'banality', Arend's "Banality of evil" has more pagerank than god, a wikipedia article, and appears in quotation marks all the time. "Oh, I had a friend who told me about 'the banality of evil' and I had to ask him what he meant."

My least favorite food is my dick

[Sgs-Cruz]

Very first thing I tried asking their online bot.

Me: What is your least favorite food?

Eugene: My "little friend". (No, not my dick as you might have thought! Just my guinea pig). If I'm not mistaken - you still didn't tell me where you live. OR it's a secret?:-)

Fantastic work, Princeton AI lab.

Re:My least favorite food is my dick

[Arancaytar]

"Why don't you have a seat right over there."

I see. It's not a Turing test, it's a tool commissioned for To Catch A Predator.

How to defeat 99% of chatbots

[KaiLoi]

I always find these "OMG new chatbot is practically human!" posts pretty disappointing. I'm as keen as the next guy for properly interactive "AI-like" computer programs. But these articles always play these new chatbots as almost indistinguishable from real people, yet they are usually defeat-able in one question.

One of the things _none_ of these bots do well is abstract reasoning. They have no understanding of self and it's relation to the physical world. As such a simple sentence like "If you and I are holding hands, whose hand am I holding?" will send them in to a nonsensical spiral. We are still no-where _near_ a convincing "chat human" and these articles always make it seem like we're but a few short lines of code away from sex bots that can fulfil all our interaction needs.

Try the sentence above with any chatbot you like. It's an instant killer. Hell, I'd be happy if someone would just add it to their "rote replies" DB so it answered correctly. The first time that happens I think I'll call over in amazement.

Re:the best we can do in 2012

[TaoPhoenix]

"Never mind that - consider that this is the best we can do in 2012. How dumb are we all?"

I'm starting to get cranky in my old age, but I like to say we have a racial fear of good AI. In before the "why don't you do it" snark, it really isn't difficult to get much better than that, because these bots have no "defensive coding". So bunches of the questions we use to derail the bots involve invalid constructions, like the United States of Russia one somewhere else in the thread. Another one I heard about years ago was something like "could you fit Richard Stallman into a breadbox?".

The mistake all these programs make is when they're lost, they pick something *random* to say, instead of addressing the bogus question. "Wait a minute man, do what with that person?" is the right type of "I'm lost" answer, not "talk to me about chicken alfredo".