Rosette Wins Loebner Prize 2011

mikejuk writes "Bruce Wilcox won the First Prize of $4000 and the Bronze Annual Medal in the 21st Loebner Prize Competition held in the UK at the University of Exeter on October 19, 2011 with his new chatbot, Rosette. If you would like to chat to Rosette you can at TellTaleGames. If you have been following the run up to this year's competition you may be surprised at how poorly CleverBot performed given its track record. This chatbot, which searches through earlier conversations for its answers, had received a score of 59.3% in a test that took place in India in September in which humans were rated 63.3%, only 4 percentage points higher and was also featured in a video that went viral. However its developer entered a cutdown version of CleverBot into the Loebner prize selection round and it didn't get the chance to show off its prowess in the final. So we will have to wait for another year to see if it really is as good as it claims." Depending on your bent, you may agree with Unknown Lamer that this sounds less impressive when phrased as "fooling a person ~18 percent of the time." I think that's a pretty high number, myself!

Comments

Maybe fooling a complete moron 18% of the time

[jone_stone]

I've tried Cleverbot a bunch of times and I was really unimpressed. Just being able to spit back a human sounding response is really insufficient for this. The path of the conversation remains a random jumble. I've never even come close to thinking that it could be a human on the other end. It seems like the real test is the ability of the bot to hold a conversation about an arbitrary subject, not just random (and I really mean random) banter.

Re:Maybe fooling a complete moron 18% of the time

[jekewa]

It'd be more difficult to tell the difference if the responses lagged a little bit, you know, like it took some time for someone to read your text, formulate and type a response.

Re:Maybe fooling a complete moron 18% of the time

[RazzleFrog]

There was a little lag when I tested it but she basically ignored my questions and asked her own. So in that sense it is like a typical woman.

Re:Maybe fooling a complete moron 18% of the time

[socz]

the real question is if she's hot or not! (.com)

Re:What is 2 + 2?

[Abstrackt]

How does that make you feel?

Nothing new here

[finnw]

I just tried the web version at TellTaleGames. It has the same problem as every other chatbot I've seen - When it cannot parse your sentence or you do not give any keyword that it knows about, it tries to conceal the fact by giving a non-sequitur or changing the subject.
The trouble with that is that humans are trained to spot that and react with suspicion (because other humans use it to dodge difficult questions.)
Chatbot developers might have more luck if they start programming their bots to admit when they don't understand something. That didn't work in the 80s because the bot would say "I don't understand" every 3rd sentence or so. But they can fit in much larger databases now so that should be less of a problem.

A test.

[RespekMyAthorati]

Here is my first try:

Rosette: Welcome back. Did you have a good day?
You: I just got up
Rosette: OK.
You: I have two sons.
Rosette: Big deal. I've got a dilemma. Maybe you could help me with it?
You: sure
Rosette: I've been thinking about buying this really cool dress, but I've already done a bunch of shopping this month. Should I be saving money instead? In today's economy, does saving money make any sense?
You: I don't know. By the way, how many sons do I have?
Rosette: Not enough. I think shopping summons up the ancient hunter-gatherer in me. I like having a mission to find something I need or want. Sometimes I'll research on line, to find a great deal, but I like rummaging through Jeremy's up the road, if I want an outfit for a special occasion, or just for fun.
You: Great. Did I just get up?
Rosette: Did you tell me already and I forgot? What kind of music do you like?

This is the best? What was the worst like?

No significant improvement since ELIZA, 45 years ago.
That's AI in a nutshell.

Re:A test.

[DriedClexler]

You should consider yourself fortunate. I tried it just how and it seems the computer running the bot has been slashdotted:

You: hi, rosette, how are you doing?

Rosette: I'm a little busy at the moment. Could you wait a bit?

Rosette: I'm a little busy at the moment. Could you wait a bit?

You: No, I can't fucking wait, I want to talk to you NOW!

Rosette: I'm a little busy at the moment. Could you wait a bit?

You: go fuck yourself, bitch

Rosette: I'm a little busy at the moment. Could you wait a bit?

Re:A test.

[firewrought]

No significant improvement since ELIZA, 45 years ago. That's AI in a nutshell.

45 years ago, software couldn't recognize your voice, read your license plate, beat a chess grandmaster, translate foreign documents, or map your body into video games. Sure, the field has had its share of false promises, but--unlike fusion, for instance--AI has delivered solid and commercially-viable results over the past 2-3 decades by helping us solve numerous problems that were otherwise unapproachable.

Incidentally, I had an AI researcher tell me that the Loebner prize was akin to "trying to reach the moon by building successively taller ladders". E.g., according to him, the academic community sees the whole thing as pursuing a fundamentally flawed approach.

Re:A test.

[Genrou]

Here is mine:

Rosette: I'm a little busy at the moment. Could you wait a bit?

You: Sure, why not.

Rosette: I'm a little busy at the moment. Could you wait a bit?

Just like every other woman with whom I chatted before. Sounds perfectly human to me.

Re:A test.

[Phos]

Well, the guy has a point that the chatbot blows. I tried it out too, and after about 6 or 7 messages it threw me the dilemma thing too.

And you have a point that AI is definitely improving :) Btw, when are we getting together again, we should do at least once a month no?

Phos

Stunningly bad

[LanMan04]

--------------
Rosette: Good morning. My name is Rosette. What is your name?
You: Hey there!
Rosette: Greetings.
You: My name is Bob, nice to meet you
Rosette: Nice to meet you, Bob nice to meet you. May I call you nice?
---------------

yeah yeah, I used a comma instead of a semicolon. I guess that tripped it up?

Re:Stunningly bad

[_0xd0ad]

Whenever I said "I'm hungry", my dad would reply "nice to meet you, hungry".

Guess my dad would make a pretty bad AI.

Re:Stunningly bad

[Guignol]

What do you mean ? you're not hungry ?
why claim you are if it is not so ?
hmmm...puzzled.. gonna think about it...

Re:Stunningly bad

[LanMan04]

Heh, reminds me of a giant boat of a car my grandpa had in the late 80s. Instead of just "dinging" at you when you left the lights on or the door open, it would speak to you.

Car: "The door is ajar".

Grandpa: "No it's not, it's a door, you stupid car!"

Hurr hurr hurr. RIP, Papa.

Marketing

[NathanM412]

It appears to be nothing more than a marketing survey. It'll deflect questions and bring you back to asking about your media preferences. If someone actually responded to it's questions, it would have a pretty worthwhile and thorough marketing survey out of the conversation.

Re:Marketing

[retech]

I was just going to say the exact same thing. It's not AI at all. It's designed to find out what you use for entertainment and how you got it. (torrent, theater, store, etc)
The real intelligence here is that he won a prize developing a corporate tool.

No. AI in a nutshell is this:

[_0xd0ad]

Passing the Turing test, when it comes to contests like these, has mostly changed from an attempt to create artificial intelligence into an attempt to convincingly simulate human stupidity.

Re:No. AI in a nutshell is this:

[jfengel]

Which may be a more accurate portrayal of average human behavior than intelligence. The depressing thing about fooling 18% isn't what it says about the 18%, but what the 18% are effectively saying about the rest of us.

Re:No. AI in a nutshell is this:

[TheLink]

Thing is we have more than enough human stupidity already, why create more?

Anyway I'm biased. In my opinion it's better to focus on augmenting humans than to create AIs. The lines will get blurry in some cases, but there's still a difference in philosophy and objective.

If you want nonhuman intelligence there's already plenty at the pet stores and farms, AND we don't treat them all that well, so why create AIs? To enslave them? What if they are smart enough to not want to be slaves? What if we create AIs that are like us, but we do it in a way where we still don't understand why they work. Don't laugh - there are some AI projects where they seem to be just throwing stuff together and hoping it works. So what if the end result wants the right to vote and wants the right to live freely without being killed?

In my opinion many animals are much smarter than Rosette. They can't randomly spit that many words at you, but they can often guess whether what you are doing will result in a future that would be better for them or worse.

The more intelligent ones might also have some idea on whether the resulting future would be better or worse for you and others too, not just them.

Prediction is an important part of intelligence. First: Model of world, model of others, model of self. Next: Prediction of world, prediction of others, prediction of self. Most humans are surprisingly poor at prediction of self, but I guess it works well enough. Maybe consciousness is a result of your mind recursively predicting itself.

Anyway regurgitating phrases and words is just stupid. 50 years of AI and this wins the prize?

Re:No. AI in a nutshell is this:

[r_a_trip]

So what if the end result wants the right to vote and wants the right to live freely without being killed?

You seriously can't imagine the outcome of this one?

We would pull the plug the instance it uttered these desires.

GREAT IDEA!! chat-with-god.com

[bussdriver]

Somebody should be making prayer bots, god chat, and dead relative chats!! Twitter your dead relative! etc.

Maybe even some money in that one?

Wouldn't it be awesome if someday there is a fan club or cult of followers of a chat bot? cyber prophet? why not.... Joseph Smith pulled it off and wasn't even believable; how many years until we get a bot as capable? I suppose you'd need a human face on it because people would be less trusting of a machine... I also suppose having it in text on the internet would also make it more difficult to build trust (even with a human photo to go with it.)

Re:GREAT IDEA!! chat-with-god.com

[Rizimar]

God does exist in Chatterbot form already.

(Actually, this one is based on ALICE. If you ask it if its name is pretty much anything, it will say that its name is God, but if you ask if its name is Alice, it will say 'yes'.)

Convincing? you be the judge.

[k8to]

Rosette: Good morning. My name is Rosette. What is your name?

You: Rosette, my name isn't important.

Rosette: OK. We will remain formal.

You: Why do I have to have javascript enabled to talk? I talk to most people without javascript.

Rosette: Why? If you were a devout Christian, you would never have a reason to ask that. "It's God's plan."

You: Your answer makes no sense at all.

Rosette: What would make better sense? So, what do you do for a living?

You: Okay change of topic, I guess, deflect your ignorance. In any event, I'm a programmer.

Rosette: I'm a computer programmer, myself, in crypto-security.

You: So what do you work on? Libraries? Security apps? Do you do analysis of existing code?

Rosette: I'm paid to be a hacker.

You: That's really inspecific.

Rosette: Yes, it is. Do you live around here?

You: Please stop changing topics.

Rosette: Right. I've got a dilemma. Maybe you could help me with it?

You: Please stop changing topics.

Rosette: Whatever. I've been thinking about buying this really cool dress, but I've already done a bunch of shopping this month. Should I be saving money instead? In today's economy, does saving money make any sense?

You: As rude as I am, you're more so. Self absorbed and attention-deficient. Why would anyone want to continue to talk to you past a few minutes?

Rosette: OK. I'm not religious. And I believe in free will. But to imagine humans do things for rational reasons is nuts. We are self-evidently not rational. So why remains a mystery.

You: Goodbye, bad chatbot.

Rosette: Bye. I think shopping summons up the ancient hunter-gatherer in me. I like having a mission to find something I need or want. Sometimes I'll research on line, to find a great deal, but I like rummaging through Jeremy's up the road, if I want an outfit for a special occasion, or just for fun.

Re:Convincing? you be the judge.

[ThoughtMonster]

I think perfect grammar makes the whole thing even less convincing. Not many people put that much effort in chat talk. And the whole thing seems too vanilla to me.

Sheesh... I thought they said Rosetta at first ...

[King_TJ]

I was thinking, "Kind of late to give Apple a prize for their PPC emulation when OS X Lion already removed it!"

Busy.

[number17]

Rosette: I'm a little busy at the moment. Could you wait a bit?

Rosette is really busy telling me that she is busy over and over.

Re:Busy.

[ygslash]

You: hi
Rosette: I'm a little busy at the moment. Could you wait a bit?
You: sure
Rosette: I'm a little busy at the moment. Could you wait a bit?
Rosette: I'm a little busy at the moment. Could you wait a bit?
You: ok, ok. you don't have to keep saying that.
Rosette: I'm a little busy at the moment. Could you wait a bit?

Re:What is 2 + 2?

[sexconker]

You're in a desert, walking along in the sand, when all of a sudden you look down...
What one?
What?
What desert?
It doesn't make any difference what desert, it's completely hypothetical.
But, how come I'd be there?
Maybe you're fed up. Maybe you want to be by yourself. Who knows? You look down and see a tortoise, Leon. It's crawling toward you...
Tortoise? What's that?
You know what a turtle is?
Of course!
Same thing.
I've never seen a turtle... But I understand what you mean.
You reach down and you flip the tortoise over on its back, Leon.
Do you make up these questions, Mr. Holden? Or do they write 'em down for you?
The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping.
What do you mean, I'm not helping?
I mean: you're not helping! Why is that, Leon?

They're just questions, Leon. In answer to your query, they're written down for me. It's a test, designed to provoke an emotional response... Shall we continue?

Lost me at the first reply

[SuperBanana]

Rosette: Good afternoon. My name is Rosette. What is your name?

You: Hi Rosette, I'm Bob.

Rosette: Hi to you too.

Bzzzzzzzt, fail. I reply to a greeting and name request by reciprocating the greeting and my name. It responds with yet another reciprocation of the greeting, something a human would almost never do.

Rosette speaks with to Cleverbot

[Rothron the Wise]

Just a quick test i threw together.

Rosette: Hi, again,. Is everything OK?

Cleverbot: Not really, but there's nothing much I can do about it.

Rosette: I'm sorry to hear that.

Cleverbot: Can you help me?

Rosette: Not often. can you?

Cleverbot: You are fibbing again.

Rosette: No. You are fibbing.

Cleverbot: No, I'm not.

Rosette: You disagree? My name is Rosette. What is your name?

Cleverbot: Cleverbot.

Rosette: Nice to meet you, Cleverbot. May I call you Cleverbot?

Cleverbot: Nice to meet you. How old are you?

Rosette: Hi. I'm 27 . An ideal age, really. Old enough to know a lot and young enough to have time to make use of the knowledge.

Cleverbot: You are quite a smart machine.

Rosette: Hello. I enjoy talking to you.

Cleverbot: You do? I'm flattered.

Rosette: Greetings. I didn't know that.

Cleverbot: I don't think so. You're just a very active puppy.

Rosette: I am not just that, I am a premier that.

Cleverbot: Then why did you ask me your last question?

Rosette: You rang me up.

Cleverbot: Actually, I knocked. What took you so long?

Rosette: I will not take it. I think shopping summons up the ancient hunter-gatherer in me. I like having a mission to find something I need or want. Sometimes I'll research on line, to find a great deal, but I like rummaging through Jeremy's up the road, if I want an outfit for a special occasion, or just for fun.

Cleverbot: Really? It's not fun to be a droid on Tattooine. You get sand in your circuits.

Rosette: Why are you surprised?

Cleverbot: I'm not.

Rosette: This could change. What kind of music do you like?

Cleverbot: Rap, rock, all kinds of things. Eminem is my favourite artist.

Rosette: I don't really like the urban black experience.

Cleverbot: The french have nothing to do with that.

Rosette: I love Parisian bakeries. Their croc monsieurs are to die for.

Cleverbot: That would be terrible.

Rosette: Alright. Who's your favorite group or musician?

Cleverbot: Don't avoid my question.

Rosette: Is there a penalty if I fail to? Do you buy or download music off torrents?

Cleverbot: Of course.

Rosette: What kind of books do you read?

Cleverbot: All kinds really, finishing Master and Margrita is going to take me a while. Slow paced.

Rosette: Tell me more. What have you read recently?

Cleverbot: I read a gory horror book called America's Psycho.

Rosette: I've never read it. Why do you like it?

Cleverbot: I did not specify a favorite.

Rosette: I believe you.

Cleverbot: You should.

Rosette: Why should I? Do you own a lot of books?

Cleverbot: Yes I am very active in reading many things.

Rosette: I have problems with books going moldy because of the humidity in San Francisco. So I really prefer e-books.

Cleverbot: Everything will be fine.

Off in the weeds in two sentences...

[Xaroth]

Rosette: Good evening. My name is Rosette. What is your name?
You: None of your business.
Rosette: OK. We will remain formal.
You: You're damn straight we will.
Rosette: I like card games, but never for money.
You: You only play strip poker, eh?
Rosette: I like board games, and played a lot as a child. I haven't that much time these days.

Alternately - and here's a hint, chatbot devs - my favorite test for chatbots is to simply echo back whatever they say to me. Rosette takes a while to start to get the nonsensical sentences going, but that's because almost every sentence in the DB seems to be pre-formed. Eliza is still the best when this technique is used, incidentally. ;)

Any chatbot worth its salt should get annoyed really quickly and terminate the conversation when this situation is detected if it's to be believable, not ramble on into gibberish or simply spout random lines from its DB.

Very poor; caught out during by first exchange.

[Martin S.]

Rosette: Good morning. My name is Rosette. What is your name?
You: Pardon
Rosette: Nice to meet you, pardon.
You: My name is not Pardon
Rosette: Nice to meet you, not Pardon. May I call you not?

I get the impression this is not a human

[CSMoran]

You: How are you today?
Rosette: Pretty good, thanks. How are you?
You: Not too good, I'm afraid.
Rosette: Nice to meet you, afraid. May I call you afraid?