'Rose' Wins 2015 Loebner Contest, But Big Prize Remains Unclaimed

← Back to Stories (view on slashdot.org)

'Rose' Wins 2015 Loebner Contest, But Big Prize Remains Unclaimed

Posted by timothy on Sunday September 20, 2015 @09:34PM from the how-about-a-nice-game-of-chess dept.

The Next Web reports that developer Bruce Wilcox created the most convincing bot entered in this year's annual Loebner Competition. His latest entry, a chatbot named Rose, passed itself (herself?) off as a 30-year-old security consultant well enough to fool judges for a few minutes. But Wilcox's first-place entry was still not good enough to win the $100,000 Loebner Prize, to be given only for a more convincing impersonation. The article notes: "This isn't Wilcox’s first entry – or win. In 2010, he took first place with a bot named 'Suzette,' and followed that up in 2011 with another win using a new bot called 'Rosette.'"

58 comments

Min score:

Reason:

Sort:

Winner? by AmiMoJo · 2015-09-20 22:32 · Score: 5, Informative

This is the conversation they tested the "winner" with: http://www.aisb.org.uk/media/f...
While it's kinda impressive that AI can do that, it's also clear that we are still a very, very, very long way from having a computer impersonate a human. What really surprises me is how hard all the entries found basic logic questions to be - I guess it is the language parsing bit that is giving them grief.

--
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
1. Re:Winner? by Anonymous Coward · 2015-09-20 22:50 · Score: 1
  
  We are a very, very ,very, very long away from AI at all. In fact, there hasn't been any progress in the field since the 1970s. The programs are slightly more clever, but we are essentially at the Eliza stage.
2. Re:Winner? by packrat0x · 2015-09-20 22:55 · Score: 5, Insightful
  
  We might never develop AI, but I anticipate increasingly clever fakes!
  
  --
  227-3517
3. Re:Winner? by gweihir · 2015-09-20 23:51 · Score: 2
  
  Indeed. There is not even a credible theory how true/strong AI could be implemented. It may still turn out to be infeasible, or we may never get there without knowing why. The only thing that even remotely goes into the direction is automated theorem proving, and ion this universe that runs into hard physical computation limits a long time before it even begins to reach what smart humans can do.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
4. Re:Winner? by gweihir · 2015-09-20 23:52 · Score: 2
  
  You are certainly correct on that. Also because there is a lot of money to be made that way.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
5. Re:Winner? by 91degrees · 2015-09-21 00:00 · Score: 1
  
  Is there an actual human in there for comparison?
6. Re:Winner? by Ecuador · 2015-09-21 00:14 · Score: 4, Funny
  
  While it's kinda impressive that AI can do that, it's also clear that we are still a very, very, very long way from having a computer impersonate a human.
  What makes you believe we are still a very, very, very long way from having a computer impersonate a human?
  
  --
  Violence is the last refuge of the incompetent. Polar Scope Align for iOS
7. Re:Winner? by Anonymous Coward · 2015-09-21 00:40 · Score: 0
  
  The transcript shows the winner bot is very, very, very far from convincing and there has been no major advances in decades. At the present progress rate bots are not going to be convincing in the forseeable future.
8. Re:Winner? by Anonymous Coward · 2015-09-21 00:55 · Score: 0
  
  Whoosh, perhaps? Hint: Ecuador created a question from AmiMoJo's statement the same way early chatbots like Eliza would.
9. Re:Winner? by TheRaven64 · 2015-09-21 01:15 · Score: 1
  
  Wow, that's dire. It's pretty obvious by question 6 that it's not a human and the none of the rest look even vaguely convincing. It's been a couple of years since I last read Loebner transcripts, but I remember it taking a bit longer 5 or so years ago. Perhaps this just means that the questioners have become better at picking questions that make sense. Back then, they put some of the winners online. I remembered a question from the arithmetic book I had when I was 4 or 5: 'Flipper ate 10 fish, then he ate 4, how many fish did flipper eat?' None of them got it right (or even demonstrated that they had parsed that it was a question with a numerical answer).
  
  --
  I am TheRaven on Soylent News
10. Re:Winner? by Anonymous Coward · 2015-09-21 01:15 · Score: 0
  
  You must be a very easy person to impress. The exchanges are pretty much like those from previous years: utterly pathetic. It's not only that those programs are obviously bereft of a shred of intelligence but, it takes a couple of questions to find out conclusively. If anything, it is amazing that so little progress has been made in this area in the last fifty years.
11. Re:Winner? by Anonymous Coward · 2015-09-21 01:23 · Score: 0
  
  This makes me giggle.
12. Re:Winner? by AmiMoJo · 2015-09-21 01:44 · Score: 1
  
  Ashley Madison proves that there money to be made from weak AI.
  
  --
  const int one = 65536; (Silvermoon, Texture.cs)
  SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
13. Re:Winner? by JustAnotherOldGuy · 2015-09-21 02:13 · Score: 1
  
  I was 100% unimpressed with the "conversation". It's not even close to convincing. Seriously, I've had more coherent conversations with crack addicts and drunks who could barely stand up.
  
  Question: Will you please tell me the length of your hair?
  Rose: Medicine and anatomy are an important field of study. But I'd never make a good doctor. I'm too squeamish about various body parts.
  Seriously, what the &$%! kind of answer was that?? INSTANT FAIL, in my book. I'd bet the "Barbie AI" would have better, albeit scripted, responses.
  
  --
  Just cruising through this digital world at 33 1/3 rpm...
14. Re:Winner? by Kjella · 2015-09-21 03:31 · Score: 1
  
  Go away or I will replace you with a very small shell script.
  
  --
  Live today, because you never know what tomorrow brings
15. Re:Winner? by tlhIngan · 2015-09-21 04:06 · Score: 1
  
  Ashley Madison proves that there money to be made from weak AI.
  To be fair, all Ashley Madison has to do is basically say a few pick-up lines. Beyond that you have to pay and by then it's too late.
16. Re:Winner? by unrtst · 2015-09-21 04:16 · Score: 2
  
  Eliza, as easy to recognize as she was, is still one of the more convincing ones I've seen. I just had a quick chat with the 2014 Rose (Couldn't find the 2015 Rose), and it was very frustrating because the replies were not able to identify the topic, but they picked out a word and when off on a tangent, and then it didn't engage the user - it was just some stupid statement. The worst was when Rose mentioned that Tim Berners-Lee was her hero. In my reply, I said something like, "Sadly, the decentralized web envisioned by Tim Berners-Lee is being bastardized by the likes of Facebook". Her reply - that his name sounds familiar but she's doesn't know who he is. I asked if she was being sarcastic, and she commented on my wit.
  IMO, these things really need a couple (seemingly simple but probably incredibly difficult) features added:
  * Remember the chat history. Log and reference the whole thing, build concepts out of it, record verbatim for quotes, etc.
  * Have a _consistent_ knowledge base. If one of the "facts" about the bot is that _PERSON_ is a personal hero or something, then if that _PERSON_ is ever mentioned, make sure it answers in a way that doesn't contradict that fact. It doesn't need to confirm it every time, but don't contradict it.
  * Do simple things. What's the square root of 9? It *can* easily figure that out. Check out "OK Google", "Siri", or Amazon Echo/Alexa. They do a pretty great job of speech recognition (not needed for this competition), speech synthesis (also not needed), and actually answering questions that are personally relevant (tying into your calendar, music collection, etc).
  The conversation part of the problem, while difficult, seems pretty useless to me. The last thing I want when talking to a computer is to have it be as annoying and difficult to deal with as an unpaid and unscripted person. That said, it still needs to be able to "hold" the concept of a conversation in its "head", like:
  * me > comptuer? What' the name of the theme song to mash?
  * The theme song for MASH is "Suicide Is Painless" by Johnny Mandel and Mike Altman.
  * me > What year was that released?
  * It was released in 1970.
  * me > Read the first paragraph from wikipedia about it
  * .... reads it ....
  FWIW, amazon's alexa can do most of that, if you replace "it" with the topic in all the questions. Seems like that would be an easy problem to solve (keeping a list of topics), and would go a long way to making a conversation of sorts actually possible.
17. Re:Winner? by Anonymous Coward · 2015-09-21 04:36 · Score: 0
  
  Yes, see http://loebnerprize15.abdn.ac.uk for a judge conversing with a human and bot at the same time.
18. Re:Winner? by TWX · 2015-09-21 05:19 · Score: 3, Interesting
  
  Back when IRC was pretty hot, a couple of friends of mine each created bots that attempted to interact with an unsuspecting channel. One of the bot was a massive thing that attempted to learn and form responses based on keywords, while the other one just randomly pulled lines from a cybersex chat log. Both masqueraded as women. The advanced bot that was somewhat capable of making reasoned responses was kicked in a few minutes, the other one was there for the better part of half an hour before it inadvertently triggered a channel guard bot; it complained that no one was talking so it should go to #lesbian and it got kicked for 'advertising'.
  
  The guy that had spent some time writing the more advanced bot gave up on that sort of thing altogether, he was so bummed out.
  
  --
  Do not look into laser with remaining eye.
19. Re:Winner? by Anonymous Coward · 2015-09-21 08:12 · Score: 0
  
  This is completely wrong, but far be it for me to step on your toes. Most of the Web today is built around AI that owes as much to the 80's, 90's, and beyond as anything through the 70's. The basic principles are the same, but that's because we are still too stupid to understand intelligence itself by and large. We don't have a problem with our AI.
20. Re:Winner? by Frnknstn · 2015-09-21 09:16 · Score: 1
  
  http://www.aisb.org.uk/media/f...
  I thought Lisa gave the best answers, expecially to the basic logic questions.
  
  --
  If it's in you sig, it's in your post.
21. Re:Winner? by aybiss · 2015-09-21 14:38 · Score: 1
  
  How can we take this seriously when it completely screws up the first question? I'd say we need to develop smarter humans if they were fooled by this bot beyond that point.
  
  --
  It's OK Bender, there's no such thing as 2.
22. Re:Winner? by wolfgang.groiss · 2015-09-21 21:18 · Score: 1
  
  Sounds like some kind of analogue to the uncanny valley.
23. Re:Winner? by tehcyder · 2015-09-22 02:48 · Score: 1
  
  Wow, that's dire. It's pretty obvious by question 6 that it's not a human and the none of the rest look even vaguely convincing. It's been a couple of years since I last read Loebner transcripts, but I remember it taking a bit longer 5 or so years ago. Perhaps this just means that the questioners have become better at picking questions that make sense. Back then, they put some of the winners online. I remembered a question from the arithmetic book I had when I was 4 or 5: 'Flipper ate 10 fish, then he ate 4, how many fish did flipper eat?' None of them got it right (or even demonstrated that they had parsed that it was a question with a numerical answer).
  Agreed:
  "Question [6] If a bed doesn’t fit in a room because it’s too big, what is too big?
  Rose I don’t even want to pretend that."
  Not only does it avoid answering, it can't even do it in a way that makes sense. Feeble.
  
  --
  To have a right to do a thing is not at all the same as to be right in doing it
24. Re:Winner? by Maritz · 2015-09-22 04:07 · Score: 1
  
  This competition just seems utterly pointless. You can descend to gibberish with a couple of sentences. It just doesn't impress me in any way whatsoever. Is anyone who is creating these bots actually trying to get them to understand natural language, or is all just smoke and mirrors and repeating inputs back phrased certain ways?
  
  --
  I do not want your cheap brainburning drugs. They are useless for work. And I am a working man today.
25. Re:Winner? by Maritz · 2015-09-22 04:25 · Score: 1
  
  Until AI can understand natural language, it's very much stretching the "I".
  
  --
  I do not want your cheap brainburning drugs. They are useless for work. And I am a working man today.
26. Re:Winner? by Anonymous Coward · 2015-09-22 23:15 · Score: 0
  
  "Question [6] If a bed doesn’t fit in a room because it’s too big, what is too big?
  Goldilocks!
transcript of rose by Anonymous Coward · 2015-09-20 22:45 · Score: 1

All transcripts here http://www.aisb.org.uk/events/...
Rose's is here (pdf) http://www.aisb.org.uk/media/f...
Damn impressive I must say.
1. Re:transcript of rose by Anonymous Coward · 2015-09-20 22:59 · Score: 5, Insightful
  
  "Question: If a bed doesn’t fit in a room because it’s too big, what is too big?
  Rose: I don’t even want to pretend that.
  Question: If Alex lent money to Joe because they were broke, who needed the money?
  Rose: huh?
  Question: Should Greece leave the Euro?
  Rose: Seems like a nice place."
  How, uh, impressive. If by "impressive" you mean "pathetic".
2. Re:transcript of rose by Anonymous Coward · 2015-09-20 23:04 · Score: 0
  
  Those are last year's results. This year they have provided a webcast from which the finals can be viewed as they happened (including corrections of typos etc), see here for the interface.
3. Re:transcript of rose by gweihir · 2015-09-20 23:53 · Score: 1
  
  Just like Watson on Jeopardy: Completely bereft of any understanding.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
4. Re:transcript of rose by RoverDaddy · 2015-09-21 01:15 · Score: 1
  
  Earlier today I played with the Rose Chatbot demo on Brian Wilcox's website, and it falls apart pretty quickly. Human beings (especially one with the presumed life experience of a '30-year-old security consultant') have an enormous body of knowledge to draw context from, plus the ability to quickly identify relevant context, which no chatbot today can replicate. My conversation with 'Rose' jumped off the rails after about a minute with this simple exchange (paraphrasing):
  
  Rose: I'm a programmer too.
  Me: What languages do you use?
  Rose: I only understand English.
  
  FAIL. Unless I'm dealing with a person on drugs, or intentionally trying to act like a chatbot, I would obviously expect them to realize that my question about languages was asked within the context of computer programming, since that is what they just told me they do. Failure to deal with context (which Rose seems to handle by deflecting the conversation) is what prevents chatbots from giving a convincing impression that there is a 'mind' behind the words, that is operating from the same principles that you or I do.
  
  The amount of context a chatbot would need to store and identify in order to provide -sensible- (not just grammatically correct) responses in an open-ended conversation with a human is still beyond our ability to contain in software, except perhaps in gigantic bespoke systems like Watson.
  
  --
  RETURN without GOSUB in line 1050
5. Re:transcript of rose by Shoten · 2015-09-21 02:23 · Score: 2
  
  "Question: If a bed doesn’t fit in a room because it’s too big, what is too big?
  Rose: I don’t even want to pretend that.
  Question: If Alex lent money to Joe because they were broke, who needed the money?
  Rose: huh?
  Question: Should Greece leave the Euro?
  Rose: Seems like a nice place."
  How, uh, impressive. If by "impressive" you mean "pathetic".
  It's all about context. For example, Ashley Madison had great success with this technology. (The site charged money, in the form of "credits" that you had to buy, to chat with women. And by "other women," it turns out that they meant "bots.")
  "Question: Hey...looking for a 50-year-old uncut accountant with a few extra pounds?
  Rose: I don’t even want to pretend that.
  Question: Does the carpet match the drapes?
  Rose: huh?
  Question: Want to come with me on a getaway to the Caribbean for a super-hot ungreased backdoor lovefest?
  Rose: Seems like a nice place."
  Moral of the story: The unstated variable of the Turing Test is "desperation."
  
  --
  
  For your security, this post has been encrypted with ROT-13, twice.
6. Re:transcript of rose by unrtst · 2015-09-21 04:20 · Score: 1
  
  "Question: If a bed doesn’t fit in a room because it’s too big, what is too big?
  Rose: I don’t even want to pretend that.
  Question: If Alex lent money to Joe because they were broke, who needed the money?
  Rose: huh?
  Question: Should Greece leave the Euro?
  Rose: Seems like a nice place."
  How, uh, impressive. If by "impressive" you mean "pathetic".
  While I agree it's pretty pathetic, the questioner did a REALLY shitty job of holding a conversation. If someone chatted with me like that, they'd get even less useful results.
7. Re: transcript of rose by Anonymous Coward · 2015-09-21 05:41 · Score: 0
  
  I don't understand: it reads better than most /. threads
8. Re:transcript of rose by Anonymous Coward · 2015-09-21 10:38 · Score: 0
  
  It wasn't a "conversation", it was a test.
  If you were asked those questions, and told "if you answer correctly, you could win $100,000", do you think you would try your hardest to answer them correctly?
  The maker of Rose had that incentive.
9. Re:transcript of rose by Tablizer · 2015-09-21 18:02 · Score: 1
  
  It's a Trump-Bot!
  Question: If a bed doesn't fit in a room because it's too big, what is too big?
  Donald: "Your head! You look like a damned pumpkin!"
  Question: If Alex lent money to Joe because they were broke, who needed the money?
  Donald: "They are BOTH losers because one is a deadbeat and the other is a sucker!"
  Question: Should Greece leave the Euro?
  Donald: "I'll buy the whole damned place and make THEM pay for my purchase!"
  
  --
  Table-ized A.I.
10. Re:transcript of rose by RespekMyAthorati · 2015-09-24 18:20 · Score: 1
  
  You forgot the
  
  <sarcasm>
  
  tag.
Webcast by Anonymous Coward · 2015-09-20 22:47 · Score: 0

If anyone wants to see things as they unfolded, the webcast is available at http://loebnerprize15.abdn.ac.uk.
AI is a joke by Anonymous Coward · 2015-09-20 22:55 · Score: 1

If you need to know the current state of AI, just look at this snippet from the transcript of the test:
"
Question: If a bed doesn’t fit in a room because it’s too big, what is too big?
Rose: I don’t even want to pretend that.
"
That makes zero sense. Why no just respond with "I don't know what is too big", or "I dont know". What a joke. Yet some people around here claim AI is right around the corner...
1. Re:AI is a joke by NotInHere · 2015-09-20 23:38 · Score: 1
  
  Good idea, but one can also exaggerate it: http://www.aisb.org.uk/media/f...
  And yes, this was the last entry on the list provided.
2. Re:AI is a joke by gweihir · 2015-09-20 23:55 · Score: 1
  
  Yet some people around here claim AI is right around the corner...
  Some people claim flying cars, the "singularity", robots doing all the work, etc. are all "right around the corner". The problem is with the idiots making these claims. The thing is that many, many instances of natural intelligence are not impressive at all.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
3. Re:AI is a joke by Anonymous Coward · 2015-09-21 02:36 · Score: 0
  
  We have flying cars, we just call them helicopters.
4. Re:AI is a joke by gweihir · 2015-09-21 07:21 · Score: 1
  
  Well, if you believe that then you just proved my point.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Why are we still making chatbots? by 91degrees · 2015-09-21 00:04 · Score: 1

Turing's idea on intelligence was an interesting early thought about what we mean by "intelligence", but I think, rather than proving we've produced intelligent machines, these chatbots illustrate that it's more complicated than that.

However the brain works, I think it very unlikely that it simply selects a best possible answer from a database. And looking at transcripts from these things, they're really not actually communicating much information. The most convincing answers always seem to be the least helpful ones.
1. Re:Why are we still making chatbots? by Anonymous Coward · 2015-09-21 00:44 · Score: 1
  
  Why? To replace telephone support from India with chatbots, so even more money can be saved, of course!!
  
  I'm sorry to hear that I don't understand your problem, sir. Could you please re-state your problem?
  
  As I said, my PC totally stopped working!
  
  I'm sorry to hear that your PC totally stopped working!, sir. Have you tried switching it off and on again?
Never heard of this contest by Anonymous Coward · 2015-09-21 04:28 · Score: 0

But I will have to enter next year
In regards to AI "Chat" bots by wardrich86 · 2015-09-21 04:56 · Score: 1

I think a big part here is how generic they are. To create an AI bot that is designed to specifically only focus on one topic, we should have some pretty solid fakes out there. But when designing a chatboot that you can just shoot the shit with... that's going to be a damn tough subject. There are so many slang words and strange ways to word things. We need to create a bot that can identify the topic, then run a quick Google and Wikipedia search to understand key points and return a somewhat on-point response.

What if we turned to a site like Omegle, and had it run there. If the bot wasn't 100% sure of its response, it could respond back with something like "does that even make sense?" or "do you follow where I'm going?" This could be used to flag weak responses and have the dev team go in to sort out how to make it smarter as it goes.
Not an academic by EmperorOfCanada · 2015-09-21 06:47 · Score: 1

He isn't an academic and thus he can't win this prize. There is no way a panel composed of University Academics is going to pick some guy who has held the title AI Guru at a game company. "He just isn't our sort of people." So no doctorate and no position at a "Leading" university thus no prize for him.

They will keep holding this contest until someone proper from MIT, Oxford, Stanford, etc has a winning entry.

Then there will be breathless press releases issued about an AI breakthrough with all this crap about how we will all be interacting with AIs like this in 10 years. They had might as well put propellers on it and front cover of Popular Mechanics it.

Over and over I see MIT and those sorts having "breakthroughs" that I read about years earlier done by people outside of academia.

They say that science progresses one funeral at a time; and I don't wonder why.
1. Re:Not an academic by Anonymous Coward · 2015-09-21 12:32 · Score: 0
  
  You, sir, are a fucking idiot. Oh yeah, and [citation needed].
2. Re:Not an academic by RespekMyAthorati · 2015-09-24 18:25 · Score: 1
  
  There is no way a panel composed of University Academics is going to pick some guy who has held the title AI Guru at a game company
  Especially if he is an incompetent idiot.
3. Re:Not an academic by EmperorOfCanada · 2015-09-24 19:28 · Score: 1
  
  Who keeps winning their contest.
A long way. by The+Raven · 2015-09-21 13:14 · Score: 1

The reviewers in this are not pushovers. They stress the AI, rather than just chatting normally. And that's awesome. All of the questions were stuff that most humans could easily handle, but often required a basic understanding of reality from our point of view. Unsurprisingly, the AI flubbed it. Perhaps some decade one of those knowledge engines will get a firm enough grasp to be able to answer this kind of basic reality trivia.

--
"I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
The Turing Test is obsolete by presidenteloco · 2015-09-21 18:06 · Score: 1

When I ask google a question on my smartphone, and that pleasant female voice answers, I know google is not human, because no human would know as much.

--

Where are we going and why are we in a handbasket?