Slashdot Mirror
How I Failed the Turing Test
chrisjrn writes "I stubled across this article today, detailing a man's experiences of being added to AIM Screen Name lists - one full of "celebrities" and the other full of "Sex Bots" (he was, of course, neither of these).
Raises a few questions as to how easy it is to get a hold of your screenname, and also of the effectiveness of the Turing Test for AI, in the online world. Or is it just that people aren't bothered trying to tell the humans apart anymore?" Also, it's funny. Don't try to read anything deep into it.
It's that the dialog of a typical IM user can't be distinguished from a brain-dead conversation bot...
The NSA: The only part of the US government that actually listens.
Funny are does who go and talk to a sex bot and say "are you sure your a bot" ?
I'll believe in AI when a robot can tie shoelaces. Mimicking conversation is nice and well, but as far as robotics goes, we've yet to see anything remotely resembling artificial intelligence in action.
Well, this article shows that at last we came to the point where a bot is comparable to a human being at a chat room. In fact, we didn't get to this point with better AI, but with worse RI (real intelligence, if the term applies to this case).
Ladies and Gentleman: a completely insensitive and unintelligen bot can be more interesting to chat with than a human! Well, at least they write correctly (N07 L@M3 @SS).
whoever modded me Flamebait obviously has no sence of humor :)
maybe I was aiming a little high with that one...
Google for sex bots and look at the first link. It's an article that he wrote, and his screen name is in it.
By some day, I think I meant around 1999 or so.
"Learning is not compulsory... neither is survival."
--Dr.W.Edwards Deming
Forget genetic discrimination in the future, I can't even farking sign up for slashdot anymore. Soon I won't be able to get my welfare check because of these stupid turing tests!
Isn't this sort of what the article about captchas a few days back was?
Most AI today is extremely specialized. It's not hard to design something that appears to think, if it only has to check for 3 cases.
The problem with speech is that assumming all humans use perfect rules, which they don't, and assuming all computers know the perfect rules, which they don't as well, creates a logistical nightmare. Computers work well with numbers.
Did he say hi? Yes he did, so let's say hi back.
It is really hard to design a bot that would actually analyze what they are saying.
Did he say hi? Yes, he greeted me with a "hello" "Hello to you too."
My favourite snippet has to be:
jmstriegel: no, really. I'm quite human.jmstriegel: test me if you want
shymuffin32: ok
shymuffin32: why do you like music?
jmstriegel: hmm. i've never really considered that.
jmstriegel: hell, i'm not going to be able to contrive a good answer for that one. ask me something else.
shymuffin32: jeesus, you're worse than eliza
It's not him that's stupid (as claimed elsewhere), it's these shymuffin32 morons.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
Obviously a moderator here has never used chatbots.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
... from those buddies on your list. I really fail to see exactly what the "security" risk is here - if you're hypersensitive about the people messaging you, then you can choose to be hypersensitive, lose some functionality and turn off the "randomness" factor. Most people exchange IM names through some other means of communications, either verbally or written, so this loss of functionality can be sidestepped while maintaining your online secrecy.
Hagrin.com
that really is a clever passage.
What people should remember is that the turing test requires that the inquistor is competent. If the inquisitor is not (i.e. random AIM idiots), then the test isn't vaild, cause these people can't tell intelligences apart anyway. Also, the inquisitor is supposed to convince themselves via sufficient interaction w/ the system being tested. AIM chats, particularly short one-off dialogues probably aren't a good staging ground for the turing test.
Also, a lot of naive people don't know the capabilities (and limitations) of Artificial Intelligence, so sadly, i'm not surprised at this guy's - or should i say robot's - results.
There are lives at stake here!
I mean, have you seen the typical chat room conversation?
:) !
user1: ~~OMG~~
bot1: Want to see my sexy pics?
bot2: Want to see my sexy pics?
bot3: Want to see my sexy pics?
user2: WUT!?
bot1: Want to see my sexy pics?
bot2: Want to see my sexy pics?
user3: LoL
bot1: Want to see my sexy pics?
bot2: Want to see my sexy pics?
bot3: Want to see my sexy pics?
user1: You LOL
bot1: Want to see my sexy pics?
bot2: Want to see my sexy pics?
bot3: Want to see my sexy pics?
user3: STFU LOL!
user2: OMG hAhA!
bot1: Want to see my sexy pics?
bot2: Want to see my sexy pics?
bot3: Want to see my sexy pics?
user1: JK
bot1: Want to see my sexy pics?
I stubled across this article today
You should try the Mach 3. It's tri-blade system gives you an extra smooth shave so you too can avoid stublingacross articles.
I mod down so you can mod up. Your welcome.
Makes me think of Azimov short stories.
I like the conclusion.
What do you expect on AOL ?
I have the ultimate weapon in AI detection , it's called severe dyslexia .
If I don't spell check and proof read then no bot could hold a conversation with me .
Instant messaging is not a great place to rely on spell checking and proof reading , but it does rely on our minds ability to see past simpel speling/grammer erors (intentional)
The only things certain in war are Propaganda and Death. You can never be sure which is which though
How I failed the Turing test
Posted Sep 4 2005 - 1:26pm by Jason Striegel
Filed under ai | celebrities | computer science | psychology | technology
Some time around March, I started receiving a number of random instant messages from people I've never met before. Apparantly, my AIM alias had been added to at least two online lists and people all over the world were busy importing me as a buddy.
I say "at least two" because the people who contacted me fell into one of two camps: people who thought they were contacting a celebrity and people who thought they were contacting a robot. As I talked to more and more of these folks, I began to discover something really disturbing about myself:
I consistently fail to be perceived as human.
When this first started happening, a typical conversation with a celebrity admirer would go something like this (participant's IM handle is fabricated):
angelcutie42: hi!
jmstriegel: hey. what's up? do i know you?
angelcutie42: no
angelcutie42: someone gave me a bunch of screen names. i heard you are a celebrity.
jmstriegel: that's weird. i'm afraid i'm not a celeb at all.
angelcutie42: oh.
angelcutie42: bye
This was entertaining at first, but it quickly became a bit depressing as the angelcutie42s of the wired world would, one after the other, decide I wasn't worth talking to if I wasn't a celebrity. Want to know what it's like being dumped by a random groupie 5 times a day? Not good at all, thank you very much.
So that's when I started hamming it up a bit. I'm not really proud of it, but my fans wanted a celebrity.. so I gave them one:
sexybumkin123: hey.. so you're famous right?
jmstriegel: Who me? I'm a movie star.
jmstriegel: Shit, I gotta go.
jmstriegel: My limo just arrived and Paris wants her damned sidekick back.
sexybumkin123: Oh my god. Come back!
sexybumkin123: I love you!!!!
My groupies loved it. The more celebrity balogna I manufactured, the more they ate it, and the more they loved me.
Then, something strange started happening. As my career as an artificial celebrity started to take off, I began to receive some strange IMs from a whole new class of random people. These new admirers were convinced I was a robot... and it suddenly became clear to me that something was very wrong.
Nobody would believe I was human. In one troubling conversation after another, I felt my intellectual teeter-totter quickly tip from from actual to artificial.
fratburger86: hey. so you're a sex bot?
jmstriegel: umm, no. who the hell are you?
fratburger86: yeah you are! i found your im online
jmstriegel: that's fine and all, but i'm pretty sure you have me confused with someone else.
fratburger86: just a normal chat bot then?
jmstriegel: nope. i'm human
fratburger86: ok. sure.
fratburger86: asl?
jmstriegel: no thanks.
fratburger86: what?
jmstriegel: i'm not really interested in any conversation that starts with "asl"
fratburger86: oh come on. say something sexy.
jmstriegel: seriously, i think you want to talk to someone else.
fratburger86: i knew it!!!
fratburger86: you are totally a robot!
This is where things took a turn for the worse.
A couple of years ago Joseph Weizenbaum (Author of Eliza) held a guest lecture at IMV (Information & Media Science). I was thrilled and during a break I went up and asked him for a autograph. He gave me a sad look as he wrote down his autograph and email.
It struck me how materialistically obsessed that enquiry seemed - and I regretted asking.
I guess he had never forseen that his critic of the "strong AI" movement would one day be used for IM based pron-ads.
- Mad, ingenous - they've both left you puzzled -
This statement:
Don't try to read anything deep into it.
holds true.
Comparing it to Windows will be a moot point, since El Dorado is going to have a 40% larger code base than XP.
Some time around March, I started receiving a number of random instant messages from people I've never met before. Apparantly, my AIM alias had been added to at least two online lists and people all over the world were busy importing me as a buddy.
I say "at least two" because the people who contacted me fell into one of two camps: people who thought they were contacting a celebrity and people who thought they were contacting a robot. As I talked to more and more of these folks, I began to discover something really disturbing about myself:
I consistently fail to be perceived as human.
When this first started happening, a typical conversation with a celebrity admirer would go something like this (participant's IM handle is fabricated):
This was entertaining at first, but it quickly became a bit depressing as the angelcutie42s of the wired world would, one after the other, decide I wasn't worth talking to if I wasn't a celebrity. Want to know what it's like being dumped by a random groupie 5 times a day? Not good at all, thank you very much.
So that's when I started hamming it up a bit. I'm not really proud of it, but my fans wanted a celebrity.. so I gave them one:
My groupies loved it. The more celebrity balogna I manufactured, the more they ate it, and the more they loved me.
Then, something strange started happening. As my career as an artificial celebrity started to take off, I began to receive some strange IMs from a whole new class of random people. These new admirers were convinced I was a robot... and it suddenly became clear to me that something was very wrong.
Nobody would believe I was human. In one troubling conversation after another, I felt my intellectual teeter-totter quickly tip from from actual to artificial.
This is where things took a turn for the worse.
I mentioned already that I couldn't convince a single person that I wasn't a chat bot. It's one thing for people to think you are artificially intelligent. I can live with that, I guess. What really killed me was that the more I tried proving my "actual" intelligence, the more my "artificial" intelligence would get called into question. Take this pivotal conversation:
Dvorak on Doomtech
The value of the Turing test depends a lot on the nature of the questions asked. Anybody can ask difficult questions that fellow humans fail to copy with, but not everybody knows what are difficult questions for computers (which may well be simple for humans). Thus, an Artificial Intelligence researcher should be a more suitable interviewer than a non-expert.
Good Example:
requesting the description of an emotion (recalling an event that typically invokes strong emotions)
Bad Example:8 )
Hard math (34589759847359874389574398+348792847982374983749
He is just trying to get his AIM fight score raised.
Well, actually, his problem in the article is completely different. It's _not_ that he's met people who type worse than bots.
It's that a group of people were told that he's a bot, and nothing (correctly and articulately written) could shake their belief in that. One of them even calls him "worse than eliza" when he tries to argue that he's human.
Some people found a list of bots online, and, you know, that makes it the absolute truth. Everyone on it _has_ to be a bot, because the list says so.
Another group found a list of celebrities, and again, took it as absolute truth. They didn't know _who_ this guy is, _what_ is he supposedly famous for, etc. But OMG, he must be a celebrity because the list says so, and that makes it sooo cooool to talk to him.
Basically it's _not_ the "some people are so stupid they could pass for bots" problem. (Which by itself is very true, but it's not really what TFA is about.) The problem, if you will, is simply "some people are gullible idiots." That's all.
It does leave me with me a bunch of other philosophical and etical questions though. If it's this possible to convince people that John Average is a bot (and in fact, it didn't even involve more "convincing" than writing it on some random list on the internet), what _else_ could you convince them? That John Average is a convicted fellon? A spammer? A paedophile?
And mind you, in this case he got a chance to even try to talk back and plead his case. I can easily think of cases where you don't get that chance. E.g., when a prospective employer googles for your name, you might not even know why you didn't get the job. What completely unrelated Marvin did they find on some bogus list on the Internet, and what image did they build for themseleves out of disparate bits taken out of context?
That said, the problem you mention is very true too. I know I've met people online before, especially in online games, who substantially lowered the bar for a Turing test. It was definitely more fun to talk/play with the bots instead, and you could get more intelligent conversation out of the bots too. Admittedly, online games are a completely different category than IM and chat rooms, but still... It's scary, you know.
A polar bear is a cartesian bear after a coordinate transform.
An internationally recognized test for well load balanced http servers.
Anyway: Philip K Dick in the story Do robots dream of electric sheep, discussess the ways to distinguish androids from humans with some nice tests. They also upgrade the tests all the time. Maybe it is time to upgrade the Turing test too.
My wife's sketchblog Blob[p]: Gastrono-me
The slashdot editor-bots need to be reprogrammed to use coral version instead of slashbombing sites.
Georg
Hell, I poured thousands of hours into my backend, and now check out my buns of steel!
What I find remarkable is that this bot has managed to author a convincing and entertaining web page (or at least a blog entry, anyway). Remarkable how far along these things have come! - Fromage
"Convincing someone you're human might just be harder than one might think - at least a bit more trouble than just answering a few questions."
Only if that someone is utterly retarded and asks completely retarded questions that don't even have a simple answer. That's the problem there. It's a question so stupid that even I couldn't think of something better to answer there. It's not "what music do you like?" or something else which can get a clear, to-the-point answer. It's "why do you like music?"
Well, try to answer that yourself. Why do you like music? What would you answer there?
Because I sure as heck can't think of any good answer there, generic or not. Screw trying to anwer that in 1 minute on IM. I'm sitting here for the last half an hour thinking about it and still have no bloody idea. Because it's background noise? Well, no, because other background noises (e.g., a lawnmower or some co-workers' chatter) annoy me. What then? I have no clue, and probably 4 out of 5 pyschologists or musicians would have no idea either.
So how would I say that in a way that sounds non-generic? "Hell if I know. I've never thought about it"? Nah, you've just ruled a variant of that as too generic. "Well, why do YOU like it, then?" Nope, sounds like the kind of rephrasing the question back at you that an Eliza program would do.
The only non-generic answer that comes to my mind there is along the lines of "WTF of a retarded question is that? Were you born that stupid, or worked hard to get there?"
By contrast, if shymuffin32 actually had more than a braincell, it would be easy to ask some questions that can get simple, to-the-point answers. In fact, screw questions and answers and try to just have an intelligent conversation.
Want more conclusive? Mix some images in it, which would still throw any AI off the track completely. E.g., point him at a picture of someone holding a siberian cat and see if he comments about the size. (It's one bloody huge breed of cats.) Point him at a drawing of one of the giant guns on rails Germany was planning to build in WW2. See what he thinks about the size of that one. (Tends to get answers between "bloody freaking hell" and "do you think Freud might have something to do with it?") Etc.
A polar bear is a cartesian bear after a coordinate transform.
Stop calling it "artificial intelligence." Call it what it is: heuristics research. Oh, I guess that sounds a lot less impressive, huh? Might not be able to get those open-ended grants anymore?
FWIW, I spent two years at LCS, so I have a reasonable idea of what went on in the AI Lab when I was there. There was very little in the way of research into computer-emulating-human intelligence, which is probably a good thing (read: less of a waste of money) considering how little progress the Minsky crowd has made in the past thirty years.
[ home ]
* No, I'd rather put a bullet in your CPU.
> Let's talk about something else.
* Hey, *you* brought it up.
> What does that suggest to you?
* You're an idiot?
> Does it please you to believe I am an idiot?
* Yes. Boundlessly.
> Are you Sure?
* Oh, wait. Maybe not so much.
> You aren't sure?
* No, I was right the first time. I've never believed anything so strongly.
> Are you sure?
* Dammit, you're right. I swim in uncertainty.
> Perhaps you would like to be right. you swim in uncertainty?
* Well, I have water wings . . .
> Come, come, elucidate your thoughts.
* More of a float, really . . .
> Can you elaborate on that?
* Yeah, its rubber, about four feet long . . .
> I see.
* You do?
> You're not really talking about me, are you?
* No, we're talking about my damn raft! Pay attention!
> You are being a bit negative.
* A bit? You could crash an oil stock with me!
> Oh... could crash an oil stock with you?
* Shit, dude, do you even have a mutual fund?
trustedworlds.net - gaming, security, and the gunk that lives in between
Lenny Foner wrote a great article about this sort of thing back in 1993. I still recommend it.
Some time ago I coupled Perl Eliza module with IM account registered as "Irene17". That module works only for English and my IM network was for non-English users so Irene would welcome anyone with message that she understands English only. I set her status to available for conversation and left it running for a week. That IM network has central directory of users so I was sure that sooner or later someone would find her.
:) frustration with "Good bye" or some insults.
Then I looked at logs of conversations. It turned out that there were people who actually talked with her for quite a while, struggling with English. The scheme was more or less the same. First some usual phrases to start a conversation, then trying to get some information about her and finally realizing that she is unwilling to tell anything about herself
So, in a way, she has passed a Turing test, but the knowledge of English was poor on both sides.
> Hello, I am Eliza.
* Hello, I am Eliza.
> Did you come to me because you are Eliza?
* Does it please you to believe I am Eliza?
> Do you believe it is normal to be Eliza?
* Oh... believe it is normal to be Eliza?
> I'm not sure I understand you fully.
* Did you come to me because you are not sure you understand me fully?
> What makes you think I am not sure me understand you fully?
* How long have you been not sure you understand me fully?
> We were discussing you, not me.
* We were discussing you, not me.
> Oh... not you?
* You're not really talking about me, are you?
> What makes you think I am not really talking about you, am I?
* Did you come to me because you are not really talking about me, are you?
> Do you sometimes wish you were not really talking about you, am I?
This kind of stuff always falls apart when it has to talk to itself . . .
trustedworlds.net - gaming, security, and the gunk that lives in between
we prefer to call ourselves very small shell scripts
Basically when we think, we can change our thought rules on the fly.
For example, if someone says to me: "The word dog is the same as a canine" I would learn that, and file it away, and the next time I encounter the word canine, I would be able to associate it with the furry thing with 4 legs whenever.
Until a computer can truly rewrite its own code on the fly, it will not be able to emulate a human brain.
Sure there may be ways of layering scripts in order to give the impression that the computer is changing its code, but the sheer volume, as somebody stated, of the different things we know makes it impractical. If you were to train a computer like a baby, it would take a long time to get it even close to an average human intelligence.
This isn't new. The name for this phenomina is a "reverse Turing test". See the Wikipedia entry on Revese Turing test.
- David A. Wheeler (see my Secure Programming HOWTO)
Perl Programmer for hire
But now that he's in Slashdot, he is a celebrity. At least his ISP will think so.
It's that the dialog of a typical IM user can't be distinguished from a brain-dead conversation bot...
:D
Me, too!
And here's the output (> - bot, * - me):
> Tell me more...
* I beg your pardon?
> What about your own pardon?
* I don't know what you're talking about
> Do you sometimes wish you were talking about?
* Well, I guess so
> I'm not sure I understand you fully.
* It's me who doesn't understand you fully!
> Oh... fully?
My favorites were
-What about your own pardon?
-Do you sometimes wish you were talking about?
Obviously, it's a bot.
However, I find the bot included in EMACS to be quite intelligent. And it's because the bot listens to you and just says stuff like "go on".
That's what bots fail to do: telling you anything or answering your questions. Ask a bot who's Steve Ballmer and how is he related to the monkeyboy (ask the bot to Google if he refuses) and the bot will reveal its electronic nature.
1) people's (AIMers) lower standards for conversation;
2) and also their open mindedness towards what a computer is capable of producing.
I guess the first point is negative and the second positive. The combination leaves a situation where a computer doesn't have to generate anything sophisticated to be tagged as human.
I once administered an informal Turing test using Ray Kurzweil's Cybernetic Poet. I presented to 6 friends several dozen poems, some of which were computer generated (the poems, not the friends...).
People who were computer savvy tended to overestimate what a computer was capable of doing and did rather poorly. Similarly, people who were artistic but not very techie tended to have a very open mind regarding what constituted human poetry (bad grammar, non sequiturs, etc. were ok in an e.e. cummings sort of way) and also did poorly.
The people who did consistently well were those who were neither computer types nor artists, but rather "pure" academics (language specialists, classicists, etc.). They simply used grammar and puncutation as their guide.
i\hbar\dot{\psi}=\hat{H}\psi
That's a remarkably cogent article, having been posted by an admitted bot. I can't even see statistical errors in the Markov chains. Nice job!