Slashdot Mirror
Turing's Original Test Played First Time Ever
aykroyd writes "Students at Simon's Rock College conducted the original test that Turing suggested in his 1950 paper, Computing Machinery and Intelligence. Often misunderstood, the Turing Test has never actually been conducted as laid out in his paper. The experiment utilized a program called A.L.I.C.E., which is designed to hold one end of an interactive conversation. The program was provided by the ALICE Artificial Intelligence Foundation. Dr. Richard Wallace, who was on hand during the experiment to troubleshoot the AI robot, later gave a lecture about it called "The Anatomy of A.L.I.C.E." and also blogged the event."
Althought I followed the blog link, I failed to find a record of the actual test like I was expecting.
If someone else finds it, I would appeciate a more detailed (direct) link.
We had an article about these AI programs, (ALice and about 30 others) last year that were competing with each other. I looked at a few, and they didn't seem anywhere ready-for-prime-time yet. Still the same 'turn the question around on the asker' type of engines. What was the difference here? The type of questions, the 4-people/1comptuer setup?
While Turing's work in the area of AI was certainly revolutionary for its time it is now generally considered by experts that his test is not an effective way to establish if an agent is intelligent. For a start there are domains beyond chatbots that we can say require proper intelligence such as vision and planning that are totally not addressed by the Turing test. That doesn't even get into the whole Searle's Chinese Room thing (whether you buy the argument or not.)
And it was proposed by a gay man, too. Turing had some preconceptions on which would be a more difficult test -- I think he assumed some traditional stereotypes about women being more emotional or social than men, meaning that imitating a woman should be harder than imitating a man.
For an amusing discussion of the Turing test and gender, see Douglas Hofstadter's "A Coffeehouse Conversation on the Turing Test" -- it's in Metamagical Themas.
I think its interesting that we think intelligence can be gauged by what we (or the AI) say in normal conversation.
I'll believe a computer is intelligent only when it acts of its own volition, not when it merely tricks a person into thinking it has acted of its own volition.
That seems to be more of a test of how gulible the person is rather than how intelligent the machine is.
You listening computer? You want to impress me, do it without me punching your buttons.
Read any good sonnets lately?
From the link, got to love this conversation.
Strangely, it seems only to willing to broadcast the virtues of Lynx.
The idea is that a computer is intelligent if it can hold a conversation with a human such that it is indistinguishable from a conversation with a real human.
RIDICULOUS.
Have you ever actually tried talking to one of these bots (including ALICE)? It is very easy to know that you're not talking to a human. Exceptionally easy. The Loebner Prize judges consistently grant the bots handicaps, acting as if they're actually being fooled. Obviously they're not, and the AI community just wants people to think that it's more advanced than it really is. Unfortnately, some members of the public *are* fooled by that.
The problem is in the Turing Test itself. It assumes that the measure of intelligence is humanoid conversational ability. I strongly disagree with that. Conversation ability is no measure of intelligence. Just for an example, I am exceptionally intelligent (statistically), but I am a poor conversationalist. Casual small-talk has always bewildered me. If I entered myself into the Loebner contest, they might think I'm a bot. Hell, ALICE might accuse me of being a bot.
Anyone who's taken an IQ test will recall that every last question has something to do with pattern recognition. You'll also recall that you were not asked to respond to any conversational questions. That's because invariant pattern recognition abilities (in a loose sense -- this also includes memory/learning and inductive reasoning) are the true mark of intelligence, and this is nearly undisputed. If they really want to test how intelligent a program is, they need to test its patern recognition ability.
Take this program -- http://www.stanford.edu/~dil/invariance/ -- for example. It's gone largely unnoticed, yet it is concrete proof of a huge breakthrough in computer intelligence. This is a little Matlab demo of a very abstract multi-layer intelligence algorithm. In this particular implementation, it is taught a set of small images. Then you can play "Pictionary" with it, drawing shapes and have it recognize them. You may say that this is unremarkable, that shape-recognition is a trivial algorithmic matter unrelated to intelligence. But the author noticed that he could draw shapes "incorrectly" -- like, the little duck picture, except with its head missing, or alphabetical symbols rotated or flipped -- and the program still recognized them. (It failed a few times, but in situations where the shape is so mangled that I would have probably failed too. How's that for a Turing test?) And this program's genius lies in not what it does, but how it does it. All of its functionality is completely abstract. It is a pattern recognizer, not a bitmap-tracer, and there are no hard-coded routines for checking if the image is flipped, rotated, etc.
This is what Palm/Handspring founder Jeff Hawkins (also the founder of new neuroscience startup Numenta, http://www.numenta.com/) calls "Real Intelligence," to distinguish it from the failed Artificial Intelligence effort. He feels that the right way to make computers intelligent is not to have them outwardly imitate human behavior, but to internally function the way the mind really works. Anyone interested should check out his book, On Intelligence http://www.onintelligence.org/. You'll wonder why you ever believed the AI hype.
Artificial Intelligence is a sham, by its very nature. Real Intelligence will be the way of the future.
Signature.
I was actually interested until I saw A.L.I.C.E. - like that chatbot will ever pass a turing test.
To be fair, this was a high school project and so I can see how a suped up chatbot would fit the bill for this "Turing test".
So no... I didn't RTFA thank you very much.
But for those interested in reading up on Wallace, A.L.I.C.Es author... Well he's a first class nut job
RIDICULOUS.
Nonsense.
The Turing test is really a very good test as it does not rely on any predefined notions of how intelligence works (as you do in later paragraphs), but on an operational test. Furthermore, it is not a measure of conversational ability (per se) so much as a measure of how well the program can seem to be human. (And, to respond to a further statement in your post, it is a pattern recognition system (as well as a pattern generation system)).
Ultimately it depends on a limiting definition, but one that is very reasonable - that the only "system" that we know that is intelligent is human, that essentially all humans communicate verbally (one way or another) and that therefore, such communication is probably an intrinsic part of human intelligence. Is this likely to be a "universal" definition of intelligence? No. But we can certainly reframe the Turing test (or discard it) when (if) we encounter alien intelligences that helps broaden the definition. Of course, if they can't (or won't) converse with us (somehow) are we even likely to notice them and believe they are intelligent?
That the Loebner contest is not a good example of a Turing test does not change the underlying validity of the test itself. Just as a rigged (for instance) footrace does not invalidate footraces in general.
The fallacy - that intelligence in programs be measured by "how it works" - is relatively common among AI researchers many of whom decide that their own particular interest is an essential ingredient in how an intelligent program would work and thus rule out anything that does not work that way.
It's not associated with artificial intelligence? Then why, if I go to the ALICE website, do I see a big banner declaring "ALICE Artificial Intelligence Foundation," and why do they call their language "Artificial Intelligence Markup Language"?
"Pattern recognition" is vastly different from blind "pattern matching." Any computer program can easily do "pattern matching" -- that's how Alice works, as you yourself admitted when you mentioned that it's an if-then sequence based on preprogrammed responses. That's nothing much more advanced than regexes.
When a human is born, he or she knows nothing of the outside world, let alone of language or vision, yet can learn to speak and recognize things within a few years, just by the process we call "learning." And whether or not ALICE can "learn," it would not be easy to teach it a language from scratch. So "pattern recognition" is the term I used to refer to human-like intelligence. This has no preprogrammed responses to anything. Maybe ALICE is good at chit-chat, but I think we can also agree that such is the least intelligent form of human communication. Anything more substantial than that can't be based on prefabricated statement/response sequences.
Pattern recognition, in this sense, requires many abilities. It requires the basic ability to observe and store patterns, but most importantly, it requires those patterns to be invariant. (See Dileep George's neuroscience research for an example of what is meant by "invariance.") And of course, it requires the ability to perform inductive reasoning on those stored patterns and on new input in order to infer information. No "chatterbot" program has ever come close to any of these.
Signature.
The idea is that a computer is intelligent if it can hold a conversation with a human such that it is indistinguishable from a conversation with a real human. RIDICULOUS. Really. I've met way too many people that have so much trouble stringing together a sentence over the phone that they would fail a turing test.
Human: Who is the Pope?
ALICE: he is John Paul II.
Human: You are wrong.
ALICE: No, I am right.
Human: You are wrong, because the Pope is Benedict XVI.
ALICE: No, I am right.
Rather stubborn isn't she?
Maybe you should educate the morons of tomorrow so they'll stop believing the leaders of tomorrow. - Dogbert
from http://jabberwacky.com/
Me: That doen't make sense.
Jabberwacky: You don't make any sense.
Me: Sorry.
Jabberwacky: It's ok, you are doing your best.
Me: No, I am sorry because now all your base are belong to us!
Jabberwacky: Don't be sorry about that, embrace it!
Me: Thank you, I WILL!!
Jabberwacky: Stop talking about it and do it.
"Some of the worst mistakes in my life have been haircuts." - Jim Morrison
I'm writing a philosophy paper which covers, in part, what intelligence is. I would say, in my completely unprofessional but well-thought opinion, that the measure of whether something is "intelligent" - or more precisely, whether something "thinks" or perhaps is "a person" (as distinguished from "a human") - is dependent on four key criteria:
Observation/Stimulation ("Senses")- the ability to receive input from the surrounding world and react to it, and to select from / focus on / filter various available inputs. All animals and some AI programs share this in common. It's the basic stimulus half of a 'stimulus-response' system.
Communication/Behavior ("Society") - the ability to interpret signals sent to it, and in kind, to transmit appropriate signals (be these electrical, audio, movement, etc). Again, animals and many computers have all got this down just fine - ALICE is doing something like this, and perhaps a little more. This is just the 'response' part of a stimulus-response system.
Intuition/Expression ("Emotion") - Pattern recognition, as you've already covered here, and the internal state-change that corresponds to it. This is one of the two criteria which differentiate a thinking entity like a human from an unthinking one like most animals or computers. It is built effectively upon recursive, internal observations and self-stimulation; observing things within oneself and behaving to achieve change in those things. In this sense, emotion is half of the criteria needed for a truly self-aware system.
Logic/Expectation ("Reason") - Symbolic manipulation, memory, and the ability to make predictions. This is the ability to manipulate symbols and link them into relationships, from which a sort of causality or rationality can be derived (if this then that; this, therefore that) to make theoretical predictions. This is built upon recursive, internal communication - in essence, talking to oneself, an internal dialogue. The ability to communicate propositions to oneself and evaluate them. This internal narration is also the basis of meaningful memory, one's personal story. As an internal, recursive process, like emotion, reason is the other half of self-awareness, and necessary for an entity to be truly a thinking thing, or a person.
All four of these criteria (really eight, if you count the dual aspects of each) may vary in their strengths within an individual person, but so long as all four are present in some degree, the entity in question could be considered a person.
The key element in all of this is recursion, the internalization of normally external stimulus-response systems. The stimulus-response must still be present (otherwise you'd have an inert box), and feeds into the self-aware part of the system, but the self-awareness is what makes the system "intelligent".
A simple stimulus-response system will always give the same responses for the same exact stimuli, but a self-aware system will change internally with every stimulus, and change in turn with every sensation of its own change, and so have a continually changing maps of certain stimuli to certain responses, the makeup of which is only very indirectly caused by the actual stimuli themselves.
This dynamic nature and self-evolving, and the fact that much of the cause of change is hidden 'inside' the system and only hinted at in its responses to certain stimuli, makes predicting the behavior of such a system extremely difficult, lending the illusion that it is free from causality and has its own "free will" or "volition" somehow separate from the rest of the universe.
But, don't take any of this as absolute truth. I'm not a programmer or a neurologist or in any way a professional about any of this. I'm just a self-aware little box sitting here running thoughts recursively through my mind until something that appears to be true comes out...
-Forrest Cameranesi, Geek of all Trades
"I am Sam. Sam I am. I do not like trolls, flames, or spam."
You miss the point, my young apprentice.
The Turing test was devised in the days of WWII. Remember Eugenics? Saving the gene pool?
If we can devise automatons that can emulate humans in functions like retail clerks and floor sweeps, we can *finally* euthanize/sterilize the half of the population with IQs below 100. Just think, a world without country music, free from rap too. We could balance the federal budget once welfare was eliminated....sigh... where have all the facists gone anyway?
"Sic Semper Path of Least Resistance"
Alice: Hiya! Me: You're in a desert walking along in the sand when all of the sudden you look down, and you see a tortoise, it's crawling toward you. You reach down, you flip the tortoise over on its back. The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't, not without your help. But you're not helping. Alice: Wadda ya mean I'm *not* helping it?!?!!??!?!/!!? Me: I mean you're not helping it.
Well, there are very few INATE differences between males and females, but the number of differences that comes out due to social learning and pressure is immense. So, yes, in a more sexist age this test would have been far more predictive than it is now. One could also foresee a time where this boy/girl test would not even be applicable.
An interesting corrolary test would be having groups of scripts used by the program written by males, and other groups written by females, and comparing the differences. You would probably want to have the person taking the test choose the female (or male... whatever) in all of the following scenarious: male/female, male/computer, male/male, female/computer, female/female just to get some baseline. Then you would also have "I can not tell" as an answer as well.
But then again, I'm not a computer scientist or a social scientist, so what do I know.