Slashdot Mirror
Turing Test 2: A Sense of Humor
mhackarbie writes "Salon has a great story, Artificial Stupidity, about the Loebner Prize, a yearly contest that for over 10 years now has offered a $100,000 prize to anyone who can create a program to pass the Turing Test. The best part is the resulting fiasco that develops between the eccentric philanthropist who started the contest and extremely annoyed AI Researchers such as Marvin Minsky."
I don't think bots are the problem... I've had several online conversations which I'd assumed were chat-bots but turned out to be real people. I guess when Turing designed his test, he probably didn't anticipate the massive advances in human stupidity that we've witnessed in the last few decades :)
-- Open Source: It's mad, but you don't have to work here to help.
Every person with whom I spoke about it said that last year's contest was an utter fiasco, with unclear rules, inconsistent judging, arbitrary fiats by an opaque prize committee, petulant prima donnas, and last-minute changes of venue that prevented most entrants from even discovering where the contest was taking place until after it had happened.
Was it held in Florida as well ? Or is it just a massive coincidence ?
(overall a good read. certainly a buttload of speculation but no more (actually probably less) than found in Wolfram's book)
On the other hand, I see nothing wrong with offering a prize for what he believes in. Heck we have the Templeton prize out there (more than the Nobel, no less) for best achievement in religion (christianity specifically, methinks), so what's wrong with offering 100G of his own money? We also have the X-BOX cracking contest - who is willing to bet that the believing in the chance of solving a 2048bit key in a few monthes is MUCH dumber than trying to shoot for some "not everybody agree as AI" AI?
My life in the land of the rising sun.
Any specified Turing Test can be defeated in much the same way as a lock-pick can defeat any specified lock, so perhaps we should move up one level of abstraction. I propose the "Meta Turing Test" which is as follows: specifying the conditions of the Turing Test (ability to lie, sense of humour, etc.) should allow a true human to design an automaton that fools the turing test, while a computer will not be able to do so.
Alternatively, why not just abandon the myth that human intelligence is some kind of mystical cloud, and see it for what it is, namely a set of thinking organs each designed (or adapted, if you prefer the 'evolution is a passive process' concept) to solve specific problems, in the same way as my hand is adapted to handling objects. Then, test each of these tools carefully. Anything - computer or human - that passes the tests can be defined as 'human'. Many beings that we today consider human will probably fail. Borg borg.
Sig for sale or rent. One previous user. Inquire within.
Why are Minsky and Shiber so upset that a sex-addicted pothead is sponsoring an A.I. prize, when the Father of Dynamite sponsors a Peace prize?
Loebner can do whatever he wants with his dough. No one is being coerced into entering his contest.
I'm not really a web designer, I just play one on the Internet.
I think that people who focus on the Turing test are missing the point, this isn't really AI and probably doesn't have much of a use outside advertising to via IRC/personal messaging etc.
The real interesting areas of research in AI are for example: in dye-master processes, where AI replaces a highly skilled human, or automating the driving of cars. These are all AI and, IMHO, much more impressive than glorified Eliza, Turing test stuff...
keyboard not found, press F1 to continue
Cheers,
Ian
the program that alters the test to fit its own capabilities. That is cheating? How more human can it get ? Humanity is constantly adapting it's surroundings to fit its own needs...
Physics of Consciousness
Building a machine to pass the Turing Test is one thing, but the nature of consciousness itself is the more profound question here. Rodney Brooks asked this question in a relatively recent Edge Online interview.
What are we missing in our computational models of living systems?
Chris
http://www.umsl.edu/~altmanc/
http://www.artilect.org/
Quantum computing / Artificial intelligence: http://www.umsl.edu/~altmanc/news.html
The author of the article appears never to have read the article by Turing where he described the so-called 'test'. It is clear that Turing was a deep and subtle thinker way ahead of his time. If you read what he is saying in context, he is arguing that first and foremost, thought can be automated in the sense of a universal computer which can compute anything that a brain can. To his critics who said that this was somehow impossible, he created a reducto-ad-absurdum argument; he said look if you are talking to this machine and it is composing sonnets which are like Shakespeare, and you *still* can't say it's intelligent, then you are an idiot. He was not proposing that this was an objective test or a desirable thing to do, he was poking fun at idiots like the author of the Salon article.
I agree that the entries are really bad-- one recent winner just said the same things no matter what the human asked. But one winner, unmentioned in Salon, was Thom Whalen, whose design was a genuine advance in the art. (Regrettably, Loebner changed the rules to exclude his approach in the future.)
What Whalen did was limit his domain to one topic, and compile a set of general answers to likely questions, which he matched by spotting keywords. So even if the answer wasn't a perfect match, it was general enough to be useful. This design should be better known and more widely used, and the Loebner contest would have been a good launchpad to bring it to people's attention if the academics weren't so prejudiced.
But the top academics get six-figure salaries for generating lots of jargon and no useful products, so a level playing-field is the last thing they want.
Turing stipulated in the Turing test (TT) that the "interrogator" specifically has the goal of trying to determine which of the contestants is human and which is the machine. Unfortunately, the way the Loebner contest is conducted, this important requirement is completely ignored (at least in the default $2000 prize). As a result, the results of the contest are completely irrelevant from the point of view of the Turing test. Claiming otherwise is incorrect and misleading, and Loebner fully deserves all the criticism he gets.
The TT is still fully valid today. We are very far from building bots that will pass it. (though Turing predicted that by 2000 we will have machines that will pass TT). In fact, the whole direction of work on the bots participating in the current day Loebner contests is irrelevant from the TT point of view. They work mostly by building enormous databases of statement-response pairs and doing minimal reasoning. Turing would have died laughing if he had known people would take this approach to passing the TT. Let me illustrate why the database idea is insufficient by itself: for a bot to pass the real TT, it would have to answer questions like "what is the integral of e^x dx". Remember that the interrogator is actively trying to find out if it is a human or a bot. The objection "but two humans in conversation wouldn't ask such question" is invalid, and this is precisely why the Loebner contest is stupid.
The reason why today's bots are so unsuccesful is not far to seek. It has long been known in the AI community that get anywhere near passing the TT, a bot would need what is known as "world knowledge". To build world knowledge, you need memory approximately the capacity of the human brain: estimated to be the order of a petabyte. And processing power to match: the brain runs something like a billion threads in parallel, and is 10^7 times as energy efficient per computation as today's computers. Of course, we aren't there yet. Thus, contrary to what most people would feel the thing that is holding AI up is hardware.
Similar to today's bot craze, there have been crazes in the past when people thought they were close to building truly intelligent machines ("expert systems" comes to mind.) However, they inevitably came up short because the hardware power wasn't there. In about 20-30 years, assuming there continue to be breakthroughs in storage technology to keep up the doubling, computers will be matching the brain's capacity, and then we'll be talking.
Summary: to hell with people who apparently popularize science and end up giving the real researchers a bad name.
Get with the program, dude: we had AI even in the UK last year. I didn't go and see it though, because it starred that irritating kid from "The Sixth Sense".
When I am king, you will be first against the wall.
In 1995, about a year after the publication of Shieber's article, Marvin Minsky, the father of artificial intelligence, posted a notice on the comp.ai and comp.ai.philosophy Usenet newsgroups. In it he drew attention to a clause in the Loebner contest rules to the effect that using the term "Loebner Competition" without permission could result in a revocation of the prize.
Minsky wrote, "I do hope that someone will volunteer to violate this proscription so that Mr. Loebner will indeed revoke his stupid prize, save himself some money, and spare us the horror of this obnoxious and unproductive annual publicity campaign. In fact, I hereby offer the $100.00 Minsky prize to the first person who gets Loebner to do this. I will explain the details of the rules for the new prize as soon as it is awarded, except that, in the meantime, anyone is free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they like, without any licensing fee."
(Minsky did not respond to e-mails requesting an interview.)
If the CACM article marked Loebner's fall from grace, the Minsky note on comp.ai marked his utter banishment into the wilds of A.I. quackery.
Can you imagine, for example, being a graduate student in computer science at a big-name school in 1996 and telling your major professor that your goal was to win the Loebner? Loebner was more "out" than Liberace.
But Loebner did not take his snubbing meekly. Loebner immediately wrote back that the best way for Minsky to get Loebner to revoke his prize was to win it. Of course Minsky had already hinted that Loebner had never made clear what the rules for winning the prize were, so that was not a very satisfactory rejoinder. But then a few days later ("while taking a nice hot bath, drinking a fine wine, about an hour after smoking a really fat joint"), Loebner came up with a more considered and clever response, one that still rattles Minsky nearly a decade later.
Minsky had announced that he would give $100 to whoever made Loebner stop his contest. But Loebner would only stop his contest when somebody won the gold medal. Therefore, Loebner reasoned, Minsky, being an honorable man, would give $100 to whoever won the ultimate Loebner competition. Therefore, Marvin Minsky was a cosponsor of the Loebner competition, simple as that. It was delicious!
Loebner promptly issued a press release saying that Marvin Minsky was now a cosponsor of the Loebner Prize, by virtue of his announcement of the "Minsky Loebner Prize Revocation Prize." What made this development so delightfully ironic was Minsky's own statement that anyone was free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they liked, which made it nearly impossible for Minsky to prevent Loebner from doing just that. Which is why Loebner continues to cite Minsky as a cosponsor of his event every chance he gets.
The image that comes to my mind whenever I think of this development is from the sublime cartoons of the late, great Chuck Jones, with Hugh Loebner in the role of Bugs Bunny, and Marvin Minsky, the father of artificial intelligence, in the role of Yosemite Sam, stamping his feet, with smoke coming from his ears. In fact, Minsky is still listed as a cosponsor of Loebner's prize on the Web site, and, as we'll see, Minsky is still stamping his feet.
I worked in a research lab that shared a building with MIT's artificial intelligence laboratory. And I have to agree with the article. The AI field is a fraud. Again and again, there would be big placards in the lobby announcing gala media events up in the AI Lab. (We lesser mortals dutifully clomped upstairs to eat the expensive, catered food.)
And yet *nothing* *ever* *happens* in the field.
Every now and then a new "hero" emerges. For a while it was Minsky. In recent years, it has been Rodney Brooks. Regardless, you can see the current hero on TV all the time, commenting on matters as an "AI expert". They don't tell you that Brooks' course is widely viewed as a complete crock; a few puerile algorithms, some linear differential equations, some finite automata, and THAT'S IT. The rest is all blabbering with no substance.
The AI community uses rotating hero-worship in lieu of progress. But it isn't like any of these guys is an actual "AI expert". There are no "AI experts", because there is no such thing as artificial intelligence in this world. They are no more experts on AI than I am an expert on Martian fruit exports. In this field, you don't need real research; an Australian accent and good sense of humor suffice.
True artificial intelligence would be amazing. But the field has made essentially zero progress in the last fifty years. Obviously, it is a really hard problem. On one hand, the AI guys do what other fields do when they're stuck (since they *must* continue to pump out graduate students, attract grants, etc.), they keep trying to change the question. But the pathetic thing is that many completely denigrate the most obviously fair benchmark-- the Turing test.
Coincidentally, a benchmark showing the complete failure of the field.
Lo and behold, what first appeared to be intelligence is now just an elaborate sequence of if-then statements. Anyone could have done it. It's not intelligence at all. It's just following a blueprint. You call this intelligence?
In other words, the lay public expects A.I. to have creativity and strokes of genius, which is much more than they expect of most humans. Or they expect it to be furry with big eyes that makes cooing noises when you pet it. As soon as one realizes that A.I. consists of a computer program, any notion of intelligence evaporates.
This just shows that we don't actualy know what we mean when we say "Intelligence". It just meant "What I am thinking about when I say Intelligence".
The Turing Test is not a pass-mark to achieve intelligence, it is an outside limit to stop argument. If something passes, completely, the Turing test, then you know you have intelligence. But that is asn extremely high benchmark. It is like saying that if you can outrun all known vehicles, I have to grant you are a fast runner. You *may* still be a fast runner when when you run a lot slower than that - but we will have to enter into a discussion about how fast is fast. Turing just set an endpoint - it it passes his test it is certainly intelligent.
There are two ways the Turing Text could be passed. One is via a special purpose machine to pass it - a human simulator. While of research interest, because building such a machine would tell us a lot about how we actually do work, this is unlikely to be a very useful machine, because it will replicate our weaknesses as well as our strengths. Why spend billions building what half an hours funa and a nine month wait can build. (One-way trips to the stars, perhaps?).
The other way is a general purpose machine which has learned how to copy humans perfectly. By any definition I can think of, this would be an awesomely intelligent machine because it would have learned to understand, and simulate, our minds by the power of pure intellect. Something like playing all the instuments in the orchestra at the same time.
While I think that the first class of machine may well be built in the fullness of time, It will not be very useful. I don't know whether the second class will ever be built - I doubt it.
Which brings us back to the "sub-Turing" class of intelligence. If Turing represents an upper limit to the grey area of where intelligence starts, there must be levels of achievement which would be regarded as intelligent by most, if not all, peoples judgement.
I then ask the question: what use is sub-Turing intelligence? Well, there are lots of tasks which we regard as needing intelligence which we would like to automate. In fact, some of them have already been automated. But when we automate them, we say "we know how that automaton works, so it can't be intelligence". Chess, for example - once regarded as the last test before the Turing test, now regarded as a nifty but essentially unimportant achievement.
We don't actually *know* what we mean when we say "Intelligence". Turing knew that, and provided an empirical rather than analytical test. However, I would say that "Intelligence" bears the same relationship to "Computer Science" as "Magic" does to "Technology" in Clarke's Law: "Any sufficently advanced technology is indistinguishable from magic".
"Any sufficiently advanced Computer Science is indistinguisahable from Intelligence" - Cawley's Law.
Or, to put it another way, Intelligence means "I don't understand how you thought that".
Which explains how Joe Luser thinks his computer is intelligent, whereas Bill Slashdot doesn't.
Consciousness is an illusion caused by an excess of self consciousness.
My favorite quote about the Turing test comes from Jaron Lanier:
"Only a fucked-up gay Englishman being tortured with hormone injections could possibly have supposed that consciousness was some kind of social exam you had to pass."
Part of the problem is that things that were once considered part of AI have moved out and become mainstream technology. Voice recognition, Expert Systems, Fuzzy Logic, Neural Nets, Chess playing computers...all of these were once considered to be unsolved AI problems but since they are now in common use, we don't consider them a part of AI anymore.
You can find plenty of twenty to thirty year old textbooks that tell you that playing chess at grand master level would be a sign of computer intelligence - now we know that all it takes are some clever heuristics and a lot of CPU power.
As soon as computers can pass the Turing Test, it'll be considered laughable that anyone ever thought it required *intelligence* to chat with a human. In a sense, this has already happened. Quite a few people were convinced by Eliza - but you can tell from just looking at the code that it's not intelligent.
The same thing is happening with animals. We used to define humans as the only tool-using animals - then they found birds breaking open clamshells by dropping rocks on them. The definition changed to humans as the only tool *making* animals...then they found chimpanzees who strip the leaves from twigs before they poke them into anthills. So then it was 'self recognition' - that also failed with dolphins who can recognise themselves in a mirror. Now it's some other thing. Animals will never be labelled intelligent" because the definition of intelligence is that thing that humans have but animals do not.
I predict that we'll never have AI. That isn't a failure of the work - it's in the nature of our definition of Intelligence as "that thing that humans have that animals and machines don't have".
www.sjbaker.org