Slashdot Mirror
Turing Test 2: A Sense of Humor
mhackarbie writes "Salon has a great story, Artificial Stupidity, about the Loebner Prize, a yearly contest that for over 10 years now has offered a $100,000 prize to anyone who can create a program to pass the Turing Test. The best part is the resulting fiasco that develops between the eccentric philanthropist who started the contest and extremely annoyed AI Researchers such as Marvin Minsky."
All Hugh Loebner wanted to do was become world famous, eliminate all human toil, and get laid a lot.
does this mean we're all considered entrepeneurs ?
Fear Breeds Knowledge
I don't think bots are the problem... I've had several online conversations which I'd assumed were chat-bots but turned out to be real people. I guess when Turing designed his test, he probably didn't anticipate the massive advances in human stupidity that we've witnessed in the last few decades :)
-- Open Source: It's mad, but you don't have to work here to help.
Every person with whom I spoke about it said that last year's contest was an utter fiasco, with unclear rules, inconsistent judging, arbitrary fiats by an opaque prize committee, petulant prima donnas, and last-minute changes of venue that prevented most entrants from even discovering where the contest was taking place until after it had happened.
Was it held in Florida as well ? Or is it just a massive coincidence ?
(overall a good read. certainly a buttload of speculation but no more (actually probably less) than found in Wolfram's book)
On the other hand, I see nothing wrong with offering a prize for what he believes in. Heck we have the Templeton prize out there (more than the Nobel, no less) for best achievement in religion (christianity specifically, methinks), so what's wrong with offering 100G of his own money? We also have the X-BOX cracking contest - who is willing to bet that the believing in the chance of solving a 2048bit key in a few monthes is MUCH dumber than trying to shoot for some "not everybody agree as AI" AI?
My life in the land of the rising sun.
Any specified Turing Test can be defeated in much the same way as a lock-pick can defeat any specified lock, so perhaps we should move up one level of abstraction. I propose the "Meta Turing Test" which is as follows: specifying the conditions of the Turing Test (ability to lie, sense of humour, etc.) should allow a true human to design an automaton that fools the turing test, while a computer will not be able to do so.
Alternatively, why not just abandon the myth that human intelligence is some kind of mystical cloud, and see it for what it is, namely a set of thinking organs each designed (or adapted, if you prefer the 'evolution is a passive process' concept) to solve specific problems, in the same way as my hand is adapted to handling objects. Then, test each of these tools carefully. Anything - computer or human - that passes the tests can be defined as 'human'. Many beings that we today consider human will probably fail. Borg borg.
Sig for sale or rent. One previous user. Inquire within.
Why are Minsky and Shiber so upset that a sex-addicted pothead is sponsoring an A.I. prize, when the Father of Dynamite sponsors a Peace prize?
Loebner can do whatever he wants with his dough. No one is being coerced into entering his contest.
I'm not really a web designer, I just play one on the Internet.
oh.
Malike Bamiyi wanted my assistance.
"... extremely annoyed AI Researchers ..."
....
Perhaps "extremely annoyed" is what distinguishes human intelligence from machine intelligence?
In John Brunner's non-novel Stand on Zanzibar, cranky sociologist Chad Mulligan declares that supercomputer Shalmaneser is now intelligent because Shalmaneser has displayed the quality of "bloody-mindedness". Not the same as "annoyance", of course, but in the same emotional realm
-kgj
I think that people who focus on the Turing test are missing the point, this isn't really AI and probably doesn't have much of a use outside advertising to via IRC/personal messaging etc.
The real interesting areas of research in AI are for example: in dye-master processes, where AI replaces a highly skilled human, or automating the driving of cars. These are all AI and, IMHO, much more impressive than glorified Eliza, Turing test stuff...
keyboard not found, press F1 to continue
Cheers,
Ian
the program that alters the test to fit its own capabilities. That is cheating? How more human can it get ? Humanity is constantly adapting it's surroundings to fit its own needs...
Physics of Consciousness
Building a machine to pass the Turing Test is one thing, but the nature of consciousness itself is the more profound question here. Rodney Brooks asked this question in a relatively recent Edge Online interview.
What are we missing in our computational models of living systems?
Chris
http://www.umsl.edu/~altmanc/
http://www.artilect.org/
Quantum computing / Artificial intelligence: http://www.umsl.edu/~altmanc/news.html
The author of the article appears never to have read the article by Turing where he described the so-called 'test'. It is clear that Turing was a deep and subtle thinker way ahead of his time. If you read what he is saying in context, he is arguing that first and foremost, thought can be automated in the sense of a universal computer which can compute anything that a brain can. To his critics who said that this was somehow impossible, he created a reducto-ad-absurdum argument; he said look if you are talking to this machine and it is composing sonnets which are like Shakespeare, and you *still* can't say it's intelligent, then you are an idiot. He was not proposing that this was an objective test or a desirable thing to do, he was poking fun at idiots like the author of the Salon article.
I agree that the entries are really bad-- one recent winner just said the same things no matter what the human asked. But one winner, unmentioned in Salon, was Thom Whalen, whose design was a genuine advance in the art. (Regrettably, Loebner changed the rules to exclude his approach in the future.)
What Whalen did was limit his domain to one topic, and compile a set of general answers to likely questions, which he matched by spotting keywords. So even if the answer wasn't a perfect match, it was general enough to be useful. This design should be better known and more widely used, and the Loebner contest would have been a good launchpad to bring it to people's attention if the academics weren't so prejudiced.
But the top academics get six-figure salaries for generating lots of jargon and no useful products, so a level playing-field is the last thing they want.
I used to work in nuclear fusion research. They've been saying it is twenty years away for almost 50 years now. The joke in the industry is, "Fusion power is the energy of the future, and always will be!". (actually, I am fairly positive on fusion power, but I think that spending the vast majority of research funds on a few large experiments is counterproductive)
Turing stipulated in the Turing test (TT) that the "interrogator" specifically has the goal of trying to determine which of the contestants is human and which is the machine. Unfortunately, the way the Loebner contest is conducted, this important requirement is completely ignored (at least in the default $2000 prize). As a result, the results of the contest are completely irrelevant from the point of view of the Turing test. Claiming otherwise is incorrect and misleading, and Loebner fully deserves all the criticism he gets.
The TT is still fully valid today. We are very far from building bots that will pass it. (though Turing predicted that by 2000 we will have machines that will pass TT). In fact, the whole direction of work on the bots participating in the current day Loebner contests is irrelevant from the TT point of view. They work mostly by building enormous databases of statement-response pairs and doing minimal reasoning. Turing would have died laughing if he had known people would take this approach to passing the TT. Let me illustrate why the database idea is insufficient by itself: for a bot to pass the real TT, it would have to answer questions like "what is the integral of e^x dx". Remember that the interrogator is actively trying to find out if it is a human or a bot. The objection "but two humans in conversation wouldn't ask such question" is invalid, and this is precisely why the Loebner contest is stupid.
The reason why today's bots are so unsuccesful is not far to seek. It has long been known in the AI community that get anywhere near passing the TT, a bot would need what is known as "world knowledge". To build world knowledge, you need memory approximately the capacity of the human brain: estimated to be the order of a petabyte. And processing power to match: the brain runs something like a billion threads in parallel, and is 10^7 times as energy efficient per computation as today's computers. Of course, we aren't there yet. Thus, contrary to what most people would feel the thing that is holding AI up is hardware.
Similar to today's bot craze, there have been crazes in the past when people thought they were close to building truly intelligent machines ("expert systems" comes to mind.) However, they inevitably came up short because the hardware power wasn't there. In about 20-30 years, assuming there continue to be breakthroughs in storage technology to keep up the doubling, computers will be matching the brain's capacity, and then we'll be talking.
Summary: to hell with people who apparently popularize science and end up giving the real researchers a bad name.
Get with the program, dude: we had AI even in the UK last year. I didn't go and see it though, because it starred that irritating kid from "The Sixth Sense".
When I am king, you will be first against the wall.
In 1995, about a year after the publication of Shieber's article, Marvin Minsky, the father of artificial intelligence, posted a notice on the comp.ai and comp.ai.philosophy Usenet newsgroups. In it he drew attention to a clause in the Loebner contest rules to the effect that using the term "Loebner Competition" without permission could result in a revocation of the prize.
Minsky wrote, "I do hope that someone will volunteer to violate this proscription so that Mr. Loebner will indeed revoke his stupid prize, save himself some money, and spare us the horror of this obnoxious and unproductive annual publicity campaign. In fact, I hereby offer the $100.00 Minsky prize to the first person who gets Loebner to do this. I will explain the details of the rules for the new prize as soon as it is awarded, except that, in the meantime, anyone is free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they like, without any licensing fee."
(Minsky did not respond to e-mails requesting an interview.)
If the CACM article marked Loebner's fall from grace, the Minsky note on comp.ai marked his utter banishment into the wilds of A.I. quackery.
Can you imagine, for example, being a graduate student in computer science at a big-name school in 1996 and telling your major professor that your goal was to win the Loebner? Loebner was more "out" than Liberace.
But Loebner did not take his snubbing meekly. Loebner immediately wrote back that the best way for Minsky to get Loebner to revoke his prize was to win it. Of course Minsky had already hinted that Loebner had never made clear what the rules for winning the prize were, so that was not a very satisfactory rejoinder. But then a few days later ("while taking a nice hot bath, drinking a fine wine, about an hour after smoking a really fat joint"), Loebner came up with a more considered and clever response, one that still rattles Minsky nearly a decade later.
Minsky had announced that he would give $100 to whoever made Loebner stop his contest. But Loebner would only stop his contest when somebody won the gold medal. Therefore, Loebner reasoned, Minsky, being an honorable man, would give $100 to whoever won the ultimate Loebner competition. Therefore, Marvin Minsky was a cosponsor of the Loebner competition, simple as that. It was delicious!
Loebner promptly issued a press release saying that Marvin Minsky was now a cosponsor of the Loebner Prize, by virtue of his announcement of the "Minsky Loebner Prize Revocation Prize." What made this development so delightfully ironic was Minsky's own statement that anyone was free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they liked, which made it nearly impossible for Minsky to prevent Loebner from doing just that. Which is why Loebner continues to cite Minsky as a cosponsor of his event every chance he gets.
The image that comes to my mind whenever I think of this development is from the sublime cartoons of the late, great Chuck Jones, with Hugh Loebner in the role of Bugs Bunny, and Marvin Minsky, the father of artificial intelligence, in the role of Yosemite Sam, stamping his feet, with smoke coming from his ears. In fact, Minsky is still listed as a cosponsor of Loebner's prize on the Web site, and, as we'll see, Minsky is still stamping his feet.
I worked in a research lab that shared a building with MIT's artificial intelligence laboratory. And I have to agree with the article. The AI field is a fraud. Again and again, there would be big placards in the lobby announcing gala media events up in the AI Lab. (We lesser mortals dutifully clomped upstairs to eat the expensive, catered food.)
And yet *nothing* *ever* *happens* in the field.
Every now and then a new "hero" emerges. For a while it was Minsky. In recent years, it has been Rodney Brooks. Regardless, you can see the current hero on TV all the time, commenting on matters as an "AI expert". They don't tell you that Brooks' course is widely viewed as a complete crock; a few puerile algorithms, some linear differential equations, some finite automata, and THAT'S IT. The rest is all blabbering with no substance.
The AI community uses rotating hero-worship in lieu of progress. But it isn't like any of these guys is an actual "AI expert". There are no "AI experts", because there is no such thing as artificial intelligence in this world. They are no more experts on AI than I am an expert on Martian fruit exports. In this field, you don't need real research; an Australian accent and good sense of humor suffice.
True artificial intelligence would be amazing. But the field has made essentially zero progress in the last fifty years. Obviously, it is a really hard problem. On one hand, the AI guys do what other fields do when they're stuck (since they *must* continue to pump out graduate students, attract grants, etc.), they keep trying to change the question. But the pathetic thing is that many completely denigrate the most obviously fair benchmark-- the Turing test.
Coincidentally, a benchmark showing the complete failure of the field.
This strikes me as true: for years and years and years, researchers have been promising AI was just around the corner... And what do we have right now? Nothing!
Well, nothing is a very relative term. We now have AI capable of counting the number of cars on a given street given a photograph of a region, and can automatically follow people / vehicles / animals as they travel around and through objects. OCR is accurate enough to be implemented professionally, and voice recognition is up to 95%. None of these were possible 25 years ago, and not just because of a lack of hardware.
While full AI is still a while away, the first major stumbling block, pattern recognition, is well on its way to being solved.
The AI in Quake 3 is much better than the AI in Pong.
-C
The ______ Agenda
Lo and behold, what first appeared to be intelligence is now just an elaborate sequence of if-then statements. Anyone could have done it. It's not intelligence at all. It's just following a blueprint. You call this intelligence?
In other words, the lay public expects A.I. to have creativity and strokes of genius, which is much more than they expect of most humans. Or they expect it to be furry with big eyes that makes cooing noises when you pet it. As soon as one realizes that A.I. consists of a computer program, any notion of intelligence evaporates.
This just shows that we don't actualy know what we mean when we say "Intelligence". It just meant "What I am thinking about when I say Intelligence".
The Turing Test is not a pass-mark to achieve intelligence, it is an outside limit to stop argument. If something passes, completely, the Turing test, then you know you have intelligence. But that is asn extremely high benchmark. It is like saying that if you can outrun all known vehicles, I have to grant you are a fast runner. You *may* still be a fast runner when when you run a lot slower than that - but we will have to enter into a discussion about how fast is fast. Turing just set an endpoint - it it passes his test it is certainly intelligent.
There are two ways the Turing Text could be passed. One is via a special purpose machine to pass it - a human simulator. While of research interest, because building such a machine would tell us a lot about how we actually do work, this is unlikely to be a very useful machine, because it will replicate our weaknesses as well as our strengths. Why spend billions building what half an hours funa and a nine month wait can build. (One-way trips to the stars, perhaps?).
The other way is a general purpose machine which has learned how to copy humans perfectly. By any definition I can think of, this would be an awesomely intelligent machine because it would have learned to understand, and simulate, our minds by the power of pure intellect. Something like playing all the instuments in the orchestra at the same time.
While I think that the first class of machine may well be built in the fullness of time, It will not be very useful. I don't know whether the second class will ever be built - I doubt it.
Which brings us back to the "sub-Turing" class of intelligence. If Turing represents an upper limit to the grey area of where intelligence starts, there must be levels of achievement which would be regarded as intelligent by most, if not all, peoples judgement.
I then ask the question: what use is sub-Turing intelligence? Well, there are lots of tasks which we regard as needing intelligence which we would like to automate. In fact, some of them have already been automated. But when we automate them, we say "we know how that automaton works, so it can't be intelligence". Chess, for example - once regarded as the last test before the Turing test, now regarded as a nifty but essentially unimportant achievement.
We don't actually *know* what we mean when we say "Intelligence". Turing knew that, and provided an empirical rather than analytical test. However, I would say that "Intelligence" bears the same relationship to "Computer Science" as "Magic" does to "Technology" in Clarke's Law: "Any sufficently advanced technology is indistinguishable from magic".
"Any sufficiently advanced Computer Science is indistinguisahable from Intelligence" - Cawley's Law.
Or, to put it another way, Intelligence means "I don't understand how you thought that".
Which explains how Joe Luser thinks his computer is intelligent, whereas Bill Slashdot doesn't.
Consciousness is an illusion caused by an excess of self consciousness.
Funny the AI researchers seem to be upset with the contest.
But I find it strange that various people keep trying to either:
1) Take part.
2) Stop the contest.
3) Tell the contest sponsor how to run the contest or spend his money.
Are they really so hard up for Loebner's money? If their stuff really works I'm sure they can get money from other people.
As far as I know none of the AI entrants so far deserve the main prize.
It's almost as if the tailors are upset that someone every year points out the emperor is naked. If indeed the emperor isn't naked why get upset?
Or they admit the emperor is naked and they are just tired of hearing about it? Well so far has any of them admitted that?
My favorite quote about the Turing test comes from Jaron Lanier:
"Only a fucked-up gay Englishman being tortured with hormone injections could possibly have supposed that consciousness was some kind of social exam you had to pass."
I particurly like how Loebner out foxed Marvin Minsky with the ammunition Minsky gave him.
.. but he still did it.
Sure the guy may be a pot head, might not want a lasting relationship with a woman, and is probally a horribly annoying git from hell.
He did however, manage to outthink the 'brightest' mind in AI research. Maybe the reasons he did were purile
As a programmer I know I was taught to think in small steps, think ahead to the probable issues my code might cause, and to double check my work before dropping it on a production box.
Apparantly Minsky forgot he was a computer scientist when he wrote that news group response.
I'm sure it was just a flame mail, a very human response to frustration and irritation. But as one of the Leading names in AI research, he should have known better.
So, if for nothing else, my hats off to the 'Disco-Floor-Maker' for out thinking one of the 'leaders' in AI research.
Its always nice to watch an acidemic geek get smacked down by someone who lives with the rest of society.
--Ne auderis delere orbem rigidum meum, non erravi pernicose!
Part of the problem is that things that were once considered part of AI have moved out and become mainstream technology. Voice recognition, Expert Systems, Fuzzy Logic, Neural Nets, Chess playing computers...all of these were once considered to be unsolved AI problems but since they are now in common use, we don't consider them a part of AI anymore.
You can find plenty of twenty to thirty year old textbooks that tell you that playing chess at grand master level would be a sign of computer intelligence - now we know that all it takes are some clever heuristics and a lot of CPU power.
As soon as computers can pass the Turing Test, it'll be considered laughable that anyone ever thought it required *intelligence* to chat with a human. In a sense, this has already happened. Quite a few people were convinced by Eliza - but you can tell from just looking at the code that it's not intelligent.
The same thing is happening with animals. We used to define humans as the only tool-using animals - then they found birds breaking open clamshells by dropping rocks on them. The definition changed to humans as the only tool *making* animals...then they found chimpanzees who strip the leaves from twigs before they poke them into anthills. So then it was 'self recognition' - that also failed with dolphins who can recognise themselves in a mirror. Now it's some other thing. Animals will never be labelled intelligent" because the definition of intelligence is that thing that humans have but animals do not.
I predict that we'll never have AI. That isn't a failure of the work - it's in the nature of our definition of Intelligence as "that thing that humans have that animals and machines don't have".
www.sjbaker.org
I think we are going to have AI when we realize that it is not only the presence of a clever algorithm that makes up AI, it is also the motivation. In other words, a human being is motivated to develop intelligence in order to survive, something that is not required from a computer.
Another big difference is that modern computers are much less powerful than the brain: the human brain's memory is equivalent to many million petabytes of memory, and the searching mechanism of the brain is straightforward pattern matching that works like a neural network (and can identify and discard many images in parallel). Our poor computers have only some terrabytes of memory and they are much slower in reading that memory in an efficient way.
Animals have the best cameras for eyes and the best microphones for ears, all made by mother nature!!! And these inputs are designed to stimulate and filter responses in a sophisticated non-digital way, rather than simply accumulate the data and convert them to binary information.
With all these big differences, please don't expect AI to surface in the near feature. It could surface though if we realized the differences of human vs machine and start building machines with human-like attributes (for example neural-network memory, and with motives to learn and expand its knowledge base, with cameras and ears, with feelings).
I'm not sure what you mean. The two sentences that I quoted seem to indicate that Christopher Reeves couldn't participate in a Turing Test. Turing's insight was that performance is the only measure that we have of intellegence. His paper actually included several hypothetical ways by which performance isn't the only measure. For example, parapsychological effects: you look at a Rhine Card and ask the testee what you're looking at. If humans consistently guess better (or worse!) than computers, then the Turing Test is invalid (and a whole new field of scientific study has opened up).
On the other hand, you could ask Chris Reeve (or a computer) to play chess with you. Either could say, "Sorry, I don't have a board handy, how about tic-tac-toe?"
As you read this, are you evaluating my competance or my performance? How do you know that I'm not really a bot from Cycorp?
Nothing for 6-digit uids?