Slashdot Mirror
Turing Test 2: A Sense of Humor
mhackarbie writes "Salon has a great story, Artificial Stupidity, about the Loebner Prize, a yearly contest that for over 10 years now has offered a $100,000 prize to anyone who can create a program to pass the Turing Test. The best part is the resulting fiasco that develops between the eccentric philanthropist who started the contest and extremely annoyed AI Researchers such as Marvin Minsky."
Any specified Turing Test can be defeated in much the same way as a lock-pick can defeat any specified lock, so perhaps we should move up one level of abstraction. I propose the "Meta Turing Test" which is as follows: specifying the conditions of the Turing Test (ability to lie, sense of humour, etc.) should allow a true human to design an automaton that fools the turing test, while a computer will not be able to do so.
Alternatively, why not just abandon the myth that human intelligence is some kind of mystical cloud, and see it for what it is, namely a set of thinking organs each designed (or adapted, if you prefer the 'evolution is a passive process' concept) to solve specific problems, in the same way as my hand is adapted to handling objects. Then, test each of these tools carefully. Anything - computer or human - that passes the tests can be defined as 'human'. Many beings that we today consider human will probably fail. Borg borg.
Sig for sale or rent. One previous user. Inquire within.
"... extremely annoyed AI Researchers ..."
....
Perhaps "extremely annoyed" is what distinguishes human intelligence from machine intelligence?
In John Brunner's non-novel Stand on Zanzibar, cranky sociologist Chad Mulligan declares that supercomputer Shalmaneser is now intelligent because Shalmaneser has displayed the quality of "bloody-mindedness". Not the same as "annoyance", of course, but in the same emotional realm
-kgj
I think that people who focus on the Turing test are missing the point, this isn't really AI and probably doesn't have much of a use outside advertising to via IRC/personal messaging etc.
The real interesting areas of research in AI are for example: in dye-master processes, where AI replaces a highly skilled human, or automating the driving of cars. These are all AI and, IMHO, much more impressive than glorified Eliza, Turing test stuff...
Physics of Consciousness
Building a machine to pass the Turing Test is one thing, but the nature of consciousness itself is the more profound question here. Rodney Brooks asked this question in a relatively recent Edge Online interview.
What are we missing in our computational models of living systems?
Chris
http://www.umsl.edu/~altmanc/
http://www.artilect.org/
Quantum computing / Artificial intelligence: http://www.umsl.edu/~altmanc/news.html
I used to work in nuclear fusion research. They've been saying it is twenty years away for almost 50 years now. The joke in the industry is, "Fusion power is the energy of the future, and always will be!". (actually, I am fairly positive on fusion power, but I think that spending the vast majority of research funds on a few large experiments is counterproductive)
Turing stipulated in the Turing test (TT) that the "interrogator" specifically has the goal of trying to determine which of the contestants is human and which is the machine. Unfortunately, the way the Loebner contest is conducted, this important requirement is completely ignored (at least in the default $2000 prize). As a result, the results of the contest are completely irrelevant from the point of view of the Turing test. Claiming otherwise is incorrect and misleading, and Loebner fully deserves all the criticism he gets.
The TT is still fully valid today. We are very far from building bots that will pass it. (though Turing predicted that by 2000 we will have machines that will pass TT). In fact, the whole direction of work on the bots participating in the current day Loebner contests is irrelevant from the TT point of view. They work mostly by building enormous databases of statement-response pairs and doing minimal reasoning. Turing would have died laughing if he had known people would take this approach to passing the TT. Let me illustrate why the database idea is insufficient by itself: for a bot to pass the real TT, it would have to answer questions like "what is the integral of e^x dx". Remember that the interrogator is actively trying to find out if it is a human or a bot. The objection "but two humans in conversation wouldn't ask such question" is invalid, and this is precisely why the Loebner contest is stupid.
The reason why today's bots are so unsuccesful is not far to seek. It has long been known in the AI community that get anywhere near passing the TT, a bot would need what is known as "world knowledge". To build world knowledge, you need memory approximately the capacity of the human brain: estimated to be the order of a petabyte. And processing power to match: the brain runs something like a billion threads in parallel, and is 10^7 times as energy efficient per computation as today's computers. Of course, we aren't there yet. Thus, contrary to what most people would feel the thing that is holding AI up is hardware.
Similar to today's bot craze, there have been crazes in the past when people thought they were close to building truly intelligent machines ("expert systems" comes to mind.) However, they inevitably came up short because the hardware power wasn't there. In about 20-30 years, assuming there continue to be breakthroughs in storage technology to keep up the doubling, computers will be matching the brain's capacity, and then we'll be talking.
Summary: to hell with people who apparently popularize science and end up giving the real researchers a bad name.
In 1995, about a year after the publication of Shieber's article, Marvin Minsky, the father of artificial intelligence, posted a notice on the comp.ai and comp.ai.philosophy Usenet newsgroups. In it he drew attention to a clause in the Loebner contest rules to the effect that using the term "Loebner Competition" without permission could result in a revocation of the prize.
Minsky wrote, "I do hope that someone will volunteer to violate this proscription so that Mr. Loebner will indeed revoke his stupid prize, save himself some money, and spare us the horror of this obnoxious and unproductive annual publicity campaign. In fact, I hereby offer the $100.00 Minsky prize to the first person who gets Loebner to do this. I will explain the details of the rules for the new prize as soon as it is awarded, except that, in the meantime, anyone is free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they like, without any licensing fee."
(Minsky did not respond to e-mails requesting an interview.)
If the CACM article marked Loebner's fall from grace, the Minsky note on comp.ai marked his utter banishment into the wilds of A.I. quackery.
Can you imagine, for example, being a graduate student in computer science at a big-name school in 1996 and telling your major professor that your goal was to win the Loebner? Loebner was more "out" than Liberace.
But Loebner did not take his snubbing meekly. Loebner immediately wrote back that the best way for Minsky to get Loebner to revoke his prize was to win it. Of course Minsky had already hinted that Loebner had never made clear what the rules for winning the prize were, so that was not a very satisfactory rejoinder. But then a few days later ("while taking a nice hot bath, drinking a fine wine, about an hour after smoking a really fat joint"), Loebner came up with a more considered and clever response, one that still rattles Minsky nearly a decade later.
Minsky had announced that he would give $100 to whoever made Loebner stop his contest. But Loebner would only stop his contest when somebody won the gold medal. Therefore, Loebner reasoned, Minsky, being an honorable man, would give $100 to whoever won the ultimate Loebner competition. Therefore, Marvin Minsky was a cosponsor of the Loebner competition, simple as that. It was delicious!
Loebner promptly issued a press release saying that Marvin Minsky was now a cosponsor of the Loebner Prize, by virtue of his announcement of the "Minsky Loebner Prize Revocation Prize." What made this development so delightfully ironic was Minsky's own statement that anyone was free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they liked, which made it nearly impossible for Minsky to prevent Loebner from doing just that. Which is why Loebner continues to cite Minsky as a cosponsor of his event every chance he gets.
The image that comes to my mind whenever I think of this development is from the sublime cartoons of the late, great Chuck Jones, with Hugh Loebner in the role of Bugs Bunny, and Marvin Minsky, the father of artificial intelligence, in the role of Yosemite Sam, stamping his feet, with smoke coming from his ears. In fact, Minsky is still listed as a cosponsor of Loebner's prize on the Web site, and, as we'll see, Minsky is still stamping his feet.
that has a chance in hell, I'm placing my bets on Cyc
It's basically a computer program that a bunch of researchers have spent 60 million dollars trying to teach it common sense. And they've had some impressive advancements. Previous slashdot story here
Alcohol, Tobacco and Firearms should be the name of a store, not a government agency.
The "AI" in Quake is virtually nonexistent. Entities don't learn, they see through walls and have their aim altered based upon difficulty and small random variations.
When you say "Pong," then, I'm curious if you mean the game or a specific implementation there of. I am fairly confident that _you_ could construct a more elaborate AI for the simple game of Pong than id did with Qauke.
How about designing a bot to play Nethack until it ascends.
The assumption is that human intelligence and human minds are really nothing more than a program in of themselves. The criticism that these AI computer programs are 'simply following a mindless program' would be responded with, "Well of course it's just following a program! So am I except that instead of code my brain is following a program of interactions among neurons and chemicals based upon the laws of physics and chemistry!"
-- An assumption of the field of AI is that all human mind and intelligence is essentially a computer program, or if not that it is a machine of some sort.
-Romanpoet
Jokes are all more-or-less the same but have different levels of abstraction.
I would say something like...
Throwing stones
Slap stick
Word play
Parody and sarcasm
Association jokes ('Why do men have one more brain cell the dogs? so they don't try to hump your leg at parties')
Parody and sarcasm (again, more the bill hicks style)
Most 'good?' stand-ups do alot of Association comedy, it builds a link with the audience, and makes things seem more funny.
an AI can easily manage throwing stones, slapstick, word play and basic parody/sarcasm since they require low levels of empathy.Higler levels require the teller to have a high level of empathy with the audience, which is currently out the ability of AI's.
thank God the internet isn't a human right.
You post is marked "Score:5 Funny", but I would mark it "Insightful".
That testers can believe that humans are computers is why it will never be a 'test'. Turing himself only ever called it the 'Imitation Game'.
If there is no way to tell humans from computers, how can you ever tell the computers from the humans?
We likes the 'turing test' not because it is scientific, but because, like intelligence itself, it is ill defined and imperfect.
I love the Loebner quote: "My reaction to intelligence is the same as my reaction to pornography, I can't define it but I like it when I see it."
I think we are going to have AI when we realize that it is not only the presence of a clever algorithm that makes up AI, it is also the motivation. In other words, a human being is motivated to develop intelligence in order to survive, something that is not required from a computer.
Another big difference is that modern computers are much less powerful than the brain: the human brain's memory is equivalent to many million petabytes of memory, and the searching mechanism of the brain is straightforward pattern matching that works like a neural network (and can identify and discard many images in parallel). Our poor computers have only some terrabytes of memory and they are much slower in reading that memory in an efficient way.
Animals have the best cameras for eyes and the best microphones for ears, all made by mother nature!!! And these inputs are designed to stimulate and filter responses in a sophisticated non-digital way, rather than simply accumulate the data and convert them to binary information.
With all these big differences, please don't expect AI to surface in the near feature. It could surface though if we realized the differences of human vs machine and start building machines with human-like attributes (for example neural-network memory, and with motives to learn and expand its knowledge base, with cameras and ears, with feelings).
But this is the key. Not that Deep Junior can take Kasparov to a draw, but the amount of work it has to do to do it. It analyzes millions, if not billions of moves, probably several plies deep on each one, and then can just barely keep up with Kasparov, who is analyzing dozens of moves at the most.
If you are a chess player, try this experiment:
Find a chess program where you can set the search depth to a maximum amount. Then see how long it takes you to beat it at one ply, two plies, three plies and so on.
If you barely know the rules, you will find that it doesn't even get challenging until about three plies. But think about that: to beat you, an average patzer (just like me), it has to analyze thousands of moves a turn. You are probably only analyzing two or three.
So this is why people don't consider todays chess programs intelligent the same way they consider humans intelligent: they are brute forcing a solution where true intelligence wouldn't need to.
This is why people generally don't consider AI as having progressed. Most of these solutions rely on brute force.
I'm not sure what you mean. The two sentences that I quoted seem to indicate that Christopher Reeves couldn't participate in a Turing Test. Turing's insight was that performance is the only measure that we have of intellegence. His paper actually included several hypothetical ways by which performance isn't the only measure. For example, parapsychological effects: you look at a Rhine Card and ask the testee what you're looking at. If humans consistently guess better (or worse!) than computers, then the Turing Test is invalid (and a whole new field of scientific study has opened up).
On the other hand, you could ask Chris Reeve (or a computer) to play chess with you. Either could say, "Sorry, I don't have a board handy, how about tic-tac-toe?"
As you read this, are you evaluating my competance or my performance? How do you know that I'm not really a bot from Cycorp?
Nothing for 6-digit uids?
I used to play with megahal alot since a buddy altered it to work over IRC. You could park it, in say, #jesus and #kkk and #rap and it would "learn" from each channel, and then when you asked it things, well, I don't have to say the rest.
But this still doesn't say what AI is.
I work in numerical physics, the sorts of problems we are trying to solve would, if you tried "brute force", involve matrices of the order of 10^100 or more variables (and that is only for a toy problem, a 'real' system would have vastly more). But, by making various approximations, you can, in some interesting cases, reduce the size of the problem to something that is actually solvable on a computer. But, the algorithms are all put in by hand, there is absolutely no 'intelligence' in the program at all. But, it is all about solving problems without brute force.
Genetic algorithms and fuzzy logic are just another class of approximation algorithms to avoid the brute force search. Why is one classed as AI and the other isn't?
For even more hypocricacy, the method of 'simulated annealing' is often included along GA's and such. But this is nothing more or less than the Metropolis algorithm of Monte-Carlo used commonly in physics, but applied to some other problem instead. I use different numerical methods and I don't think anyone has tried before to apply them to "AI"-type problems, but it would not suprise me if it was possible. Would that suddenly propel me into the exalted ranks of the AI gods?
Everything you describe above came about because of some *algorithm*: someone comes up with an algorithm for ranking pages based on links to/from other pages (google), someone comes up with an algorithm to evaluate promising lines in a decision tree (chess) etc etc. Sure, there is a vast amount of intelligence here, but it is all in the algorithm design, the computer itself is pretty dumb. Lots of human intelligence, to make up for lots of artificial stupidness.
Perhaps neural networks and such self-modifying/learning algorithms form a distinct class, that can (provably?) solve some problems better than non-learning algorithms. I've never seen (or heard of) a proof this, and the notable lack of neural networks in real calculations suggests that their success is maybe not quite as general as reports would have us believe.
I think that we are no closer to REAL artificial intelligence than we were 50 years ago. Perhaps further away in fact, in the sense that we can now at least formulate (partially) the questions properly: eg, is AI an emergent property of a system having simple rules but complex behaviour (the 'strong' AI hypothesis), or is there something not yet understood about how the brain works that means we cannot (with today's knowledge of physics) reproduce? (This is the argument of Penrose.)
But even this question is, in some ways, older than computers, since it really comes to the heart of how to 'interpret' quantum mechanics, and the measurement problem, which has been an issue for almost 100 years. So, how far have we come?
Looks like you got it, though I interpret that the grand prize requirement is arbitrary audiovisual input rather than ASCII art. Pretty steep.
Whalen has some invaluable musings and observations on the contest and his second entry. I remember the generalist strategy from the Alice CHAT simulation in the early nineties (linked in the grandparent post), and it doesn't look like that was really the problem - Wientraub's winning entry end-runs it with smooth non-sequiturs. In many ways that does point out the weakness in the contest, and even in the Turing test itself (weak versions anyway). Whalen's work with CHAT and TIPS has always been geared to actually delivering information (ie. being useful instead of merely clever), so I'm not surprised he didn't use that same strategy.
You can chat with Whalen's entries at the telnet site.