Slashdot Mirror
Turing Test 2: A Sense of Humor
mhackarbie writes "Salon has a great story, Artificial Stupidity, about the Loebner Prize, a yearly contest that for over 10 years now has offered a $100,000 prize to anyone who can create a program to pass the Turing Test. The best part is the resulting fiasco that develops between the eccentric philanthropist who started the contest and extremely annoyed AI Researchers such as Marvin Minsky."
(overall a good read. certainly a buttload of speculation but no more (actually probably less) than found in Wolfram's book)
On the other hand, I see nothing wrong with offering a prize for what he believes in. Heck we have the Templeton prize out there (more than the Nobel, no less) for best achievement in religion (christianity specifically, methinks), so what's wrong with offering 100G of his own money? We also have the X-BOX cracking contest - who is willing to bet that the believing in the chance of solving a 2048bit key in a few monthes is MUCH dumber than trying to shoot for some "not everybody agree as AI" AI?
My life in the land of the rising sun.
Why are Minsky and Shiber so upset that a sex-addicted pothead is sponsoring an A.I. prize, when the Father of Dynamite sponsors a Peace prize?
Loebner can do whatever he wants with his dough. No one is being coerced into entering his contest.
I'm not really a web designer, I just play one on the Internet.
the program that alters the test to fit its own capabilities. That is cheating? How more human can it get ? Humanity is constantly adapting it's surroundings to fit its own needs...
The author of the article appears never to have read the article by Turing where he described the so-called 'test'. It is clear that Turing was a deep and subtle thinker way ahead of his time. If you read what he is saying in context, he is arguing that first and foremost, thought can be automated in the sense of a universal computer which can compute anything that a brain can. To his critics who said that this was somehow impossible, he created a reducto-ad-absurdum argument; he said look if you are talking to this machine and it is composing sonnets which are like Shakespeare, and you *still* can't say it's intelligent, then you are an idiot. He was not proposing that this was an objective test or a desirable thing to do, he was poking fun at idiots like the author of the Salon article.
I agree that the entries are really bad-- one recent winner just said the same things no matter what the human asked. But one winner, unmentioned in Salon, was Thom Whalen, whose design was a genuine advance in the art. (Regrettably, Loebner changed the rules to exclude his approach in the future.)
What Whalen did was limit his domain to one topic, and compile a set of general answers to likely questions, which he matched by spotting keywords. So even if the answer wasn't a perfect match, it was general enough to be useful. This design should be better known and more widely used, and the Loebner contest would have been a good launchpad to bring it to people's attention if the academics weren't so prejudiced.
But the top academics get six-figure salaries for generating lots of jargon and no useful products, so a level playing-field is the last thing they want.
I disagree, these bots are made to parse language and make sense out of it. They can be (if people think outside the realm of IRC bots) important research in developing a conversational interface to a computer or robot.
The Anti-Blog
I have the distinct feeling that "worthy" AI objectives are defined by the AI community as "those things we think we can do reasonably well at the moment." In my opinion, the AI community disparages Turing Test-like objectives because they've been unsuccessful at achieving them. To me that makes AI less like science and more like selling Florida time-share condos. Kinda tough on high-profile PhD's, but what the hell: I don't actually know any of them anyway.
I've met (and work with) people who would not be able to conceive of a computer that chats with you. They would assume the weird answers were from a weird person and wander off in the same manner as they usually would... I guess this Turing Test proves that the "intelligence" of the computer can only be judged by reviewing the intelligence of the User???
Come and see the violence inherent in the system!
I worked in a research lab that shared a building with MIT's artificial intelligence laboratory. And I have to agree with the article. The AI field is a fraud. Again and again, there would be big placards in the lobby announcing gala media events up in the AI Lab. (We lesser mortals dutifully clomped upstairs to eat the expensive, catered food.)
And yet *nothing* *ever* *happens* in the field.
Every now and then a new "hero" emerges. For a while it was Minsky. In recent years, it has been Rodney Brooks. Regardless, you can see the current hero on TV all the time, commenting on matters as an "AI expert". They don't tell you that Brooks' course is widely viewed as a complete crock; a few puerile algorithms, some linear differential equations, some finite automata, and THAT'S IT. The rest is all blabbering with no substance.
The AI community uses rotating hero-worship in lieu of progress. But it isn't like any of these guys is an actual "AI expert". There are no "AI experts", because there is no such thing as artificial intelligence in this world. They are no more experts on AI than I am an expert on Martian fruit exports. In this field, you don't need real research; an Australian accent and good sense of humor suffice.
True artificial intelligence would be amazing. But the field has made essentially zero progress in the last fifty years. Obviously, it is a really hard problem. On one hand, the AI guys do what other fields do when they're stuck (since they *must* continue to pump out graduate students, attract grants, etc.), they keep trying to change the question. But the pathetic thing is that many completely denigrate the most obviously fair benchmark-- the Turing test.
Coincidentally, a benchmark showing the complete failure of the field.
This strikes me as true: for years and years and years, researchers have been promising AI was just around the corner... And what do we have right now? Nothing!
Well, nothing is a very relative term. We now have AI capable of counting the number of cars on a given street given a photograph of a region, and can automatically follow people / vehicles / animals as they travel around and through objects. OCR is accurate enough to be implemented professionally, and voice recognition is up to 95%. None of these were possible 25 years ago, and not just because of a lack of hardware.
While full AI is still a while away, the first major stumbling block, pattern recognition, is well on its way to being solved.
The AI in Quake 3 is much better than the AI in Pong.
-C
The ______ Agenda
Marvin Minsky is the biggest scientific fraud of the 20th century. The magnitude of his deception dwarfs the cold fusion scientists. He has promised over an over intelligent machines right around the corner to unsuspecting dupes who pour hundreds of millions of dollars into misguided AI research, chosen for flash value rather than potential scientific returns.
But as PT Barnum said, there is a sucker born every minute, so the Minsky machine rolls on...
What is worse the silence from the rest of the scientific community is deafening. Nobody seems willing to take this clown head on.
Lo and behold, what first appeared to be intelligence is now just an elaborate sequence of if-then statements. Anyone could have done it. It's not intelligence at all. It's just following a blueprint. You call this intelligence?
In other words, the lay public expects A.I. to have creativity and strokes of genius, which is much more than they expect of most humans. Or they expect it to be furry with big eyes that makes cooing noises when you pet it. As soon as one realizes that A.I. consists of a computer program, any notion of intelligence evaporates.
you know, the company that apparently everyone wants to DIE ? what other online magazine would/has run this ?
This just shows that we don't actualy know what we mean when we say "Intelligence". It just meant "What I am thinking about when I say Intelligence".
The Turing Test is not a pass-mark to achieve intelligence, it is an outside limit to stop argument. If something passes, completely, the Turing test, then you know you have intelligence. But that is asn extremely high benchmark. It is like saying that if you can outrun all known vehicles, I have to grant you are a fast runner. You *may* still be a fast runner when when you run a lot slower than that - but we will have to enter into a discussion about how fast is fast. Turing just set an endpoint - it it passes his test it is certainly intelligent.
There are two ways the Turing Text could be passed. One is via a special purpose machine to pass it - a human simulator. While of research interest, because building such a machine would tell us a lot about how we actually do work, this is unlikely to be a very useful machine, because it will replicate our weaknesses as well as our strengths. Why spend billions building what half an hours funa and a nine month wait can build. (One-way trips to the stars, perhaps?).
The other way is a general purpose machine which has learned how to copy humans perfectly. By any definition I can think of, this would be an awesomely intelligent machine because it would have learned to understand, and simulate, our minds by the power of pure intellect. Something like playing all the instuments in the orchestra at the same time.
While I think that the first class of machine may well be built in the fullness of time, It will not be very useful. I don't know whether the second class will ever be built - I doubt it.
Which brings us back to the "sub-Turing" class of intelligence. If Turing represents an upper limit to the grey area of where intelligence starts, there must be levels of achievement which would be regarded as intelligent by most, if not all, peoples judgement.
I then ask the question: what use is sub-Turing intelligence? Well, there are lots of tasks which we regard as needing intelligence which we would like to automate. In fact, some of them have already been automated. But when we automate them, we say "we know how that automaton works, so it can't be intelligence". Chess, for example - once regarded as the last test before the Turing test, now regarded as a nifty but essentially unimportant achievement.
We don't actually *know* what we mean when we say "Intelligence". Turing knew that, and provided an empirical rather than analytical test. However, I would say that "Intelligence" bears the same relationship to "Computer Science" as "Magic" does to "Technology" in Clarke's Law: "Any sufficently advanced technology is indistinguishable from magic".
"Any sufficiently advanced Computer Science is indistinguisahable from Intelligence" - Cawley's Law.
Or, to put it another way, Intelligence means "I don't understand how you thought that".
Which explains how Joe Luser thinks his computer is intelligent, whereas Bill Slashdot doesn't.
Consciousness is an illusion caused by an excess of self consciousness.
Not hardly. As it turns out, one of the more frustrating aspects of AI is that once some particular computation that would appear to be correlated with intelligence can be performed, then it invariably doesn't count as AI anymore. So there are lots of practical systems out there today that can prove theorems, do symbolic algebra, play chess better than 99.999% of all people, a whole bunch of stuff. But hardly any of this strikes us as AI anymore. On the other hand, there are lots of horribly difficult problems out there whose solutions we really can't expect to get at within 10 years, and those are all "good" AI problems. Now, one thing that makes them good problems is that we know they contain many different thesis-sized projects that correspond to sub-goals for the "real" problem, and because it is possible that knocking off some of these subgoals could yield some real insights.
Now the interesting thing to notice here is that Turing was a *very* smart guy, and any program that successfully passes the strong version of the Turing Test has almost by definition solved every hard problem that confronts AI, and all of the subproblems that compose those problems, and... It's a truly gargantuan task, and one where even your most advanced programs are almost guaranteed to look really bad in competition.
Having said that, I do still think there is some point in holding contests like the Loebner, not for what they will tell us about the state of how fast AI is progressing, but because the programmers who compete at this point really are trying to scam the system and "get away with" producing a program that is NOT intelligent but that might LOOK like they are intelligent. Understanding how clever these deceptions can be, and why we fall for them, is itself an interesting by product of the competition. So the importance of ELIZA in the end was not that it was a great piece of code or introduced techniques that we could build on directly, but because it taught us a *lot* about people's implicit assumptions about a conversational partner, and how you could generate conversational situations that could finesse the hard stuff. So people don't go out to talk to ELIZA with the goal of determining that it is just a program; they don't go looking for the disconfirming evidence. That's a pretty key point in itself.
Babar
Funny the AI researchers seem to be upset with the contest.
But I find it strange that various people keep trying to either:
1) Take part.
2) Stop the contest.
3) Tell the contest sponsor how to run the contest or spend his money.
Are they really so hard up for Loebner's money? If their stuff really works I'm sure they can get money from other people.
As far as I know none of the AI entrants so far deserve the main prize.
It's almost as if the tailors are upset that someone every year points out the emperor is naked. If indeed the emperor isn't naked why get upset?
Or they admit the emperor is naked and they are just tired of hearing about it? Well so far has any of them admitted that?
> Thus, contrary to what most people would feel
> the thing that is holding AI up is hardware.
Uh? Not only the hardware!
Let's suppose that you have a computer as powerfull as a brain: I give it to you and say now try to pass the Turing test, would you be able to do it?
No, because you would be missing:
1) the software 2) the database.
We have very little clue about how to do the software right now.
And even if you had a software which could be interesting, you'd still have to build a HUGE database if you want to have an interesting result..
And the funny thing is that to really know if your software is interesting or not, first you have to invest a lot of time and money to build the database..
And if a computer is better than another (with the same hardware to simplify comparison) would it be because it has a better software or a better database?
Also I disagree with you that making a competition with the Turing test is only to give researchers bad name: human vs computer chess competitions existed also back when human beat computers without effort and nobody protested that it was giving AI researchers a bad name.
Of course in the end, it seems that beating human has been made thanks to advance in computer power but caused very little progress in AI researches.
I hope that Go competitions between man and machine will be more interesting for AI researchers.
I particurly like how Loebner out foxed Marvin Minsky with the ammunition Minsky gave him.
.. but he still did it.
Sure the guy may be a pot head, might not want a lasting relationship with a woman, and is probally a horribly annoying git from hell.
He did however, manage to outthink the 'brightest' mind in AI research. Maybe the reasons he did were purile
As a programmer I know I was taught to think in small steps, think ahead to the probable issues my code might cause, and to double check my work before dropping it on a production box.
Apparantly Minsky forgot he was a computer scientist when he wrote that news group response.
I'm sure it was just a flame mail, a very human response to frustration and irritation. But as one of the Leading names in AI research, he should have known better.
So, if for nothing else, my hats off to the 'Disco-Floor-Maker' for out thinking one of the 'leaders' in AI research.
Its always nice to watch an acidemic geek get smacked down by someone who lives with the rest of society.
--Ne auderis delere orbem rigidum meum, non erravi pernicose!
Part of the problem is that things that were once considered part of AI have moved out and become mainstream technology. Voice recognition, Expert Systems, Fuzzy Logic, Neural Nets, Chess playing computers...all of these were once considered to be unsolved AI problems but since they are now in common use, we don't consider them a part of AI anymore.
You can find plenty of twenty to thirty year old textbooks that tell you that playing chess at grand master level would be a sign of computer intelligence - now we know that all it takes are some clever heuristics and a lot of CPU power.
As soon as computers can pass the Turing Test, it'll be considered laughable that anyone ever thought it required *intelligence* to chat with a human. In a sense, this has already happened. Quite a few people were convinced by Eliza - but you can tell from just looking at the code that it's not intelligent.
The same thing is happening with animals. We used to define humans as the only tool-using animals - then they found birds breaking open clamshells by dropping rocks on them. The definition changed to humans as the only tool *making* animals...then they found chimpanzees who strip the leaves from twigs before they poke them into anthills. So then it was 'self recognition' - that also failed with dolphins who can recognise themselves in a mirror. Now it's some other thing. Animals will never be labelled intelligent" because the definition of intelligence is that thing that humans have but animals do not.
I predict that we'll never have AI. That isn't a failure of the work - it's in the nature of our definition of Intelligence as "that thing that humans have that animals and machines don't have".
www.sjbaker.org
I predict that we'll never have AI. That isn't a failure of the work - it's in the nature of our definition of Intelligence as "that thing that humans have that animals and machines don't have".
In general I agree with the points you make -- especially that the problem with developing A.I. is that it is a moving target. As you point out, lots of things that used to be holy grails of A.I. have been achieved and dismissed. Remember the article on slashdot awhile back about the walking robot that "figured out" how to escape from the lab? Is that A.I.? Probably not, but it does make you stop and go "Wow, that's kind of neat!"
What I don't agree with in your post is how to seem to reserve the word "intelligence" for human beings. I really don't think most people defines intelligence as "that thing that humans have but animals do not." I think we should consider the goal of A.I. as not trying to copy or better a human, but just successfully achieving some form of independent, creative thought probably on the level of a mammal. You use the example of chimps utilizing twigs to collect ants for eating. I think if a computer program could demonstrate tool-making and tool-using capabilities like that, it should qualify as A.I. Getting a computer to act indistguishably from a human is a pretty tough goal, but if it can demonstrate characteristics of animals with reasonable thought processes (as opposed to brute instinct), I think it would generally be hailed as a milestone in the quest for true A.I.
GMD
watch this
I think you don't know about recent successes of AI just because it is no longer a fashionable buzzword, which in part is due to the fact that AI people *stopped* promising heaven after the 80's. The excessive hype with expert systems caused a backslash -the so called "AI winter"- and researchers learned their lesson. Today, AI focuses on slowly extending the frontiers of solvable problems, with a much more rigorous experimental evaluation of algorithms. Much less hype, but impressive progress.
People often assume that computer intelligence must work the same way as human intelligence, but there's no reason why it should. Indeed nobody expects a computer to do arithmetic *in the same way* as we do; we just care that its circuitry gives the correct answer, no matter how it is designed.
Further we shouldn't expect computers to be best where humans are, or viceversa. Computers excel in (of course) calculation, and (more relevantly for this discussion) combinatorial problems which are impossible to handle for a human, whereas attempts to imitate the intelligence of a five-year old kid fail miserably (this includes many things: natural language understanding, including understanding of context, sensory -e.g. image interpretation - and motor activities, and plain old common-sense knowledge).
Search is a central technique of artificial intelligence since its beginnings (and in any modern textbook as well). There's no point in dismissing search or heuristics as unintelligent, for two reasons: search is often the fastest way of solving problems by computer, and heuristics and related tricks may easily turn a problem from unsolvable (in reasonable time) to very easily solvable. The problem with more human-like computer chess players is simply that they perform much worse. In other words, it is often *not* intelligent for a computer to behave like a human.
Artificial intelligence has progressed inmensely in the last decade, specially in reasoning and learning. Data-mining is simply the buzz-word for AI learning techniques applied to industry.
As for reasoning, in a sense AI still uses 1960's techniques. This is simply because these techniques, with a number of crucial improvements, have proven the best for solving all sorts of industrial problems. Take as an example the modeling of problems as (propositional) logical formulas. There is a huge gap between solvers for this kind of problem from the early 90's and the most recent solvers. Tons of problems which were then very hard are now trivial, and not just because of hardware advances. Still, the basic search framework used by old and new solvers was defined in a still widely cited paper from 1963. Brute force, you say? Well, it is more refined than that, but in the end there's not that much difference conceptually, except that the ways to prune search trees are much more effective now. And now it is possible to control NASA space flights, or verify very large and complex hardware designs, by "just" encoding the problem in propositional logic and solving it with a generic solver. Similarly tons of applications in constraint satisfaction are making their way in industry, for all sorts of problems in production planning, scheduling, optimization, etc. These applications also use search, but are advertised by the task they solve, not as AI. But they are AI.
Hardware is the least of our worries.
To emulate the human brain, one must first understand how the human brain actually works. We know next to nothing about the framework that allows human consciousness. Sure, we know quite a bit about neurons and synapses and such, but we don't know how all of this comes together to form an object that can think.
Even with a massively powerful computer, one hundreds of times more powerful than the human brain, we would be unable to make that computer 'think' because we don't know what it MEANS to 'think.' And even if we did, there is the problem of programming that computer to emulate those brain processes in such a way that the computer can become conscious.
Overall, I don't see a computer being able to think for a long, long time.
"a bunch of researchers have spent 60 million dollars trying to teach it common sense."
Link says:
"The military, which has invested $25 million in Cyc"
Now if you knew the absolute bollocks the military has spent money on, you'd ignore that figure entirely. The military has paid Ingo Swann (or was it P&T?) much more than that for bullshit remote-viewing "research" which has so far proclaimed great success, but been unalbe to reproduce anything under independent scrutiny. Go read James Randi and Martin Gardener, as they've written a fair bit on the absolute nonsense the military has spent its money on.
YAW.
Your head of state is a corrupt weasel, I hope you're happy.
No it can't. Why then has no-one won the gold Loebner Prize yet?
The specification can be extremely simple. Here's mine: Take a panel of 10 computer scientists, a human volunteer and 11 computers. The volunteer and the AI software must both attempt to convince the panel that they're human, in IRC chat or something.
Most AI programs would be exposed as frauds in about 30 seconds or less.
That's why the Turing Test is so good. It's hard - because it's general, not specific. If you think it's specific to a certain task I think you have the wrong idea about what the Turing Test is.
Female Prison Rape in NY
But this displays an interesting limitation of the Turing test - why would you want to hobble a real AI, other than to pass this test? A machine intelligence simply won't be a human intelligence and it doesn't necessarily make sense to ask it to be. A human can't do an imitation of a cat that will convince another cat, but does that really mean anything?
When the first self-aware machine intelligence comes along, I bet the last thing it'll be able to do is convincingly imitate the complexities of a human being, even though it might be at least as complex itself. A machine intelligence is an alien intelligence. To ask it to be able to "pass" as human means you'll need something vastly more intelligent than that. Different doesn't equate to stupider.
As another analogy to the above, consider a deaf person who's perfectly able to express herself (and understand others) in Sign, but unable to communicate with speech. Here it's not even modes of thought (which will be similar) but modes of communication. For a long time other people would assume such a person was "dumb". Now we know better.
That's not an opinion; that's an incorrect factual statement. As for why it's incorrect: the Turing Test implicitly defines intelligent as `indiscernable from a human'. By Leibniz' principle, this means `a human'. So, a computer can never acheive the Turing Test's definition of `intelligence'. Of course, the AI community believes computers can be intelligent, so they have to reject the Turing Test, in much the same way that the practitioners of any field have to reject standards that implicitly outlaw their field. To give an analogy, requiring AIs to hold up under Turing Test conditions would be like requiring theories of evolution to satisfy hard-core Bible-thumpers. Scientists (quite rightly) don't accept those conditions, but no one says that ``makes biology less like science and more like selling Florida time-share condoes''.
There are reasons why democracy does not work nearly as well as capitalism.
-- David D. Friedman