Slashdot Mirror

I am sure that Professor Jefferson [a critic of AI] does not wish to adopt the extreme and solipsist point of view. Probably he would be quite willing to accept the imitation game as a test. The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has "learnt it parrot fashion." Let us listen in to a part of such a viva voce:

Interrogator: In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?

Witness: It wouldn't scan.

Interrogator: How about "a winter's day," That would scan all right.

Witness: Yes, but nobody wants to be compared to a winter's day.

Interrogator: Would you say Mr. Pickwick reminded you of Christmas?

Witness: In a way.

Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.

Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.

And so on, What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance."

That's an example of what Alan Turing expected of the "Turing Test." And the issue isn't knowledge of sonnets or English lit here or whatever -- it's being able to parse and understand and respond reasonably to demonstrate such understanding. That was Turing's definition of AI. The kind of AI that he predicted by the year 2000 would be able to fool a skilled "interrogator" specifically trying to trip up the AI and identify the computer when an AI would be put up against a human in the "imitation game" test.

When a chatbot can do this, call me. Otherwise, all of this talk about "artificial intelligence," "deep learning," "neural networks," etc. is just fancy words for slightly more powerful statistical tools and adaptive algorithms. Maybe chaining billions of such things together could eventually lead to something that could carry on a conversation like Turing's example, but I've never encountered a chatbot with anything close to that. Most chatbots can't understand a pronoun reference to the previous sentence, let alone make abstract connections as shown in the above quotation.

It is not a test of whether an AI can fool an average person, but whether it can fool an expert.

You are not allowed to redefine the test just because it makes you more comfortable to do so. The original paper simply said "A man, a woman, and an interrogator". It did not qualify that interrogator as an expert, but simply the one who poses the questions (thus, an interrogator)

Well, please re-read the original paper.

You are correct that the original test did not specify an AI expert as interrogator. On the other hand, read the types of dialogue Turing offers as examples. It's very clear that he is imagining "interrogators" (note that word -- it implies someone with a strong drive to ask probing questions) who are not only quite intelligent but also keep asking very probing questions designed to test the intellect of the person/thing on the other side.

The standard is clearly NOT, "Gee, can I have a nice small talk conversation?" Instead, the "interrogator" uses questions varying from computational problems to chess problems to questions about composing a sonnet to detailed discussion of subtle linguistic meanings in English, related in abstract ways to classic literature.

That doesn't sound like your "average Joe" interrogator to me. Does it to you? I'm sure Turing didn't expect all his interrogators to be so intelligent, but they were clearly expected (based on his sample dialogues) to understand how to probe intelligence at a pretty sophisticated level.

The pronoun disambiguation is a good test, because AI does that poorly, and humans do it well. But that is not a replacement for the Turing Test, that IS the Turing Test.

Indeed. Here's an excerpt from Turing's original paper that described the "imitation game," replying to a possible objection that his test would not be able to be used to gauge true understanding as a human might:

Probably [the objector to the test] would be quite willing to accept the imitation game as a test. The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has "learnt it parrot fashion." Let us listen in to a part of such a viva voce:

Interrogator: In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?

Witness: It wouldn't scan.

Interrogator: How about "a winter's day," That would scan all right.

Witness: Yes, but nobody wants to be compared to a winter's day.

Interrogator: Would you say Mr. Pickwick reminded you of Christmas?

Witness: In a way.

Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.

Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.

And so on, What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance."

THAT is the sort of standard of AI that Turing was envisioning could be passed in his "test." It isn't a computer pretending to be a non-responsive teenager with an attitude problem who doesn't really speak the same language as the interrogator (as some chatbots might claim).

It's an idea of AI as something that could debate word replacement in a Shakespearean sonnet, would understand and be able to process poetic scansion, understand the subtle word meanings and connotations in language, and be able to synthesize these various things together while applying such concepts to evaluations of classic literary references.

Turing's test then assumes an AI competent enough to have a flawless conversation on the level of a bright university student or even a colleague of Turing's. Now, granted, we might find the literature quiz a little unnecessary, but in a more general sense this example gets at the idea of probing the AI's understanding of concepts, connecting disparate uses of things together (like a literary character to an abstract concept to a matter of style or poetic form), and in general a fluent and adaptive recognition of linguistic meaning.

I think we would all agree that the various chatbots that have claimed in recent years to have "passed the Turing test" are NOWHERE near this level.

This is the kind of standard Turing himself explicitly mentioned in his original article on the test. And frankly, if I encountered an AI that could have a conversation this fluid and wide-ranging (even if not on literature specifically) in flawless English, I'd be happy to declare it "intelligent." But we don't have anything close to that -- and pretending the "Turing test" is obsolete and needs to be more strict is misunderstanding the ridiculously high expectations Turing himself set out many decades ago.

Interrogator: In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?

Witness: It wouldn't scan.

Interrogator: How about "a winter's day," That would scan all right.

Witness: Yes, but nobody wants to be compared to a winter's day.

Interrogator: Would you say Mr. Pickwick reminded you of Christmas?

Witness: In a way.

Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.

Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.

I think the problem is that the way Turing was picturing the test, the human interrogators would be as smart as Turing and his friends, people who actually know how to ask probing questions. When you look at the conversation above, you see that he had in mind a program that does things which is decades beyond of what chatbots can do today. Everybody is dissing the Turing test, and if it has a problem, it's in that Turing overestimated people, in assuming that they actually know how to have conversations of significance. I still think there is something deeply significant about the Turing test, but in the one that I'm picturing, the interrogators must all be broadly educated experts on natural language processing with specific training in how to expose chatbots. And there should be money on the line for the interrogators: $1000 bonus for each correct identification, $2000 penalty for incorrect identification, no penalty for "not sure". If the majority of such experts can be fooled by an AI under these circumstances, then I think we should all be impressed.

First, thank you, everyone, for the feedback. There are some wonderful stories that I recognize and others that I look forward to reading.

Second, because the solicited essays and fiction will only be a small part of the course, I will have to rely on short stories (including novellas) instead of entire novels. That is part of what makes it hard to research. It's much easier to find out about novels, which have more readers and are better publicized than short stories, especially recent ones that have not yet been widely reprinted.

Third, to those of you who think I am being too lazy to do my research myself, gathering information is part of the research process, and I'd be remiss in not making use of the hive mind if it has useful information that I might not. I would much rather be called a negligent teacher than to be one. Academics study one another's reading lists and syllabi all the time. Believe me, plenty of work remains in deciding what material to include, how present it, etc.

Fourth, thank you for letting me know the history of the word "futurism". The sense I used it ("concern with events and trends of the future or which anticipate the future") is the first one in some dictionaries and is widely used at kurzweilai.net, The Foresight Institute, and other sites I have used, but I will certainly let my students know that some people prefer the word "futurology". For those who are interested, here's a Google n-gram view of "futurism", "futurist", and "futurology".

Fifth, some commenters suggested using primary sources and biography. Agreed. I was already planning to include Turing's Computing Machinery and Intelligence, Vannevar Bush's As We May Think, and the stories of Khan Academy, Iqbal Quadir, Sugata Mitra, and others.

Sixth, it was also suggested that I look at past predictions of the future. Also agreed. I assembled such a reading list for a previous course. It hadn't occurred to me to include in my question what I didn't need, because I'd already assembled it, but I see now that it would be helpful.

Thank you again for the suggestions and even for the criticisms. Soliciting opinions from Slashdot is always a story in itself.

As my AI prof often said, "AI is whatever we haven't taught a computer to do yet."

To anyone with nightmares about metallic Terminator-like machines with eerie red glowing eyes taking over the Earth, I direct you to the current winner of the Loebner Prize, the Elbot. If you still think computers are 'alive,' you may want to consider upgrading your wife/girlfriend/significant other and replacing them with a plastic dildo.

I have to concur, being an actual researcher in the field. That guy is literally unknown by the community.
Not in any conference program committee. Has not written a single paper at an AI conference. Ever. Not even a reviewer
anywhere that I am aware of. check http://www.ijcai.org/ for instance.
or his own homepage : http://www.loebner.net/
quite scary actually ! (check the patent list)

From the article ("Subject" is the AI):

Subject: What is the subject I wonder if I think? I know that it's good to thonk about. Think deeplt about mysterious subjects.

KW: What mysterious subjects?

Subject: Think deeply about mysterious dubjects.

Apparently, these chatbots still have problems with spelling basic english words! The article says that it is a conversation with UltraHAL, which won last year's Loebner prize.

Turing did mention chess as a possible starting point in Computing Machinery and Intelligence. However, even he did not go as far as the grandparent poster in claiming to replicate sentience in anything that way.

Your post is an interesting example of a comment that would be highly unlikely to have been written by a computer. The nuanced observations and extrapolations seem very human indeed. But if you are a computer then your programmer deserves the Loebner prize.

They should run a Perl version with prizes for actually clear code. But I suppose it would be too difficult. Or maybe it would end up as a Loebner contest-style contest: lots of entries, no winners. Oh well.

The authors of this Dvorak bot should really get their act together and fix it they want it to last more than 5 minutes during the next Turing Test challenge.

who will be checking the authenticity of those upcoming Olympic medals?

Is there a defintion of a test that a robot would have to pass to be a convincing human? All the senses would need to be covered, sight, sound, touch, smell and even taste! The Turing Test does the job for sound, the Loebner Prize rewards anyone that can accomplish this. Perhaps there should be prizes for the other senses as well?

I was about to say the same thing. I even entered a program into the Loebner competition once (with the full expectation of coming in last, and I was not disappointed).

Information can be seen here: http://www.loebner.net/Prizef/loebner-prize.html

There may be some subtle differences between this and Turing's original paper, but the spirit of it is the same I think, and it's been held annually for many years now.

I think I first heard of ALICE as one of the entrants. It's won several of the Loebner competitions.

"into an annual competition that might further fuel imaginations."

Yeah, like the Loebner Prize?

Or maybe not.

A group at MIT is using RDF for an integrated data management system. It's sorta like Outlook (or Kontact, if you prefer ;-) on steroids. It's called Haystack.

I have to say, their ideas are intriguing, but after using it... I think the big shortcoming is that it's tough to come up with a generalized user interface for manipulating any data thrown at it. Haystack tries at this, and I think, fails at providing any kind of cues or context that tells you what your are dealing with. In Haystack, every task and piece of information you deal with looks very much like every other piece of data, because, as a design choice, Haystack every piece of data has the same rank as every other piece of data.

Having different applications for different types of data usually make sense, if only to limit the amount of options presented to the user so they can make an intelligent decision about what action they want to perform. See this article on Slashdot about how users need limited since it makes decision-making too difficult psychologically.

Inevitably, discussions around RDF and metadata always devolve into hand-wavy discussions on how the computer will be able to "magically" do smart things based on the metadata. But it really isn't magic and it isn't automatic at all. Equivalencies and mappings have to be created by humans along with the rules about what to do.

RDF uses many concepts from AI research. Anybody who has read about this branch of computer science knows that the discipline has pretty much given up on creating AI in the 'sci-fi' sense as an impractical dream. That's what makes the Loebner prize so controversial. I don't expect that computers will be intelligent enough able to relieve users of too much of the burden in assigning metadata.

RDF is a promising approach, but if you read the article, it makes a lot of assumptions about what needs to happen to make the benefits real. Among them are establishing standards for what metadata fields apply to different types of objects: photos, people, music, etc. That kind of standardization won't happen overnight, if at all.

The computer also needs to know what to do when it encounters that kind of data. The article mentions MIME and browsers and, in effect, says the browser can make a rational decision even if it hasn't seen a particular MIME type before. That isn't really true.. you have to install a plugin that tells the browser what to do, or have a registry that someone has put together where the browser can install the right plugin at the right time.

That said, KDE's unification of contact information and passwords does show some of the promise of metadata efforts. And Apple's Spotlight looks like a good solution as far as it goes. I guess I'm just trying to make the point that the magic of metadata needs to be taken with a fairly large hunk of salt.

"In 1990 Hugh Loebner agreed with The Cambridge Center for Behavioral Studies to underwrite a contest designed to implement the Turing Test. Dr. Loebner pledged a Grand Prize of $100,000 and a Gold Medal for the first computer whose responses were indistinguishable from a human's. Each year an annual prize of $2000 and a bronze medal is awarded to the most human computer. The winner of the annual contest is the best entry relative to other entries that year, irrespective of how good it is in an absolute sense."

Further information on the development of the Loebner Prize and the reasons for its existence is available at Loebner's web site.

AI bots are getting pretty sophisticated. I wouldn't be surprised to know that some large companies may be implementing them as Tier-I support nowadays...

That's practically no use at all. The backlash from the open-source community would be enormous. It would take years to work from that. Remember that the catalyst drivers are for many, many chipsets, integrated into one. Many assignment codes for different chipset capabilities would be gibberish without the documentation.

If you can make a program to eliminate only secret comments, though, and let all the others through, I expect the same program will be able to run and win the Loebner Prize. As far as I know, the only way to figure out if there are any trade secrets in the comments is to have humans read them.

Remember that secret comments are not prefaced with "// THIS IS A SECRET:" They're only secrets for a little while. The engineers who wrote the code may not have really known it was a secret. There may be no secrets at all! But if I were a non-coding executive, I'd be worried, and I might doubt the advisability of open-sourcing it for little or no gain for my company.

On the other hand, if I were starting a new company, I'd set a policy of making all comments open-safe, in case we wanted to open-source the drivers, or in case a court-order required that we expose our code to legal scrutiny.

... if it really is something interesting, it's engine will probably been given a Loebner prize. Other than that, it's just a bunch of flashy marketing talk and some sleek graphics.

As a comparison just play a cool game on the C64 and tell me why that has to be any worse than the sleekest, most graphically attracting game on the pc with the most boring gameplay.

It's all about the contents baby.

For "great and seminal" it's hard to beat Alan Turing's 1950 (!) paper on AI.

Programs like Eliza have been around since and have not increased in their ability to pass the Turing Test signifigantly. The results Loebner Prize do not show a significant improvement, at least not one that correlates to the improvement in speed/storage. Therefore, I don't think that one can infer that a, future, "fast" computer would be able to pass the turing test based on engineering advances. This will have to come from biological/psycological/neurological/algorithmic advances that are not assured.

Presumably you mean that "Unix" brought this on itself? I want to be clear about this: you're claiming that an operating system brought a lawsuit on itself? Does Hugh Loebner know about this?

So you're a geeky cybernetics guy? And no good at sports I presume?
Not to worry, if you can beat this challenge you're better than any olympic competitor.
"Loebner Prize Gold Medal
(Solid 18 carat, not gold-plated like the Olympic "Gold" medals)"

Loebner's home page

As someone who spent a lot of time working with an actual Turing Test bot, I'd just like to say that the term "Turing Test" really only applies to a "conversation" between computers and people, not emulating behavior in games, etc. I mean, when Turing wrote Computing Machinery and Intelligence, this isn't really what he envisioned to be a Turing Test.

From what I could gather, this is a lot closer to a programming tournament rather than a Turing Test...

The problem here is that ALICE is easily determined to be non-intelligent by the average person. ALICE can only pass for an intelligence under conditions so severely constrained that what ALICE is emulating is merely a narrow and relatively trivial part of intelligent behavior. Humans cry out when they are injured -- I don't see anyone claiming that an animal, a rabbit for example, that screams when it's injured is intelligent.

The average person does have trouble determining that Alice is not intelligent, when they have nothing to compare it against. Most people can do it, just not easily. The problem is that a person who is ignoring you is almost indistinguishable from a recording of a person who is ignoring you.

Turing originally suggested that a machine be pitted against a human, with a second human trying to determine which is which. Most of the chatter bots would last about 2 sentences in such a contest, Alice might make 5 if it were lucky.

If the Loebner prize actually used this format, instead of the bastardized version they do run, then we might see some real developement.

-- this is not a .sig

The programs that have competed seem to have received far less attention.

I'm with Shieber on this. Anyone interested in this competition should have read Lessons from a Restricted Turing Test and the answer in In response to lessons from a restricted Turing test (which I found unconvincing, but YMMV).

Its not so much that a computer can fool you into thinking its intelligent, but rather that you can not distinguish which of two candidates is the computer (given that one is human).

I'm sorry, but that is completely false. Yes, certain people did believe that Eliza was a human being - but that wasn't a Turing test, because:

• It didn't involve comparing Eliza to another human being in the same setting (dubious relevance, I admit, but this is a requirement of the Turing Test)
• It didn't involve a panel of judges (this is absolutely necessary to avoid setting the bar too low)
• And, most importantly of all, the people using Eliza were not told to try and reveal whether it was a computer or not! That makes all the difference - to whether the questions are easy or impossibly "hard" for the computer to understand.

The Turing Test has still never been passed. There is a cash prize on offer - the Loebner Prize - to anyone who writes a program which passes the Turing Test. (A much smaller prize is also given to the "most human-like" software at each Loebner Prize competition) Of course, no-one is anywhere near winning the main prize - including Cyc, which is quite capable of spewing out nonsense when confused.

In my opinion - speaking as a computer scientist in training - the Turing Test is an excellent test of AI.

It never got off the ground.

The question I have is for timothy and the other slashdot operators. Why did you pick McKinstry to answer questions about telescopes? I gather that he happens to work at an observatory as a night assistant. Perhaps he has a good deal of knowledge about giant telescopes, but none of the supplied links demonstrate that.

As for hacking consciousness, his idea of minimal intelligence appears to be anything that responds in a non-random fashion. I propose the sequence "11111111111111..."

And we're supposed to ask him serious quesitons? Might as well ask Lars.

I guess I do have a question for Chris. He says that he entered a program in Loebner's Turing Test but withdrew because "Hugh Loebner stated that to win, a program must respond to audio/visual input and not just text." But that is only to win the $100,000 grand prize. Why wouldn't Chris leave his program to compete for the $2,000 (text-only interface) prize?

Steve Robbins

The new form of the problem can be described in terms of a game which we call the "imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B.

The Turing Test is a slight adaptation of this imitation game. There are three players: an interrogator, a human being and a computer. The interrogator is connected to one person and one machine via a terminal, therefore can't see them. The interrogator's task is to find out which of the two candidates is the machine, and which is the human, only by asking them questions. If the machine can "fool" the interrogator, it is intelligent.

FYI, there is also another competition called the Loebner Contest, where a $100,000 prize is offered to the author of the first program to pass this Turing test. Bots have entered this competition often, with varied degrees of results.

From what I read, most people working in AI don't treat them as something worth while. It's fairly obvious that programs won't be able to pass the turing test for some time (decades, maybe centuries), and the results of such tests only make it less likely that people working on valid AI projects will be taken seriously.

The Loebner Prize has it's own homepage. Chech out the transcripts of the conversations. The most 'clever' programs simply look for keywords, some insist on asking all the questions, some are 'whimsical' and use metaphores while constatnly switching topics, none show any comprehension whatsoever.