Slashdot Mirror
The Lovelace Test Is Better Than the Turing Test At Detecting AI
meghan elizabeth writes If the Turing Test can be fooled by common trickery, it's time to consider we need a new standard. The Lovelace Test is designed to be more rigorous, testing for true machine cognition. An intelligent computer passes the Lovelace Test only if it originates a "program" that it was not engineered to produce. The new program—it could be an idea, a novel, a piece of music, anything—can't be a hardware fluke. The machine's designers must not be able to explain how their original code led to this new program. In short, to pass the Lovelace Test a computer has to create something original, all by itself.
Maybe they mean the "Linda Lovelace" test?
That is all.
When was the last time the average person created something original?
if a human cannot determine if they just got a hummer from a machine or another human?
That is a flatly ludicrous requirement, far in excess of what we would ever even consider applying to determine if even a human being is intelligent or not. Hell, if you were to apply that standard to human beings, ironically, many extremely intelligent people would fail that metric, because in hindsight, you can very often identify precisely how a particular thought or idea came out of a person.
File under 'M' for 'Manic ranting'
It was passed as defined: 10 out of 30 judges (lay people) thought they were talking with a human when they were talking with a machine in 5 minute chat sessions. Whether passing this is any way significant is up for debate, but the test was passed.
I do recall reading a while back experiments done with AI in which programs compete for resources by generating programs to do tasks given to it (computing sums etc). Some programs did generate code that were completely unexpected.
It raises the question programs that are evolved are designed by the programmer or the program, or the process of evolution. And it also raises the philosophical question about whether we should be more humble and accept that our "creativity" that we think is what makes humans intelligent could be nothing more than a process of the evolution of ideas (I hesitate to use the word meme) that we don't actually originate nor control.
If we consider programs that can create things through evolution as "intelligent", that would ironically make natural selection intelligent, since DNA is a digital program that is evolved into complex things over time that can't be reduced to first principles.
Those who do not learn from commit history are doomed to regress it.
The machine's designers must not be able to explain how their original code led to this new program.
Whoa, whoa, whoa. I have severe problem with this. This is like looking at obscurity and declaring it a soul. The measure of intelligence is that we can't understand it? Intelligence through obfuscation? There should be no way for a designer to not be able to figure out why their machine produced what it did given enough debugging.
It was passed as defined
The Turing Test was not passed, and the only people who claim it was are ignorant reporters looking for an easy story with a catchy headline and tech morons who also believe Kevin Warwick is a cyborg.
The test was rigged in every way possible:
- judges told they were talking to a child
- that doesn't speak English as a primary language
- which was programmed with the express intent of misdirection
- and only "fooled" 30% of the judges.
And, even after all that, Cleverbot did a much better job back in 2011 with a 60% success rate.
This Eugene test outcome was a complete farce -- something to remind everyone that Warwick still exists and to separate the ignorant and sensational tech news trash rags from the more legitimate sources of information.
"What do you despise? By this are you truly known." --Princess Irulan, Manual of Muad'Dib
/)
Most of the programs I write produce stuff I can't explain.
The machine's designers must not be able to explain how their original code led to this new program.
If I'm not mistaken, this has already happened when evolutionary algorithms were applied to hardware design: some slides. The author of the program has no idea how the resulting circuit worked.
Ezekiel 23:20
Oddly computer chess programs may already meet this criteria. The programs usually apply a weight or value to a move and a weight and a value to the consequences down stream of the move. But there are times when the consequences are of equal value at some event horizon and random choices must be applied. As a consequence sequences of moves may be made that no human has ever made and the programmer could not really predict either. As machines have gotten more able the event horizon is at a deeper level. But we might reach the point at which only the player playing white can ever hope to win and the player with black may always lose. We are not in danger of a human ever being able to do that unless we alter his brain.
That's because they keep shifting the goalposts.
They are shifting them again. This new test includes this requirement: The machine's designers must not be able to explain how their original code led to this new program. So now anything we understand is not intelligence??? So if someone figures out how the brain works, and is about to describe its function, then people will no longer be intelligent? Intelligence is a characteristic of behavior. If it behaves intelligently, then it is intelligent. The underlying mechanism should be irrelevant.
Just give me the blow by blow account.
Alas, the test that was "passed" was not actually the test Turing proposed.
So it passed the Turingish test.
http://en.wikiquote.org/wiki/I... Detective Del Spooner: Human beings have dreams. Even dogs have dreams, but not you, you are just a machine. An imitation of life. Can a robot write a symphony? Can a robot turn a... canvas into a beautiful masterpiece? Sonny: [With genuine interest] Can you?
Just because someone sets some random people up for a five minute interview with a chatbot doesn't mean they're running a Turing Test.
Give people enough time to conduct a proper conversation, hell give them time to ask the chatbot for some original content. Do that and you'll be running a real Turing Test.
The reason you keep hearing about these simplified Turing Tests is those are the only tests people run because those are the only tests computers can pass. But passing a true Turing Test is still a great standard for detecting real AI, and something no one can even approach doing yet.
I stole this Sig
That's deep, man.
That's because they keep shifting the goalposts.
I don't think "a chatbot isn't AI and hasn't been since the 1960s when they were invented, whether you call it a doctor or a Ukrainian kid doesn't make any difference" counts as shifting the goalposts.
Furthermore, reproducible results are an important part of science. Let him release his source code, or explain his algorithm so we can reproduce it. Anything less is not science.
"First they came for the slanderers and i said nothing."
Gives a whole new meaning to, "My computer went down on me..."
Do not look into laser with remaining eye.
No.
you describe "behaviorism" which is a thoroughly discredited and reductive theory
the ***whole conversation*** is about ***the underlying mechanism***
the "Lovelace Test" is more rigorous, but how it will affect computing I cannot say, because the Turing Test itself is a time-wasting notion.
the problem: questions of "what is intelligence" are Philosophy 101 questions...not scientific or computing questions...and we hurt our industry when we overlap the two
just because we can prod a human to make them do something, or dose them with a chemical or whathaveyou, doesn't mean we have disproven the existence of "free will"
we will map every neural connection in the human brain soon, this doesn't mean all humans will become remote controlled techno-zombies
people take other's freedom by many means:
by gunpoint
emotional manipulation
through blackmail
too much alchohol
the Frey Effect
threats of loss of work
so learning how neurons work is just another potential addition to that list
the point: humans have free will and it can be subverted in many ways, this does not have any implications in computing
Thank you Dave Raggett
It was passed as defined
The Turing Test was not passed, and the only people who claim it was are ignorant reporters looking for an easy story with a catchy headline
Indeed. There's a lot of misinformation out there about what Turing originally specified. The test is NOT simply "Can a computer have a reasonable conversation with an unsuspecting human so that the human will not figure out that the computer is not human?" By that standard, ELIZA passed the Turing test many decades ago.
The test also doesn't have a some sort of magical "fool 30%" threshold -- Turing simply speculated that by the year 2000, AI would have progressed enough that it could fool 30% of "interrogators" (more on that term below). The 30% is NOT a threshold for passing the test -- it was just a statement by Turing about how often AI would pass the test by the year 2000.
So what was the test?
The test involves three entities: an "interrogator," a computer, and a normal human responder. The interrogator is assumed to be well-educated and familiar with the nature of the test. The interrogator has five minutes to question both the computer and the normal human in order to determine which is the actual human. The interrogator is assumed to bring an intelligent skepticism to the test -- the standard is not just trying to have a normal conversation, but instead the interrogator would actively probe the intelligence of the AI and the human, designing queries which would find even small flaws or inconsistencies that would suggest the lack of complex cognitive understanding.
Turing's article actually gives an example of the type of dialogue the interrogator should try -- it involves a relatively high-level debate about a Shakespearean sonnet. The interrogator questions the AI about the meaning of the sonnet and tries to identify whether the AI can evaluate the interrogator's suggestions on substituting new words or phrases into the poem. The AI is supposed to detect various types of errors requiring considerable fluency in English and creativity -- like recognizing that a suggested change in the poem wouldn't fit the meter, or ir wouldn't be idiomatic English, or the meaning would make an inappropriate metaphor in the context of the poem.
THAT'S the sort of "intelligence" Turing was envisioning. The "interrogator" would have these complex discussions with both the AI and the human, and then render a verdict.
Now, compare that to the situation in TFS where the claim is that the Turing test was "passed" by a chatbot fooling people. That's crap. The chatbot in question, as parent noted, was not even fluent in the language of the interrogator, it was deliberately evasive and nonresponsive (instead of Turing's example of AI's and humans having willing debates with the interrogator), there was no human to compare the chatbot to, the interrogators were apparently not asking probing questions to determine the nature of the "intelligence" (and it's not even clear whether the interrogators knew what their role was, the nature of the test, whether they might be chatting with AI, etc.).
Thus, Turing's test -- as originally described -- was nowhere close to "passed." Today's chatbots can't even carry on a normal small-talk discussion for 30 seconds with a probing interrogator without sounding stupid, evasive, non-responsive, mentally ill, and/or making incredibly ridiculous errors in common idiomatic English.
In contrast, Turing was predicting that interrogators would have to be debating artistic substitutions of idiomatic and metaphorical English usage in Shakespeare's sonnets to differentiate a computer from a real (presumably quite intelligent) human by the year 2000. In effect, Turing seemed to assume that he would talk to the AI in the way he might debate things with a rather intelligent peer or colleague.
Turing was wrong about his predictions. But that doesn't mean his test is invalid -- to the contrary, his standard was so ridiculously high that we are nowhere close to having AI that could pass it.
So now anything we understand is not intelligence?
I heard a great anecdote about this from an MIT proffessor on youtube. Back in the 80's the professor developed an AI program that could translate equations into the handful of standard forms required by calculus and solve them. A student heard about this and went calling to see the program in action. The professor spent an hour explaining the algorithm, when the student finally understood he exclaimed, "That's not intelligent, it's doing calculus the same way I do".
It could be argued that neither the student nor the computer were intelligent since they were simply following rules, but if that's the case the only those handful of mathematicians who discovered the standard form are intelligent. It should also be noted that since that time computers routinely discover previously unknown mathematical truths by brute force extrapolation of the basic axioms of mathematics, however none of them have been particularly useful for humans.
When people dispute the existence of AI what they are really disputing is the existence of artificial consciousness, we simply don't know if a computer operating a complex algorithm is conscious and quite frankly it's irrelevant to the question of intelligence. For example most people who have studied ants agree an ants nest displays highly intelligent behaviour, they have evolved a more efficient and generally better optimised solution to the travelling salesman problem than human mathematics (or intuition) can provide, yet few (if any) people would argue that an ant or it's nest is a conscious being.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
You're describing Core War. You can still get the source.
"the server sucked my job right in"
Some drink at the fountain of knowledge. Others just gargle.
I think Watson would be able to give it's real age by finding the information rather than recalling it, although it might get confused by progressive versions. AI can also produce a picture of a generic rabbit, or cat as the case may be.
The thing that Watson (and AI in general) has difficulty with is imagination, it has no experience of the real world so if you asked it something like what would happen if you ran down the street with a bucket of water, it would be stuck. Humans who have never run with a bucket of water will automatically visualise the situation and give the right answer, just as everyone who read the question have just done so in their mind. OTOH a graphics engine can easily show you what would happen to the bucket of water because it does have a limited knowledge of the physical world.
This is the problem with putting AI in a box labeled "Turing test", it (arrogantly) assumes that human conversation is the only definition of intelligence. I'm pretty sure Turing himself would vigorously dispute that assumption if he were alive today.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
The easy way to resolve the question is to figure out what algorithm the human brain runs. After that, the question of intelligence is trivial.
"First they came for the slanderers and i said nothing."
One of Feynman's memoirs includes the haha-only-serious observation that mathematical theorems are either unproven or trivial, and this is simply a re-statement of the same principle.
And actually, there's a lot of speculation about whether colonies exhibit intelligence or consciousness (eg Hofstadter's Aunt Hillary, but also Jack Cohen & Ian Stewart's Heaven - they also did the Science of the Discworld series with pterry).
Turing was wrong about his predictions. But that doesn't mean his test is invalid
Imho it is.
Suppose we manage to create a strong AI. It's fully conscious, fully aware, but for some quirk we cannot understand, it's 100% honest.
Such an AI would never pass the Turing test, because it would never try to pass off as human, and any intelligent human could ask it questions that only a machine could answer in limited time.
Turing was wrong about his predictions. But that doesn't mean his test is invalid
Imho it is.
Suppose we manage to create a strong AI. It's fully conscious, fully aware, but for some quirk we cannot understand, it's 100% honest.
Such an AI would never pass the Turing test, because it would never try to pass off as human
That sounds like a legit point at first, but think about it for a sec. Programming a computer to lie and be evasive about its nature is easy, and many chatbots can already do that. Asking a strong AI "are you a computer?" or "what did you have for breakfast?" would not be useful for evaluating intelligence. Getting the AI to debate an intellectual topic, on the other hand, will be less likely to require deception but would be a better measure of intelligence. That's another fundamental point people miss: The point of the Turing test was to imitate human INTELLIGENCE, NOT to pretend to be a physical human.
A knowledgeable interrogator trying to evaluate intelligence would thus likely be more interested in asking intellectual questions, rather than queries just designed to test whether the computer can make up some nonsense about itself.
Given toys for sale and various videos across the Internet, I don't believe most people care whether it was a human or machine that just got them off.
"Lack of speed can be overcome. In the worst case by patience." --Znork
Good points. I would add one more- people lie. There is nothing to stop the human comparison from lying and saying he was a computer as well. If both say they are a computer that should level the playing field so that they both need to judged on the merits of the debate.
I have always respected both Ada Lovelace's and Alan Turing's genius, but the "Turing Test" has always seemed too simplistic for me. For my purposes in discussing the matter I use what I call the Alan Turing "Surprise" Test: Can a computer produce relevant responses with an unexpected but relevant response (aka "surprise") in them? Examples include puns, twists-of-phrase, sarcasm, and other artifacts of a quick-thinking conversationalist. (And, for the record, I don't consider Trolls as members of any of these classes; their range of responses is severely limited in context and devoid of any pretense of humanity. Some of you can prove that in the responses here.)
Eliza and its' various successors have never qualified, and so far only rudimentary steps have been made toward the elementary Turing Test. However, the goal is to determine whether a human can distinguish between another human's responses and a computer's responses. I'd put my Turing-Surprise test right in between the traditional definition of the Turing Test and the Lovelace Test.
Nice servers don't go down.
I can see the fnords!
Why is it called the Lovelace test?
Maybe it's because Ada envisioned that the machines that would become computers would one day be capable of all kinds of useful things, as opposed to Babbage who saw them strictly as number crunchers.
Ada Lovelace was just someone that translated a book for the worlds first programmer.
Hardly. She didn't translate the book for a programmer, she translated the book for a machine. She was the programmer.
"Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black