Slashdot Mirror
Upgrading the Turing Test: Lovelace 2.0
mrspoonsi tips news of further research into updating the Turing test. As computer scientists have expanded their knowledge about the true domain of artificial intelligence, it has become clear that the Turing test is somewhat lacking. A replacement, the Lovelace test, was proposed in 2001 to strike a clearer line between true AI and an abundance of if-statements. Now, professor Mark Reidl of Georgia Tech has updated the test further (PDF).
He said, "For the test, the artificial agent passes if it develops a creative artifact from a subset of artistic genres deemed to require human-level intelligence and the artifact meets certain creative constraints given by a human evaluator. Creativity is not unique to human intelligence, but it is one of the hallmarks of human intelligence."
Lovelace 2.0
So, you are blindfolded and have to figure out if it's a human deep-throating you, or the latest Flashlight?
There's nothing wrong with the Turing test, but it needs to have some thought put into the set up and execution, plus competent judges.
This is just making the "Turing test" into a moving target. The Turing test makes sense, and if you have a long enough test you can eventually rule out the "abundance of if statements."
There's a Forest Service joke that the problem with designing trash cans is that the smartest bear is smarter than the dumbest tourist.
'In the following sentence: "Ann gave Sue a scarf. She was very happy to receive it." Does "she" refer to Ann? (yes/no)'
A series of similar and increasingly difficult inference questions like this one can usually knock over an AI pretty easily, while not being too difficult for humans.
Too lazy to RTFP (read the fine PDF), but I assume the point is that some humans can pass the Lovelace test, whereas few or no machines currently can.
Just another wannabe fantasy novelist...
All I can think of while reading up on the Turing and related tests is how many humans would fail such a test.
With the many assumptions made about what constitutes 'true' intelligence, how sure are we of the assumption that a human being of at least average intelligence would pass it? What's the research telling us there so far?
Or are human and artificial intelligence somehow considered to be mutually exclusive?
Site & blog: http://www.mayaposch.com
We will never have "real" AI because every time we approach it, someone moves the bar as to what is required. It's been happening since the mid-late '80s. We *have* what would have qualified as AI according to the rules of '86-'87.
I do not fail; I succeed at finding out what does not work.
The Turing test works just fine. No one has passed it. A bunch of if statements will not pass it either.
The Turing test doesn't require judges per-se, it just requires no limits on the subject matter.
So yet another article on Turing test which completely misses the point... First of all computer scientists never considered Turing test valid test of "artificial intelligence". In fact, there's practically no conceivable reason for a computer scientist to test their artificial intelligence by any other way than making it face problems of its own domain.
Perhaps there will come a day where we really have to ask "is this entertainment droid genuinely intelligent, or is it only pretending", possibly for determining whether it should have rights, but this kind of problem still doesn't lie in the foreseeable future.
On the Other hand, as Turing himself put it in the paper where he introduced his thought-experiment, from Wikipedias phrasing: "I propose to consider the question, 'Can machines think?'" Because "thinking" is difficult to define, Turing chooses to "replace the question by another, which is closely related to it and is expressed in relatively unambiguous words." Turing's new question is: "Are there imaginable digital computers which would do well in the imitation game?"
In other words, the Turing test does not seek to answer the question of whether machines can think, because Turing considered the question meaningless, and noted that if a machines thinking was outwardly indistinguishable from human thinking, then the whole question would become irrelevant.
There is a further erroneous assumption at least in the summary - as of present times, even the most advanced computers and software are basically simply an abundance of if-statements, or for the low-level programmers among us, cmp and jmp mnemonics. If, on the other hand, we expand our definition of a "machine" to encompass every conceivable kind, for the materialistic pragmatic it becomes easy to answer whether machines can ever think - yes of course, the brain is a machine that can think.
The Turing Test has flaws.
Firstly, it requires a human-level of communication. One cannot use the it to determine whether a crow (for example, or cat or octopus) is intelligent since they cannot communicate at our level. Even though these creatures demonstrate a surprising level of intelligence. Watch this video and be astonished.
The extended video shows the crow taking the worm to it's nest, then returning to grab the hooked wire and taking that back to the nest! Can we use the Turing Test to determine whether the crow is intelligent?
Secondly, it conflates intelligence with human intelligence. There's no spectrum of measurement, no "ruler" which can be laid down to measure the level of intelligence in an entity, or to determine whether one entity is more (or less) intelligent than another. Are crows more intelligent than cats? Can the question be resolved using the test? Could the test be used to determine which of two humans is the more intelligent?
But most importantly, the Turing Test has no predictive value: it cannot be used to guide research or development of intelligence.
Consider trying to build a fizzbin, and whether you are successful will be determined by a yes/no decision from a jury of professionals. With no description of what a fizzbin actually is, how hard would it be?
Consider trying to deliver a package, given that you have a GPS system with a broken display. The GPS still works, and the LED will light when you are at the delivery address, but otherwise you have no idea where to go. The address could be in NYC or Tokyo, or anywhere else.
The fundamental problem with the Turing Test is that it doesn't define intelligence(**). Defining something as a test works in mathematics where there is no time or effort to make the axiom of choice on the set of all objects (ie - the universe), but intelligence isn't a purely mathematical concept. It's partly based on a real-world measurement (being: information), and as such is more closely akin to physics.
Instead of a fizzbin, consider trying to build a car. A car can be defined as a body, frame, 4 wheels, engine, and seats, and the purpose is to transport people from place to place (*). A wheel can be further described as a tire on a rim with brakes, a tire can be described as a loop of rubber with steel wires and a valve-stem, a valve-stem as a tube with a schrader valve, a schrader valve is... and so on.
This is a constructive definition: an object is made of simpler objects, each of which is composed of even simpler objects. Math is full of these (a field is a ring plus some stuff, a ring is a group plus some stuff, a group is a set plus some stuff... and so on.)
With the constructive definition, one could build a car directly; or at least, know how to make the attempt. You can determine whether something is a car; and if not, know what needs to be changed.
In my opinion (I'm an AI researcher) the Turing test and the Lovelace test have little value. The tests don't show where to look or how to proceed.
(*) A simplified definition to not lose sight of the position.
(**) This is an academic position. I am a great admirer of Alan Turing and his many brilliant results, including the Turing Test.
I think judging creativity is much more problematic than the classic Turing test.
Modern art, abstract art, dumping some random squares on a canvas. Should that count as creativity?
And how to judge whether the computer has truly created a work or has just reshuffled, or picked at random, some stuff that the programmer has put in. That would require inspection of the source code, not merely the output.
Wasn't there just an article about an AI that developed some magic tricks for stage magicians?
But a machine that could act as a human with IQ 90 would be lauded as a great success. Make the test too hard and you'll only consider something AI when it's smart AI.
That's easy to solve. Shoot the bears that are wearing neck ties.
This test is rather silly, it's easy to come up with a chaotic system that is "beautiful".
The Turing and Computability Function paradigm for computing is (finally) being rightly and fully criticized (ironically, as we get a Turing hollywood movie)
Ada Lovelace's theories ***do indeed*** provide the theoretical ground work (along with others like Claude Shannon) to cleans ourselves of Turing Test nonsense
However...this test...in TFA is not the test.
It's just a variation on the Turing test that still has the same tautology...it's a test of fooling a human in an artificial, one time only environment...which has nothing to do with actual computing
We need to stop pretending we can make a machine that thinks like we do...
It's a tautology and a waste of resources.
Machines follow the instructions we give them via code. End.
Thank you Dave Raggett
exactly...it's all based on a tautology...a faulty ontology. The Computability Function is not a computing paradigm, it's reductive.
'AI' is complex machines following instructions. That's what it is. The rest is people projecting their own emotions onto inanimate objects.
When I say "it's a tautology" what I mean is, it's based on linguistic distinctions only. Not actual, functional distinctions.
A tautology says, "If people think a pile of shit is a steak dinner, then it becomes a steak dinner"
That's an extremem example, but it's actually not that far off from what 'teh singularity' crowd are doing with 'ai'
I'm a telecommunications engineer and cyberneticist...my MS is in Information & Comm Science and I'm ABD in System Science
I'm working on promoting the *Cybernetic* ontology as the foundational paradigm for computing.
Cybernetics for computing would be a combination of Claude Shannon, Norbert Weiner, Lovelace, and others...
In the cybernetic paradigm, the Atanasoff-Berry Computer is the prototypical advancement.
ahem...i'm tweeting about this using the #cybernetics hashtag...it's my way of trying to promote the idea
Thank you Dave Raggett
exactly the point/problem with the Turing and 'teh singularity' paradigms
that's the correct analysis here
Thank you Dave Raggett
There are many criticisms of the Turing test...from many angles.
You address none of them, you just simply stated the negative.
That's the problem...supporting the Turing paradigm means constantly avoiding the question (litterally and figuratively if you think about it)
Thank you Dave Raggett
It was Turing's first attempt to answer the question "what makes a machine intelligent?". As a mathematician he wanted an empirical answer so he felt that the Turing Test would be a good test. A decent idea, but remember, computers had only been around for a few years. I don't know if he'd ever written a program.
But what he had was a user requirements list. He didn't have a working implementation. He had "computer must be able to respond like a human to questions asked", so we have software that fits those requirements. But it's not obvious that it's intelligent. Personally I think computer chess shows more signs of intelligence. It requires imagination, prediction and abstraction. These seem much more important than ability to communicate with a human.
you prefer man-gays in flannel shirts?