Slashdot Mirror
Was Turing Test Legitimately Beaten, Or Just Cleverly Tricked?
beaker_72 (1845996) writes "On Sunday we saw a story that the Turing Test had finally been passed. The same story was picked up by most of the mainstream media and reported all over the place over the weekend and yesterday. However, today we see an article in TechDirt telling us that in fact the original press release was just a load of hype. So who's right? Have researchers at a well established university managed to beat this test for the first time, or should we believe TechDirt who have pointed out some aspects of the story which, if true, are pretty damning?"
Kevin Warwick gives the bot a thumbs up, but the TechDirt piece takes heavy issue with Warwick himself on this front.
I want to talk to these AIs myself! Give me a webpage or irc chatroom to interact with it directly. It sounds fascinating even if its only 'close' to passing the test.
I have successfully written a chatbot that convinces people that it is a slime mold. It had to tell people it was a slime mold to make them do the mental gymnastics necessary to wave away all the absurd replies. But, it did manage to convince 90% of its conversational partners that it had the mental capacity of a slime mold. This is a striking success.
They got 30% of the people to think they were texting with a child with limited language skills. I don't think that's what Alan Turing had in mind.
I always thought of it as more a philosophical question or thought experiment. How do you know that anything has an internal consciousness when you can't actually observe it? I can't even observe your process, I just assume that you and I are similarly in so many other ways (well I assume, you could be a chatbot, whreas I know I am definitely not)....and I have it, so you must too, aferall, we can talk.
So.... if a machine can talk like we can, if it can communicate well enough that we suspect it also has an internal cosciousness, then isn't our evidence for it every bit as strong as the real evidence that anyone else does?
"I opened my eyes, and everything went dark again"
It has nothing to do with actual artificial intelligence and everything to do with writing deceptive scripts. It's not just this incident, it's a problem with the goal of the Turing test itself. I always found the Turing test a kind of stupid exercise due to this.
Yes. TechDirt's points 3 and 6 are basically the same thing I wrote here the other day:
First, that the "natural language" requirement was gamed. It deliberately simulated someone for whom English is not their first language, in order to cover its inability to actually hold a good English conversation. Fail.
Second, that we have learned over time that the Turing test doesn't really mean much of anything. We are capable of creating a machine that holds its own in limited conversation, but in the process we have learned that it has little to do with "AI".
I think some of TechDirt's other points are also valid. In point 4, for example, they explain that this wasn't even the real Turing test.
Actually, that's not the whole point -- it's not even the point at all, which is what most people here are pointing out.
The test CAN be beaten without clever tricking: it can be beaten with a program that actually thinks.
This was Turing's original intent. He didn't think, "I'm going to make a test to find someone who can write a program to trick everyone into thinking the program is intelligent." He thought, "I'm going to make a test to find someone who has written a program that is actually intelligent." See the difference?
The only reason we're in this stupid mess with the Turing test right now is that most laypeople (including reporters) can't see the difference between those two positions.
(posting AC because I lost my password)
The first time I saw ELIZA in action, I realized that the Turing test is basically meaningless, as it fails on two fronts. We are not good judges for it, as we are hard-wired to assume intelligence behind communications, and Turing's assumption that the ability to carry on a reasonable conversation was a proof of intelligence was wrong.
This is not to fault Turing's work, as you have to start somewhere, but, really, after all of these years we should have a better test for intelligence.
So if there were an AI system which genuinely had the intellect and communication capabilities of a 13-year-old Ukrainian boy (conversing in English), you would not consider it intelligent?
Not until I posed questions in Ukrainian.
This is a good point. I'm guessing every single one of the entries into these Turing test competitions since 'Eliza' has been an attempt by the programmer to trick the judges. Turing's goal, however, was that the AI itself would be doing the tricking. If the programmer is spending time thinking of how to manufacture bogus spelling errors so that they bot looks human, then I'm guessing Turing's response would be that this is missing the point.
Kevin Warwick is a narcissistic, publicity seeking shitcock.
The problem is that priming the judges with excuses about why the candidate may make incorrect, irrational, or poor language answers is not part of the test.
If the unprimed judges themselves came to the conclusion they were speaking to a 13 year old from the Ukraine, then that would not be a problem. But that's not what happened.
So.... if a machine can talk like we can, if it can communicate well enough that we suspect it also has an internal cosciousness, then isn't our evidence for it every bit as strong as the real evidence that anyone else does?
Not even close, because our conclusion about other humans is based on a huge amount of non-verbal communication and experience, starting from the moment we are born. AI researchers (and researchers into "intelligence" generally) conveniently forget that the vast majority of intelligent behaviour is non-verbal, and we rely on that when we are inferring from verbal behaviour that there is intelligence present.
Simply put: without non-verbal intelligent behaviour we would not even know that other humans are intelligent. Likewise, we know that dogs are intelligent even though they are non-verbal (I'm using an unrestrictive notion of "intelligent" here, quite deliberately in contrast to the restrictive use that is common--although thankfully not universal--in the AI community.)
With regard to the Turing test as a measure of "intelligence", consider it's original form: http://psych.utoronto.ca/users...
Turing started by considering a situation where a woman and a man are trying to convince a judge which one of them is male, using only a teletype console as a means of communication. He then considered replacing the woman with a computer.
Think about that for a second. Concluding, "If a computer can convince a judge it is the human more than 50% of the time we can say that it is 'really' intelligent" implies "If a woman can convince a judge she is male more than 50% of the time we can say she is 'really' a dude."
The absurdity of the latter conclusion should give us pause in putting too much weight on the former.
Blasphemy is a human right. Blasphemophobia kills.
What should the program have claimed to have been?
I don't care. What I care about is what the organisers of the "test" told the judges. I was under the impression they had told the judges it was a 13 years old boy from the Ukraine. Now I look again, it's not clear who told them that. Which brings another problem: we don't know what the judges were told. Given the effort to invite a celebrity to take part as one of the judges, you'd have thought there would be video of the contest. But no.
If you've been around tech for a while, you will have come across some of Kevin Warwick's bullshit claims to the press before. He's a charlatan. So therefore we need more than his say so that he conducted the test in a reasonable way.
We also need independent reproduction of the result. You know, the scientific method and all that.
That was my general understanding of it too. It was less about devising an actual test for computer intelligence, and more about making a philosophic point that we never directly observe intelligence. We only observe the effects-- either in words or actions-- and then guess whether something is intelligent by working backward from those effects. For example, my coworker is sitting next to me, and I see him talking in sentences that appear to make sense. I ask him a question, and I get a response back. When I listen to his response, I analyze it and decide whether it seems like an appropriate or insightful response to my question. As I result, I guess that he's reasonably intelligent, but that's the only thing I have to go on.
So in talking about machine intelligence, Turing suggested that it may not be worthwhile to dwell on whether the machine is actually intelligent, but instead look at whether it can present behavior capable of convincing people that it's intelligent. If I can present questions to a machine and analyze the response, finding that it's as appropriate and insightful as when I'm talking to a human, then maybe we should consider that machine to be intelligent whether it "actually" is intelligent or not.
Still, to me it seems like there's some room for debate and room for problems. For example, do we want to consider it intelligent when a machine can convince me that it's a person of average intelligence, or do we want to require that it's actually sensible and smart? It may be that if an AI gets to be really intelligent, it starts failing the test again because it's answers are too correct, specific, and precise.
There's a further problem in asking the question, once we have AI that we consider "intelligent", will it be worth talking to it? Maybe it will fool us by telling us what we want to hear or expect to hear.
I'm not sure Turing had the answers to whether an AI was intelligent any more than Asimov has the perfect rules to keeping AI benign.