Slashdot Mirror
Turing Test Competition At CalTech
Charles Dodgeson writes "The Turing Tournament at Cal Tech wants to
know if you
can program an emulator that will play games like a
human, or if can you write detector that can correctly sort the wetware from the software.
Before you get too excited, the "games" are very limited things. But there is a $10,000 prize for the winner. You can read the gory
details."
Exactly how not needed do we want to make actual people?
Searching for AOL, VB, Bonzi, etc in a binary could easily detect if it is wetware or not. Nothing without those keywords can afford to force out competitors for a profit!
You can't judge a book by the way it wears its hair.
"[The competition] is about as productive as promoting biology by offering a prize to the designer of the most convincing silk flower..." - Steven Pinker
What does this contribute to the field?
Would someone please translate this to something than can be parsed by non-math types?
We're just trying give the humans more time to play tetris
Jason
ProfQuotes
Just make it run around shooting stuff and saying things like "lol u camping fagot!!!!" ;)
Oh, and "my new vidcadr r0x ur world".
Honey, I shrunk the Cygwin
According to The Jargon Dictionary wetware is:
/wet'weir/ n. [prob. from the novels of Rudy Rucker] 1. The human nervous system, as opposed to computer hardware or software. "Wetware has 7 plus or minus 2 temporary registers." 2. Human beings (programmers, operators, administrators) attached to a computer system, as opposed to the system's hardware or software. See liveware, meatware.
wetware
I didn't know what it meant... figured other people may not either.
sig.
its "caltech" NOT "Cal Tech"
So I guess that if you can't understand their algebra in the 2nd part of the explanation, you aren't elible to join..maybe some /.'er could translate that for those of us who haven't graduated grade school yet?
At least the war on the environment is going well
Hang on..so the TT wants me to write something that will play a very limited game in a very limited way? Is that it?
Sounds pretty damn easy to me!
Well! Never mind, then.
Fight with computer brings SWAT team
...if you can program an emulator that will play games like a human...
What? You mean make a bot to miss every shot using the railgun in Quake 3? I think I can whip up some AI for ya!
It's "Caltech", no intercaps, not "Cal Tech", or "CalTech" or whatever other odd spelling that the press came up with last week because no-one pays attention. "Caltech."
Sorry; it's a relative minor gripe from an ex-Lloydie...
"The Turing Tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior"
:-)
humanator2()
{
while (sex=="false")
for (0:ii:4294967296) {
if (ii mod 100!=0) {
call dwnld_porn(kiddie);
else
call mstrbte();
end;
ii++;
}
}
Note the function requires no parameters...and eventually self-destructs
-Chris
--an unbreakable toy is useful for breaking other toys--
From the site:
... ... ...
:-(
An emulator is computer program that takes as input a stage game file, and gives as output a dataset file.
The input file is in the file game.nfg, and the output file should be written in the file dataset.txt, both of which should reside in the same directory as the executable program. Note that the file game.nfg will be written into each emulator's directory by the tournament program prior to running your emulator.
This is exactly the way ACM ICPC contests are conducted, except that if the systems crash for any reason, you're not given extra time to make up for it
And sadly, from the site:
Languages supported:
The computer program that you submit (for either an emulator or a detector) must be written in a combination of one or more of the following languages:
C or C++
java
Perl
Mathematica
Gambit GCL
I would have expected them to atleast add shell-scripting to this - very useful under such conditions to do some Q&D work, or would be taken for granted that since the shell can be a part of the OS, you are free to use it?
Also, would have been nice if they'd added Python to the list, and more importantly Forth (yes, despite what you've heard, Forth is indeed useful, just look at Arthur T Murray's Mind Project).
others have felt that way before
As someone who spent a lot of time working with an actual Turing Test bot, I'd just like to say that the term "Turing Test" really only applies to a "conversation" between computers and people, not emulating behavior in games, etc. I mean, when Turing wrote Computing Machinery and Intelligence, this isn't really what he envisioned to be a Turing Test.
From what I could gather, this is a lot closer to a programming tournament rather than a Turing Test...
I never understood this. Why would I want a computer that ridicules my hair, dress, and generally pathetic life? If I wanted that, I'd just get married!
Alito: A vote for Alito is a punch in the eye to put that bitch back in her place!
The rules are written in a very obscure minimalist fashion, so it took me a while to figure this out but: the game has not been defined! Your program is to get an input file and process it in a manner similar to a human. Currently the website is lacking examples of human output. Therefore, from an Information Theory perspective, we know absolutely nothing about the game.
Now, since I wasted my time figuring this out, I also decifered the instructions: basically the pool consists of a bunch of humans and "emulators" (programs). Each one is given a set of input files that they are supposed to transform into output files. Then the set of output files is run through a detector (human or machine?) that gives the participant some score. You win if your score is most like the humans' scores.
A flash version of the game is available here for those too lazy to download.
Enjoy!
The Turing Test was developed by Alan Turing as one of his "free time endeavors." (I swear, that man had too much time on his hands.) Either way, the Turing Test has been the unsurpassed test for true AI and a human-wetware algorithm. The Turing Test dictates that only when a computer is fully capable of handling a conversation with a human, being able to respond to questions in logical, grammatically correct formats AND being able to learn new slangs and vocabularies - that is true AI (according to Turing). I think this CalTech game is about the same - it's not merely making a computer make mistakes and saying human-like taunts. It's truly.. being human. Those once-in-a-while lucky shots, beginner's luck, fresh leg advantages, the works. That's what they're looking for. Besides, it's CalTech. I think they deserve a little more credit than just merely modifying existing AI. =)
REM HUMAN V2.3
10 BUILD STUFF
20 WANT MORE STUFF
30 BUILD MORE STUFF
40 WANT STUFF YOU CANT HAVE
50 BUILD WEAPONS TO TAKE STUFF FROM WHO DOES HAVE
60 GAIN ENEMIES
70 BUILD BIGGER WEAPONS FOR DEFENSE
80 BUILD SUPER WEAPON
90 DESTROY SELVES
99 GOTO 10
--an unbreakable toy is useful for breaking other toys--
The Turing tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior, and on the other hand, the best computer programs to detect the difference between machine and human behavior. Two types of submissions will be accepted: an emulator, which mimics human behavior, or a detector, which detects the difference between human and machine behavior.
So, I suppose we could say by evaluating the success of response (as would be weeded out by whomever *actually* turns out an entry), we will have achieved our research, VOILA! It's a successful research incentive, the prize that is.
Whaddya think? no? heck of a fight though wasn't it? :P
Amazing. They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language.
Where the hell are the good AI languages? Functional languages? Lisp? Scheme? Caml? SML? Hell, I'd settle for Python.
Justin Dubs
CHECK IT OUT\\\
http://buckland.resnet.gatech.edu/
HE KNOWS WHAT HES TALKIN ABOOT
The 1st annual Turing Tournament was held.
.
With 8 of the world's most lifelike AI programs. But . .
Mr. X, the sponsor of the tournament, took control of the programs and began to take over the world.
MegaEliza: Mr. X, how do you feel about why!?
Mr. X: It's time to tell you the truth. I have been manipulating that fool, DrBoy Neal from the beginning.
Mr. X: But now I no longer need DrBoy Neal's help. Come and face my power, MegaEliza!!
MegaEliza: How do you feel about me stopping you? Follow me, CmdrTaco!
Mega
Eliza
VI
> Press Start
Pass Word
(c) Slashcom 1993
Page sucks!
Oh, er, hmm. Sorry about that.
As part of a "branding" attempt after around WWII, California Institute of Technology refers to itself as "Caltech", not "Cal Tech".
See this Caltech Institute Archive.
Its been awhile sence I read about the subject, but isn't the Turing test just putting people in front of a terminal to talk to either a real person or a AI, and then asking which is the real person. When the same number of testers chose the AI as the real person, then the AI passes the test. Sence when did the Turing name apply to every AI competion? Am I wrong? This isn't a flame just a question about definitions...
Business News and Resources: www.usasource.net
isn't a game a simple form of communitation.
thank God the internet isn't a human right.
You gave a link to a Puzzle Bobble clone. Though Slashdot recently ran an article about Snood, another Puzzle Bobble clone, this article is about a different game.
Was this post written by wetware or hardware?
In all seriousness, the Turing test is really old hat. We've had Eliza the shrink for years, hell, I remember in a CS class a story about an app that emulated a paranoid schitzophrenic(sp) that talked with Eliza.
We've passed the test. It doesn't really mean much as far as AI goes, what's the point? Emulation is NOT simulation. A parrot can emulate a human, that doesnt mean he thinks like one.
In the ICPC, the set of available languages is restricted because the participants are using secured computers (to keep them from getting on the Net and causing trouble on the LAN). You'd think it wouldn't be a problem if participants could submit a binary compiled at home. However the whole point of this contest is for Caltech to see what ideas everyone else has come up with -- so they want to make sure they can read your source code.
you must have missed this node and possibly this one as well
Anyone know what happened to metamod.pl? It's 404 right now.
The guy of the post before me (I say guy because of the tone I got from the post, what? I can't use common sense anymore to form a deductive statement based on the US' political correctness? bs) had a good point of the other applications such a contest can provide value to.
I mean, maybe if you had tens of millions of dollars to analyze how humans do specific tasks compared to computers you might be able to code something up, but just guessing? I don't believe anyone can come up with solutions to this without doing huge amounts of research.
It's one thing to try to write a program that does things as well as humans, it's a whole other things like humans. And it's impossible without an operative definition of 'humanness'
Especially given that there is such a huge range of how 'well' people do things. I mean, some people are idiots, and some are geniuses. How can anyone write a program that can tell the difference between Deep Fritz and a grandmaster, and a program making lots of intentional errors and someone who doesn't know chess? And every class of player in between?
autopr0n is like, down and stuff.
After reading the specs really quick, here comes a simplified summary of how the games work. Don't sue me if this is wrong, I did not look too carefully and it has been more than a year since I left Caltech.
Basically, the two players get to choose numbers with different pay-offs for themselves and the other player.
If you are short-sighted, it is easy to write an 'optimal' program, you always pick the max payoff for yourself.
However, you can also cooperate so that you maximize the average payoff for both players. Hopefully your opponent will realize this and also start picking numbers in this way, if not it is probably wise to go back and be short-sighted and selfish.
Now, the real object of this is to make (or detect) human-like opponents. I would guess that real human players are very irrational, for example they can get pissed if the other player is too selfish and then demand revenge, even though it does not maximize their own payoff.
Tor
Maybe it's just me, but "Before you get too excited" and "gory details" don't really match up with the heading "Turing Test Competition At CalTech".
Things that should have never been allowed to live: GCC "2.96" (really a CVS snapshot done by Redhat as they wanted something more up-to-date than the lovely 2.95.3). This means, that my code (which will probably be running using -Wall -Werror -ansi -pedantic), may/may not compile, depending on such things as the phases of the moon.
WTF is going on? Why are they using the broken gcc? Can the Caltech ppl who set this pass a Turing Test? I doubt it....
Beware the psychokinetic mimes!
The following is an example game file your program is supposed to output a dataset for that is "most human". I give you:
NFG 1 R "game1" { "1" "2" } { 2 2 }
21 3 3 5 3 5 5 3
What is the most human response? Anyone? Anyone?
sad /sad/ adj. 1. Bereft of happiness. 2. Human beings (programmers, operators, administrators) so attached to a computer system that their hardware will never encounter any wetware.
I dont know if this can take a slashdot, but...
http://www.strong.dk/girl.php
" 'the fuck? "
Cover your eyes and click this link!
If you'd read the article (even a little of it) you'd realize that the competition is not a Turing test at all. Unless, of course, you don't know what a Turing test is, in which case you certainly shouldn't be calling the concept "old hat".
old hat adj.
Behind the times; old-fashioned: Last year's styles will be old hat soon.
Overused; trite: That prank is old hat.
So -- exactly how is the Turing test "old hat" in any way? Being around a long time does not necessarily cause something to be less compelling or lose its usefulness (i.e. Fermat's Last Theorem).
Is it behind the times? If so, what's the modern AI test? C'mon, a better one please.
Is it trite? (Lemme help ya, just in case:)
trite adj. triter, tritest
Lacking power to evoke interest through overuse or repetition; hackneyed.
Archaic. Frayed or worn out by use.
If you think either of those apply, then you truly do not understand what a Turing test is.
everything in moderation
I don't think Caltech is the best place to determine what is "human" behavior.
Life at Caltech.
Also, notice that this contest is being held by the "Division of Humanities and Social Sciences".
They SHOULD have said:
The Turing Tournament at Cal Tech wants to distribute $20,000 of the school's funds to their 'internet friends', Caltech friends, and other IVY league schoolers whom they would explain the contest to in human terms.
The contest strives to limit the accessibility of other people by not using human terms whenever possible, and by making the contest information as uninteresting and hard to follow and understand as possible to regular internet visitors to keep the pool as low as possible.
This decreases the chance high school kids or other 2 and 4 year college students would know about this since their professors would not read the dready explanations that they would then have to interpret and explain to students, when the rules and explanations should have been obviously simple to understand already.
Since the attempt is clearly to have a small amount of entries, this makes the competition mostly garbage."
Seriously, their page says it was made with LaTeX2HTML, but they only wrote that because they seriously knew they wouldn't have fooled anyone with telling us the directions were written by a human.
Cover your eyes and click this link!
"Everything seems happier, there's all these new faces,"
You'd think you would have to know grammar to be in a school like Caltech...
Cover your eyes and click this link!
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
This 'contest' reminds of the scene in Ghostbusters where Bill Murray manipulates pre-cog cards to get women. Really, if a ESP buff were to present something like this to the Amazing Randi he would not accept it on the grounds that it was too easy to manipulate.
First and foremost, there is a large sum of money being bandied around. The participants are incented to win by monetary payouts, and two payouts of $10,000 dollars are at stake as well. When games with this high of stakes are being played, great caution is generally used by the house.
But look at the rules of the game...there basically are none. Participants are identified by e-mail address; no rule is specified about the number of entries per person. Also, no rule is specified about collusion between entrants (detectors and emulators). It doesn't take an einstein to figure out how to bias the results of the experient by making enough colluding entries. It is funny this, given that the games themselves are *about* collusion. Its a joke.
Next, notice that there really isnt any way for anybody to tell if the results of the experiement are meaningful, or if they have been manipulated. Its not based on a falsafible proposition and is not scientific. Its no different from any method used by psychic hoaxers of the past. The creators of the 'contest' can manipulate the data, and direct the winnings to their confederates.
Finally, the bizarre nature of the contest should raise some flags. They are giving out $10,000 for a program that would have been hacked together in a couple of months at most. What kind of value could you expect from a contest like this?
There is a good reason for this. If the game (or a small finite set of games) were pre-defined, it would be easy to have a bunch of human subjects play it and then have the emulator regurgitate such a "book". Most entries, I suspect, will be from people or teams who are familiar with studies of how real people do play such games.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
As a recent alum (2002), I can shed some light on some things.
:-) )
Why is this in the Deparment of Humanities and Social Sciences?
From the URL, this Tournament is being run by the Social Sciences Experimental Laboratory (SSEL), not the CS department. The SSEL has been one of the leaders in experimental economics research (read: actually testing all those crazy theories you hear in economics classes).
Why is there money involved?
All experiments by the SSEL involve money. As an undergraduate, I participated in many experiments, mostly involving trading "commodities" in simple (and sometimes not-so-simple) markets. We were paid based on our performance. If I had an off night, I got paid $5 for 2 hours of the experiment. If I had a good night, I could make upwards of $80. Yes folks, this is real money we're talking about here. Since the point is to test people's economic thinking, you must make your decisions based on a real outcome, otherwise the data gathered is invalid.
Why then are they doing this test?
I don't work for the SSEL (and never have), but here's why I think they're doing this: Since they're interested in not only individual human behavior, but also how individuals interact and make choices based on the actions of other individuals, it would be useful to design a computer program that mimics other human's behavior. If other humans think this program acts like a human, then you can do two things: you can take the specifications of the computer program and figure out what qualities of the program humans have. As well, you can then replace humans with the computer program in real experiments (this not only allows you to test the limits of the program, but also to save money
And as an alum (who was not too fond of his time there, but still feels compelled to defend Caltech), it's *Caltech*, not *Cal Tech* or *Cal-Tech* (but if you're feeling lazy, *caltech* is all right too).
nak
(define humanator2 (lambda sex pr0n)
(if(and sex (not (isKiddy pr0n))
(masterbate sex pr0n)
0)))
autopr0n is like, down and stuff.
The only game I want to play is Global Thermonuclear War.
The more entries they get, and I expect there should be quite a few, the more difficult it becomes to come up with a workable strategy for collusion among multiple entires. You'd have to submit a very large number of entries for it to work. I'm not sure if it's practical. If so, they might just modify the rules to prohibit it. They do get to look at the source code, you know.
Whether the results will be meaningful I have no idea. Could be - or maybe not. You'd need to look at a lot of sample human data in very different ways than they're usually looked at for this kind of game to even begin to judge whether it's an interesting challenge.
Four months is more than enough time for something like this. I don't see any problem there.
You seem to assume than anything involving $thousands must be a scam... maybe it is, but the reasons you give for that belief are mostly wrong. I'm sure you'd make an interesting test subject in the game.
Languages supported:
The computer program that you submit (for either an emulator or a detector) must be written in a combination of one or more of the following languages:
* C or C++
* java
* Perl
* Mathematica
* Gambit GCL
How am I supposed to do a combination of one? Stupid tech schools...
go away or i'll replace you with a very simple shell script.
$hungry = 0;
$falls_asleep=0;
$fav_tvshow_ison = 0;
until($hungry || $falls_asleep || $fav_tvshow_ison){
play snood;
}
Obviously I didn't read closely enough. Thanks to everyone who covered my err.
Er, I'm assuming you're a college student. Or high school student. $10,000 isn't that much money. Even, as you say, for a program "hacked together in a couple of months." Let's say it's just two months. That's $5,000/month for the winning entry. That's below what a good programmer makes.
/. banner ads do you think $10K would buy, with what return? What is the value of getting hundreds or thousands of /. types to think about this, and write comments, and generally stir up interest in the contest and company?
And what are they going to do with the program once they get it? Nothing. This isn't about buying programming time. Like most contests, they're betting that the contest itself will buy them more than $10K worth of promotion. How many
It's not much money, but they have already demonstrated a pretty savvy approach to spending it.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
It is not merely the possibility of scamming that is the problem.
There is something very deeply wrong with the theory and the experiment. You say yourself you would have to look at a lot of sample human data to judge this game; think about that for a moment. Wouldn't collecting a large amount of human data and fitting it with a neural network or whatnot be a more straightforward approach that leads to a scientific result?
There is feedback in this contest method that would lead to an unbounded number of refinements to emulators and detectors(and ever more research grants no doubt!) Its a tail chaser.
I am a retired software engineer.
If you believe this contest is really about marketing CalTech rather than a scientific approach to sociology or intelligence, I won't argue.
Note that my post just one way to approach the flaws riddling this contest...
Heh, distributed.net spent years brute-forcing RC5-64 for ... $10000 total. With only $2000 going to actual participants.
There is a flaw that makes this not a true turring test: The competition is set up so that your program only plays itself. Your program does not get a chance to play other programs or real humans. So it is sorta like a turring test where the computer is only allowed to talk with itself.
The true guts of the turring test come from telling if an unknown entity is a human or machine through its interaction with a known human, not its interaction with a copy of itself or another machine.
(sometimes the ineracting human decides, and sometimes an over the sholder observer tries to decide. depends on the setup)
To realy make it work, you need programs that can be input an arbitrary sequence of moves that can then decide the next move. Programs and humans can then be run against humans (and even other programs) randomly. The detectors job would be to decide wich players were human instead of which sets of games were human.
"You saved 1968." - Ms. Valerie Pringle to the crew of Apollo 8
The premise behind this competition is similar to Ai's Learning Machine Challenge. See www.a-i.com for more.
So they have numbers for games? Does chess have numbers too? Would you like to have a match in the game 202039832908130...
so the algorithm should produce data sets similar to human data sets. then it would be much easier to have a human detector before developing a human emulator... otherwise, how to test the emulator (before going to contest)??
for humans (i think) there are a lot of possible strategies (e.g.,revenge or even stupidity, intended 'randomness',always cooperative)... a neural network or similar would learn a 'general' strategy that would not really represent every one of the human strategies...
If my Calc II prof is any indication, I guess they are encouraging non-programmers to participate . . .
That would be trivial for a MMORPG. Just write a program to make the character stand around at popular spots, like vendors or quest dispensers, casting doubt on the manhood of passerby, and it will be indistinguishable from your typical 13 year old gamer. It could even be done with an existing program like Eliza, if its conversation database were tweaked to generate juvenile insults.
8-) Im happy
Singularity: a belief in the "God" idea with the "demiurge" relation inverted.
Go Beavers!
Caltech is a private university and not part of the UC system. To refer to it as "Cal Tech" or "CalTech" is not only wrong and misleading, but does a disservice to the people associated with that fine institution.
That said, I think the stuff happening there is very cool.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
He is right, the web site shouldn't be so technical (not all of us are in humanities, so they got some technical dude to write this up for us 'cs' people, when they could have written a better explanation themselves.)