Slashdot Mirror
Turing Test Competition At CalTech
Charles Dodgeson writes "The Turing Tournament at Cal Tech wants to
know if you
can program an emulator that will play games like a
human, or if can you write detector that can correctly sort the wetware from the software.
Before you get too excited, the "games" are very limited things. But there is a $10,000 prize for the winner. You can read the gory
details."
Exactly how not needed do we want to make actual people?
Would someone please translate this to something than can be parsed by non-math types?
We're just trying give the humans more time to play tetris
Jason
ProfQuotes
Just make it run around shooting stuff and saying things like "lol u camping fagot!!!!" ;)
Oh, and "my new vidcadr r0x ur world".
Honey, I shrunk the Cygwin
According to The Jargon Dictionary wetware is:
/wet'weir/ n. [prob. from the novels of Rudy Rucker] 1. The human nervous system, as opposed to computer hardware or software. "Wetware has 7 plus or minus 2 temporary registers." 2. Human beings (programmers, operators, administrators) attached to a computer system, as opposed to the system's hardware or software. See liveware, meatware.
wetware
I didn't know what it meant... figured other people may not either.
sig.
Well! Never mind, then.
Fight with computer brings SWAT team
...if you can program an emulator that will play games like a human...
What? You mean make a bot to miss every shot using the railgun in Quake 3? I think I can whip up some AI for ya!
"The Turing Tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior"
:-)
humanator2()
{
while (sex=="false")
for (0:ii:4294967296) {
if (ii mod 100!=0) {
call dwnld_porn(kiddie);
else
call mstrbte();
end;
ii++;
}
}
Note the function requires no parameters...and eventually self-destructs
-Chris
--an unbreakable toy is useful for breaking other toys--
From the site:
... ... ...
:-(
An emulator is computer program that takes as input a stage game file, and gives as output a dataset file.
The input file is in the file game.nfg, and the output file should be written in the file dataset.txt, both of which should reside in the same directory as the executable program. Note that the file game.nfg will be written into each emulator's directory by the tournament program prior to running your emulator.
This is exactly the way ACM ICPC contests are conducted, except that if the systems crash for any reason, you're not given extra time to make up for it
And sadly, from the site:
Languages supported:
The computer program that you submit (for either an emulator or a detector) must be written in a combination of one or more of the following languages:
C or C++
java
Perl
Mathematica
Gambit GCL
I would have expected them to atleast add shell-scripting to this - very useful under such conditions to do some Q&D work, or would be taken for granted that since the shell can be a part of the OS, you are free to use it?
Also, would have been nice if they'd added Python to the list, and more importantly Forth (yes, despite what you've heard, Forth is indeed useful, just look at Arthur T Murray's Mind Project).
As someone who spent a lot of time working with an actual Turing Test bot, I'd just like to say that the term "Turing Test" really only applies to a "conversation" between computers and people, not emulating behavior in games, etc. I mean, when Turing wrote Computing Machinery and Intelligence, this isn't really what he envisioned to be a Turing Test.
From what I could gather, this is a lot closer to a programming tournament rather than a Turing Test...
I never understood this. Why would I want a computer that ridicules my hair, dress, and generally pathetic life? If I wanted that, I'd just get married!
Alito: A vote for Alito is a punch in the eye to put that bitch back in her place!
The rules are written in a very obscure minimalist fashion, so it took me a while to figure this out but: the game has not been defined! Your program is to get an input file and process it in a manner similar to a human. Currently the website is lacking examples of human output. Therefore, from an Information Theory perspective, we know absolutely nothing about the game.
Now, since I wasted my time figuring this out, I also decifered the instructions: basically the pool consists of a bunch of humans and "emulators" (programs). Each one is given a set of input files that they are supposed to transform into output files. Then the set of output files is run through a detector (human or machine?) that gives the participant some score. You win if your score is most like the humans' scores.
Does anybody actually read the stuff pointed to before posting?
This actually is useful, but not for AI. There is a whole branch of what is called "experimental game theory". Getting something that plays these games like humans is interesting (well, to me at least).
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
The Turing Test was developed by Alan Turing as one of his "free time endeavors." (I swear, that man had too much time on his hands.) Either way, the Turing Test has been the unsurpassed test for true AI and a human-wetware algorithm. The Turing Test dictates that only when a computer is fully capable of handling a conversation with a human, being able to respond to questions in logical, grammatically correct formats AND being able to learn new slangs and vocabularies - that is true AI (according to Turing). I think this CalTech game is about the same - it's not merely making a computer make mistakes and saying human-like taunts. It's truly.. being human. Those once-in-a-while lucky shots, beginner's luck, fresh leg advantages, the works. That's what they're looking for. Besides, it's CalTech. I think they deserve a little more credit than just merely modifying existing AI. =)
REM HUMAN V2.3
10 BUILD STUFF
20 WANT MORE STUFF
30 BUILD MORE STUFF
40 WANT STUFF YOU CANT HAVE
50 BUILD WEAPONS TO TAKE STUFF FROM WHO DOES HAVE
60 GAIN ENEMIES
70 BUILD BIGGER WEAPONS FOR DEFENSE
80 BUILD SUPER WEAPON
90 DESTROY SELVES
99 GOTO 10
--an unbreakable toy is useful for breaking other toys--
The Turing tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior, and on the other hand, the best computer programs to detect the difference between machine and human behavior. Two types of submissions will be accepted: an emulator, which mimics human behavior, or a detector, which detects the difference between human and machine behavior.
So, I suppose we could say by evaluating the success of response (as would be weeded out by whomever *actually* turns out an entry), we will have achieved our research, VOILA! It's a successful research incentive, the prize that is.
Whaddya think? no? heck of a fight though wasn't it? :P
Amazing. They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language.
Where the hell are the good AI languages? Functional languages? Lisp? Scheme? Caml? SML? Hell, I'd settle for Python.
Justin Dubs
Page sucks!
Oh, er, hmm. Sorry about that.
As part of a "branding" attempt after around WWII, California Institute of Technology refers to itself as "Caltech", not "Cal Tech".
See this Caltech Institute Archive.
Its been awhile sence I read about the subject, but isn't the Turing test just putting people in front of a terminal to talk to either a real person or a AI, and then asking which is the real person. When the same number of testers chose the AI as the real person, then the AI passes the test. Sence when did the Turing name apply to every AI competion? Am I wrong? This isn't a flame just a question about definitions...
Business News and Resources: www.usasource.net
isn't a game a simple form of communitation.
thank God the internet isn't a human right.
In the ICPC, the set of available languages is restricted because the participants are using secured computers (to keep them from getting on the Net and causing trouble on the LAN). You'd think it wouldn't be a problem if participants could submit a binary compiled at home. However the whole point of this contest is for Caltech to see what ideas everyone else has come up with -- so they want to make sure they can read your source code.
you must have missed this node and possibly this one as well
I mean, maybe if you had tens of millions of dollars to analyze how humans do specific tasks compared to computers you might be able to code something up, but just guessing? I don't believe anyone can come up with solutions to this without doing huge amounts of research.
It's one thing to try to write a program that does things as well as humans, it's a whole other things like humans. And it's impossible without an operative definition of 'humanness'
Especially given that there is such a huge range of how 'well' people do things. I mean, some people are idiots, and some are geniuses. How can anyone write a program that can tell the difference between Deep Fritz and a grandmaster, and a program making lots of intentional errors and someone who doesn't know chess? And every class of player in between?
autopr0n is like, down and stuff.
After reading the specs really quick, here comes a simplified summary of how the games work. Don't sue me if this is wrong, I did not look too carefully and it has been more than a year since I left Caltech.
Basically, the two players get to choose numbers with different pay-offs for themselves and the other player.
If you are short-sighted, it is easy to write an 'optimal' program, you always pick the max payoff for yourself.
However, you can also cooperate so that you maximize the average payoff for both players. Hopefully your opponent will realize this and also start picking numbers in this way, if not it is probably wise to go back and be short-sighted and selfish.
Now, the real object of this is to make (or detect) human-like opponents. I would guess that real human players are very irrational, for example they can get pissed if the other player is too selfish and then demand revenge, even though it does not maximize their own payoff.
Tor
Maybe it's just me, but "Before you get too excited" and "gory details" don't really match up with the heading "Turing Test Competition At CalTech".
The following is an example game file your program is supposed to output a dataset for that is "most human". I give you:
NFG 1 R "game1" { "1" "2" } { 2 2 }
21 3 3 5 3 5 5 3
What is the most human response? Anyone? Anyone?
sad /sad/ adj. 1. Bereft of happiness. 2. Human beings (programmers, operators, administrators) so attached to a computer system that their hardware will never encounter any wetware.
I don't think Caltech is the best place to determine what is "human" behavior.
Life at Caltech.
Also, notice that this contest is being held by the "Division of Humanities and Social Sciences".
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
This 'contest' reminds of the scene in Ghostbusters where Bill Murray manipulates pre-cog cards to get women. Really, if a ESP buff were to present something like this to the Amazing Randi he would not accept it on the grounds that it was too easy to manipulate.
First and foremost, there is a large sum of money being bandied around. The participants are incented to win by monetary payouts, and two payouts of $10,000 dollars are at stake as well. When games with this high of stakes are being played, great caution is generally used by the house.
But look at the rules of the game...there basically are none. Participants are identified by e-mail address; no rule is specified about the number of entries per person. Also, no rule is specified about collusion between entrants (detectors and emulators). It doesn't take an einstein to figure out how to bias the results of the experient by making enough colluding entries. It is funny this, given that the games themselves are *about* collusion. Its a joke.
Next, notice that there really isnt any way for anybody to tell if the results of the experiement are meaningful, or if they have been manipulated. Its not based on a falsafible proposition and is not scientific. Its no different from any method used by psychic hoaxers of the past. The creators of the 'contest' can manipulate the data, and direct the winnings to their confederates.
Finally, the bizarre nature of the contest should raise some flags. They are giving out $10,000 for a program that would have been hacked together in a couple of months at most. What kind of value could you expect from a contest like this?
There is a good reason for this. If the game (or a small finite set of games) were pre-defined, it would be easy to have a bunch of human subjects play it and then have the emulator regurgitate such a "book". Most entries, I suspect, will be from people or teams who are familiar with studies of how real people do play such games.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
As a recent alum (2002), I can shed some light on some things.
:-) )
Why is this in the Deparment of Humanities and Social Sciences?
From the URL, this Tournament is being run by the Social Sciences Experimental Laboratory (SSEL), not the CS department. The SSEL has been one of the leaders in experimental economics research (read: actually testing all those crazy theories you hear in economics classes).
Why is there money involved?
All experiments by the SSEL involve money. As an undergraduate, I participated in many experiments, mostly involving trading "commodities" in simple (and sometimes not-so-simple) markets. We were paid based on our performance. If I had an off night, I got paid $5 for 2 hours of the experiment. If I had a good night, I could make upwards of $80. Yes folks, this is real money we're talking about here. Since the point is to test people's economic thinking, you must make your decisions based on a real outcome, otherwise the data gathered is invalid.
Why then are they doing this test?
I don't work for the SSEL (and never have), but here's why I think they're doing this: Since they're interested in not only individual human behavior, but also how individuals interact and make choices based on the actions of other individuals, it would be useful to design a computer program that mimics other human's behavior. If other humans think this program acts like a human, then you can do two things: you can take the specifications of the computer program and figure out what qualities of the program humans have. As well, you can then replace humans with the computer program in real experiments (this not only allows you to test the limits of the program, but also to save money
And as an alum (who was not too fond of his time there, but still feels compelled to defend Caltech), it's *Caltech*, not *Cal Tech* or *Cal-Tech* (but if you're feeling lazy, *caltech* is all right too).
nak
(define humanator2 (lambda sex pr0n)
(if(and sex (not (isKiddy pr0n))
(masterbate sex pr0n)
0)))
autopr0n is like, down and stuff.
The only game I want to play is Global Thermonuclear War.
go away or i'll replace you with a very simple shell script.
Obviously I didn't read closely enough. Thanks to everyone who covered my err.
Er, I'm assuming you're a college student. Or high school student. $10,000 isn't that much money. Even, as you say, for a program "hacked together in a couple of months." Let's say it's just two months. That's $5,000/month for the winning entry. That's below what a good programmer makes.
/. banner ads do you think $10K would buy, with what return? What is the value of getting hundreds or thousands of /. types to think about this, and write comments, and generally stir up interest in the contest and company?
And what are they going to do with the program once they get it? Nothing. This isn't about buying programming time. Like most contests, they're betting that the contest itself will buy them more than $10K worth of promotion. How many
It's not much money, but they have already demonstrated a pretty savvy approach to spending it.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
It is not merely the possibility of scamming that is the problem.
There is something very deeply wrong with the theory and the experiment. You say yourself you would have to look at a lot of sample human data to judge this game; think about that for a moment. Wouldn't collecting a large amount of human data and fitting it with a neural network or whatnot be a more straightforward approach that leads to a scientific result?
There is feedback in this contest method that would lead to an unbounded number of refinements to emulators and detectors(and ever more research grants no doubt!) Its a tail chaser.
I am a retired software engineer.
If you believe this contest is really about marketing CalTech rather than a scientific approach to sociology or intelligence, I won't argue.
Note that my post just one way to approach the flaws riddling this contest...
That would be trivial for a MMORPG. Just write a program to make the character stand around at popular spots, like vendors or quest dispensers, casting doubt on the manhood of passerby, and it will be indistinguishable from your typical 13 year old gamer. It could even be done with an existing program like Eliza, if its conversation database were tweaked to generate juvenile insults.
Caltech is a private university and not part of the UC system. To refer to it as "Cal Tech" or "CalTech" is not only wrong and misleading, but does a disservice to the people associated with that fine institution.
That said, I think the stuff happening there is very cool.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.