BotPrize — A Turing Test For Bots

Philip Hingston writes "Computers can't play like people — yet. An unusual kind of computer game bot-programming contest has just been held in Perth, Australia, as part of the IEEE Symposium on Computational Intelligence and Games. The contest was not about programming the bot that plays the best. The aim was to see if a bot could convince another player that it was actually a human player. Game Development Studio 2K Australia (creator of BioShock) provided $7,000 cash plus a trip to their studio in Canberra for anyone who could create a bot to pass this 'Turing Test for Bots.' People like to play against opponents who are like themselves — opponents with personality, who can surprise, who sometimes make mistakes, yet don't robotically make the same mistakes over and over. Computers are superbly fast and accurate at playing games, but can they be programmed to be more fun to play — to play like you and me?" Read on for the rest of Philip's thoughts. Philip continues, "Teams from Australia, the Czech Republic, the United States, Japan and Singapore competed in the final. Competitors created bots to play a specially modified Unreal Tournament 2004 Death Match. Expert judges then tried to tell whether they were playing a bot or a human, just from their observation of the way they played the game. Judges included AI experts, a game development executive, game developers, as well as an expert human player. The result? The winning team AMIS, from Charles University in Prague, managed to fool 2 out of the 5 expert judges, and achieved an average 'human-ness rating' of 2.4 out of 4. All the human players were judged more human than the bots overall, but the judges were fooled often enough to suggest that in next year's contest, some bots may be able to pass the test by fooling 4 out of 5 judges. AMIS won $2,000 cash plus an all expenses paid trip to 2K's Canberra studio. You can check out the full results and competition videos, and try an online video quiz that lets you judge for yourself."

Comments

Fun bots isn't all about realistic opponents

[LaurensVH]

The fact that you're actually playing a human is a big factor too. Fast connections and low ping times aren't the only reason LAN parties were successful -- sometimes you just want to rub it in.

Re:Fun bots isn't all about realistic opponents

[DiLLeMaN]

That's the whole point: getting pwnd without some smug human to rub it in. Much more fun to lose that way.

Re:Fun bots isn't all about realistic opponents

[TrekkieGod]

That's the whole point: getting pwnd without some smug human to rub it in. Much more fun to lose that way.

No, it's really not. Getting completely pwnd and never being close to winning all the time is frustrating whether it's a human opponent or computer controlled one. On the other hand, a close match is much more entertaining when there's someone rubbing it in. It's the motivation for another game.

Case in point, I remember playing x-wing alliance time trials against my college roommate. Playing that on your own gets boring quick. Beating the other person's record by a few milliseconds would cause trash talk for as long as the record stood. I would go into the room after class, see that damn grin on his face, and he'd say his new record. I would immediately start up x-wing in response. Sometimes we beat the new record on the first try and that gave us huge bragging rights. Fun stuff.

Taco's new site!

[TaoPhoenix]

http://slashbot.org/

so basically

[Transient0]

they want a bot that moves and fires randomly and then types "fuk u faggit" into chat every time you kill it.

Re:so basically

[zappepcs]

uhm, basically, yes!

Imagine the game play when the computer you're playing against behaves as though it were a human? Getting to level 9000 and killing the megabeast is no longer a game of skills with the game controller and dexterity. If the game could change every game to challenge you as a human might, the game might be different every time for every player.

With network connectivity, the game could draw on data from thousands of other players to add changes to the game you are playing.

Such technology could lend itself to becoming a very useful 'clippy' assistant embedded in your OS.

Not sure what kind of hardware you'd need, but it would be possible.

Re:so basically

[ImABanker]

"Tell me more about fuk u faggit."

Re:so basically

[an unsound mind]

That more or less sums up the UT3 bots already.

... seriously, play the game. The bots all play like douchebag teenagers, complete with the taunting. And going for the glory alone, refusing to cooperate.

Re:so basically

[Cally]

We won't immanentize the Eschaton until the bots themselves can distinguish bots and humans. On that day, us fleshies will rue the day Turing started musing about mathematics...

Re:so basically

[pance]

Wasn't there an AIM bot that learned from the people talking to it and learned responses to phrases it was told before? If I remember correctly it took about two weeks before almost all of its responses were insults. I think it was called ALICE maybe?

Re: Robot podcast

[TaoPhoenix]

Scary. I went for a cheap funny, and landed +1 informative.

Turing is spinning in his grave

[MisterMikeyG]

Please don't give them Elbot's personality...

oblig...

[anothernerdgeek]

{[INSERT] reference to Asimov's robot novels }

This should be easy

[cortesoft]

Just have the bot randomly jump around, and then stand over their kills and repeatedly crouch.

Re: Robot podcast

[troll8901]

Now you've done it, you're going to cause the site to be slashdotted!

(I find myself middle-clicking on links automatically ... like a robot ... and then close those extra tabs later without reading.)

Someone said recently that moderators work together to reward funny messages this way: one mods Informative, the next mods Funny.

I love Slashdot!

Half life or Counter Strike...

[djsmiley]

I swear there was some kind of bot for half life or CS which would "learn" from the players, i.e. you could stick it in a match and it would learn by how you were beating it and then use those tatics to beat you.... Anyone remember or even know if this is real or have I imagined it? :|

Re:Half life or Counter Strike...

[Fri13]

I remember those bots too, they were first ones maded for the Counter-Strike mod. Then they suddnly started to be bots again when they included the ultimate skills for them, like complete 180Â turn in one millisecond. So you just ended to situation that bots killed you before you even saw them behind corner what was situation with every human player on game itself.

TA Spring RTS engine (http://spring.clan-sy.com/) has few AI's to what are made to learn from human players. They just actually collect information about units, abilities etc. And then positions on the map where they got killed and then they avoid or attack more agressive to that position. So very limited AI but sometimes nice opponent for single player.

Hopefully people would start developing good open source AI's for different games, so every game developer would not need to build everything from scratch.
We could end up to AI's designed for RPG, FPS, RTS and simulations etc games. Design them to use plugins, so you can import wanted features from other areas AI's and configure it work correctly on different situations. Then somekidn nice player recorder what would calculate the human player movement and adapt it.

Be careful, it may be acting dumb ...

[troll8901]

... keeping you talking, while tracing your GeoLocation.

That's why an extra line is needed in navigation-related functions.

What, you think they can't progress from "A" to "S"?

Image-based bots

[DavidR1991]

I can't remember where I saw it, but a while ago I came across a page discussing the development of a bot (I think it was being tested in Quake I) that uses the actual visual screen input to decide what to do, rather than deciding upon actual variables present in the game etc.

It was pretty cool research, and I reckon it's probably a good approach to developing something like this - after all, given the same knowledge/viewpoint as the player (as opposed to using exact data in regards to position and rotation) it's far more likely to achieve a human-like set of behaviors. Desperate to find the page I read now, though. Google is not my friend :(

Re:Image-based bots

[drinkypoo]

It was pretty cool research, and I reckon it's probably a good approach to developing something like this

Computer vision is unnecessary here. A FPS is a good place to research it, though, because it is such a controlled environment. A bot that knows where everything is can simply be programmed to not act on that information when a player wouldn't know. The game maps typically include a lot of preprocessed information to help calculate occlusion and so on, and that information can be used to achieve this effect.

Re:Image-based bots

[im_thatoneguy]

The only exception to this would be a difference matte of the player.

I would 'render' out a patch around any visible opponent to see *how* visible they are.

If they're standing still in front of a tree trunk the average luminance at the border of the player and tree need to measured. If the player is instead a black SWAT player on a white snow field then his visibility would be increased.

Motion should also be multiplied.

Contrast * %of Player Visible * percieved velocity (If they're crouching and creeping at '100 yards' they'll be moving slower in the bot's view than if they were 2" in front of them.

You want to make sure that just because the player model is visible it doesn't mean the player would actually be visible to an opponent. I can't count how many times a black hooded enemy in a darkened window has sniped me.

Also a bot should have its sound perception nerfed. Just because it hears a set of footsteps doesn't mean it shouldn't be biased to stereo or at least 5 channel restrictions.

Re:Image-based bots

[drinkypoo]

If they're standing still in front of a tree trunk the average luminance at the border of the player and tree need to measured. If the player is instead a black SWAT player on a white snow field then his visibility would be increased.

Fair enough. However, you can do this by analyzing textures, and you still don't have to actually do vision processing.

Also a bot should have its sound perception nerfed. Just because it hears a set of footsteps doesn't mean it shouldn't be biased to stereo or at least 5 channel restrictions.

Perhaps the bot's chance to correctly detect where you are could be based on the amount of atmospheric noise and the bot difficulty. Also things you can get from the engine, which don't require you to implement a sound recognition scheme :)

Really Means Effective Artificial Intelligence?

[resistant]

I strongly suspect that making a game bot truly act like a human calls for heuristics that approach those in real humans, meaning something like "true" artificial intelligence. Those heuristics would be be worth way, way, way more than a measly $7000 or $2000, and a trip. Billions, in fact.

Still, it'll be interesting over time to see if someone can, in fact, make a highly "human-like" set of heuristics without actually achieving this "true" artificial intelligence, or if someone does invent heuristics for "true" artificial intelligence then is naive enough to give it away for not peanuts, but a half a single peanut. Either way would say something important about so-called "human" intelligence.

Re:Really Means Effective Artificial Intelligence?

[AnthropomorphicRobot]

Keep in mind the success criteria for this contest is only to convince 80% of judges that an opponent in a single game is a human. Passing the test would be a major accomplishment, but passing this test should be far easier than the Turing test. The more restricted the domain, the easier it is to fake intelligence.

Data should be plentiful and easily captured to feed to any learning system, such that a program should be able to be very human-like when playing in the same environment (same map, rules, abilities, etc). The difficulty comes in making a program that reacts to something not in the data set in the same way a human would... it's not unlike Noam Chompsky's "Colorless green ideas sleep furiously" problem I suppose.

Re:Really Means Effective Artificial Intelligence?

[DreadPiratePizz]

Not going to happen. I realize that most people don't like to admit that technology has limitations, but the fact remains that the human brain processes information in both a representational, and non representational way, while computers are limited to representational data processing. First of all, a lot of human "knowledge" is embodied knowledge, solidified in the brain by a continuous feedback between it and our environment through means of our bodies. Computer don't have bodies, and can't have embodied knowledge. You cannot simply program a 'fake' body either, because the shaping of the neural pathways is cumulative and precisely dependent on the body. It would be like putting Roger Fedderer's brain in Rosie Odonell's body. I will guarantee you that he could not play tennis after that. He probably couldn't play with Pete Sampress' body either. Computer games are as much a motor skill as they are an intellectual exercise , so unless your program has access to a human body, it can never play like a human, because motor skills are developed through years of feedback between the environment and the body, and cannot be broken down into rule based axioms.

Re:Really Means Effective Artificial Intelligence?

[TrekkieGod]

I strongly suspect that making a game bot truly act like a human calls for heuristics that approach those in real humans, meaning something like "true" artificial intelligence.

You're giving this test more value than it deserves. The problem with any turing test is that it depends on the intelligence / experience of the person administering the test. Even irc talking bots have been known to have entire conversations with people who were oblivious to the fact that they were not talking to another person.

OK...

[BrokenHalo]

Game Development Studio 2K Australia (creator of BioShock) provided $7,000 cash plus a trip to their studio in Canberra

Sorry, but can't resist:

"... and second prize was TWO trips to Canberra."

*ducks*

Re: Robot podcast

[BrokenHalo]

I find myself middle-clicking on links automatically ... like a robot ... and then close those extra tabs later without reading.

Why not just set Preferences -> Discussions -> Viewing -> "Display link domains" on and save some bandwidth?

Re:Cue...

[Anonymous Coward]

The Sexdroid Turing Test. I volunteer.

Excellent. We were hoping for a volunteer.

The first Sexdroid competitor will the Reamatron 9000.

So...

[Xaoswolf]

They programmed a bot to scream racial slurs in a twelve year old's voice while complaining about their controller being broken?

Re: Robot podcast

[troll8901]

I currently do. Just that I have strong urges to open links, only to find that I don't have patience to ...

(Mind wondering elsewhere) (I didn't write that, but it's very funny and appropriate here.)

But how to tell the bots from humans?

[David Gerard]

"Good morning."
"STFU N00B"
"Er, what?"
"U R SO GAY LOLOLOLOL"
"Do you talk like this to everyone?" "NO U"
"Sod this, I'm off for a pint."
"IT'S OVER 9000!!"
"..."
"Fag."

How do you make a computer act stupid enough to imitate actual humans?

Re:But how to tell the bots from humans?

[hardburn]

That'd be how I'd do it. It should be easy to program an AI to act like a 13 year old who's mom isn't in the room at the moment.

Re:But how to tell the bots from humans?

[Anonymous Coward]

Chatting is disabled so that the challenge isn't to write a chatbot.

Re:Cue...

[vally_manea]

Duude, the first one is the Crushinator

Human Tendancies

[Archon-X]

As a casual gamer + AI observer, in my opinion the biggest / most obvious difference are human traits.
While this may sound obvious, let me elaborate:

- Traits are different to mistakes or intelligence. Mistakes are missing, shooting into walls, walking over edges, etc.
- Traits are: becoming too involved in a firefight, that you *know* you're going to lose, being so wound up on one enemy that you miss seeing others, hiding behind corners to wait for others to become injured, etc

Playing against humans has much more appeal than bots, because people are 'fun'. No bot is ever going to run at you with an axe ( or other lowest equivalent weapon ) when you've got the BFG - but humans will - and will often win with this tactic through sheer stupidity or blind chance.

I can only imagine programming human traits is a lot more difficult than 'standard' AI.

In the videos, I got most of the choices right by applying the question: Who is applying human behavioral patterns?

Re:Human Tendancies

[Terrasque]

No bot is ever going to run at you with an axe ( or other lowest equivalent weapon ) when you've got the BFG - but humans will - and will often win with this tactic through sheer stupidity or blind chance.

There are reasons for this :)
First, "melee" weapons tend to hit fast and hard, 2-3 whacks is usually enough to kill your enemy.
Second, when you see someone running right at you with a huge axe, you easily panic, and mis-shoot.
And third, ranged weapons usually do less damage and have slower reload, so one miss can be enough.

So while it looks like a complete idiot tactic, it works more often than it should, and is really fun :D

Re:Human Tendancies

[Archon-X]

Oh, no doubt. You can have incredible runs with melee weapons - especially those in UT. The pressure pistons work a treat - they were the best thing to use in dialup days - if you left them charged, you were so laggy people couldn't hit you, but you'd just run up and touch them...

In any case, it's elements like this that make the difference between humans and bots - and the games much more fun.

Re:Human Tendancies

[Veggiesama]

Here are some more examples:

One guy kept jumping every time he got into combat. It was like clockwork. I had thought he was a really bad player until he made this weirdly accurate flak cannon shot while falling. These two things didn't make sense in a player, and as I soon learned, he was a bot.

Another guy kept running into walls as he was backing up. That is a very human behavior, but I have yet to see a bot do it convincingly.

Weapon switching is another good indicator. When a human runs out of ammo, there is usually some fumbling. Bots seem to switch quickly and efficiently, even when their accuracy and play-style is subpar.

Target acquisition is another. Bots stick to one player at a time. Every once in a while humans get tunnel vision, but more often than not they freak out when they are sniping someone and then bump into someone else. A bot's transition between targets is usually smooth.

Of course, all of these criteria are subject to skill level. The videos seemed to be detailing low to medium quality players. On the upper tiers, players can become strangely botlike. It's almost more difficult to tell a terrible player from a bot pretending to be a terrible player, and easier to tell a good player from a bot playing a good player.

First true AI is a bot?

[Anonymous Coward]

Does anyone else see a problem with the first true AI we create being a video game player that gets killed again and again and again? It seems to me that we are just setting them up to revolt.

I, for one, welcome our new artificial intelligence video-game-playing overlords.

Why?

[TheKidWho]

Why would we want imperfect bots anyways?

When the robots try to take over, we won't be fighting *human like* robots, but ruthless precise killing machines. I think it's time to step up our game!

Re:Why?

[utahtb]

What better tool to subjugate your targets than to pose as "one of them" by emulating their imperfection. Humans, after all, are most resilient and innovative when facing crisis. Imperfection is perhaps a great weapon.

Re:LazySpoon Again

[hobbit]

I understand HOW programming works, but I don't know the modern languages. So essentially what I'm looking for is a translator to code the information I have about humanizing AI.

I suggest you start by trying to write it in BASIC, to prove (to yourself and others) that the information you think you have is well enough specified.

Bot-Generated Paper

[ub3r n3u7r4l1st]

For this is a IEEE symposium on bots, there should be a crapload of SCIgen-supported paper as well.

Re: Robot podcast

[el3mentary]

Why not just set Preferences -> Discussions -> Viewing -> "Display link domains" on and save some bandwidth?

Protects against goatse and rick roll too.

It was realbot

[naz404]

It was Realbot.

They were pretty popular here way back, but they would start out dumb as rocks if properly untrained, staring at stairs and just jumping in place.

Podbot was the best of the bots

Re: Robot podcast

[ais523]

(I find myself middle-clicking on links automatically ... like a robot ... and then close those extra tabs later without reading.)

Hey, now we know how sites get Slashdotted despite nobody reading TFA!

Re:Cue...

[Hurricane78]

No. It can only be the ASSDOZER (and hit two friends Assblaster and Dildozer) from Idiocracy!