Student Creates On-Line Poker Playing Program

Psyber Samurai writes "Online-Athens has this story about Paul Apostolik, a cognitive science and computer science student at the University of Georgia that has created a program that plays the risk-free online video poker sites. The program takes over the screen and actually reads the cards off the screen then determines which cards to hold and which cards to discard. I've run it, and it is extremely fun to watch it play. Yes, it does win over the long term as it collects many, many tokens."

Comments

ARGH! - this is not A.I.

[jamiemccarthy]

I'd like to blame it on Spielberg's latest, but the fact is, movie or no movie, reporters will take any damn program that someone writes and turn it into the latest example of "artificial intelligence." Yes, this is my pet peeve. Do reporters think nobody will be interested in reading their story unless it mentions computers and "intelligence" in the same sentence?

In this case, the program reads the screen and moves the mouse to click on the proper places. Pretty clever I suppose. But once you get past the fancy I/O code, it's just applying simple formulas to decide what cards to hold. There's nothing "intelligent" about calculating that holding the pair will give an expected return of $0.21, while going for the flush will give an expected return of $0.08.

The only thing about poker vaguely related to "intelligence" is its condensation of the social relationships of zero-sum games into the formal interface of betting. Placing bets is much more amenable to mathematical analysis than the fuzzier stuff we do when we're playing similar zero-sum games, say, negotiating the price for a used car: furrowing our brows, clearing our throats, poking fingers at each other's chests, yelling, pretending to think over an offer, saying "my manager will have to approve this," and so on.

When you simplify the interface between entities to just a few operations (raise, call, fold), suddenly mathematics can play a role.

John von Neumann was one of the first to recognize this, and it was the game of poker that he focused on, in his study of game theory:

The nominal inspiration for game theory was poker, a game von Neumann played occasionally and not especially well. (A 1955 Newsweek article appriased him as "only a fair-to-middling winner" at the game.) In poker, you have to consider what the other players are thinking. This distinguishes game theory from the theory of probability, which also applies to many games. Consider a poker player who naively tries to use probability theory alone to guide his play. The player computes the probability that his hand is better than the other players' hands, and wagers in direct proportion to the strength of the hand. After many hands, the other players will realize that (say) his willingness to sick twelve dollars in the pot means he has at least three of a kind. As poker players know, that kind of predictability is bad (a "poker face" betrays nothing).

Good poker players do not simply play the odds. They take into account the conclusions other players will draw from their actions, and sometimes try to deceive the other players. It was von Neumann's genius to see that this devious way of playing was both rational and amenable to rigorous analysis.

Though much more interesting to analyze than, say, blackjack, playing a good game of poker still does not qualify as "artificial intelligence." It's been pretty well analyzed since, if I'm not mistaken, the 1920s:

[A particular type of deception] resembles bluffing in poker. Poker can be quite complex, in part because it usually has more than two players. Von Neumann analyzed a simplified form of poker. In outline, his conclusions apply to the real game. He showed that you should always bid aggressively when you have a strong hand. With a weak hand, you should sometimes bluff (bid aggressively anyway).

Von Neumann distinguished two reasons for bluffing. A player who never bluffs misses many chances to call the other player's bluffs. Suppose that both you and your opponent have bad hands. You don't bluff; your opponent does. That means you fold and your opponent wins without a showdown. Had you also bluffed, your lousy hand would have been compared with his lousy hand, and you might have won. The bluffer can exploit the nonbluffer; ergo, von Neumann's rational player must bluff. [...]

At the most abstract level, game theory is about tables with numbers in them -- numbers that entities are efficiently acting to maximize or minimize. It makes no difference whether you picture the entities as poker players who want to win as much money as possible...

(Quotes from Prisoner's Dilemma, William Poundstone, 1992, pp. 40, 60-61.)

A program that applies von Neumann's minimax theorem to place poker bets is still just doing simple math. But Apostolik's program -- cool though it is -- doesn't even do that. He hasn't incorporated betting yet. "That would be an interesting thing to play around with," he says. True! But even that still wouldn't be A.I.

Jamie McCarthy

Re:Debunking online casinos.

[Anonymous Coward]

But this article is talking about *video poker*, which if you haven't been to Vegas, is played using electronic machines even in the casinos.

This is a common misconception. The video poker machines are actually miniature kiosks with midgets inside who shuffle and deal the cards. Surveilance cameras are set up inside so that casino employees can look in on and make sure the midgets aren't cheating, but if you're lucky you can find a machine which if you whisper "Hey midget, i'll split my winnings with you" into you will find yourself winning a lot more than probability will allow for. Of course you don't actually have to split your winnings with the midget, he's a midget what's he gonna do? beat you up? Ha!

I did this with real money, and lost.

[GlenRaphael]

I wrote a program that played online video poker several years ago when online casinos were just starting to become popular. I had been making money at blackjack and this seemed like a potentially worthwhile time and money investment. It was very easy to write the program. I tailored my program to a site called InterCasino because at the time they were offering a video poker game that was positive expectation for players with perfect basic strategy.

I only had a Mac handy but the casino client software was windows-only, so I ran the client under SoftWindows emulation figuring this would make it harder for the client to determine something untoward is going on.

Recognizing the cards involved nothing more complicated than hit-testing for colored points along lines with certain offsets. The card pictures never changed, so it was just a matter of partitioning the set appropriately. For instance, a "9" of anything has a pip in the top-left corner; an ace does not. The only thing that was at all tricky there was differentiating a couple of the face cards. For instance, I seem to recall that their Jack of Spades and Jack of Clubs looked very similar to my algorithm. Differentiating spades and clubs generally was hard but it was especially hard with the face cards.

As for the play, that also required no original work. There are people who play video poker professionally; you can order basic strategy charts for all the major games that detail what payout schedules have what expected value and tell you how to play the hands. Some common casino machines are positive expectation and the casinos don't care because the average player makes enough strategy errors to be a net loser. My program had to recognize the card values, categorize the hand, then run down a chart (either a table lookup or a series of if-thens) to determine what strategy to employ. Then it had to simulate a mouse click on the appropriate "hold" buttons, then on the "draw" button.

To tune and debug the program, I had it play the hands but NOT hit draw until I verified the right cards were selected. I played through many hundreds of hands, discovering and fixing several bugs along the way. There were a couple of card-recognition bugs and a couple of play-related bugs, but eventually I got to the point where every time I questioned a play and looked it up in the rules, the computer was right. My program could play faster and more accurate video poker than I could.

InterCasino had two modes. You could play just for fun with fake money, or you could play for real money. To shake out the bugs, I played in the "just for fun mode", letting it run through the night while I slept. I had it preprogrammed to pause at random intervals for random amounts of time to simulate bathroom breaks or whatever, but mostly it just played.

In the morning, I'd won about a thousand dollars of fake money. As long as I played in "free" mode, I seemed to be earning about what was expected, occasionally hitting the big payoffs that make up for the long term negative grind.

So I switched to the real-money mode. I figured it was easily worth investing $500 or so of my blackjack earnings to see if I could institute a real money machine. The dream was to have a bank of PCs in my closet raking in the cybercash while I sleep, eat, read, or go to work... So I deposited the money and set the thing to work. Again I ran it in semi-interactive mode for a while before putting it into automatic.

And it lost. So long as I played with real money, it just wasn't hitting the big payoffs as often as I think it should have. I switched back to free mode and won again, switched back to paid mode and lost again. After losing several hundred dollars I decided to pull the plug.

I did not play enough hands to be sure I didn't have a run of bad luck. Video poker is a game with a very small edge and almost all of the edge you have comes from hitting that once-in-a-blue-moon royal flush or equivalent. Maybe if I'd been willing to lose $10k I'd have eventually gone positive. But there's no way to prove that the game is honest, so I couldn't count on the laws of probability being in my favor. My only protection was to have a reasonable stop-loss figure, and I hit it.

So I won't be retiring to the carribbean just yet. :-)

(In fact, I'm currently unemployed due to a recent dot.bomb event. Need a lead QA engineer, software engineer or otherwise just generally smart guy? My resume is here; check it out. Especially if you're doing something with handhelds.)

One exception - Go

[invenustus]

Computers playing games are always much better than humans if their AI is done correctly.

Actually, computers have, as of now, never been able to master the game of Go. For an intro to the game from the angle of Game Theory, check out http://eksl.cs.umass.edu/~heeringa/home/doc/go_pap er/. A human being with a mastery of the basic strategy can beat most of the best artificial intelligence simulations. Fascinating stuff....
----
"Here to discuss how the AOL merger will affect consumers is the CEO of AOL."

yawn

[QuantumG]

I know at least 5 people who have shown me code which does this. It's no biggy. What is interesting is that looking at their statistics it is clear that online casinos cheat when they detect perfect play, but it is very very hard to prove.

Not bad...here's some better stuff.

[ruebarb]

The only thing new about this is it's interacting with online gambling. Interesting if it could be used in a monetary sense - like Texas Hold Em or 7 Card Stud to actually win money. (if you're playing video draw poker, you're playing to lose. The house has the best odds) - In fact, I half wonder if it's being done right now...people playing online and feeding odds and stats into a computer to see if they've got the odds to win.

FYI - the most commonly recognized poker playing software is Wilson Software - www.wilsonsw.com - they design poker programs with very detailed programmable profiles...That's how I improved my game...a thousand hands a day no problemo. (it's also good for simulation - wondering if you'd save or lose money raising AK preflop? Run 100000 hands and find out!)

The most interesting thing is this...Poker is actually a lot like a casino. You think you're playing each other, but if you get the ten best players together, then all you have is 10 steady declines of bankrolls. The house rake sucks it all away at 10% a hand. The best way to make money at poker is to find a fish - which I'm thinking he could program this thing to beat no problemo

But ...

[psergiu]

> and actually reads the cards off the screen
...
> Yes, it does win over the long term as it collects many, many tokens.

The question is: can this program Punch the monkey to win ?
--

Very cool book about using computers for roulette

[Anonymous Coward]

The Eudaemonic Pie, by Thomas Bass, about these chaos theorists in the 80's who built computers in their shoes at a time when that was technically a difficult feat (no pun intended). They used a 6502 microprocessor potted in a blob of epoxy along with toe-operated switches and vibrating pins that buzzed different parts of the wearer's foot to tell him/her what numbers to play, and the circuit was stuck into the hollow heels of custom made shoes originally designed for dope smugglers.

The way it worked was casinos used to let you place bets when the roulette ball was already rolling. By clicking the toe switches the first few times the ball went around, the shoe computer could calculate the deceleration rate of the ball and predict where it was going to end up. It couldn't predict perfectly of course, but it was enough to give a substantial advantage over the house.

The circuitry was flaky but worked well enough to prove the concept was sound. The builders claimed they didn't actually win much money with it, but they sound a little bit coy. More interesting is the book's description of the looney, communal environment that the hackers lived in when they built the thing.

Bass's later book "The Predictors" is about the same guys using similar techniques to program computers to automatically pick stock market trades (www.predict.com). They were the first to do that and were apparently pretty successful, but now I think all the brokerages are doing it and there's not much advantage to be had any more from those methods.

Re:Real Poker

[s390]

"So is this kind of like Big Blue, I mean if I were to play some of the best poker players with this little programs assistance would I win a majority of matches?"

In college we played hi-low poker for table stakes and it was... interesting. $100 might change hands in a single "tap" - a lot of money, then. Later I played 5-stud in $2 limit games (that's a loser to the house rake). Win some, lose some, but I do like poker. Haven't played much for years, though. If it's recreation, then don't count the chips. If it's for real money, it takes as much focused concentration as any profession. And as professions go, it's a fairly stressful one. Hacking MVS, networks, Linux is lots easier.

It's also an unfortunate fact that people do cheat at poker. I sat in a low-ball game once where the guy to my left said "watch this" before dealing me a low wheel (A2345, perfect hand). I raised the open and everyone folded; I took the pot and left. When there's real money on the table, the odds of people cheating in various ways go way up. Be aware. There's even more danger of cheating online in virtual casinos, where you don't know which "players" are shills and which aren't; the server (house) knows your cards.

Poker isn't like chess, in that it's not just a deterministic game. A simple poker 'bot will be too predictable (below 50% fold, above 75% raise, etc.). The reason it's an interesting game is that human strategies, emotions, and intuition all play at some times in any session. Poker, like playing the stock markets, is 50% money management and 50% picking plays. The similarities are very striking.

Poker 'bots have been written and played against each other by academics for decades. Best results are obtained by heuristic routines that analyze the moves of all other players and learn from the progression of play, building a 'book' on each from which to calculate odds about bluff or hold. There are many books out about poker (most published to scam wannabe winners out of a few dollars), and not a few academic papers - use Google to find them. Probabilities are easy, strategies are hard, for many reasons.

But poker is a complex game, especially with human factors added. The complexities are infinite and arguably incalculable. There is no perfect way to play. One of the best session strategies is to bluff on small pots early, have 'em on big ones, then later "on towards morning" reverse the strategy - but like all poker, it's risky and won't work repeatedly with the same group of players, like most poker session strategies. Or, one could reverse the above strategy. These are the two basic possibilities for any one game. If one plays consistently though, others will see it and learn when to duck or push. There are levels within levels, in each session and venue.

I've been killed at poker, and I've also shut down a small poker parlor by taking every player until no one would sit down with me (400% profit). One good piece of advice I'll offer is... don't play for blood with friends, or especially, coworkers. Those relationships are more valuable than money, and some people take lasting offense at a sandbag. If they give you hands, OK - but don't extract it.

Re:Exists...

[nachoworld]

Your basic strategy chart based on number of decks, DAS, DA2, etc.
The best blackjack site for info.
The most informative site on counting cards for the beginner

Just a few points to help you on the field. Insurance is considered a good bet when the true count is 3 or above. This is regardless of your hand, of course, because whether the dealer has a face in the hole has nothing to do with your hand.

You're right about the odds just barely being in the house favor (at Foxwoods, CT, where I play, they're 0.33% on the 6-deck high limits and 0.36% on the 8-decks).

Which brings me to my last point. Fewer decks is considered better, but what matters most by far to a card counter is penetration, where the dealer places that colored card to signal shoe reshuffle. A penetration that's closer to the front of the shoe, the worse it is for the counter.

BTW, simple increment counting systems, while simple and difficult to get good at, are the systems that the vast majority of counters use.

---

programs that play well against humans

[Silmaril]

Darse Billings and others at the University of Alberta Computer Poker Research Group have done quite a bit of research on computer poker. Their program, Poki (source here) plays a reasonably good game of Texas Hold-Em. This is the primary game at the famous World Series of Poker.

java golf

[Perdo]

While not as great an accomplishment, I wrote a macro to play a java golf game online. Since there was some randomness to the game and my reflexes are not that good, I figured it would be a way to get my name in the top ten. after three days running the macro, I had all ten of the top ten spots. The game was pulled on the third day. It has never been back. Computers playing games are always much better than humans if their AI is done correctly. I realized that I had ruined the game for everyone else. Deep blue beat Kasparov, My AMD box beat java golf, Aimbots beat me at TFC, Paul Apostolik beat pogo poker, and the power producers are beating California (with a great weather model/consumpion prediction algorithm) on the energy trading floors.

Are we are all cheating? Cheating only applies to games and applying enourmous computer power to commodity markets is simply shrewd? Will we one day wake up with zero value in stocks because some bright day trader wrote a miraculous piece of code that makes a billion perfect trades in the course of a day? Cheating by attacking protien folding with petaflop power?

What are the moral causes for the application of analytical force? When is the right time to spray for moths? When is the right time to let a hummingbird fly over japan? (obscure reference, chaos theory: the ammout of air a hummingbird moves in flight over japan is all that is required to generate a hurricane a month later in florida)

With IBM predicting a teraflop available on your (or Saddams) desktop by 2004, these questions need to be addressed yesterday.

Possibly more common than you think

[Anonymous Coward]

I have a friend who has written several perl scripts to automatically play online games, the kind that reward you with pseudo-money. (Sometimes you can trade the pseudo-money for prizes, sometimes you can use it in auctions.)

Some of his programs are quite primitive; he randomly 'clicks' on links that could pay off, and a small percentage do. He wins simply through the massive number of clicks.

But he has at least one program that plays a trivia game. If the question is new, it guesses, but it "learns" when it has won the question. If the question ever comes up again, it now knows the answer. Pretty impressive for a few hundred lines of perl. (And he's not even that good of a perl programmer!)

He has been caught several times, and usually gets a 'cease-and-decist' letter (for violating the terms of service you have to sign to get an account), but has never gotten in any real trouble. (A key to not getting caught is to never do TOO good; never be the top winner of the day.)

Moral of the story: Programmatic attacks on games go on all the time, and my friend has dozens of free t-shirts to prove it.

Posting anonymously to protect the guilty.

Impressive

[sg3000]

> After losing several hundred dollars I decided to pull the
>plug.

>I did not play enough hands to be sure I didn't have a run
>of bad luck. Maybe if I'd been willing to lose $10k I'd
>have eventually gone positive.

Wow, it sounds like not only did you write a program to simulate video poker playing, but also to simulate a gambling addiction!

My Gambling Adventures

[Jesus IS the Devil]

Once a blue moon I had an idea. Here where I live casinos aren't allowed to play as the bank (win people's money), so they have to let the people be the bank, rotating every few turns. These casinos make $ by taking in a "collection fee" of $1 per hand.

Anyways, some of the games definitely gave the "bank" an advantage. Pan-9 because you could see if people are hitting or not. If they're not, and you have a low-scoring hand, you'll know to hit. I'm too lazy to get into the details, but trust me, there's an advantage for the bank.

Then there was Pai Gow. Here the bank has this "copy advantage". If a player has equal rank with the upper 2-card hand as the bank, the bank will win that set. Blah blah...

One more thing. You'd also have to factor in the collection fee into your chances. You'd need to put in enough money per hand to offset the disadvantage created by this fee or you'll lose. Also, I made a vow to NEVER tip those greedy-ass, good-for-nothing, I'll-sell-your-mom-just-to-make-a-dime dealers.

Anyways, I wrote this program to simulate playing the game from beginning to finish. I ran in on an Apple II GS. Yeah very primitive heh? That's all I had at the time. I did this just to prove that I was right that the bank did have an advantage.

Since that computer was slow and lame and didn't have a hard drive, I had to have everything printed out to me for record-keeping.

So day and night this Apple ran. A few days go by so I decided to analyze the results.

What do you know? There were distinct patterns repeating over and over again. Then it occurred to me, the damn computer's random number generator wasn't really random after all! I had wasted my time with that cheap computer.

So anyways, off I went to the casino to test my theory out. With $3000 in my pocket I started gambling away day after day... Up, down, up, down.

I was actually up $5000 for a while (and this was after I'd logged over 150 hours of gambling), but then I got greedy and started betting too big. This proved to be my fatal mistake. Wham in one night it was all gone. Just like that... After a month of hard work it all disappeared. >(

Anyways, I'd gotten sick and tired of the constant smoking and rude atmosphere. Besides, I wasn't learning anything and it was a HELL of a job (if I can call it that). So I quit cold turkey and went back to school.

To this day I'm sure there's an advantage there. Maybe one day when I'm really really bored or really really rich I'll try that again.

What a life.



---------
Did you just fart? Or do you always smell like that?

My cheating experience..

[Sarin]

I used to play a game called alien online where you raise stats on your alien by battling other peoples aliens etc. like a bbs game there were many sub-games where you could make money to buy better equipment. There was a nice hangman game, and after playing it a couple of times I realized I sucked at it. So I wrote a program (costed me a couple of days of freetime) in PHP using the excellent Snoopy class file (to navigate over the pages) that could beat any hangman game not by storing all answers but by generating an answer using a large dictionary and making intelligent guesses, my success rate was 85%, not bad.. After running the script for 2 hours I realized their dictionary was very limited and there were only 500 guesswords so I could only make something like 1500 gold.
I hadn't thought of that, I would had made far more when I just played regular. But my friends were impressed and we had a good laugh.

All this talk of gambling addictions...

[nick_davison]

All this talk of gambling addictions, I'm concerned I may have a slashdot addiction when the first thing that comes in to my head is... "I wonder if I could write a PHP script that analyses the topic of each new subject then generates a karmawhoring reply for me?"

Definitely time to go cold turkey. I'll just read one more story before I go.....