Student Creates On-Line Poker Playing Program

Psyber Samurai writes "Online-Athens has this story about Paul Apostolik, a cognitive science and computer science student at the University of Georgia that has created a program that plays the risk-free online video poker sites. The program takes over the screen and actually reads the cards off the screen then determines which cards to hold and which cards to discard. I've run it, and it is extremely fun to watch it play. Yes, it does win over the long term as it collects many, many tokens."

Comments

ARGH! - this is not A.I.

[jamiemccarthy]

I'd like to blame it on Spielberg's latest, but the fact is, movie or no movie, reporters will take any damn program that someone writes and turn it into the latest example of "artificial intelligence." Yes, this is my pet peeve. Do reporters think nobody will be interested in reading their story unless it mentions computers and "intelligence" in the same sentence?

In this case, the program reads the screen and moves the mouse to click on the proper places. Pretty clever I suppose. But once you get past the fancy I/O code, it's just applying simple formulas to decide what cards to hold. There's nothing "intelligent" about calculating that holding the pair will give an expected return of $0.21, while going for the flush will give an expected return of $0.08.

The only thing about poker vaguely related to "intelligence" is its condensation of the social relationships of zero-sum games into the formal interface of betting. Placing bets is much more amenable to mathematical analysis than the fuzzier stuff we do when we're playing similar zero-sum games, say, negotiating the price for a used car: furrowing our brows, clearing our throats, poking fingers at each other's chests, yelling, pretending to think over an offer, saying "my manager will have to approve this," and so on.

When you simplify the interface between entities to just a few operations (raise, call, fold), suddenly mathematics can play a role.

John von Neumann was one of the first to recognize this, and it was the game of poker that he focused on, in his study of game theory:

The nominal inspiration for game theory was poker, a game von Neumann played occasionally and not especially well. (A 1955 Newsweek article appriased him as "only a fair-to-middling winner" at the game.) In poker, you have to consider what the other players are thinking. This distinguishes game theory from the theory of probability, which also applies to many games. Consider a poker player who naively tries to use probability theory alone to guide his play. The player computes the probability that his hand is better than the other players' hands, and wagers in direct proportion to the strength of the hand. After many hands, the other players will realize that (say) his willingness to sick twelve dollars in the pot means he has at least three of a kind. As poker players know, that kind of predictability is bad (a "poker face" betrays nothing).

Good poker players do not simply play the odds. They take into account the conclusions other players will draw from their actions, and sometimes try to deceive the other players. It was von Neumann's genius to see that this devious way of playing was both rational and amenable to rigorous analysis.

Though much more interesting to analyze than, say, blackjack, playing a good game of poker still does not qualify as "artificial intelligence." It's been pretty well analyzed since, if I'm not mistaken, the 1920s:

[A particular type of deception] resembles bluffing in poker. Poker can be quite complex, in part because it usually has more than two players. Von Neumann analyzed a simplified form of poker. In outline, his conclusions apply to the real game. He showed that you should always bid aggressively when you have a strong hand. With a weak hand, you should sometimes bluff (bid aggressively anyway).

Von Neumann distinguished two reasons for bluffing. A player who never bluffs misses many chances to call the other player's bluffs. Suppose that both you and your opponent have bad hands. You don't bluff; your opponent does. That means you fold and your opponent wins without a showdown. Had you also bluffed, your lousy hand would have been compared with his lousy hand, and you might have won. The bluffer can exploit the nonbluffer; ergo, von Neumann's rational player must bluff. [...]

At the most abstract level, game theory is about tables with numbers in them -- numbers that entities are efficiently acting to maximize or minimize. It makes no difference whether you picture the entities as poker players who want to win as much money as possible...

(Quotes from Prisoner's Dilemma, William Poundstone, 1992, pp. 40, 60-61.)

A program that applies von Neumann's minimax theorem to place poker bets is still just doing simple math. But Apostolik's program -- cool though it is -- doesn't even do that. He hasn't incorporated betting yet. "That would be an interesting thing to play around with," he says. True! But even that still wouldn't be A.I.

Jamie McCarthy

I did this with real money, and lost.

[GlenRaphael]

I wrote a program that played online video poker several years ago when online casinos were just starting to become popular. I had been making money at blackjack and this seemed like a potentially worthwhile time and money investment. It was very easy to write the program. I tailored my program to a site called InterCasino because at the time they were offering a video poker game that was positive expectation for players with perfect basic strategy.

I only had a Mac handy but the casino client software was windows-only, so I ran the client under SoftWindows emulation figuring this would make it harder for the client to determine something untoward is going on.

Recognizing the cards involved nothing more complicated than hit-testing for colored points along lines with certain offsets. The card pictures never changed, so it was just a matter of partitioning the set appropriately. For instance, a "9" of anything has a pip in the top-left corner; an ace does not. The only thing that was at all tricky there was differentiating a couple of the face cards. For instance, I seem to recall that their Jack of Spades and Jack of Clubs looked very similar to my algorithm. Differentiating spades and clubs generally was hard but it was especially hard with the face cards.

As for the play, that also required no original work. There are people who play video poker professionally; you can order basic strategy charts for all the major games that detail what payout schedules have what expected value and tell you how to play the hands. Some common casino machines are positive expectation and the casinos don't care because the average player makes enough strategy errors to be a net loser. My program had to recognize the card values, categorize the hand, then run down a chart (either a table lookup or a series of if-thens) to determine what strategy to employ. Then it had to simulate a mouse click on the appropriate "hold" buttons, then on the "draw" button.

To tune and debug the program, I had it play the hands but NOT hit draw until I verified the right cards were selected. I played through many hundreds of hands, discovering and fixing several bugs along the way. There were a couple of card-recognition bugs and a couple of play-related bugs, but eventually I got to the point where every time I questioned a play and looked it up in the rules, the computer was right. My program could play faster and more accurate video poker than I could.

InterCasino had two modes. You could play just for fun with fake money, or you could play for real money. To shake out the bugs, I played in the "just for fun mode", letting it run through the night while I slept. I had it preprogrammed to pause at random intervals for random amounts of time to simulate bathroom breaks or whatever, but mostly it just played.

In the morning, I'd won about a thousand dollars of fake money. As long as I played in "free" mode, I seemed to be earning about what was expected, occasionally hitting the big payoffs that make up for the long term negative grind.

So I switched to the real-money mode. I figured it was easily worth investing $500 or so of my blackjack earnings to see if I could institute a real money machine. The dream was to have a bank of PCs in my closet raking in the cybercash while I sleep, eat, read, or go to work... So I deposited the money and set the thing to work. Again I ran it in semi-interactive mode for a while before putting it into automatic.

And it lost. So long as I played with real money, it just wasn't hitting the big payoffs as often as I think it should have. I switched back to free mode and won again, switched back to paid mode and lost again. After losing several hundred dollars I decided to pull the plug.

I did not play enough hands to be sure I didn't have a run of bad luck. Video poker is a game with a very small edge and almost all of the edge you have comes from hitting that once-in-a-blue-moon royal flush or equivalent. Maybe if I'd been willing to lose $10k I'd have eventually gone positive. But there's no way to prove that the game is honest, so I couldn't count on the laws of probability being in my favor. My only protection was to have a reasonable stop-loss figure, and I hit it.

So I won't be retiring to the carribbean just yet. :-)

(In fact, I'm currently unemployed due to a recent dot.bomb event. Need a lead QA engineer, software engineer or otherwise just generally smart guy? My resume is here; check it out. Especially if you're doing something with handhelds.)

Not bad...here's some better stuff.

[ruebarb]

The only thing new about this is it's interacting with online gambling. Interesting if it could be used in a monetary sense - like Texas Hold Em or 7 Card Stud to actually win money. (if you're playing video draw poker, you're playing to lose. The house has the best odds) - In fact, I half wonder if it's being done right now...people playing online and feeding odds and stats into a computer to see if they've got the odds to win.

FYI - the most commonly recognized poker playing software is Wilson Software - www.wilsonsw.com - they design poker programs with very detailed programmable profiles...That's how I improved my game...a thousand hands a day no problemo. (it's also good for simulation - wondering if you'd save or lose money raising AK preflop? Run 100000 hands and find out!)

The most interesting thing is this...Poker is actually a lot like a casino. You think you're playing each other, but if you get the ten best players together, then all you have is 10 steady declines of bankrolls. The house rake sucks it all away at 10% a hand. The best way to make money at poker is to find a fish - which I'm thinking he could program this thing to beat no problemo

But ...

[psergiu]

> and actually reads the cards off the screen
...
> Yes, it does win over the long term as it collects many, many tokens.

The question is: can this program Punch the monkey to win ?
--

Re:Real Poker

[s390]

"So is this kind of like Big Blue, I mean if I were to play some of the best poker players with this little programs assistance would I win a majority of matches?"

In college we played hi-low poker for table stakes and it was... interesting. $100 might change hands in a single "tap" - a lot of money, then. Later I played 5-stud in $2 limit games (that's a loser to the house rake). Win some, lose some, but I do like poker. Haven't played much for years, though. If it's recreation, then don't count the chips. If it's for real money, it takes as much focused concentration as any profession. And as professions go, it's a fairly stressful one. Hacking MVS, networks, Linux is lots easier.

It's also an unfortunate fact that people do cheat at poker. I sat in a low-ball game once where the guy to my left said "watch this" before dealing me a low wheel (A2345, perfect hand). I raised the open and everyone folded; I took the pot and left. When there's real money on the table, the odds of people cheating in various ways go way up. Be aware. There's even more danger of cheating online in virtual casinos, where you don't know which "players" are shills and which aren't; the server (house) knows your cards.

Poker isn't like chess, in that it's not just a deterministic game. A simple poker 'bot will be too predictable (below 50% fold, above 75% raise, etc.). The reason it's an interesting game is that human strategies, emotions, and intuition all play at some times in any session. Poker, like playing the stock markets, is 50% money management and 50% picking plays. The similarities are very striking.

Poker 'bots have been written and played against each other by academics for decades. Best results are obtained by heuristic routines that analyze the moves of all other players and learn from the progression of play, building a 'book' on each from which to calculate odds about bluff or hold. There are many books out about poker (most published to scam wannabe winners out of a few dollars), and not a few academic papers - use Google to find them. Probabilities are easy, strategies are hard, for many reasons.

But poker is a complex game, especially with human factors added. The complexities are infinite and arguably incalculable. There is no perfect way to play. One of the best session strategies is to bluff on small pots early, have 'em on big ones, then later "on towards morning" reverse the strategy - but like all poker, it's risky and won't work repeatedly with the same group of players, like most poker session strategies. Or, one could reverse the above strategy. These are the two basic possibilities for any one game. If one plays consistently though, others will see it and learn when to duck or push. There are levels within levels, in each session and venue.

I've been killed at poker, and I've also shut down a small poker parlor by taking every player until no one would sit down with me (400% profit). One good piece of advice I'll offer is... don't play for blood with friends, or especially, coworkers. Those relationships are more valuable than money, and some people take lasting offense at a sandbag. If they give you hands, OK - but don't extract it.

java golf

[Perdo]

While not as great an accomplishment, I wrote a macro to play a java golf game online. Since there was some randomness to the game and my reflexes are not that good, I figured it would be a way to get my name in the top ten. after three days running the macro, I had all ten of the top ten spots. The game was pulled on the third day. It has never been back. Computers playing games are always much better than humans if their AI is done correctly. I realized that I had ruined the game for everyone else. Deep blue beat Kasparov, My AMD box beat java golf, Aimbots beat me at TFC, Paul Apostolik beat pogo poker, and the power producers are beating California (with a great weather model/consumpion prediction algorithm) on the energy trading floors.

Are we are all cheating? Cheating only applies to games and applying enourmous computer power to commodity markets is simply shrewd? Will we one day wake up with zero value in stocks because some bright day trader wrote a miraculous piece of code that makes a billion perfect trades in the course of a day? Cheating by attacking protien folding with petaflop power?

What are the moral causes for the application of analytical force? When is the right time to spray for moths? When is the right time to let a hummingbird fly over japan? (obscure reference, chaos theory: the ammout of air a hummingbird moves in flight over japan is all that is required to generate a hurricane a month later in florida)

With IBM predicting a teraflop available on your (or Saddams) desktop by 2004, these questions need to be addressed yesterday.

Impressive

[sg3000]

> After losing several hundred dollars I decided to pull the
>plug.

>I did not play enough hands to be sure I didn't have a run
>of bad luck. Maybe if I'd been willing to lose $10k I'd
>have eventually gone positive.

Wow, it sounds like not only did you write a program to simulate video poker playing, but also to simulate a gambling addiction!