Artificial Intelligence in Poker

Markian Hlynka writes "The University of Alberta's research into Poker AI is featured in this New York Times article. There is also detailed discussion of the game of Poker, and the 'new breed' of players who have honed their abilities online. See the U of A's poker project for more information."

Comments

Article Text

[daBum]

For those who don't want to subscribe:

WHEN an accountant named Chris Moneymaker won $2.5 million in the World Series of Poker last May, the chatter in the poker world wasn't focused on his skillful bluffing, his tremendous luck or even the aptness of his surname. Everyone wanted to know how a man who had never before sat down at a tournament table could clean out so many skilled professionals.

While the Las Vegas hype machine focused on the rags-to-riches tale of a man who parlayed a $40 entrance fee into a huge pot, many poker players recognized that the amateur's success signaled the arrival of a new age in the game. Mr. Moneymaker may never have been in the same room as other players in a tournament of Texas Hold'em poker, but he had played extensively online, where the game is faster but the money is just as real. He was as much a rookie as Ichiro Suzuki, who joined the Seattle Mariners after nine years in the Japanese major leagues.

The online poker saloons that nurtured Mr. Moneymaker, 27, are just the beginning. Many players hone their craft with simulation software that allows them to test strategies by playing out thousands or even millions of hands. Some researchers are building software opponents that use sophisticated concepts from economics and artificial intelligence to seek out the best strategy, then use the knowledge to beat human players. The experience of playing thousands of games in roadhouses and casinos is being eclipsed by a cyborg-like intelligence produced by humans weaned on machine play.

The changes in the nature of the game are both subtle and striking. The advantages of some well-understood strategies are being tuned, and others are being abandoned. Some online enthusiasts, for instance, are even suggesting that the value of any information gleaned from watching the opponent's body for telltale tics or gestures is overrated. These so-called tells are too easily manipulated. More information comes in the pattern of bets, raises and calls. The money, they say, talks.

The biggest factor propelling change may be the speed of technology. Players do not wait while someone shuffles and deals. Chips do not need to be counted or watched. Computers handle the accounting, often finishing hands in as little as 30 seconds.

Steve Badger, the editor of the Web site playwinningpoker.com and winner of the 1999 World Series in a game called Omaha Hi-Lo, says that online poker halls are appealing because of their convenience.

"You could play them every day," he said. "You're able to play two games at the same time. Or you can sit and read or vacuum or do any infinite number of things while waiting for the next hand."

The online halls also offer substantially better rates. Most casinos pay for the lights and the dealer by subtracting either a fixed amount or a percentage from the pot. This levy, known as the rake, is often about $3 to $5 a hand in physical casinos, but about $1 or less online.

The rake depends on the stakes, which can be lower than those at physical casinos. Some online tables have minimum bets as low as 25 cents, an amount that makes learning the game cheaper. The speed of the game, however, ends up raising the amount at risk because 60 to 100 hands can be played in an hour. Higher minimum bets of $5, $10 or more are also common at tables with the better players.

Gautam Rao, a well-known Canadian player, said he stopped going to casinos in 2000, not long after his daughter was born, "because of the smoke and distance.''

"I told my wife I had to find a way to play online," he said. Now, he is able to play every night between 10 p.m. and 3 a.m. while his daughter sleeps in the next room.

"The rake is much less," he said. "The number of hands is much more. There are never any misdeals. There are never any issues related to tipping. The average cost of winning a pot is so much less. It's so much more efficient."

The speed of play lets players work through the thousands of apprentice hands faste

It's all about the odds

[Katamai]

To create a poker AI you just have to figure out what the odds are of getting something after your first hand. Then based on odds the computer will decide which cards to get rid of and which to go for. This will typically result in the computer not going for the long shot royal flush and instead going for the safer full house which is more likely to happen.

A computer theoretically could be as good or even better than the average human at poker. It is able to calculate the odds of winning and is therefore able to make the best choice possible.

What would be really amazing would be if the computer was able to calculate based on how many cards other players turn in and adjust itself as neccesary.

Can you imagine having to try to look at the computer and imagine if it is bluffing? Talk about poker face...

Re:It's all about the odds

[pizen]

Your thoughts on odds are valid. However, the AI here plays Hold'em, which is very different from Draw. In addition to calculating the odds it has to account for the betting of each player on the different streets. It would be interesting if it could tell if a player was playing slow or just limping in. It would probably be difficult to figure from just one hand but over the course of a session the computer (just like real players) is going to pick up on the betting habits of the players. When I get off work I'm going to go try this thing out. If it's better than me it can only help improve my game.

Re:It's all about the odds

[djp928]

Well, if he won't, I will.

Hold 'Em (also known as Texas Hold 'Em) is a variation of stud poker that uses community cards on the board to complete the hands of the people playing.

In the first round, everyone is dealt two cards, face down. These are known as your hole cards. The first round of betting takes place after the hole cards are dealt, with the person left of the "dealer" (in casinos, this is represented by a button placed in front of the player which rotates clockwise around the table with each hand) going first. In the first round, nobody knows what anybody else has, obviously, but watching the patterns of checks, raises, and calls can give you a good idea of how much the person likes the two cards given them.

After the first round of betting, the dealer turns up three cards face up in the middle of the table. This is known as "the flop", and these are community cards. You get to play these cards in combination with the two you hold in your hand to make the best five-card poker hand possible.

After the second round of betting, a fourth card is added to the board, known alternately as "the turn" or "fourth street". Another round of betting takes place, then the fifth and final community card is revealed, called either "the river" or "fifth street".

At this point, everyone still in the pot has seven cards to work with to make a five-card poker hand. After the final round of betting, all remaining players show their hands, and the best hand wins.

The reason this variation of poker has so much more strategy involved in it is directly related to the community cards. You can tell, at each step of the game, what the best possible hand (called "the nut") is for that set of cards. Say the flop is Ac Ks 5d. The best possible hand so far, including the players two hole cards, is three aces. But, if nobody bet before the flop, then it's a fairly good bet that nobody has three aces, because anyone with bullets in the hole before the flop would probably bet/raise before the flop. With this same flop, say the turn hits Js. NOW the nut hand is a straight, ace high, but that assumes somebody stayed in the pot on 10 Q, which, depending on the betting pattern and the reletive skill of the players, may or may not be likely. Then, say the river turns up As. Again the absolute nut hand is four aces. But by this time, you can probably tell if anyone has that or not by the pattern of their betting.

If you have two spades in the hole, you have a flush now. It may or may not have been a good idea to have stayed in the whole way with your two spades, but at this time you think you're looking good, right?

Well, you may have missed that someone with A K, A J, or A 5 has a full house now. Again, you're in a good position to judge at this point whether someone has A K in the hole by the pattern of their betting. A K, especially of same suit, is a strong hole hand, so that person probably would have bet it before the flop, and after catching two pair on the flop, may have felt good about continuing to bet, depending on the pattern of other bets.

So, instead of just betting on the reletive strength of your hand, you have to take into account what you know and what you can divine of the other player's hands. Hold 'Em is much more strategy oriented than draw or straight stud poker.

-- Dave

Re:It's all about the odds

[TexVex]

In Hold 'Em poker, the position left of the dealer posts a small blind bet, and the position to the left of that posts big blind. These "blinds" are forced bets, and as the deal switches clockwise with each hand, over time everybody must make the blind bets. The blinds guarantee action but do it in a very different way from ante.

Each player is dealt two face-down cards. Starting with theh position to the left of big blind, each player in turn must choose to either fold, call the blind bet, or raise. Play continues around the table until all players have either folded or called the blind bet or any raises. (But the big blind is given an opportunity to raise if other players only called (or folded out). Some hands end where all but one fold out before this first betting round is over.

There's your first difference between Hold 'Em and varieties of poker where you ante. You have to start making decisions about the strength of your hand very early.

Once the pot's right, three cards are dealt face-up in the middle of the table. This is the "flop". Each player will build his final poker hand using any combination of his own cards and the community cards in the middle. After the flop, another betting round ensues. Then the "turn" (a fourth community card) is dealt, and more betting happens. Finally, the fifth community card (the "river") is dealt, and after one final betting round the top hand wins.

There's the other big difference between Hold 'Em and Stud or Draw. The community cards. (Omaha also uses community cards but has more complex hand building rules). When you have shared cards, then your own hand's strength becomes relative. If three cards of the same suit are showing in the middle, and you got a pair of the same suit, then you have a flush. But there are reasonable odds that someone else at the table also has a pair of cards in the same suit. In stud or draw a flush is a very strong hand -- but in Hold 'Em it's not all that rare to see a showdown with two or three flushes in the same suit. (It's not actually possible in Hold 'Em for two players to get flushes in different suits in the same hand, because the majority of the community cards have to be in the flush's suit.) When that happens, the flush with the highest cards in it wins. The inexperienced often reraise aggressively with a low flush, only to lose to the other guy's higher hole card.

Re:Poker AI? riight...

[rkent]

Yes and no. For limit hold 'em, there is an (approximately) "correct" way to play based on pot odds, likelihood of drawing the hand you need, etc. Early rounds of no-limit tournies approximate this, with the occasional all-in raise to shake things up, but I'd have to agree with you that in later rounds against clearly competent players, no-limit is very much a social game.

Which leads to the larger issue: poker is a game of incomplete information; you don't KNOW what your opponent holds. You can make estimations based on past play and game conventions (eg, a bet from early position usually indicates AA, KK, or AK), but you don't know for sure, and this raises the possibility of deception.

The problem with that is, game theory generally models strategies to combat players who are playing (rationally) to win. Not all players play like this, or at least not apparently based on the strength of cards. I think most emulators are going to get screwed on bluffs.

But still, in lower-limit games, people are loose enough that bluffing doesn't really help (Lee Jones: "generally, you're going to have to show down with the best hand to win"), so a decent AI could at least maintain a winning margin, and so could an actual human who played tightly enough to take advantage of this. I don't know. You sure can't make zillions playing cards online, and it's definitely a while before the "deep blue" of poker.

Re:Tells

[zdislaw]

You read the article, right? The guy who won the last World Series of Poker had previously only played on the internet. I think that may qualify him for "climbing the ranks." Or maybe he just went right to the top, so techincally did not climb.

That may not be the "true art of poker" but I'm sure he's not letting it bother him too much.

Re:Poker AI? riight...

[tpengster]

Poker is not a card game, it's a people game (aka don't play the cards, play the people). It's all about bluffing and reading other people's bluffs. I'm baffled that people even bother playing poker on the internet. Even with webcams the game wouldn't be the same at all.

While it's true that poker involves reading people, it is primarily a game of calculating odds and using them to your advantage. For example: if you have a 1 in 4 chance of hitting your flush on the next card, then you should only pay to see the next card if there are $4 in the pot for every $1 you must pay. Etc. This gets more complicated with bluffing.

It's also easy to read people on the internet. People still bluff and call with patterns that reveal their hand, and observant players can take advantage of this. That's why the same players always win at the high-stakes table. That's why, in the recent World Series of Poker, the winner was an internet player who had never played a live game before

For more information on internet poker, please see PokerTips.org . This is a great site that has lots of strategy information and reviews of online casinos so you know which ones you can trust.

Re:Poker AI? riight...

[Idarubicin]

He then ran some numbers and proved that with that, they were open to an exploit of the Martingale system.

What sort of 'exploit' of Martingale is this? Martingale is ultimately always a losing strategy unless 1) there is no house limit and 2) you have an infinite supply of cash.

How Computers can win at poker

[tpengster]

Poker is primarily an odds game, that is to say it is all math. There are three places where a decision has to be made. The first decision is, "Should I pay to see the next card?" This is called Drawing. The second decision is, "I have a decent hand, but my opponent raised me. Is he bluffing?". The third decision is, "Should I try to bluff?".

Odds come into play everywhere. When you are Drawing, you must have the correct odds or else you will lose money in the long run. That is to say, if you have a 1 in 4 chance of hitting your straight on the next card, you must have at least 3:1 odds to Draw. (The pot must have $3 for every $1 you pay). There is also the concept of "implied odds" - predicting how much will be in the pot at the end of the hand and not just at the present.

When deciding whether or not to bluff, you must know the odds of your bluff succeeding, and add that to the odds of you hitting your out on the next card. At that point the calculation becomes the same pot-odds calculation described above. This involves some reading of your opponent; you have to know how often he will call, and how often the bluff will be sucessful. Luckily, computers can be pretty good at modeling and seeing patterns, probably much better than humans. It seems that Neural nets and other well-developed AI techniques would be very good at modeling these behaviors and predicting future ones. Calling bluffs will require the same type of knowledge.

Some have asked how it's possible to read patterns on the internet. Some people don't really have patterns in their game, they just call everything. These people will lose because they put too much money in the pot, they don't have the odds for the bets and calls they're making. Mostly, decent players have patterns in how they bet, for example they will bet when they only have 4 out of 5 flush cards. (A Semi-bluff). Computers have an advantage here because they can introduce a random element that humans cannot reproduce.

The recent winner of the World Series of Poker, Chris Moneymaker, had never played in a live game until the WSOP, he had only played internet games. This probably gave him good fundamentals in reading people based on their bets, and good math fundamentals.

Some have also questioned the wiseness of playing internet poker, since it is just "gambling". Well I'll tell you a little secret, poker isn't really gambling, poker is a skill game and especially with so many bad players out there who think it is just luck, hoping they'll get lucky, it's easy to win money. That's why the same players consistently win thousands of dollars online. For more information on poker strategy and reviews of online casinos, see this site: PokerTips.org

Texas No-Limit Hold Em

[Ridge]

Poker has had quite a resurgence as of late, both due to online play and media coverage. Chris Moneymaker was on Letterman one night and the World Poker Tour can be caught on the Travel Channel and sometimes older ones on ESPN or ESPN2. It's actually fairly enjoyable to watch. Nothing like seeing a guy go all in with for 800 grand on a stone cold bluff whilst his opponent has a pair (you see the hole cards on the tv broadcasts, at least on the Travel Channel). Some talk smack, some wear sunglasses, some play conservative, though most are fairly aggressive. I've learned quite a bit by watching them play. As others have mentioned and as the article alludes to, the game is more mathematical than reading your opponents, but that's not really what I've seen from watching the pros on the WPT. It's all about the people, the chips, the cards, and the math all seem secondary to your read on your opponent and your style and reputation. It's all very interesting, I suggest you catch a broadcast.

Re:Poker AI? riight...

[rudedog]

The point is not that you can actually expect to win with a bluff, the point is that if you develop a reputation for "never bluffing", then you never get called when you have the goods. The occasional bluff will not win the pot, but if you get caught in a bluff, you will be more likely to get called when you actually have a hand.

This advertising strategy is one of the few that work in low-limit poker as well, because people tend to remember bluffers, and they have this irrational need to "keep them honest".

Re:For non-robots, a simplified poker method

[Rudeboy777]

Good effort, but the Wizard of Odds site you linked to uses mathematical exhaustion as opposed to you general heuristics.

There is an outstanding product out there to train you to play "optimal" video poker (that is, make the mathematically best choice for any 5-card deal. A crippled shareware version can be found here, but this proprietary software is worth many times its' price: WinPoker (sorry Windows only, and no I have no financial incentive to plug)

Never forget that Poker (against other people online or in person) and Video Poker (where you try to complete the best hand to win a fixed payout) are COMPLETELY different games with very little shared strategy.

Loki

[Ka0s23]

There was a large push made by some great mathematicians in coordination with poker proffesional to make a poker "robot" they called Loki. What they found was over a large period of time it could be profitable at low limit tables, playing limit hold'em. However, once the game changed to no-limit, there was simply no way for it to deal with the fluctuating betting amounts. It could pick up on patterns and beat many online limit players. I would be willing to bet that they could design a sophisticated enough program to beat even higher-limit limit hold'em games, but I'm guessing right now they weren't quite confident enough to trust it with a serious bankroll! A poker bot which could play in no-limit tournaments would be almost impossible to program, as anyone who has played in no-limit tourneys know, its very different from cash games. Sorry if anyone allready posted about this, I didnt have time to read everything...

Re:Sigh....

[mugnyte]

Correction: Nothing stops the NYT from showing ads to us without the reg'd required. The registration bit is a farce, and everyone pumps nonsense into it. Why put it up as mandatory? Does the NYT try to spam its readers from that input? Wouldn't it be our duty to combat spam by intentially filling it with junk?

I let a lot of sites hog MY paid-for bandwidth with ads. This doesn't mean i have to type "Mr Goober" as my registration name on each one. The NYT is way behind in web concepts on this, IMO.

FYI: The google cache and the NYT's own backup copy are other sources of articles - registration free.