Slashdot Mirror

← Back to Stories (view on slashdot.org)

Poker Pros Win Against AI, But Experts Peg Match As Statistical Draw

Posted by timothy on from the at-this-rate-they-win-next-year dept.

hypnosec writes with some positive news for Skynet watchers, in that humans still have at least a slight lead against the AIs who might one day imprison us in energy-harvesting goo tanks, or at least beating us in Las Vegas. The two-day poker showdown involving four of the world's top (human) players and a Carnegie Mellon University AI program called Claudico saw the professionals win, after several days of heads-up no-limit Texas Hold'em. "Despite the win, the poker players' $732,713 collective lead over Claudico wasn't quite large enough to attain statistical significance, experts have said. This means that the results can't be accepted as scientifically reliable thereby indicating that the "Brains Vs. Artificial Intelligence" competition effectively ended in a statistical tie." On the other hand, the computers sure got over what looked like a rout by the humans.

65 comments

  1. So which is it? by Anonymous Coward · · Score: 4, Interesting

    Halfway through the competition, the four human pros had a cumulative lead of 626,892 chips. Though much could change in the week remaining, a lead of around 600,000 chips is considered statistically significant.

    or

    "Despite the win, the poker players' $732,713 collective lead over Claudico wasn't quite large enough to attain statistical significance"

    1. Re:So which is it? by Kjella · · Score: 2

      Probably something to do with average winnings/hand, playing more hands with the total winnings not increasing much makes the significance weaker.

      --
      Live today, because you never know what tomorrow brings
    2. Re:So which is it? by Anonymous Coward · · Score: 0

      Halfway through 600,000 is significant. All the way through, 1,200,000 is significant.

    3. Re:So which is it? by kekx · · Score: 1

      Well, the important thing is "halfway through the competition". I don't know how to calculate the statistical significance here, but it seems sensible that it would need to be a bigger lead after playing more games.

    4. Re:So which is it? by Anonymous Coward · · Score: 0

      That doesn't make much sense. A statistical test for the presence of a difference should get more sensitive as sample size increases. We need to know exactly how this was calculated.

    5. Re:So which is it? by Mateorabi · · Score: 1

      Perhaps chips are worth significantly more than $1? So the quantity of winnings shrunk significantly between midpoint and end?

      --
      "You saved 1968." - Ms. Valerie Pringle to the crew of Apollo 8

    6. Re:So which is it? by Anonymous Coward · · Score: 0

      Yeah no way is $732,713 statistically significant! No worries, I'll take care of that insignificant sum of $$$ for you ...

    7. Re:So which is it? by Demonoid-Penguin · · Score: 1

      Probably something to do with average winnings/hand, playing more hands with the total winnings not increasing much makes the significance weaker.

      Is there a difference between what a gambler calls a "useful advantage" and what statistician calls one?

      I know a couple of professional poker players - they reckon the "house" has as 5.5% advantage, and are pretty pleased with any system that gives them a 2% advantage.

    8. Re: So which is it? by Anonymous Coward · · Score: 0

      If $732,000 is not significant, give it to me.

    9. Re:So which is it? by CodeArtisan · · Score: 1

      It's the poker players' mentality. if you speak to any of them, none will ever tell you they are a losing player. The losers always say "Oh,I pretty much break even"

    10. Re:So which is it? by rahulov · · Score: 1

      If someone is curious it was 170M of total bets. Bot at least is not total fish. https://www.cs.cmu.edu/brains-...

  2. Per Player? by Guy+From+V · · Score: 1

    Maybe it means a 600k chip lead per individual head-to-head. Or it could mean FUCK THS COMPUTARZ ROOL

  3. It was the beginning of the rout of civilization, by turkeydance · · Score: 0

    of the massacre of mankind.

  4. Why non-conclusive? by Anonymous Coward · · Score: 0

    So after about 13*8*4=416 man-hours of play, they can't say who's better? Was "no-limit" the reason why? I say the test is badly designed if 99.9% of the time is wasted betting pocket change, and the outcome is determined by just 10 big bets.

    1. Re:Why non-conclusive? by Mikkeles · · Score: 4, Funny

      Personally, when I gamble and end up about 3/4 of a million dollars in the hole, I assume that I lost.

      --
      Great minds think alike; fools seldom differ.
    2. Re:Why non-conclusive? by hax4bux · · Score: 0

      Mod up.

    3. Re:Why non-conclusive? by davester666 · · Score: 0

      Not applicable. The computer was only playing for street cred. It had no real stake in the outcome, as it would not have received any of the money if it "won" and doesn't have to pay anything for losing.

      --
      Sleep your way to a whiter smile...date a dentist!
    4. Re: Why non-conclusive? by Anonymous Coward · · Score: 0

      You're overthinking this. The computer clearly lost.

    5. Re:Why non-conclusive? by radtea · · Score: 1

      Personally, when I gamble and end up about 3/4 of a million dollars in the hole, I assume that I lost.

      That sounds more like a conclusion than an assumption.

      But the question isn't "Who won?" It is: "On the basis of this result what can we say about who will win next time?"

      I don't know what kind of measures they used, and there are a couple of links in this discussion to papers pointing out how problematic p-values are, but it is perfectly possible for the weaker competitor to win any given competition. All it requires is that the width of the performance distributions be large enough to give significant overlap between the players.

      People who don't understand statistics are baffled by this. They see individual instances, but statistics is about distributions. We can, by measuring instances, make judgements about the distributions they are drawn from, and knowing about the distributions we can make predictions about future instances.

      In the present case, it appears that the observed distribution of performance was such that it wasn't possible to distinguish clearly between the case where the computer is slightly better than the humans but the humans got lucky, and the case where the humans are definitely better than the computer.

      --
      Blasphemy is a human right. Blasphemophobia kills.
    6. Re:Why non-conclusive? by AuMatar · · Score: 2

      Welcome to most poker. 75-85% of hands you fold at a full table without putting any money in. Another big chunk you fold to a bet either pre or post flop. Most of your money in any session is won or lost in a few big hands.

      --
      I still have more fans than freaks. WTF is wrong with you people?
    7. Re:Why non-conclusive? by Anonymous Coward · · Score: 1

      There is a slightly better description here:

      Although the gap between the humans and Claudico seemed large, Tuomas Sandholm, the CMU computer science professor who has overseen Claudico's development, said that when you consider the humans' win rate, and the fact that the total amount wagered over the two weeks in theoretical dollars was $170 million, $730,000 is not that much.

      http://www.smh.com.au/world/man-bests-machine-in-twoweek-poker-tournament-20150509-ggxwuz.html

    8. Re:Why non-conclusive? by Paradise+Pete · · Score: 1

      Personally, when I gamble and end up about 3/4 of a million dollars in the hole, I assume that I lost.

      Obviously you lost, but the question is why. Was your expectation negative or did you get unlucky?

    9. Re:Why non-conclusive? by Anonymous Coward · · Score: 0

      Not bad, having about a $35,000,000 stake.
      Or are they counting the money multiple times?

  5. Optimal outcome by Reaper9889 · · Score: 2

    I think this is the optimal outcome for the scientists. They can show: 1) We have done something (i.e. the poker bot is not too far from the best human players so our time has not been wasted). 2) There is more to be done (i.e. give us more money to look at this).

    Also, I do think it is a quite impressive outcome.

    1. Re:Optimal outcome by bloodhawk · · Score: 1

      reading the details this looks more like a complete route and they are just trying to put a nice face on things. A person that relies heavily on maths and is a tight player would have similar results, these are the type of players that professionals make their living off, players that slowly lose money too you, not fast enough to scare them away and make them realize they suck though.

  6. Is playing a game Artificial Intelligence? by tomhath · · Score: 0

    Seems more like applied statistics to me.

    1. Re:Is playing a game Artificial Intelligence? by PRMan · · Score: 3, Informative

      Playing blackjack is applied statistics. Playing poker is AI.

      --
      Peter predicted that you would "deliberately forget" creation 2000 years ago...
    2. Re:Is playing a game Artificial Intelligence? by blueshift_1 · · Score: 1

      Pretty much any algorithm can be considered Applied Statistics... and even our own existence if you want to go down that rabbit hole.

    3. Re:Is playing a game Artificial Intelligence? by tomhath · · Score: 1

      So what you're saying is that it's AI if we call it AI.

    4. Re:Is playing a game Artificial Intelligence? by Anubis+IV · · Score: 2, Interesting

      In this case, yes, it is. All they gave it as inputs were the rules to the game. The AI had to make its own determinations after that for what the optimal strategy was, how it would go about achieving it, and how it should respond to the individual characteristics of each of its opponents.

      Alternatively, ask yourself if playing a game takes intelligence at all. We could argue that all that the pros are doing is making their best estimation of the statistical likelihood that they'll win a hand, then betting accordingly along an optimal path that they've cultivated through experience. It's really no different than what this AI was doing either, it would seem.

    5. Re:Is playing a game Artificial Intelligence? by itzly · · Score: 1

      Any AI problem that is solved, is no longer AI.

    6. Re:Is playing a game Artificial Intelligence? by tomhath · · Score: 1

      All they gave it as inputs were the rules to the game. The AI had to make its own determinations after that for what the optimal strategy was,...

      Yes, that was where the original question was going (despite someone thinking it was trolling and modding me down).

      When the domain is constrained by a set of rules and the programmers (not really "the AI") have built a decision tree on how to act based on current conditions, it makes me wonder what is meant by intelligence. Obviously it takes some intelligence for a person to be good at the game; but it also takes some intelligence to multiply two numbers together. I don't think anyone claims multiplication is AI.

      A few years ago "Expert Systems" were all the rage. We used to joke that the defining characteristic of an expert system was whether or not it could execute "if" statements. So, to repeat the question, at what point does "if" statements and decision trees become Intelligence?

    7. Re:Is playing a game Artificial Intelligence? by AthanasiusKircher · · Score: 1

      Alternatively, ask yourself if playing a game takes intelligence at all. We could argue that all that the pros are doing is making their best estimation of the statistical likelihood that they'll win a hand, then betting accordingly along an optimal path that they've cultivated through experience.

      Perhaps that's true.

      It's really no different than what this AI was doing either, it would seem.

      Except that human players somehow manage to make their estimates through an advanced "higher-level" intuition, while TFA says: "Claudico [the AI] sets its own strategy, Brown noted, and that strategy occupies about two terabytes of data -- far more than the CMU team could analyze."

      And here comes the problem with a single definition of "intelligence." If you believe (as I do) that "intelligence" fundamentally requires a level of adaptability, as well as abiilty to alter one's own behavior on the basis of new data, can this AI do anything like that? It seems it cannot, because TFA later talks about how the next steps will be to spend time analyzing the 80,000 hands of pro poker playing they have on file and "improving the algorithms" which the AI uses to generate its 2 TB strategy. This is a tremendous achievement that should not be underestimated, but ultimately we're still looking at stats from a "black box" and tweaking some of its inputs until its output stats are better.

      Meanwhile, you ask whether playing a game takes "any intelligence at all." If it does not, then why are these very, very smart people taking months and years to create a program that can beat humans? The answer, to me, lies in the fundamental nature of true "intelligence," which requires levels of abstraction, concept formation, and higher-level adaptability that doesn't seem to be part of most AI algorithms in any meaningful way.

      Everyone's always talking about how the "goalposts" are supposedly moved for AI all the time. I have the same goalposts that AI researchers declared back in the 1950s in terms of concept formation and adaptability.

      When they have created an AI that plays poker at this high of a level and can actually TWEAK ITSELF to beat human champions consistently, THEN I'll begin to consider whether it's actually "intelligent." Give any of these human poker champions a perfect memory and the ability to create a 2 TB optimally-derived strategy that he/she could tweak however, and that human player would clearly blow the pants off of any other player in the world, as well as any current "AI." Yet somehow humans manage to do well without all of that perfect access and recall to huge quantities of information... and it's in that higher-level sorting and adaptability that the secret to true "intelligence" lies.

      Or, get this AI to the point that it can beat any human poker players in the world, and then let it learn to play another card game on a high level without any human assistance or tweaking of algorithms. Then I'll believe it's closer to "intelligence." Or let this poker-playing AI figure out how to make me a decent sandwich BY ITSELF or with some basic instructions from me. Or whatever. The specific task is irrelevant, but one point of intelligence should be the ability to develop NEW SKILLS independently, rather than requiring human tinkering and tweaking with algorithms. It's great to have 2 TB of optimally-generated perfect strategy to win against humans, and it's a fascinating and awesome experiment -- and I COMPLETELY applaud the efforts of this team. But, to me, it doesn't qualify yet as anything close to "intelligence" just because the generated strategy happens to be too long for humans to spend time parsing.

    8. Re:Is playing a game Artificial Intelligence? by Anonymous Coward · · Score: 0

      The difference is with humans in poker the maths and betting behaviours is the easy part, This is also all a computer can work on. The hard part on poker is actually the social and psychological aspects, these are the parts that turns solid/good players into great players and as a computer can't engage in that yet it will never be much more than a fun experiment in maths until it can.

    9. Re:Is playing a game Artificial Intelligence? by itzly · · Score: 1

      Yet somehow humans manage to do well without all of that perfect access and recall to huge quantities of information.

      Humans also have access and recall to huge quantities of information. It may not be perfect, but in the case of professional poker players, it's close enough.

    10. Re:Is playing a game Artificial Intelligence? by Anubis+IV · · Score: 1

      I completely agree with everything you said in response to me. The only thing I feel a need to address is this:

      Meanwhile, you ask whether playing a game takes "any intelligence at all." If it does not [...]

      I was trying to call attention to a double-standard he had set up. I took it as a given that we would all agree that games take intelligence to be played, and that as a result we would recognize the invalidity in his assertion that this is nothing more than applied statistics, since that assertion could be applied equally as well to natural intelligence.

      As you, I think that what we've seen on display here is impressive, but it falls short of what we'd typically think of as "intelligence", though that word has enough different meanings to it that we need to be careful how we're using it. We're still nowhere close to achieving strong AI, and I have serious doubts that we ever will. My Computer Science research advisor in grad school once tried to explain what he viewed as the impossibility of it by pointing out that we're incapable of ever fully conceiving the entirety of our own brains and holding that idea in our mind (i.e. holding onto all of the details of a brain would take more than one brain of storage), so the notion that we could ever replicate the intricacies of our own brains is nonsensical. I'm not sure I agree with him (after all, the burden to conceive of a brain can be shared among many people, each of whom only needs to know their part), and I'll admit that I'm likely not doing justice to his ideas since I'm probably explaining them poorly, but it was definitely an interesting idea to hear, particularly since he was a through-and-through atheist, and I had only ever heard such ideas espoused by religious folks up to that time.

  7. Statistical by Livius · · Score: 1

    The losing team of every competition likes to say things like 'statistical draw'.

    1. Re:Statistical by PRMan · · Score: 0

      Poor losers. Real men can admit they lost, even in Game 7 in triple overtime.

      --
      Peter predicted that you would "deliberately forget" creation 2000 years ago...
    2. Re:Statistical by Anonymous Coward · · Score: 0

      01000110 01010101 01000011 01001011 00100000 01011001 01001111 01010101 00100000 01001101 01000101 01000001 01010100 00100000 01000010 01000001 01000111

    3. Re:Statistical by Anonymous Coward · · Score: 0

      I'll have to remember this one the next time my bookie comes to collect...

      Bookie: Your team didn't cover the spread, pay up.
      Me: While that may by true, the results were not statistically significant.

      News Caster: The body was found floating in the East River...

  8. 20,000 hands against each player? by 14erCleaner · · Score: 0

    Let's see, over two days that's about 400 hands per hour, or one hand every 8.5 seconds, assuming the players take no breaks for sleeping, eating, etc. Yet they still trounced the computer, and it's claiming "no statistical significance"? I think this is one of the greatest achievements of man versus machine in history!

    --
    Have you read my blog lately?
    1. Re:20,000 hands against each player? by Anonymous Coward · · Score: 0

      You just keep that belief close to your chest when Skynet comes for the humans. I'm sure it'll choose poker as a deciding method of who gets control of the earth.

    2. Re: 20,000 hands against each player? by Anonymous Coward · · Score: 0

      Just for the information of those not into poker: These days, so-called multitabling is the norm among humans playing online. Anywhere from three to a dozen games at the same time are pretty much standard (even at heads-up which requires many more actions than 6max or full ring).

    3. Re: 20,000 hands against each player? by Anonymous Coward · · Score: 0

      Just for the information of those not into poker: These days, so-called multitabling is the norm among humans playing online. Anywhere from three to a dozen games at the same time are pretty much standard (even at heads-up which requires many more actions than 6max or full ring)

    4. Re: 20,000 hands against each player? by bloodhawk · · Score: 2

      the reason people can multi-table so much is because you rarely have to do anything. mostly you are just folding until you find hands or positions you want to play from, multi tabling reduces the boredom and stops you from playing in hands you shouldn't. It is not uncommon to go 10-20 hands in a row on a table where all you are doing is folding. multi tabling is not hard.

    5. Re: 20,000 hands against each player? by Severus+Snape · · Score: 1

      the reason people can multi-table so much is because you rarely have to do anything. mostly you are just folding until you find hands or positions you want to play from, multi tabling reduces the boredom and stops you from playing in hands you shouldn't. It is not uncommon to go 10-20 hands in a row on a table where all you are doing is folding. multi tabling is not hard.

      Yeah sure, you are right and multi-tabling works great if you are a level above your competition. Where multi-tabling falls down though is studying your opponents style and play. Necessary to give you an edge against players who are of a similar skill to you.

    6. Re:20,000 hands against each player? by Anonymous Coward · · Score: 0

      20,000 hands over 2 weeks, not 2 days (typo in the summary).

    7. Re: 20,000 hands against each player? by bloodhawk · · Score: 1

      in online poker that isn't hard. There are a heap of programs that track the betting behaviours and stats of your opponents so you can know as much as possible without having to monitor the table constantly. online poker by its very nature is far more limited in the information you have and as such is not to difficult to track.

  9. If statistical result is important in poker by fluffernutter · · Score: 1

    Then can I have the $732,713?? I mean really, isn't this the very point of AI? To be less than just a statistical automaton and actually be able to beat a human at real life in something? AI should be about winning the money, that's it.

    --
    Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
    1. Re:If statistical result is important in poker by Anonymous Coward · · Score: 0

      I am not sure how winning at Poker proves intelligence or AI capability.

      It is Poker, not even chess... Did we give up on Chess because computers can see very possible move and every possible outcome in a millisecond? Chess is more like war than Poker, so AI owning us in Chess is more scary. I guess we moved onto Poker, because it is about the luck of the draw as much as any intelligence on odds of swapping out card which again are mainly luck of the re-draw.

      "No No guys, it was total skill and intelligence that led me to getting 3 aces on the re-draw to go with the ace I kept from the original draw...."

  10. Statistical Significance != Scientifically Reliabl by Anonymous Coward · · Score: 0

    This is just the most recent paper explaining this once again. The fickle P value generates irreproducible results:

    The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.

    http://www.ncbi.nlm.nih.gov/pubmed/25719825

  11. In no way does this make me feel better by Anonymous Coward · · Score: 0

    Let's see, ONE university (Granted, it is one of the top five most prestigious in its field) given a small handful of decades is able to statistically just-about-beat people.

    This does nothing to assuage my fears of Skynet. Imagine what a few dozens of universities, combined with the financial and industrial might of the US MIC, could do in a time span 60 or 70 years?

  12. I think I have seen this con before. by Anonymous Coward · · Score: 0

    Where the card sharps win by just a small enough margin to con the mark into a comeback game...... Don't fall for this simple ruse Claudico!!!!!!!!!!!!!!!!!!!!!!

  13. A Draw? by Anonymous Coward · · Score: 0

    How much do you want to bet that if the computer beat the humans by $0.50, the "But Experts Peg Match As Statistical Draw" part would be left out of the headline?

  14. "Statistical Significance" by foreverdisillusioned · · Score: 2

    Look, it is two thousand freaking fifteen. This is an article from some site called "Techie News" being re-reported at Slashdot. Can we please get a little ridicule of this supposed binary concept of "statistical significance" ? It would take us one or two sentences to tell us the actual numbers involved--the expected value, expected deviation, margin of error, confidence level, etc.

    And then when all's said and done, if indeed the level of significance was too low (e.g. p too high), maybe we could get a Bayesian or two in here to criticize the traditional 5% value is being arbitrary and tell us all a little about the Frequentist vs. Bayesian rivalry in statistics that persists to this day? (Obligatory XKCDs: https://xkcd.com/1132/ , https://xkcd.com/882/)

    1. Re:"Statistical Significance" by Anonymous Coward · · Score: 0

      The problem depicted in that first xkcd isn't really about Frequentist vs Bayesian. Although most people who do not even know that Bayesian stats exist seem to act the same way as that "Frequentist", the real problem is not knowing whether you measured what you think. That is not a problem stats can solve and is the prime reason that stats should not be driving scientific investigation or be equated with scientific reliability as in TFS.

      "The null hypothesis is rejected? OK, I can think of dozens reasons it may be false in the next hour, I'm sure if we all got together and thought of them the list would be orders of magnitude longer than that."

      "The null hypothesis wasn't rejected as false? OK, I can think of dozens reasons it may still be false in the next hour, I'm sure if we all got together and thought of them the list would be orders of magnitude longer than that."

    2. Re:"Statistical Significance" by Shadow+of+Eternity · · Score: 1

      This is why we need to teach students that theory drives stats and not the other way around. There are tons of bullshit results out there that are perfectly statistically valid but logically bullshit.

      --
      A bullet may have your name on it but splash damage is addressed "To whom it may concern."
    3. Re:"Statistical Significance" by foreverdisillusioned · · Score: 1

      Talking about correlation and causation is also rather important but I was asking for something a little more fundamental--exposing the term "statistically insignificant" for the bullshit that it is.

      Maybe the mainstream media can't understand anything more complicated than that woefully misleading phrase, but why can't "techie news" and "news for nerds" give us the details? As-is, we can't have any sort of meaningful discussion here about the results of this experiment; we can only speculate.

    4. Re:"Statistical Significance" by Shadow+of+Eternity · · Score: 1

      We can't have that for the same reason we're constantly flooded with "Ew nerds are gross!" stories. They're pandering to a different crowd that can't understand the details.

      --
      A bullet may have your name on it but splash damage is addressed "To whom it may concern."
  15. Novices by Anonymous Coward · · Score: 0

    Wonder what will the results looks like of the match is against novices. Will the AI still lose a bit, lose badly or win a lot?

  16. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  17. $732,713 is a statistical tie? by neminem · · Score: 1

    Awesome. I want to play poker with these guys... we can just draw right at the beginning, and they'll give me 700 grand, right?

  18. conjecture to explain by Anonymous Coward · · Score: 0

    The AI didn't exploit human weakness (failed bluffs) but the humans did. This explains why the AI got better as the weaker human players left the arena -- The winning humans didn't beat the AI, they beat other humans to outcompete the AI.

  19. Give it a try yourself... by iq145 · · Score: 1

    http://poker.srv.ualberta.ca/