OpenAI's Bots Defeated Former Pro E-Sports Players At Dota 2 (theverge.com)

On August 5th, OpenAI's bots defeated four former professional Dota 2 players in two of the three matches played. "There were a few conditions to make the game manageable for the AI, such as a narrower pool of 18 Dota heroes to choose from (instead of the full 100+) and item delivery couriers that are invincible," reports The Verge. "But those simplifications did little to detract from just how impressive an achievement [the] win was." From the report: The OpenAI Five triumphed in convincing fashion in the first game, not allowing the human players to even destroy one of their defensive towers. The humans recovered a little in game two, conquering one tower, but they still got demolished. And finally, in a game three played purely for pride, the humans managed to squeeze out a win. What stands out when you watch the matches is the apparent intelligence of the AI's decisions and the inhuman absence of any indecision. The typical Dota 2 game, even on the professional tier, involves quite a bit of equivocation about whether to engage in a fight, try and shift it to a more favorable battleground, or run away from it completely. The OpenAI team just doesn't need the processing time that humans require, which made its play appear unnatural -- but only in the speed and crispness of the decision-making, not in the content of those decisions.

Comments

Sample size too small

[SuperKendall]

So wait a second, the pros beat the bots on round three...

Doesn't that seem to mean that the pros simply needed time to learn how the bots thought, and reacted to different situations? And that they did so by the third game, where after two sound thrashings they pulled out a win? Even if close that is quite a performance leap they managed.

I would have been a lot more interested in the results of a ten-game series where the pros had adequate time to understand the inhuman patterns of play the bots had.

Re:Sample size too small

[SirMasterboy]

For games 1 and 2 the AI got to draft it's own choice of 5 heroes out of the allowed 18 in an alternating picking sequence with the human opponents as is traditional. When each game started, the AI announced that it had a 95% confidence that it would win based solely on the heroes it had choosen against the human's choices.

For game 3 the audience picked 5 "random" heroes that had little to no synergy as a DotA team lineup and not good options for each of a DotA team's "roles" (pushing, ganking, tanking, supporting, etc). At the start of game 3 the AI announced that it calculated only a 2.9% chance that it would win against the human team with the heroes it was given.

Even with the severely gimped hero choices, it did an admirable job at trying to make it work by doing some unusual strategies. The professional casters even commented on how the AI was seemingly doing all the right things that it could have done given its poor position and that it actually did better in the match than anyone had expected it to do given the heroes that it was.

Team of casters

[Tough Love]

Those are hardly professional players, that is a team of casters who never practice together. It is doubtful that any of them could get onto even a second tier team today (well, moon, but then again maybe not). Still an impressive benchmark for AI and top tier pros are sure to get beaten sooner or later too, but what is the point of misrepresenting what actually happened?

Game three was just for fun

[martyros]

One thing the summary didn't point out was that game three they didn't let the AI choose their own heroes. The humans basically conceded defeat; so for the third game they were just experimenting: The let the "audience" choose the heroes, and they chose heroes specifically which would be poor at playing the style which the computers had played so far, just to see if it could change its playing style to adapt to the new heroes. And the human team chose exactly the heroes that the AI chose for the first two games. After that draft, the AI's rating of its own chance of winning was 2.3%, based only on the draft. The AI adapted somewhat, but not much; and near the end of the game, the AI seemed to be doing a bunch of fairly sub-optimal things; like, it knew it couldn't really win, so it didn't know what to do except random micro.

So, it was an interesting data point -- particularly the importance of choosing the right set of heroes. But it was certainly not a victory for humans. The AI soundly trounced them except when it was purposely crippled.

Matches, post-game commentary, and other information available on the OpenAI Blog about the match.

Bots Cheating

[Luthair]

It should be noted that the bots are cheating and have a different level of access to the information in the game that players do not have. e.g. they have no need to select players to know inventories, precise health, mana etc. In addition to perfect possible information they also have perfect control input.

They also make a big point about how the draft is considered to be one of the more difficult parts of dota and how impressive it is their bots can draft... but restrict the pool from the normal 115 to 18 and eliminate bans rendering it entirely unlike dota drafting. They also restricted 2 items, no summons/illusions, 5 invulnerable couriers (instead of a team having 1, which can be killed) and no scanning (a time-limited radar like ability).

Don't get suckered into the news hype cycle until the bots use a screen with USB input and play the actual game instead of an arbitrary one.