Slashdot Mirror
Can DeepMind's AI Really Beat Human Starcraft II Champions? (arstechnica.com)
Google acquired DeepMind for $500 million in 2014, and its AI programs later beat the world's best player in Go, as well as the top AI chess programs. But when its AlphaStar system beat two top Starcraft II players -- was it cheating?
Long-time Slashdot reader AmiMoJo quotes BoingBoing: It claimed the AI was limited to what human players can physically do, putting its achievement in the realm of strategic analysis rather than finger twitchery. But there's a problem: it was often tracked clicking with superhuman speed and efficiency.
Aleksi Pietikainen writes "It is deeply unsatisfying to have prominent members of this research project make claims of human-like mechanical limitations when the agent is very obviously breaking them and winning its games specifically because it is demonstrating superhuman execution."
"It wasn't an entirely fair fight," argues Ars Technica, noting the limitations DeepMind placed on its AI "seem to imply that AlphaStar could take 50 actions in a single second or 15 actions per second for three seconds." And in addition, "This API may allow the software to glean more information... " After playing back some of AlphaZero's back-to-back 5-0 victories over StarCraft pros, the company staged a final live match between AlphaStar and [top Starcraft II player Grzegorz "MaNa"] Komincz. This match used a new version of AlphaStar with an important new limitation: it was forced to use a camera view that tried to simulate the limitations of the human StarCraft interface. The new interface only allowed AlphaStar to see a small portion of the battlefield at once, and it could only issue orders to units that were in its current field of view....
We don't know exactly why Komincz won this game after losing the previous five. It doesn't seem like the limitation of the camera view directly explains AlphaStar's inability to respond effectively to the drop attack from the Warp Prism. But a reasonable conjecture is that the limitations of the camera view degraded AlphaStar's performance across the board, preventing it from producing units quite as effectively or managing its troops with quite the same deadly precision in the opening minutes.
Long-time Slashdot reader AmiMoJo quotes BoingBoing: It claimed the AI was limited to what human players can physically do, putting its achievement in the realm of strategic analysis rather than finger twitchery. But there's a problem: it was often tracked clicking with superhuman speed and efficiency.
Aleksi Pietikainen writes "It is deeply unsatisfying to have prominent members of this research project make claims of human-like mechanical limitations when the agent is very obviously breaking them and winning its games specifically because it is demonstrating superhuman execution."
"It wasn't an entirely fair fight," argues Ars Technica, noting the limitations DeepMind placed on its AI "seem to imply that AlphaStar could take 50 actions in a single second or 15 actions per second for three seconds." And in addition, "This API may allow the software to glean more information... " After playing back some of AlphaZero's back-to-back 5-0 victories over StarCraft pros, the company staged a final live match between AlphaStar and [top Starcraft II player Grzegorz "MaNa"] Komincz. This match used a new version of AlphaStar with an important new limitation: it was forced to use a camera view that tried to simulate the limitations of the human StarCraft interface. The new interface only allowed AlphaStar to see a small portion of the battlefield at once, and it could only issue orders to units that were in its current field of view....
We don't know exactly why Komincz won this game after losing the previous five. It doesn't seem like the limitation of the camera view directly explains AlphaStar's inability to respond effectively to the drop attack from the Warp Prism. But a reasonable conjecture is that the limitations of the camera view degraded AlphaStar's performance across the board, preventing it from producing units quite as effectively or managing its troops with quite the same deadly precision in the opening minutes.
This is not really beating a human fairly. If you could click that fast then sure, but otherwise it's not a fair fight.
Just cruising through this digital world at 33 1/3 rpm...
When AlphaZero was pitted against Stockfish, the best chess AI, they set the match up with an outdated version of stockfish, bizarre time controls that removed stockfish's edge in time management (a static time per move was enforced), stockfish didn't get its opening books (a mini database containing information about the best moves to start with), nor did it get endgame tablebases (another mini database of information about moving at the end of games) and it was limited to a very small amount of ram (only 1GB when it should've had 64GB or more). Deepmind will CONTINUE to mislead people about what they've accomplished at every opportunity.
... to test AI. Since RTS games already have a bad UI where the bottneck is the human being in the chair, aka trying to control many units with a limited UI v ia keyboard and mouse is cumbersome at best. It was even back in the Warcraft 2 days when you tried to bloodlust ogres or heal paladins -- healing paladins being damn near impossible. While warcraft 3 'fixed' the issue with impossible casting /w large numbers of units using autocast.
The main problem being is that games like starcraft can be played perfectly because it's really an action game masquerading as a strategy game, aka the actions take place in real time. So for a computer like deepmind, the human appears super slow. Imagine if you ropponent appeared retarded in terms of their reflexes. That's basically deepmind vs any human opponent in an RTS. So a computers perfect information and perfect reflexes mean making 99% accurate micromanaging decisions for units everywhere at once.
You can't do that as a human player. Deepmind for an RTS is like having an aimbot in quake. Not really impressive since we already know making bots that can win against humans is trivially easy.
I've seen a couple of comments already where folks are talking about DeepMind being able to micro and click faster than a human. While that's neat, that's not entirely the goal here with DeepMind. The makers already put an artificial limit on the actions DeepMind can execute to 600/minute. In comparison the humans are executing at around 250-300 actions per minute. Now that indeed made DeepMind's micro game strong, what was the real tipping point was that DeepMind could see anywhere where a unit was located. Humans however can only see a "screen at a time". When DeepMind's makers went back and implemented "screen at a time" limitation, DeepMind was easily fooled again. And that's the thing here. Not the "can I beat a human?" but "can a human fool me?". As soon as the amount of information coming IN to DeepMind was reduced, the data coming OUT couldn't compensate and the humans were able to slowly figure out how to trick the AI into an unwinnable situation.
There's a continual fallacy on Slashdot where pure research like DeepMind is confused for "who's jerb can it take and reasons why it can't take that jerb." The media here is presenting in the terms of "Hey look! Something AI can do better than us worthless puny humans!" but DeepMind is mostly research first. The entire point here isn't, "Hey can I pawn this guy?" It's why did limiting the input allow the human to so easily fool the machine? Because researchers aren't sure why the AI was so easily fooled where when it had a wider field of view, it could not be so easily fooled. That question has a lot more wider ranging implications than how great the micro game is for DeepMind.