Slashdot Mirror
Can DeepMind's AI Really Beat Human Starcraft II Champions? (arstechnica.com)
Google acquired DeepMind for $500 million in 2014, and its AI programs later beat the world's best player in Go, as well as the top AI chess programs. But when its AlphaStar system beat two top Starcraft II players -- was it cheating?
Long-time Slashdot reader AmiMoJo quotes BoingBoing: It claimed the AI was limited to what human players can physically do, putting its achievement in the realm of strategic analysis rather than finger twitchery. But there's a problem: it was often tracked clicking with superhuman speed and efficiency.
Aleksi Pietikainen writes "It is deeply unsatisfying to have prominent members of this research project make claims of human-like mechanical limitations when the agent is very obviously breaking them and winning its games specifically because it is demonstrating superhuman execution."
"It wasn't an entirely fair fight," argues Ars Technica, noting the limitations DeepMind placed on its AI "seem to imply that AlphaStar could take 50 actions in a single second or 15 actions per second for three seconds." And in addition, "This API may allow the software to glean more information... " After playing back some of AlphaZero's back-to-back 5-0 victories over StarCraft pros, the company staged a final live match between AlphaStar and [top Starcraft II player Grzegorz "MaNa"] Komincz. This match used a new version of AlphaStar with an important new limitation: it was forced to use a camera view that tried to simulate the limitations of the human StarCraft interface. The new interface only allowed AlphaStar to see a small portion of the battlefield at once, and it could only issue orders to units that were in its current field of view....
We don't know exactly why Komincz won this game after losing the previous five. It doesn't seem like the limitation of the camera view directly explains AlphaStar's inability to respond effectively to the drop attack from the Warp Prism. But a reasonable conjecture is that the limitations of the camera view degraded AlphaStar's performance across the board, preventing it from producing units quite as effectively or managing its troops with quite the same deadly precision in the opening minutes.
Long-time Slashdot reader AmiMoJo quotes BoingBoing: It claimed the AI was limited to what human players can physically do, putting its achievement in the realm of strategic analysis rather than finger twitchery. But there's a problem: it was often tracked clicking with superhuman speed and efficiency.
Aleksi Pietikainen writes "It is deeply unsatisfying to have prominent members of this research project make claims of human-like mechanical limitations when the agent is very obviously breaking them and winning its games specifically because it is demonstrating superhuman execution."
"It wasn't an entirely fair fight," argues Ars Technica, noting the limitations DeepMind placed on its AI "seem to imply that AlphaStar could take 50 actions in a single second or 15 actions per second for three seconds." And in addition, "This API may allow the software to glean more information... " After playing back some of AlphaZero's back-to-back 5-0 victories over StarCraft pros, the company staged a final live match between AlphaStar and [top Starcraft II player Grzegorz "MaNa"] Komincz. This match used a new version of AlphaStar with an important new limitation: it was forced to use a camera view that tried to simulate the limitations of the human StarCraft interface. The new interface only allowed AlphaStar to see a small portion of the battlefield at once, and it could only issue orders to units that were in its current field of view....
We don't know exactly why Komincz won this game after losing the previous five. It doesn't seem like the limitation of the camera view directly explains AlphaStar's inability to respond effectively to the drop attack from the Warp Prism. But a reasonable conjecture is that the limitations of the camera view degraded AlphaStar's performance across the board, preventing it from producing units quite as effectively or managing its troops with quite the same deadly precision in the opening minutes.
This is not really beating a human fairly. If you could click that fast then sure, but otherwise it's not a fair fight.
Just cruising through this digital world at 33 1/3 rpm...
When AlphaZero was pitted against Stockfish, the best chess AI, they set the match up with an outdated version of stockfish, bizarre time controls that removed stockfish's edge in time management (a static time per move was enforced), stockfish didn't get its opening books (a mini database containing information about the best moves to start with), nor did it get endgame tablebases (another mini database of information about moving at the end of games) and it was limited to a very small amount of ram (only 1GB when it should've had 64GB or more). Deepmind will CONTINUE to mislead people about what they've accomplished at every opportunity.
... to test AI. Since RTS games already have a bad UI where the bottneck is the human being in the chair, aka trying to control many units with a limited UI v ia keyboard and mouse is cumbersome at best. It was even back in the Warcraft 2 days when you tried to bloodlust ogres or heal paladins -- healing paladins being damn near impossible. While warcraft 3 'fixed' the issue with impossible casting /w large numbers of units using autocast.
The main problem being is that games like starcraft can be played perfectly because it's really an action game masquerading as a strategy game, aka the actions take place in real time. So for a computer like deepmind, the human appears super slow. Imagine if you ropponent appeared retarded in terms of their reflexes. That's basically deepmind vs any human opponent in an RTS. So a computers perfect information and perfect reflexes mean making 99% accurate micromanaging decisions for units everywhere at once.
You can't do that as a human player. Deepmind for an RTS is like having an aimbot in quake. Not really impressive since we already know making bots that can win against humans is trivially easy.
I've seen a couple of comments already where folks are talking about DeepMind being able to micro and click faster than a human. While that's neat, that's not entirely the goal here with DeepMind. The makers already put an artificial limit on the actions DeepMind can execute to 600/minute. In comparison the humans are executing at around 250-300 actions per minute. Now that indeed made DeepMind's micro game strong, what was the real tipping point was that DeepMind could see anywhere where a unit was located. Humans however can only see a "screen at a time". When DeepMind's makers went back and implemented "screen at a time" limitation, DeepMind was easily fooled again. And that's the thing here. Not the "can I beat a human?" but "can a human fool me?". As soon as the amount of information coming IN to DeepMind was reduced, the data coming OUT couldn't compensate and the humans were able to slowly figure out how to trick the AI into an unwinnable situation.
There's a continual fallacy on Slashdot where pure research like DeepMind is confused for "who's jerb can it take and reasons why it can't take that jerb." The media here is presenting in the terms of "Hey look! Something AI can do better than us worthless puny humans!" but DeepMind is mostly research first. The entire point here isn't, "Hey can I pawn this guy?" It's why did limiting the input allow the human to so easily fool the machine? Because researchers aren't sure why the AI was so easily fooled where when it had a wider field of view, it could not be so easily fooled. That question has a lot more wider ranging implications than how great the micro game is for DeepMind.
We don't know exactly why Komincz won this game after losing the previous five
You could know if you'd watch the games. In the first set, DeepMind won with inhumanly superior micro. It was really cool, but computers have been better at micro for a long time. Speed and precision are things computers are good at, that's why we have aimbots.
In the second set, the human readjusted, and thought of strategies that would defend against the superior micro (by building more powerful units), while taking advantage of the computer's weaknesses (poor knowledge of army compositions, weak knowledge of positioning, and seemingly no object permanence: once enemy units are out of view, it has no idea where they are or if they exist).
"First they came for the slanderers and i said nothing."
The difference between Jeopardy and the Starcraft AI is that the Starcraft AI wouldn't have come close to beating a human if it weren't for the inhuman precision.
The strategies the computer came up with were lousy (and very map-dependant), but it was able to compensate by having extremely precise movements.
"First they came for the slanderers and i said nothing."
Watch the pros play: how many of the 15 clicks per second are actually accurate.
Pros don't click 15 times per second, that's probably impossible physically. When they get an APM that high, it's by spamming on the keyboard in combination with clicks (something holding down a key to get the repeat, or spinning the mouse wheel).
"First they came for the slanderers and i said nothing."
That's like saying a fat runner is optimized to use a car. Stockfish isn't really optimized for opening books, it just sucks without them, mainly because the difference between a good and poor move in the opening may not manifest itself in a concrete eval difference far beyond the search horizon. As shown in some of the games, Stockfish doesn't care if its bishop gets trapped behind its own pawns. A bishop is still a bishop. It may get a penalty for limited mobility, but it doesn't get a penalty for being stuck for 40 moves. And the heuristic eval that Stockfish uses is just too simple to recognize these concepts.
Still I would conclude that Google's Deepmind only showed that it does not need an opening library.
I would say a neural network just combines the advantages of a database and of calculating moves ahead. The weighting of different connections in a neural network seems pretty equivalent to me to storing a library of good and bad starting moves.
Therefore it has pretty much a database function, and it is not surprising that it is superior to a software without one.
The big difference is that an opening book contains literal moves, whereas a neural net represents generalized patterns, similar to how a human grandmaster's brain has these patterns. If you give AlphaZero a position that's not in any of the games it played, it will still find appropriate patterns and use them to evaluate the position.
it is not surprising that it is superior to a software without one
If you take a weak engine with an opening book, then Stockfish is still going to be superior, because as soon as it plays a non-book move, the weaker engine is on its own. Even if the move was technically a mistake, it's unlikely that a weaker engine is going to be able to exploit it against Stockfish. The engine would actually have to recognize that the Stockfish move was bad, and understand how to exploit it.
For example, if Stockfish makes a bad move that potentially traps its bishop, the opponent needs to understand what moves to play to keep the bishop trapped, and why those are important. With specific patterns for trapped bishops, that's not going to happen.
I'd beat AlphaZero in chess even though I would play under the same parameters (Single core, 386 with 4MB of RAM)
Let me get this clear. You are arguing that your brain is roughly equivalent to a single core 386 ?