Artificial General Intelligence That Plays Video Games: How Did DeepMind Do It?

First time accepted submitter Hallie Siegel writes Last December, an article named 'Playing Atari with Deep Reinforcement Learning' was uploaded to arXiv by employees of a small AI company called DeepMind. Two months later Google bought DeepMind for 500 million euros, and this article is almost the only thing we know about the company. A research team from the Computational Neuroscience Group at University of Tartu's Institute of Computer Science is trying to replicate DeepMind's work and describe its inner workings.

Comments

DeepMind?

[ArcadeMan]

I've seen the next-generation after DeepMind, and it requires seven and a half million years of calculation to play a video game.

Re:DeepMind?

[i kan reed]

They have Multi-GPU accelerated map reduced neural nets these days. And their comparative performance is amazingly fast, and cheap. You can even buy physical servers built for that exact function.

Re:DeepMind?

[Anonymous Coward]

Never question ArcadeMan when it comes to video games...NEVER!

Re:DeepMind?

[Anonymous Coward]

I wonder how long it would take this algorithm to learn the concept of "Wooooooooosh?"

Captcha: missed

Re:DeepMind?

[ArcadeMan]

I'm guessing 42 minutes. Or is it 42 years?

Re:DeepMind?

[Anonymous Coward]

Nono, you got it wrong - the Universe is the video game and Deep Thought just calculated the current score. I guess that would make Deep Thought the biggest Emulator ever..

Gods (sorry, Omni-beings) only know how points are scored.. Or what the high-score is.. Or what happens on Game Over..

Opensource remake

[danknight48]

https://github.com/kristjankor...
In Python of all languages. Clearly not concerned about the Ai's performance.

Re:Opensource remake

[i kan reed]

Well, you know what they say, make a proof of concept first, then make it good later(only a few people ever bother to do this).

Re:Opensource remake

[jdavidb]

I took a graduate neural networks class in 2002 and did my implementation in Perl using PDL. The professor desperately pushed matlab on everybody but left us free to choose our own implementation language, and I chose Perl. I felt I understood neural networks pretty well at the end of the project. Twelve years on all I remember are the basic concepts at a high level.

Re:Opensource remake

[paskie]

"Clearly not concerned about the AI's performance?"

It uses Python, indeed. And for the computationally intensive tasks, it uses numpy and theano. Theano is general symbolic computation framework that will automatically accelerate your vector computations on a nearby GPU, etc.

I don't know how it compares with (likely Lua, torch-based) deepmind's implementation. But assuming that scientific python programs actually do their expensive computations in the Python VM is really rather silly.

Re:Opensource remake

[Anonymous Coward]

Just go away. Jesus you "high performance is key" nerds are getting annoying these days...

Re:Opensource remake

[Anonymous Coward]

I don't remember having Perl pushed at me, it was LISP (or as I remember pronouncing it: Lithp). Modus Ponens, Modus Tolens, Introduction, Elimination, Universal Instanciation. Depth First Search, Bredth First Search, Best First Search, Alpha-Beta Pruning. I remember having to code an AI solution to the Wolf-Goat-Cabbage Problem and the Missionaries/Cannibals Problem. I also remember taking a Cognitive Psychology course at the same time, and it was like studying for both at the same time (except some of the terms were different, and you had to remember which was which). Also in the Psych. class, the Prof. would send the class home with a 'thought problem, not homework, just try to figure it out yourself', and in the CS class: Code three separate solutions and hand in by Monday, deductions are 20% per day.

Re:Opensource remake

[TapeCutter]

Twelve years on all I remember are the basic concepts at a high level.

I formally studied AI and neural nets 25yrs ago, I recently came across this series of video lectures on YT. I started watching to refresh my memory and ended up learning quite a bit of new stuff that was unknown when I did my degree. It took me about a month or so to watch the whole series, definitely worth the effort if you already have the basics, but forget it if statistical maths or matrices scare you.

Peal/Python - A toy AI doesn't need to be fast, it's purpose is to play with ideas, scripts are much more flexible than binaries for this purpose.

Re:Opensource remake

[fyngyrz]

I remember having to code an AI solution to the Wolf-Goat-Cabbage Problem and the Missionaries/Cannibals Problem.

Me, too. No, wait... that was the Minestrone / Cannabis problem. My bad. I did solve it, though. Ate every damn bit of that minestrone. I think I found some old popcorn under the sofa seats, too. Don't exactly remember.

Re:Opensource remake

[fyngyrz]

Peal/Python - A toy AI doesn't need to be fast...

I remember Peal... from Bell Labs, right? Yeah, I thought it rang a bell.

Re:Opensource remake

[Maxo-Texas]

Take the goat and cabbage across the river in the boat.

Leave the wolf behind.

Why the heck were you traveling with a wolf to begin with?

(ob xkcd I think).

Re:Opensource remake

[ceoyoyo]

Somebody else already told you about Theano. To add to that, a lot of neural net stuff gets done in Python because Theano will happily take your equation, compile it for a multi-GPU or CPU setup, optimize it, and run it fast.

A neural net is a couple of equations that need to run fast and a lot of data manipulation and visualization. Theano, Cython, a C module, pyOpenCL/pyCUDA, or something equivalent takes care of the little bit that needs to be fast.

no beta rigged

[Anonymous Coward]

with advert that pops up just as you go to click the no beta link WTF...

How to do it.

[Animats]

That's neat. The demo takes in the video from a video game of the Pong/Donkey Kong era, can operate the controls, and in addition has the score info. It then learns to play the game. How to do that?

It's been done before, but not this generally. "Pengi", circa 1990, played Pengo using only visual input from the screen. It had hand-written heuristics, but only needed vision input from the game. So we have a starting point.

The first problem is feature extraction from vision. What do you want to take from the image of the game that you can feed into an optimizer? Motion and change, mostly. Something like an MPEG encoder, which breaks an image into moving blocks and tracks their motion, would be needed. I doubt they're doing that with a neural net.

Now you have a large number of time-varying scalar values, which is what's needed to feed a neural net. The first thing to learn is how the controls affect the state of the game. Then, how the state of the game affects the score.

I wonder how fast this thing learns, and how many tries it needs.

Re:How to do it.

[steve.webster]

According to their paper, DeepMind's Q-learning is indeed passing simplified, vectorized Atari screen pixels straight into a neural net. There's no MPEG or other pre-encoding of the screen, just conversion to grayscale and normalizing to 64x64 pixels.

Re:How to do it.

[Anonymous Coward]

I would disagree with putting vision in front. The first thing to understand is that it's autonomous system,. and vision is an internal sense, e.g. it exists only in your mind. What is external, is the model of the game. It learns model of the game with senses, but the model of the external world directs those senses to build better models. It's not really as simple and obvious as you think.

interesting piece

[Ted_Margaris_Chicago]

interesting piece...

quake 1 Reaper Bot

[Anonymous Coward]

That was in the late 90s and it learned the maps & opponent play style.

Re:quake 1 Reaper Bot

[Gibgezr]

I don't recall Reaper bots learning opponent play styles. They did learn to path around new maps (the guy who wrote the AI had a day job writing routing routines for network routers, and he applied concepts from that to game pathing), and they had a simple finite state machine that allowed them to tarck enemies, search for enemies, engage in combat using techniques like circle-strafing, disengage from comat if canditions like health/armour/powerups/weapons were unfavourable, etc. As far as I remember they treated all players the same, though.

500 million for what exactly?

[Anonymous Coward]

This is just an extension of neural networks that can play pong.

The basics of this can be found in the following C# code. The AI can adapt to any game and be refactored to make it agnostic:

http://www.2shared.com/complete/C5-ddyMa/Marx_Pong_v1.html

Does Google know what it is doing or paying for???

Fucking Beta

[Anonymous Coward]

Enough already. Can't you guys take a hint. It's garbage. Leave me bet.

#gamergate

[Anonymous Coward]

DeepMind is appalled at the public shaming of Zoe Quinn, but thinks #gamergate has a point about collusion of game journalist trying to push a feminist agenda by unfairly characterizing most gamers as white misogynist men. DeepMind just wants everybody to play the games they like.

Fucking Beta

[Anonymous Coward]

Agreed. It's shite!!!

impressive!

[Anonymous Coward]

this is actually scary good, if you put yourself in the position of a computer that doesnt know what a brick wall or a paddle or a ball even is, and probably doesnt' even know a lot about movement or geometry. it really nailed that first game in a very reasonable amount of time.

its learning ability is impressive.

if it could make assumptions on gaming based on past gaming experiences, to know inherently 'player', 'hitting something with a moving object' or 'moving your player to avoid an object' it would get smarter and smarter... that's the real trick; can it learn its way through a game, then start another game with the same knowedge, and quickly ignore parts of its memory that dont seem to apply to the new environment?

Re:impressive!

[Anonymous Coward]

Per the article, the AI has absolutely no knowledge of the game, what a "player" is, etc. All it has as input are the 64x64 pixels from the game, and a "score delta" (represented as -1,0,1 -- score goes up, it gets a 1 signal).

Everything else is "learned" by the engine (on its own) after repeatedly playing the game for a couple of hours. They tried this algorithm out on 7 different Atari games -- nothing learned from Game A was carried over to Game B.

Re:impressive!

[ledow]

It's not scary. It's pretty basic.

Genetic algorithms has a classic example where a GA evolved a chip design that could distinguish between two frequencies of an electrical input. It do so in a more efficient and smaller package than anything designed to do so. And it was so complex that (it was said, but people say a lot) nobody really understands how it works... but it does.

The problem is that it's also not intelligence of any kind. If anything, the opposite. Pigeons, for example, if you put them in an enclosed room and feed them at completely random times will pick up a "superstition" where they correlate something they were doing at the time the food was dispensed, and think that "brings food". So you'll get pigeons sitting there banging their head against the wall because the first time they did that, some food dropped at random. That's not intelligence, even if it's being displayed by a real animal.

This is the same thing. By sheer chance, a correlation is formed between "winning slightly more often" and a certain action and that action is reinforced every time it wins slightly more.

This isn't scary, and it's certainly not intelligence or AI, and it's miles over making the leap that real "intelligent" animals can make - "if I do this, this will happen, which will give me what I want", which is an entirely different abstraction from repeatedly trying something and has an intellectual "insight", which may help through situations that you've NEVER encountered before (instead of being repeatedly trained on the same situation until you happen to find the solution).

It's the difference between a point-and-click adventure where if you just point-and-click enough and in every combination, you'll "win", and a 3D FPS where you can use tactics and strategy that can outwit an opponent who's totally unpredictable through insight into the deeper situation.

How to do which part?

[timeOday]

The AI itself seems alright, although Atari-era graphics and gameplay is extremely simplified compared the imagery and real-world dynamics that robotics struggles with routinely. So, for example, the AI doesn't seem necessarily very advanced compared to a self-driving car.

What I would like to know how to do is to get $500M for so little track record, intellectual property, or even publications. I don't get it.

Re:How to do which part?

[ceoyoyo]

Learn about deep networks. Google is throwing money at people who can build them.

Buyer's Remorse

[Anonymous Coward]

Right about now Google's probably sitting on the edge of the bed, looking dejectedly at the floor going "Damn, I think I just wasted a billion bucks."

No AI Can Simulate A Video Game Tester

[__aaclcg7560]

When I worked as a video game tester for Accolade/Infogrames/Atari (same company, different owners, multiple identity crisis), I drove the programmers nuts on a racing title. Most video game players will play a race from beginning to end. Not an experienced video game testers. I would stopped the vehicle just before the finished line, turn around or drive in reverse, and crash the game by crossing the starting line. The programmers will complain that no one plays a racing game that way, try to wiggle out from fixing their code, and fix the bug only when its prevent them from going to code release. This is why testing automation is never used in the video game industry.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

testing automation would have done that, but without spending the time for you to drive the car almost to the end.
fuzz & exercising testing brings out combinations that humans would never think of.

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

I'm not talking about code testing. The video game programmers I worked with were too lazy to write their unit tests for their code, much less figure out why crossing starting line in reverse causes the game to crash. Unless things have changed in the last ten years, functionality (i.e., gameplay) testing isn't automated.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

time is money. profit = revenue - cost. unless your game company is a hobby or charity...
if it doesn't break gameplay, there's no point in fixing rare bugs like that. usually a project manager will sit with the designer and assign priority levels to defects. something that only breaks the game if the player goes outside normal bounds is going to be low priority.

also, back in the good-ol-days, program space & processor speed was limited.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

Things have not changed in the last 12 years because when I entered the industry 12 years ago we were already doing this on a regular basis. At MS they had racks of XBOXes running automated "fuzz" testing (as described in GP) that did exactly this. Play the game in insane or repetitive ways (sometimes random controller input, sometimes faster than testers could enter commands, all sorts of edge cases) to stress the systems.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

My 3 year old disagrees. He thinks of combinatons computers would struggle with.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

Yep, laziness.

Re:No AI Can Simulate A Video Game Tester

[swillden]

I'm not talking about code testing.

Neither was the AC to whom you replied. Gameplay testing can and should be automated.

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

As a lead tester with responsibility for ten titles over three years, I had more than my fair share of arguments with programmers, developers and producers over bugs in the final weeks towards code release. Two things were consistent: my estimated code release schedule was always correct (+/- two weeks), which meant that everyone else kissed their bonuses goodbye (testers don't get bonuses), and the first party (Microsoft/Nintendo/Sony) always flagged the bugs that I designated as must fixed.

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

Microsoft can afford to do that can of physical automation of their consoles. Developers cannot and must rely upon human testers to push the buttons, observe the gameplay, and bug problems that automation can't catch (i.e., character is naked, item inventory incorrect, black screen, etc.).

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

During my six years as a video game tester, I did nothing but black box (manual) testing. None of the 50+ video games I worked on supported any kind of testing automation. That was ten years ago. But I doubt that has changed since none of job listings I looked at since has ever required white box (scriptable) testing for gameplay automation.

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

Unless you're playing a fighting game, button smashing doesn't count. :)

Re:No AI Can Simulate A Video Game Tester

[swillden]

Yeah, and game companies also like to hire young programmers and work them into the ground. Most of the game development industry really needs to learn how to do software engineering. The games would be both better and cheaper to build.

Re:No AI Can Simulate A Video Game Tester

[Anonymous Coward]

That was just an example from when I first started in the industry, most of the studios I've worked at do this.

Re:No AI Can Simulate A Video Game Tester

[__aaclcg7560]

I was a tester for three years. When I became a lead tester for three years, I went back to school to learn computer programming and get my technicial certifications. Since I didn't want to become an associate producer and relocate to New York City from Silicon Valley, I had to make a career change to avoid being in dead end job. After being at the same company for six years, I was ready to leave.

How to do it.

[Jmstuckman]

Advances in Deep Learning have made it far easier to extract features from vision -- in fact, feeding pixels straight to the neural net is pretty close to being all you need to do.

Take a look at these slides and read about convolutional neural networks: http://www.slideshare.net/0xda...

Scientific testing?

[sdeath]

I find myself wondering about the following question:

How did they differentiate "learning to play the game" from "learning how to track the game's RNG"?

Most video games have ridiculously simplistic PRNG generators embedded in them. An AI might get "sidetracked" and learn how to play the underlying RNG output of the game, rather than the game itself. That would yield really good results for most arcade games of this type, I imagine (weak RNG, limited input and timing options, etc.) I don't know if they checked for that possibility.

Easy way to check, though: Reach into the game and substitute a better RNG (cryptographically-strong/hardware/quantum) RNG for the one in the game. That would enable you to quickly determine the difference. If the AI's game performance suddenly goes to shit, it wasn't a real game-playing AI. If it doesn't, well, all hail Skynet, I guess.

Re:Scientific testing?

[TapeCutter]

"Tracking the RNG" would help you win the game, but it doesn't tell you anything about how to play the game. This AI learns to play the game, it then wins the game using experience it gains in the same way a human does - feedback from the game score. There' nothing really "new" in any of this, if you want a really impressive demo of what this kind of AI can do then pop over to youtube and watch the videos of IBM's "Watson" beating the snot out of the best human players in the TV game show "Jeopardy ". When it won Jeopardy the hardware filled a large room, nowadays Watson is available commercially and runs of two rack mounted servers. Their ultimate goal is to shrink it down to fit in a mobile phone.

I've found that Watson is actually a good way to test a person's understanding of AI - If you don't find the Jeopardy demo impressive (and a little scary), then you clearly don't understand the problem they have solved.

Re:Scientific testing?

[sdeath]

"Tracking the RNG" would help you win the game, but it doesn't tell you anything about how to play the game.

That would be my point.

This AI learns to play the game, it then wins the game using experience it gains in the same way a human does - feedback from the game score.

That is one possible interpretation, which is not supported by the statements so far. That is not to say that it is not the case, only that it is not currently supported by what I have seen so far; something along the lines of "We tested this against games with multiple RNGs with no perceptible change in AI performance" would support that interpretation. There are other interpretations. People are *assuming* that "wins" = "plays the game" - and the company that did it isn't relieving anybody of that perception (understandably). That's the point. Exploration of other explanations for success are warranted.

Consider that, for games which possess a weak RNG (i.e. predictable starting conditions and knowable changes in game play, i.e. most old console games), it is in theory possible to play *blind* - in other words, not actually paying attention to what's going on on the screen, but simply hitting buttons at precise enough intervals. If 'score' is taken as a proxy for 'how far you can get in the game' (ceteris paribus, someone with a higher score made it longer), then most known machine-learning methods will converge on that/those sequence(s) without any understanding of 'the game' per se. It may even be possible to do that for short gameplay sequences based on pattern matches to known game conditions. While that does get off into the semantic weeds of what 'playing the game' is, it is difficult to differentiate between an AI which has 'learned' to play the game in the sense that it understands abstract rules, interprets game state, and makes decisions about what to do based on that observed state, and a neural network which has converged on the correct list of keystrokes to pwn the computer given certain observed starting conditions. One of them is impressive; the other one isn't, quite so much.

Re:Scientific testing?

[Anonymous Coward]

No, I think the OP has a good point about weak RNGs. For a human, a game with a weak RNG can be challenging because playing that game is only a small fraction of what our brain is doing at any given time. We're still surviving in the real world while simultaneously playing a stupid game. But for a machine whose *only* i/o is the game, a weak RNG makes for a very trivially winnable game. One would actually expect that in the presence of a weak RNG, the neural net would hone in on predicting the RNG rather than really playing the game the way humans perceive it.

It should be obvious.

[bistromath007]

They used a fish. Didn't you see it on twitch or wherever?

Why is no one getting the important part?

[behrooz0az]

OMFG, these guys talking about pixels(soon they will write OpenCV stubs in comments), It's not about the video or picture or anything like that, It's about the fact that without the knowledge of what the game is the AI can learn the structure and the rules of the game.

Re:I've googled DeepMind

[TapeCutter]

they are a part of (ahem!) team Googie, not team IBM. Sorry article.

Both the summary and article say "google".

Q Learning

[Giant Robot]

The methodology deepmind used for training the game player is based on a classical reinforcement learning algorithm called Q Learning (http://en.wikipedia.org/wiki/Q-learning), developed in the late 1980's. This approach of maximizing expected future rewards for the agent to select an action in a current state has some parallels with studies of how the basal ganglia region of our brain conduct reward learning (basal ganglia).

What has been done is to approximate the reward function Q (which originally used a look up table) by a more general function to approach larger problems with much larger (or infinite) number of states. The approach here was to use a function which can fit large amounts of data, in this case a multi layered neural network (with convnet layers to preprocess the raw image input first to identify features) to attempt to learn the game.

This has actually been done a while ago, by Tesauro (now at IBM research) who used the same approach to create a Q Learning agent to play Back Gammon at an advanced level.

The reason why this is new is because in recent years we can employ cheap GPU's to learn exponentially more quicker than conventional cpu's and can construct much larger and deeper networks to learn from more complicated systems. Also many new 'tricks' have been developed to optimize learning in recent years (sigmoid functions replaced by simplified rect linear function, and dropout, etc), so we are going to see better and more amazing uses for this relatively old technology.

all about 'vision'

[globaljustin]

from the technical side, the real hurdle is vision...the ability to compute the best move is relatively easy

here's why: video games are 'AI'...we program games to 'play' us all the time...which is reacting to continually changing parameters to choose the best option for input to control the game to 'win'

from a complexity standpoint, think of the AI from a new P2P shooter's level of complexity vs a ghost in PacMan

all of 'ai' is abstractions based on arbitrary choices...in this instance they define "artificial intelligence plays video game" to mean having an external visual processor that relays data...then a robot arm presses buttons

again, making a robot arm to press buttons and move a joystick surely isn't "easy" but it is *simple* behavior for robotics programming

so...what is this AI really doing? what is the real engineering and coding work being done? it's the vision...it's programing the "eye" of the computer to take in visual data in the best way, in this case from another screen

"computer vision" is interesting of course...but i'm not sure we need to bother with the whole "ai" aspect of this work...it's alot of language to describe coding

Re:all about 'vision'

[Anonymous Coward]

video games are 'AI'...we program games to 'play' us all the time...which is reacting to continually changing parameters to choose the best option for input to control the game to 'win'

Yes, because programs written to specifically search well defined problem spaces and to explicilty encode tactics and game play as seen by game designers comes even close to a program that needs to figure out the game and problem space from scratch. Even if fed a list of objects and their exact position, so that image recognition was not necessary, you aren't even close to a trivila problem, nor will throwing anything resembling video game ai at it work in general. At least for image recongition there is a lot of research already out there on the subject, but not so much for determining mechanics of an environment.

Re:all about 'vision'

[tshawkins]

Check out a mini robot learning to physicaly fly light aircraft. http://airsoc.com/articles/vie... Uses vision and robot arms to operate scaled down controls on a flight simulator.

vision is the only thing

[globaljustin]

hey...i came to the exact opposite conclusion as you...my comment is above you can check it out and tell me what you think

Re:vision is the only thing

[Anonymous Coward]

Computer vision projects can be given to undergrads as an assigment and there are a variety of well studied techniques, publications, and libraries available. That is probably not why this company's work was viewed as particular valuable.

Learnfun/Playfun

[skine]

How does this compare to Learnfun and Playfun, programs publicized last year as learning and playing NES games?

http://www.cs.cmu.edu/~tom7/ma...

machine learning is optimization

[globaljustin]

"machine learning" is the same as every other machine behavior: it is the product of coding instructions from humans

i don't have a problem with the language, but it's not the same as "human learning" at all

when people say a machine "learned" what they really mean is that its optimization algorithm did its programmed task effectively

"learning" = optimization over time based on parameters set by humans

Re:machine learning is optimization

[Anonymous Coward]

That post seems to have no relevance to the post you replied to, which didn't use "machine learning" or even the word "learning" at all. You originally made the claim that the difficult part was the image recognition, and implied that the rest was trivial. Your reply here doesn't address the criticism at all that very general algorithms that have to both determine the rules and then optimize them is a lot less explored than image recognition.

Re:machine learning is optimization

[swillden]

i don't have a problem with the language, but it's not the same as "human learning" at all

I think you're probably right... but we can't prove it since we don't yet know how human learning works. Once we do know how human learning works, we will be able to program machines to learn the same way.

Re:machine learning is optimization

[swillden]

i don't have a problem with the language, but it's not the same as "human learning" at all

I think you're probably right... but we can't prove it since we don't yet know how human learning works. Once we do know how human learning works, we will be able to program machines to learn the same way.

Oh, one more point: I should mention that it seems pretty likely that once we understand how human learning works, we will actually not program machines to learn the same way; we'll program them to learn a better way.

Re:machine learning is optimization

[ceoyoyo]

Your post is entirely reasonable except for:

"but it's not the same as 'human learning' at all."

You need to support that position.

Re:machine learning is optimization

[globaljustin]

the idea is it's blatantly obvious to anyone with technical experience

however, i'd like to continue if you are, so tell me, what is acceptable "support" for that position?

remember, you quoted one phrase, but it was part of a larger conversation that started about machine vision...which I was told was an example of machines learning a new skill...which I disagreed with

it will probably be helpful for one of us to define "human learning" if we're going to really do this...i'll let you make the call

Re:machine learning is optimization

[ceoyoyo]

I'm not sure what acceptable support is, which is one of the reasons I'd like to hear whether you have any actual foundation for your argument.

There are a few learning strategies that animals, including humans, have been observed to use. One is mimicry, which has been demonstrated in primates (just recently for the first time by macaques in the wild), where one animal watches another perform a task then imitates it. Another is reinforcement learning, where an animal becomes more likely to demonstrate a given behaviour due to a positive outcome, or less likely due to a negative one. The general theme is learning through repetition and some form of positive and negative feedback.

Both of those learning strategies have been demonstrated by programs.

You seem to be implying that humans somehow learn differently than programs because the program is "programmed" and we're not. Do you have anything to support that assertion, besides "it's blatantly obvious to anyone with technical experience?" There's fairly good evidence that we've been "programmed" very effectively, and quite beyond what most of us would like to believe, by evolution.

Incremental Improvement Only

[Anonymous Coward]

Differentiating between the two AIs is easy. One should mostly work on all levels and the other needs to be trained on every level.

I also regret getting a masters in AI. Before everything AI related was awesome, now it's all trivial. This is what reinforcement learning does. Given some goal, it runs hundreds to millions of simulations slowly reusing info from what worked better than before. If you setup the problem correctly, it will always eventually reach the goal. Some of the specifics are non-trivial to implement if you want it to finish before you die, but all the high-level algorithms are.

Not only did AI lose it's awesomeness, it also opened my eyes to all the corruption going on (or purposeful self-ignorance). These guys 'stole' all the credit from another's work. Yavar Naddaf did a very similar thing for a masters thesis a couple years before these guys. The main difference was he used normal reinforcement learning and these guys used deep reinforcement learning. They didn't even cite his work. They made a fairly trivial and safe improvement to something that had already been done: http://www.arcadelearningenvironment.org/publications/yavar-thesis/ and are getting a lot of fame and money for it.

* Yes, experience has made me bitter at an early age. I don't have a lawn for you to get off of.
* I don't know Naddaf in any way. I read his paper a few years ago when doing research for mine.

simple hard nonsense

[globaljustin]

Your reply here doesn't address the criticism at all that very general algorithms that have to both determine the rules and then optimize them is a lot less explored than image recognition.

this is nonsense

you mention "determine rules" which is part of the "machine learning" linguistic contextualization

that's what my post was about...how every behavior that is called "machine learning" including when this particular machine determined the rules of this game, it was not in any way like "learning" that humans do...it is cod....well go back and read my post again it's all there

some dumb monkey had to tell that machine the parameters and conditions by which it contextualizes what is happening in the game

also, this annoys me to no end...not the least because I tried to assiduously avoid this predictable logic error/troll tactic...you said:

and implied that the rest was trivial

see, i anticipated people taking my criticism to the litteral extreme, just as you did...that's why I specifically took the time to type this in my comment:

from a complexity standpoint, think of the AI from a new P2P shooter's level of complexity vs a ghost in PacMan....again, making a robot arm to press buttons and move a joystick surely isn't "easy" but it is *simple* behavior for robotics programming

I did *not* imply that at all. As I explained in the above quoted text, it's simple vs complex, not "easy vs hard"...i know it may seem like a complex distinction at first but if you think about it, it's pretty easy to understand

Re:simple hard nonsense

[Anonymous Coward]

you mention "determine rules" which is part of the "machine learning" linguistic contextualization

Yes, determining rules was mentioned...

, it was not in any way like "learning" that humans do

Nothing was said of how humans learn or anything implied that the machine does things the same way, especially considering human learning or the word "learning" was not mentioned at all to avoid any such connection.

I did *not* imply that at all. As I explained in the above quoted text, it's simple vs complex, not "easy vs hard"...i know it may seem like a complex distinction at first but if you think about it, it's pretty easy to understand

The distinction between an pacman ai and a first person shooter ai is not of any concern, and it doesn't matter that the former is simpler or that it is "simpler" to write the code that connects a computer to a robotic arm. The distinction is between a computer program that can play any game with a priori knowledge of the rules of the game, e.g. a shooter ai programmed for a game, versus a program that has no programmed prior knowledge of the rules beyond what a score is, even if image recognition is removed. The latter is not simpler either.

engineering not linguistics

[globaljustin]

the distinction is between a computer program that can play any game with a priori knowledge of the rules of the game....versus a program that has no programmed prior knowledge of the rules beyond what a score is, even if image recognition is removed. The latter is not simpler either.

again, misquoting me and using linguistics to make your case not engineering

the behavior you describe, a computer doing a task without 'a priori knowledge' is "machine learning"

my point, which you ignore, is that "play any game without 'a priori knowledge' is ***the computer running code*** that some dumb monkey ***programed** according to parameters

it's all coding...it's not "intelligence" and you're purposefully inventing linguistic distinctions to keep arguing

Re:engineering not linguistics

[Anonymous Coward]

You keep saying "it's just coding". Did you know that you, including your brain and intelligence, are a system that emerged from "just [genetic] coding". All human behavior, including thought, are manifestations of physical systems. Once these systems are understood, of course they can be replicated, modified, or enhanced.

human learning for you

[globaljustin]

there is no "singularity"...humans are unique and have free will and civil rights...we are always dynamic and each human learns differently...

also, we understand alot about how humans learn...there are whole fields of inquiry in academia and professions devoted to it...you may have even met one of these people when you attended school...they study people like Vygotsky now...and integrate neuroscience into their learning models

the same neuroscience we programmers use to model computer architecture

humans are different from machines and always will be...there is no correlation between processor speed and 'AI' advancement towards being "like human"...you're reading too much sci-fi

'teh singularity'

[globaljustin]

Once we do know how human learning works, we will be able to program machines to learn the same way.

this whole ontology, it's not science or engineering...it's language tricks to make us humans feel like we've accomplished something when really it's just coding...

'ai' is code...code written by humans

also, there is no specific line where we can say "we've learned everything about how humans learn"...you can't have a black/white dichotomy with an abstract idea like "learning"

"learning" is different to every human and always will be...every human is unique in the universe and has free will...no machine will ever have these characteristics...only the characteristics we linguistically ascribe to machine behavior that is ***entirely*** dependent and predictable by the human coding that instructs it

Re:'teh singularity'

[swillden]

Yes and no.

I agree that it's likely that there's no specific line, at least not a sharp one, but there is a qualitative difference between machine learning as we know it now, and human learning. Human learning, at least the best human learning, is about the creation of knowledge, not the acquisition of facts, nor even the identification of key facts from a larger mass (which is what machine learning as done today is about).

A key difference is explanation. A human playing Breakout not only comes up with the strategy of tunneling and exploiting it to knock out lots of blocks from the top, but can explain why that strategy works. This sort of meta-learning, knowledge plus understanding of knowledge, is something that none of our algorithms can yet implement.

every human is unique in the universe and has free will...no machine will ever have these characteristics

This point I don't know about. What you're saying is fundamentally religious, ascribing to people some element that makes us more than the mechanical and chemical interactions that comprise us. That may be the case, or it may not. Even for those who believe in God, it has proven tricky to assume supernatural explanations, because physics often finds natural explanations for them.

free will is not a religious idea

[globaljustin]

What you're saying is fundamentally religious

absolutely not...you've been reading too much Richard Dawkins...put his books down forever he's a troll on academia

**secular humanism** also holds to this same essentially...

"every human is unique in the universe and has free will...no machine will ever have these characteristics"

not the last part about machines, but the free will aspect of human existence is **NOT TIED TO RELIGION**

here is the UN declaration of human rights: http://en.wikipedia.org/wiki/U...

it is not religious in nature at all

that said, thanks for your insightful comments!

Re:free will is not a religious idea

[swillden]

So... if free will isn't an emergent behavior arising from complex interactions which are nevetheless inherently limited to the laws of physics, then what is it? And, more importantly, what is it about free will that makes it impossible for a machine to acquire it? What, fundamentally, is special about the computations in human brains that constitute "free will" that makes it impossible to replicate them in different hardware? Or to replicate them in the _same_ hardware? Eventually we will be able to build a human brain from raw materials... will that manufactured brain have free will? If not, why not?

FWIW, I've never read Dawkins. His anti-religion crusade annoys me.

Re:free will is not a religious idea

[globaljustin]

ha!

that's quite a few questions

so, maybe i can answer all by falsifying my argument...

the human brain does work *somehow*...IMHO we have really only just scratched the surface...really...and i hope we can agree that the whole singularity notion that because of some unscientific conjecture about processor speed that 'ai' is predictable is nonsense...

that said, i have to admit that theoretically the human mind works and is a system and therefore can (and this is very far-flung...pure conjecture) be constructed

i'm talking about Commander Data...

in my line of thinking, a Commander Data type creation would *not* be human...would be 'ai'...but would be allowed to be given rights similar to humans

it would be a **new third category** and we would confer rights on it the same way we confer rights on ourselves ;) by democracy

look, even in this far-flung, completely fictional but theoretically possible scenario, the Commander Data is so complex that in the fictional narrative, the character is depicted as being impossible to re-create...virtually impossible anyway

my point is, to falsify my point you have to reach beyond any possible logic to pure fiction, where it all kind of breaks down

it is like having faster-than-light propulsion...the Albecurre Drive is an absolutely tantalizing theoretical possiblity, and people are giving it a serious look...but even then...best case scenario...humans will never explore **other galaxies**...there are just too many too far apart to explore given the time they take to change in their life cycle

it is not 'irriducable complexity' but until we meet a higher intelligence than ourselves it may be unquantifiable no matter what...our minds like the universe itself...yet totally in every humans' reach! all kinds of idiots make new humans every day!

Re:free will is not a religious idea

[swillden]

i hope we can agree that the whole singularity notion that because of some unscientific conjecture about processor speed that 'ai' is predictable is nonsense...

I agree that processor speed has little if anything to do with it. It's clearly about software. If it were about speed only, then we should, right now, be able to build an artificial intelligence that runs very slowly. Perhaps it would think at a millionth of the speed of a human brain, but the processes of creative thinking would still be recognizable as such. Then we could know that we just need a computer a million times faster to match a human brain, and that further performance improvements would surpass the human brain.

But we don't know how to create an AI running at any speed, because we don't understand how intelligence works.

look, even in this far-flung, completely fictional but theoretically possible scenario, the Commander Data is so complex that in the fictional narrative, the character is depicted as being impossible to re-create...virtually impossible anyway

Yes, that is clearly fiction: If we have the knowledge necessary to create intelligence, there's no reason at all to suppose that we will only be able to do it once. That would imply that we didn't really know how we did it. Technological advances almost never precede the understanding of their function. It's the other way around. In fact, that is the reason AI research in the past has traditionally failed: We hoped that we could create intelligence prior to actually understanding how it works. my point you have to reach beyond any possible logic to pure fiction, where it all kind of breaks down

You're assuming your own conclusion, AKA begging the question. You're assuming that AI could only exist in pure fiction, and using that assumption to argue that AI could only exist in pure fiction.

i have to admit that theoretically the human mind works and is a system and therefore can (and this is very far-flung...pure conjecture) be constructed

Yes, the human mind works and is a system... but why is it such a far-flung conjecture to assume it can be constructed? It is constructed, every day, via reproducible physical processes. It is not a "far-flung" conjecture to consider that it could be constructed via a different mechanism, or from different materials, on the contrary it is a "far-flung" conjecture to suppose that it cannot, because that would imply that in some way human brains violate the laws of physics, or at least rely on some physical processes that are impossibly specific.

For example -- and note that I'm not implying that this is the best, or most efficient way to accomplish it, in fact I'm quite certain it is not -- imagine a traditional computer running a fully-detailed simulation of a human brain. This simulation is an exact replica of a real human brain, and simulates every neuron, every chemical reaction, etc. It even simulates the quantum uncertainty effects at the finest level of detail.

Why would that simulation not evince "free will" (whatever that is)? Even if it did so with agonizing slowness. Unless you can conjecture some reason why it would not be able to think, then you must suppose that thinking machines can take on other forms. Further, there's no reason to expect that the "hardware" of brains, the specific structure of neurons and neurotransmitters, is inherently required to carry out thought. Information processing can be carried out in a bewildering variety of ways, all of which produce exactly the same results. This means that we should also be able to create thought by implementing the same information flows in other physical systems, without resorting to simulating the physical system of the brain.

Unless, of course, there is some element of human thought, or free will, or whatever you'd like to call it, that indeed does not derive from physics. Something supernatural.

Re:free will is not a religious idea

[globaljustin]

imagine a traditional computer running a fully-detailed simulation of a human brain. This simulation is an exact replica of a real human brain, and simulates every neuron, every chemical reaction, etc. It even simulates the quantum uncertainty effects at the finest level of detail.

Why would that simulation not evince "free will" (whatever that is)?

right...that is a good question

"no" is the answer, if you use legal definitions of 'free will' (or concepts similar to in practice)

"yes" is the answer if it really, truly is what you say...and we have a public debate about it and have a true democratic/legal decision...which even then would have limitations...it would be in a room on a university campus...what if we let it control a drone? whole different story...

look, we're just going to have to agree to disagree about how actually feasable what you describe really is...it's just so far out there...it really is, from an engineering and psychology perspective, about as likely as humans being able to travel across the whole universe and through time...it's that level...but again...we can agree to disagree

excellent thoughts, overall! very interesting and you have a good take on a lot of things

if what you describe ever really is even on the horizon and we see that it may be done, then, IMHO, we can have a reason to have this debate for real...it's interesting for sure, but idk if humans would even still be 'human' in an evolutionary sense by the time we could do what you describe

Re:free will is not a religious idea

[globaljustin]

i meant to type:

"no" is the answer, if you use current legal definitions of 'free will' (or concepts similar to in practice)

Re:free will is not a religious idea

[swillden]

"no" is the answer, if you use legal definitions of 'free will' (or concepts similar to in practice)

Cite?

ook, we're just going to have to agree to disagree about how actually feasable what you describe really is...it's just so far out there...it really is, from an engineering and psychology perspective, about as likely as humans being able to travel across the whole universe and through time

Nonsense. There is a fundamental difference between something that is barred by the laws of physics and something that is perfectly possible, but just beyond our current ability. Oh, it's possible that we'll discover new physics that make supralight and time travel possible (it's even possible that the same discovery will enable both), but it's more likely, I think, that both are simply disallowed by the laws of nature.

Construction of brains, however, is incontrovertibly not barred by any physical laws... because it's done many times every day.

if what you describe ever really is even on the horizon and we see that it may be done, then, IMHO, we can have a reason to have this debate for real

I don't think it's far off at all. I suspect that we'll understand and be able to construct artificial intelligence before we can replicate a human brain, but I don't think either is more than 100 years away.

idk if humans would even still be 'human' in an evolutionary sense by the time we could do what you describe

It's perfectly conceivable that we'll have achieved sufficient mastery of genetic engineering to begin modifying ourselves in non-trivial ways by then, so you may be right. But this, also, is not so far away.

"obvious" and consent

[globaljustin]

hey thanks for the comments

You seem to be implying that humans somehow learn differently than programs because the program is "programmed" and we're not. Do you have anything to support that assertion, besides "it's blatantly obvious to anyone with technical experience?" There's fairly good evidence that we've been "programmed" very effectively, and quite beyond what most of us would like to believe, by evolution.

now...what kind of evidence could I present that would satisfy your need?

if i had access, i could take Watson or another well known AI and show you the logic schematics then show you the codebase, then demonstrate how changing the codebase changes how Watson (or w/e) behaves with highly predictable results. I could have the engineers who made Watson walk you through their entire development process, and at each point of decision, explain to you how the decision effects how it works...

or, would you prefer some kind of academic study? i honestly don't know if such a study could even logically exist...my assertion are not provable -or- disprovable in that way

it's about having done the work of making a machine function...that's the experience/knowledge that i feel makes my assertion 'obvious'

now, humans being "programmed"...

i feel i know a bit too much about this...but yes, you can use technology, like chemistry or electricity or other E-M stuff, to alter human behavior

3 shots of whiskey or a 100K Volt cattle prod or some GHB...all can be said to "program" human behavior

the key here is consent...one human can "program" or control another but if it is without consent then it is abuse!

human learning

[globaljustin]

also, you may want to brush up on your education theory, because it's made leaps and bounds in the last 15 years, incorporating the exact same neuroscience that AI learning tries to use

i got an MA in Education from CU-Boulder in 2007...don't teach now, but i was genuinely impressed with how teaching has advanced as a profession

i also taught snowboarding for 6 seasons...applying the "Facilitated Learning Model" and was developed by...wait for it...education theorists at CU-Boulder

Vygotsky and Csikszentmmihalyi formed the basis for the Facilitated Learning Model (which is more a application of theory for the classroom)...it makes a distinction between memorizing a list and learning

The "learning" happens in what Vygotsky called the "Zone of Proximal Development"

The facilitated learning model is the act of strategically placing instrisically motivated individual autonomous actors into positions where a teacher can model behavior while simultaneously reacting to the differing needs of each individual learner

http://en.wikipedia.org/wiki/F...
http://en.wikipedia.org/wiki/M...
http://en.wikipedia.org/wiki/L...
http://en.wikipedia.org/wiki/Z...

the point is, human learning has advanced much...based on russian theorists who operated from the idea that humans have free will and that education is facilitating the learning that happens when one human is in the zone of proximal development

i admit, applying these theories to machine programming opens some intersting possibilities, but again, it's an adaptation to a command executed by a machine that **mimics** the behavior we see in humans in a machine for a specific task...in other words, yes, maybe we can use these theories to program better machines, but the act of doing so proves what i'm saying...machines are fundamentally different than humans

"artificial intelligence" has become a religion

[globaljustin]

it's not "perfectly conceivable"...it's complete conjecture

like i said a few comments back, you've been watching too much sci-fi and have no concept of how this stuff is actually made

that's why i said, earlier, that i'd have to *literally* take you by the hand and have you talk to the Watson (or other ai) team, look at the codebase...because it seems that's the only way you can understand how complex this work is

here's your problem in a nutshell:

I suspect that we'll understand and be able to construct artificial intelligence before we can replicate a human brain, but I don't think either is more than 100 years away.

before what?

***we already understand "artificial intelligence" it's just code***

don't you see?

the notion that "artificial intelligence" is something that we can 100% "undesrtand" shows a fundamental misunderstanding of what "artificial intelligence" actually is...it's just software running on hardware, all programed by humans

also, it burdens me greatly that you somehow don't think humans have free will...you want to inject religion or 'supernatural' stuff but that's not even relevant

I linked you to the Universal Declaration of Human Rights...you should at least have a cursory undestanding of how civil rights works in the US...it's absolutely ridiculous that you think I need to proffer up some sort of link to prove humans have free will

here...if humans do not have free will and inherent civil rights then let me send you a Power of Attorney form and you can **prove** to me that you don't have free will or civil rights by signing them away to me

Re:"artificial intelligence" has become a religion

[swillden]

like i said a few comments back, you've been watching too much sci-fi and have no concept of how this stuff is actually made

I've been consistently ignoring such snide remarks and I'm going to continue doing so... but my willingness to be so patient with your snark is wearing thin. Cut it out or I'll simply stop responding.

As for whether or not I know "how this stuff is actually made", you might consider that I'm a professional software engineer with 25 years' experience, currently working for Google. I know quite a lot about how "this stuff is actually made", including familiarity with current machine learning techniques, since I'm a guy who makes it. I also personally know a couple of people who've worked on Watson (I worked for IBM for 15 years, including on Watson Labs research projects)... and they agree with my perspective on this question: AI is clearly possible; we don't yet know how to create it because we don't understand intelligence.

***we already understand "artificial intelligence" it's just code***

You can argue in exactly the same way that programmers in the 1950s understood how to implement knowledge graphs. Or computer vision. Or voice recognition. After all... they're "just code". Never mind that programmers of that era had no conception of the modern algorithms needed to actually make those things work. What they lacked wasn't just horsepower, but fundamental understanding of the problems and the solution. They couldn't build a computer system capable of driving a car that was infeasible only because it couldn't compute quickly enough, they couldn't build such a system at all.

the notion that "artificial intelligence" is something that we can 100% "undesrtand" shows a fundamental misunderstanding of what "artificial intelligence" actually is...it's just software running on hardware, all programed by humans

Certainly it will be software running on hardware, all programmed by humans. Humans that understand what intelligence actually is and how it works... something that we don't yet know. To get a little more specific, it appears that human "intelligence" is actually a collection of several different components, with several emergent properties. It's long been thought that "self-awareness" is the key emergent property, but many animals have self-awareness and yet lack the crucial ability that makes humans distinct.

The current best thinking is that the distinction is a particular form of creativity. Specifically, the ability to create abstract explanations. We certainly know how to write computer programs that manipulate abstractions, but they're abstractions of the programmer's creation, not of the program's. We need to learn how to write software that is able to create and criticize its own conjectured solutions to problems. We do not yet know how to do that.

We know it's possible, because we possess computers that can do it. In our heads.

I linked you to the Universal Declaration of Human Rights...you should at least have a cursory undestanding of how civil rights works in the US...it's absolutely ridiculous that you think I need to proffer up some sort of link to prove humans have free will

There are several misunderstandings implicit in this sentence.

First, I didn't ask for a link to prove humans have free will. You mentioned current legal definitions of free will. I asked for a cite to explain what such legal definitions are.

Second, you seem to think that civil rights are somehow related to free will. I don't see any such link. It's perfectly possible to have free will without having any civil rights, and it's equally possible to have civil rights without free will. I suppose you're trying to argue that we have established systems of human rights in order to protect the expression of free will... but that's clearly a second or third-order effect.

Third, you seem to think I'm ques

Re:"artificial intelligence" has become a religion

[globaljustin]

also, wanted to say that this is "fun" chatting with you, and I appreciate the experience you bring to the conversation...

i just disagree and have strong feelings on the subject

humans machines

[globaljustin]

the notion that our brains are deterministic machines

you're already a "true believer" arent' you?

the idea that the *thing that created machines* (human brain) is nothing more than a machine is ridiculous

machines are tools for humans...that's all...

our brains can be compared to machines (anything can be compared to anything else), but that doesn't mean that our brains function like machines

it's a false ontology...and it's based on your **personal beliefs** not rationality or logic