New 'Deep Learning' Technique Lets Robots Learn Through Trial-and-Error

jan_jes writes: UC Berkeley researchers turned to a branch of artificial intelligence known as deep learning for developing algorithms that enable robots to learn motor tasks through trial and error. It's a process that more closely approximates the way humans learn, marking a major milestone in the field of artificial intelligence. Their demonstration robot completes tasks such as "putting a clothes hanger on a rack, assembling a toy plane, screwing a cap on a water bottle, and more" without pre-programmed details about its surroundings. The challenge of putting robots into real-life settings (e.g. homes or offices) is that those environments are constantly changing. The robot must be able to perceive and adapt to its surroundings, so this type of learning is an important step.

Comments

"Deep Learning"...??

[taiwanjohn]

This seems more like basic-level stuff... learning from your mistakes. That strikes me as the sort of thing that would be "hardwired" in everything from nematodes to primates. Why is this news?

Re:"Deep Learning"...??

[ColdWetDog]

Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so.

-- Douglas Adams

Re:"Deep Learning"...??

[zaxus]

...Why is this news?

Because they couldn't do it before....

Re:"Deep Learning"...??

[ShanghaiBill]

That strikes me as the sort of thing that would be "hardwired" in everything from nematodes to primates. Why is this news?

Because it isn't a nematode or a primate. It is a robot. A living thing that can learn and adapt is not news, because that's what living things do. A non-living think that can learn and adapt is news because that's what living things do.

Re:"Deep Learning"...??

[Ol Olsoc]

This seems more like basic-level stuff... learning from your mistakes. That strikes me as the sort of thing that would be "hardwired" in everything from nematodes to primates. Why is this news?

Because you haven't learned what is news yet. But by trial and error, you'll catch on

Re:"Deep Learning"...??

[ColdWetDog]

I don't want to see us talk about machines having "agency" when most people struggle with it for themselves.

And you think the NSA is run by humans? Have you ever seen pictures of the top brass at the NSA? I've seen toasters with more anthropomorphic features.

Re:"Deep Learning"...??

[Richard Kirk]

It is a good question, and there are several answers...

Artificial Intelligence has been seen as a goal since Ada Lovelace was a lass. In the fifties, it was hoped that computers fed with parallel translations could learn the rules of languages, and provide fought translations of (say) technical documents on aeronautics from Russian to English, where sufficiently skilled and positively vetted engineers were rare. There were later attempts in the sixties and seventies to learn to walk, recognise objects, or solve puzzles. There was the constant hope that the next hardware would be a bit more powerful, and you could throw problems at it, and intelligence would somehow boot up. After all, that is how it must have started last time. However, intelligence failed to boot up, or maybe it always lost out to other brute force techniques which regular computers are good at.

The nematode has a simple. pre-programmed brain. It is good for being a nematode, but it doesn't really learn. Our brains have a lot of structure when they are formed, which means that our language centres, our vision centres, the parts that are active when we are solving spatial problems, or composing music, turn up in the same places most of the time; but we don't seem to run an actual program as such. We are born with very little instinct when compared to most other complex animals, but I suspect even they are not really running a program either.

The trick seems to be to provide the robot with enough plastic design to nudge it in the general direction of intelligence: too little design and it never gets its act together, while too much design means it is just doing what you programmed it to do. There are interesting times where computers are getting the complexity and the connectivity and plastic re-programmability to rival animal brains; but the spontaneous self-evolving problem solving spark just isn't there yet. But I hope we may see it in our lifetimes.

Re:"Deep Learning"...??

[tmosley]

" An automaton can be neither benevolent nor have free agency."

But Anon, humans are automata.

"I don't want to see us talk about machines having "agency" when most people struggle with it for themselves."

Birds struggle with flying above a certain ceiling, or beyond a certain speed. Human made flying machines breach those limits easily. The same will likely hold true with human made thinking machines. These will not be a slave to evolution, though they may utilize evolution as a force for optimization. If we don't talk about these things now, they are going to happen, and very, very fast (30 years from Kitty Hawk to jet aircraft--only computers advance a LOT faster than engineering). By the time computers reach human level in all aspects, they will be so far beyond us in some that they will look more like gods than men.

Re:"Deep Learning"...??

[ShanghaiBill]

An automaton can be neither benevolent nor have free agency.

Why not? Unless you believe that brains are magic, or created by the intervention of a deity, there is no reason to believe that computers have any inherent limitation that living things do not have.

Re:"Deep Learning"...??

[ShanghaiBill]

Hmm... sorry, this has been happening for many years already... back to the 80s... This is nothing new.

No, it was not happening in the 1980s. The fundamental algorithm behind deep learning networks was worked out by Geoffrey Hinton in 2006. Before that, training a NN more than 2 layers deep was intractable.

Re:"Deep Learning"...??

[Anonymous Coward]

Deep neural networks are only faster at learning than normal neural networks. A regular neural network can eventually compute any function. The only 'advancement' here is speed and scale, and it doesn't sound like they used any custom chips so they didn't even make any advancements there. The algorithms are well known and the outcome should have been easily predicted by the developers. If not, they haven't spent enough time in their library.

They did a good job, but this is only interesting if you know nothing about AI and it doesn't advance the state of the art in the field. It could later, but doesn't now.

Re:"Deep Learning"...??

[Lennie]

One thing I wonder about is: will machine learning systems being to transmit their experiences over the Internet to have other machine learning systems learn from that.

They can't now, but how long will until they can ?

An other is: can they take snapshots of what one system learned and transmit that to an other ?

You remember how they learned new skills in the Matrix ?

Re:"Deep Learning"...??

[itzly]

An other is: can they take snapshots of what one system learned and transmit that to an other ?

If they run on general purpose software, you can just clone the entire program. Matrix style learning is a lot more difficult, because it has to be integrated in what the person already knows.

Re:"Deep Learning"...??

[tmosley]

But those people are all idiots. All people are idiots compared to the ASI. The only way that something can make high quality decisions for a group of others is for that person to be smarter than all of those people PUT TOGETHER. This is not possible with humans beyond the level of nuclear family. When you have an ASI that has an IQ higher than the rest of humanity put together, then it becomes possible.

The thing about an ASI running your economy is that 99.99999% of interactions will be invisible. Good things "just happen", whether you find yourself standing in line next to just the right person to help you start your business, or just happen to have your car break down right in front of the house of the person who will be the love of your life. Luck that was previously neutral becomes overwhelmingly good, for everyone, simply by subtle manipulations of people and objects by a device with unimaginable predictive power.

Re:"Deep Learning"...??

[ShanghaiBill]

Deep neural networks are only faster at learning than normal neural networks. A regular neural network can eventually compute any function.

If "eventually" is exponential, that doesn't mean much. A computer can eventually solve the traveling salesman problem for a thousand cities. But in the meantime, all the black holes in the universe will evaporate through hawking radiation, and there will be nothing left but cosmic radiation at a few nano Kelvins. It will be hard to power a computer with that.

Re:"Deep Learning"...??

[Whiteox]

You can't experience the experience of others (paraphrase) J.D. Lang.
OTOH when I read the OP, I immediately thought of 'Deep Thought' and a couple of philosophers who were too highly trained to be useful.

Re:"Deep Learning"...??

[Lennie]

"Matrix style learning is a lot more difficult, because it has to be integrated in what the person already knows."

This is exactly why I worded it that way. ;-)

Actually found a talk by the people working on this project, here he talks about where/how to get data:

https://www.youtube.com/watch?...

Re:"Deep Learning"...??

[Whiteox]

but we don't seem to run an actual program as such.

Perhaps an interdisciplinary pov might be of help here. We do run programs based on hard wired (unconscious) programming.
Principally it is self-preservation, from biological respiration to environmental choices. That's the core programming from which all other extensions spring from. Replication is group preservation, so is war for survival, hunting and gathering, society, friendship, love, art, recording of knowledge etc.
The fact that AI is not concerned with that basic tenant is bemusing to me.

Re:"Deep Learning"...??

[c9brown]

"Deep learning" refers a family of machine learning techniques (such as neural-networks, convolutional neural-networks, stacked-autoencoders, etc.) that have a multi-layer architechture, typically allowing the system to learn highly non-linear functions of many variables. Each layer can be thought of as a simple learned function whose output is fed into the next layer. Such systems can often have thousands or millions of parameters to learn and thus require a LOT of training data and a fair bit of computing power/ runtime to train. But if you look at some area (e.g. object reccognition in computer vision), deep networks are currently the top techniques by a fair margin.

This seems more like basic-level stuff...

The devil is in the details. How do you best represent learning mathematically and computationally? What are mistakes and or what are the objectives? How do you encode these and how to you penalize making these mistakes in the future? These are all challenging questions.

That strikes me as the sort of thing that would be "hardwired" in everything from nematodes to primates.

Machine learning approaches have often taken inspiration from biology, however the exact neurological mechanisms of learning are not yet entirely understood. Its difficult to replicate nature. Its even more difficult when you don't yet understand nature.

Re:"Deep Learning"...??

[penguinoid]

If you believe evolution your argument makes no sense. Random mutations accomplished already what you claim is unlikely to occur until the theorized heat death of the universe. How likely is that?

Is the universe infinite?

PS: Evolution does not rely on random mutations.

Re:"Deep Learning"...??

[penguinoid]

An automaton can be neither benevolent nor have free agency.

Sure it can, you just have to program it to have free agency.

Re:"Deep Learning"...??

[Richard Kirk]

I think there is more here than just learning to imitate humans, exciting though that is.

Let us take 'Deep Blue' as an example of a machine that does not think. It was able to come up with some dramatic solutions. Its typical successes were mates involving an improbably sequence of sacrifices that gave a mate in 6 or 7, which was about the brute force look-ahead of the time. It also had weighting models that give suggestions of which were 'good' moves and which were 'bad' ones. Moving a bishop to a centre square is good because it threatens more squares, but if it was in front of your king then you may want to leave it where it was. Deep Blue could alter the weights in its model depending on the games it had seen, but it did not really have any understanding of 'edges' and 'centre' any more than a pocket calculator understands the nature of numbers and multiplication.

Let us now take a problem that Deep Blue possibly has not seen: you have two bishops and a king against a king. If you have just taken another piece then you have fifty moves to get a mate, otherwise the game is a draw. Now most of these extreme endgame solutions are known, and Deep Blue probably had the solution hard-coded. If Kasparov had got into the losing position, he would probably have given up the game because he knows it is hopeless.

Can you force a mate in less than 50 moves? Yes, you can. The two bishops can make a diagonal 'wall' of squares that the king cannot jump, so you can slowly heard it into a corner. However, the king can still take one of the bishops, so you have to either protect them with your king, or move them to the other end if the diagonal. As you get towards the corner, the diagonal becomes shorter, and this becomes harder to do. Eventually, you have to protect one bishop and move the other out of the corner entirely. There is then a tricky bit where you may have to waste a move so the other king is forced to move off the better of the squares left to it, and then move the other bishop. It can take 48 moves but it can be done.

Supposing Deep Blue had not got a hard-coded solution. 48 moves is well away from its brute force limit. Its tables for 'good' moves are not optimised for the extreme end-game, and the winning strategy seems to 'change' as you get into the corner. It has no understanding of corners and diagonals, so it might heard the king into a corner from 'instinct' (probably not the 'right' word, but it sort-of works). So, we might win because we can use our knowledge of herding sheep to get the king in a corner, the understanding of the other king's want to survive by attaching the bishops, the knowledge that the bishop can be anywhere along the diagonal to counter this be flipping to the other end, the appreciation that this strategy will not work all the way into the corner and will have to be changed for something at the last minute, and so forth.

Note, this 'Deep Learning' free problem solving ability that we use, and can probably duplicate in a machine one day, is not necessarily linked with self-awareness, will to survive, altruism, creativity, and all the other things we usually identify with intelligence. We could probably make something that could explore other planets which can work for its own survival, and determine what is interesting and worth reporting on the planet, without giving it a concept of 'self' or a fear of its own death. Indeed, it may well be better off being designed without all the baggage that comes with evolution. Maybe it will develop some of these of itself, maybe not. But I doubt it will attack its creators in its struggle to survive, in the classic sci-fi tradition, unless we deliberately train it to do so.

Some say it may have something new and wholly alien to us instead of 'free will'. I rather doubt this, but I allow there might be other radically different solutions for 'how to live'.

Apologies for the long reply, Words are tricky with this topic, but wordy illustrations can avoid some of the worst ambiguities.

Re:"Deep Learning"...??

[penguinoid]

I think if we are talking about heat death then the universe is not infinite. I would be interested in reading someone else's application of probability/frequency to a question like this though. It does not seem straightforward.

Infinite universe (spatial) means you have infinite chances of something happening. Infinite universe (temporal) means you have infinite chances of something happening and can also perform arbitrarily long calculations. There's a decent chance that the universe could be infinite in either sense, also that there could be an infinite number of different universes. (however, if our universe is temporally infinite it is likely to have certain difficulties making use of said infinity, due to entropy or data loss during a cycle)

Also, evolution is supposedly driven by random mutations. The selection of mutations that are passed down is not random, but the mutations are (supposedly, which is fine until proven otherwise since that is the simplest scenario).

The mutations are not random in at least a few ways:
1) Mutation rate is based on population size, so permutations of more successful individuals are more common than permutations of less successful individuals.
2) Survival is based partially on genetics, which means that useful mutations persist far more than harmful mutations.
3) Many organisms have a method to increase mutation rate when they are stressed. This is an "intentional" feature in that disabling certain genes will negate the increased mutation rate when stressed. The result is that individuals maladapted to their environment mutate at a faster rate.
4) Certain genes or parts of genes have higher mutation rates (for example, those involved in immune function).

Re: "Deep Learning"...??

[billdale]

Taiwanjohn: the only way you could make, such an inane statement is if you had never had the task of doing anything remotely challenging... only someone that has never taken on a difficult task such as that could, ever make such a ridiculous comment. Try something far, simpler... build a go - kart from scratch, not from a kit or a set of directions, like from Popular Mechanics... or, maybe design and build a simple drone, again without directions... those would be monstrously simpler achievements... do that, and then come back and tell us how teaching robots is so simple. You can be completely sure that a genuinely intelligent, creative, accomplished person would never make such a statement as you have made, and the vast majority of readers realize that. You have been outed, fella, as having a bully mentality!

Re:"Deep Learning"...??

[Whiteox]

I understand you post and was written with a great example. I do not deny that Deep Learning has a long way to go, yet I (as a Philosophy sub-major) can't leave my original contention alone. So with my technical knowledge I can build a do-able A.I. machine which can have elements of Deep Learning if I understand it correctly.
In this case let's use our imagination:
The simplest A.I. is a feedback circuit - like a thermostat. It always tries to control temperature within a set range. It finds it difficult to operate if the external environment is beyond it's mechanical capacity to operate. I.E. if the external temp is too cold (if it is a heat pump) or too warm (as a fridge). Uncontrolled, it eventually burns the compressor out. This damage can be avoided if another temp sensor on the compressor/motor allows it to shut itself off. So it protects itself from damage, but it still wants to work, that is achieve its purpose-of-being.
So let's give it mobility and more sensors around its environment to find a more environmentally friendly position to operate. So as soon as the temp becomes out of range, it moves to a position where it can operate longer.
Now for an example:
Imagine a split system air conditioning compressor motor that can move on rails around the corner of a house. In very hot temperatures it will move towards the cooler regions around the corner in the shade. In very cold conditions, it will move to a warmer part of the exterior. Remember, this is do-able and relatively easy to implement.
The Learning aspect:
Now let us give the machine the ability to store its own movement data based on time, days, months. It can now 'predict' where it can go and does so much earlier. It can move to a better position for better efficiency and it gives itself a +1 for purpose-of-being. It can even do statistical operations on the probability that a certain location at a certain time would be better for it.
Now give it the internet.
What machine intelligence is needed before it can work out how to log into a weather forecasting service? Although it can monitor local wind speed and temperature and find the optimal position all the time, it can also determine if the house/structure inhabited, determine power draw and communicate with other 'house A.I.' to route warmth and cool to inhabited areas.
But the big question is to hard code the weather service. Supposing it goes down? Can it find another? What other services would it require? Maybe we can get the House A.I. to crawl the net or Cortana/Siri to find an alternative or maybe associative services we have not considered when the machine was first set up.
As long as it knows that the new service may/would give it a +1 - or prevent it from getting a -1
So 'preservation' drives this machine and it has a sense of purpose.

Re:"Deep Learning"...??

[tmosley]

Sort of. There is a lot of overlap, such that a deep intellect of, say, IQ 2000 could provide insights that would allow each member of humanity to make better decisions, such that humanity with a collective IQ of 700,000,000,000 + ASI IQ of 2000 is more economically effective than humanity with 14 billion people and 100 IQ average. But for ASI to make better decisions than everyone else put together, you need it to have linear computing power greater than all of humanity combined (where higher IQ scores may require exponential advances in computing power).

The Skynet is Falling!

[zaxus]

Cue the Skynet / Matrix references in 3...2...1...

Re:The Skynet is Falling!

[ArcadeMan]

Do you want Skynet? Because this is how you get Skynet.

Genetic Algorithm Re-framed?

[Anonymous Coward]

There is a Genetic Algorithms textbook from 1989 that covers generational learning and "mutating" the parameters until you get to the end state in the best way possible. My AI knowledge isn't great but I wouldn't be surprised if there are ideas that pre-date the '89 text.

Does anyone know what the software controlling the robot is doing under the hood that's different?

Re:Genetic Algorithm Re-framed?

[tmosley]

I think we are doing much the same, it's just that computers have caught up to theory and are able to perform now. Now it is no longer a question of theory, but one of technique, and what is described in the article is a new technique--one that will likely have many, many applications in the near future.

In the late 80's/early 90's, they were able to use some of their theory, but it just wasn't super-robust because things just took too darn long. You couldn't have your system analyze at a million images in a minute, hence allowing them to go through hundreds of generations in a day.

At least that is my view, as an educated layperson.

Re:Genetic Algorithm Re-framed?

[Enokcc]

Here are the papers: http://rll.berkeley.edu/deeple...

Roderick and Roderick at Random

[hughbar]

I really recommend these two books by Sladek: http://en.wikipedia.org/wiki/R... they're very funny satire about a naive, learning robot in a cruel illogical world. This is what our little friend here can expect.

How about learning to 'fly'?

[mjensen]

Reminds me of http://www.newscientist.com/ar... from 2002. Robots goal was to raise its altitude without knowing its actuators ahead of time.

Re:This is what else matters.

[tmosley]

I don't want humans, who are subject to weird, petty shit, to have ultimate power. I'd rather start fresh with an impartial God. Of course, a deified human would be less likely to have some insane value function that causes it to completely wipe out humanity, so that would be the hedged bet.

I would bet that we get an ASI long before we can raise ourselves up to that level, unless we as a species push to delay the former and focus on the latter. But when have we as a species ever come together to do anything without some direct incentive?

Deep

[Livius]

Better known as 'learning' to everyone not trying to exaggerate an claim of artificial intelligence.

It's excellent progress, which is why I don't think it should be watered down by being compared to the simple algorithms.

New?

[ichabod801]

People were doing this when I was an undergrad, almost 20 years ago. I specifically remember a six legged robot that had to figure out how to walk by itself.

Re:This is what else matters.

[Whiteox]

There has never been a benevolent godlike human in any culture without fault. That I postulate would be impossible.
One approaching fallacy is that humans have free will without constraints. That is obviously not true and humans in their environment have finite responses for any real situation. They are no different to robots. We all operate within natural law.
For humans to be other than that which they are would mean some kind of transformation and thought and philosophy has totally explored most of that for thousands of years, rehashed it countless times with pretty much no result, either in thought or reality.
Philosophically, the origin of this was the Garden of Eden, the story of how humanity became separated from Godhead, or so they say. The end of this lies in the future. In the meantime, we create robots and give them intelligence because of some ingrained impulse?
Personally that's why I became interested in computers and automata. A machine created by humans to do work that humans can do.

What could possibly go wrong?

[Show me I'm wrong]

Self-correcting curious entities. If there's teeth on them, we're bound to become enemies.

Re:Sorta Off Topic

[Ol Olsoc]

But can't we get people to mod down the now incessent "Why is this news" or "Why is this on Slashdot?" Posts?

They are becoming the 2015 equivalent of "Frist Post, or "Welcome from the Golden Girls".

Amazing how many people are wasting mod points calling an admitted Offtopic Post as Offtopic. Captain Obvious is smiling upon thee.