Slashdot Mirror
IBM Researchers Propose Device To Dramatically Speed Up Neural-Net Learning (arxiv.org)
skywire writes: We've all followed the recent story of AlphaGo beating a top Go master. Now IBM researchers Tayfun Gokmen and Yurii Vlasov have described what could be a game changer for machine learning — an array of resistive processing units that would use stochastic techniques to dramatically accelerate the backpropagation algorithm, speeding up neural network training by a factor of 30,000. They argue that such an array would be reliable, low in power use, and buildable with current CMOS fabrication technology.
"Even Google's AlphaGo still needed thousands of chips to achieve its level of intelligence," adds Tom's Hardware. "IBM researchers are now working to power that level of intelligence with a single chip, which means thousands of them put together could lead to even more breakthroughs in AI capabilities in the future."
With this technology, chatbots can become neo-nazi holocaust deniers in less than two hours!
Really? I haven't seen anything yet that I would classify as non-hype.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Knowing that they can possibly speed it up to this extent? I might have to bother thinking about what may come, if true. I never had a concern about AI, since making a strong one has always been in the realm of fantasy, where we are just scratching at the toes of the giant.
I have always thought that AI techniques lacked elegance, but I never put forth the effort to sort it out and look for a better method, other things I wanted to do. This may be part of the answer to that problem that annoyed me at that time.
I do have one question, for those that want to read and think about it some more.
Would this technique be used to bring it closer to making a human-like mind, or simply a better mind? Those two have always been the goals of AI research. Replicate what we have as a study of the mind or make something better than what we have.
That's what cloud computing is for. Do all the processing remotely. Most speech recognition on phones is done this way currently.
"I bless every day that I continue to live, for every day is pure profit."
There are some major limitations with the design they have gone with for deep learning. You may think that thousands of chips will soon shrink to fit in a phone. (~15 years if moore's law holds). But thermodynamics won't let this happen, you can't flip an arbitrary number of bits for zero energy. There is a minimum amount of energy necessary for a register to perform a simple operation, and the amount needed for a deep learning system of this scale is more than you would want to comfortably power in your pocket.
This is for training. Once the training is done, the model can be used in a cell phone.
Case in point, voice recognition.
The human brain runs on about twenty watts. The computational power required to match it is barely imaginable.
Clearly we are a long, long way from the limits imposed by the laws of physics.
No. Not learning. comprehension. Without comprehension, the "I" in "AI" is void.
I've fallen off your lawn, and I can't get up.
This is for training. Once the training is done, the model can be used in a cell phone.
Case in point, voice recognition.
Correct me if I'm wrong, but isn't voice recognition performed by the cloud? The phone simply records and transmits your voice to the cloud for processing.
And thanks to this, all you say is now monitored and analyzed by three letters agencies around the world.
Votez ecolo : Chiez dans l'urne !
Qualcomm have visual recognition networks that will run offline on their tablets. https://www.youtube.com/watch?...
The human brain runs on about twenty watts. The computational power required to match it is barely imaginable.
AlphaGo required megawatt-hours of energy to learn to play Go well enough to beat Lee Se-dol. But how much did Lee Se-dol's brain consume in the ~20 years that he spent learning, not to mention the energy expended by the brains of his opponents (remember that much of AlphaGo's education was from playing against itself)? Supposing Lee Se-dol spent 2000 hours per year on Go for 20 years, that's about 800 kWh, plus some more for the energy expended by his opponents. AlphaGo's education required more energy input than Lee Se-dol's, but it's probably an order of magnitude more, maybe two. Not three or four. Switching from general-purpose to special-purpose hardware will probably get us to the same order of magnitude.
That said, my guess is that you're right that we're still a long way from physics-imposed limitations. My guess is that current technology would already be capable of building something vastly more efficient than a human brain... if only we knew what to build. We're learning.
define those terms, learning and comprehension for me please.
As far as I am concerned, comprehension (of a situation) could be defined as: extracting facts about the current situation, combining those with previously known facts, selecting which are relevant, and then applying rules when working out how to respond or making new inferences.
Please tell me how your comprehension differs to this process that the machines are already doing. Just because they are not self aware, does not mean they don't comprehend their domain of knowledge incredibly well.
We don't understand how our brains work. So continually denying that these machines are intelligent is in danger of "no true scotsman" if we can't precisely pin down what intelligence is.
'cause this is how we get skynet
Well, you can easily have learning without comprehension. You can learn to predict that a pattern will occur without knowing anything else about it. Comprehension is required if you are supposed to act in response to it in a "useful" way.
And that's not understanding. Understanding requires that you construct a model relating multiple streams of input, and comprehend what those streams mean for the model's reaction.
And THAT's not sufficient. (Google's robot has exhibited that kind of understanding.) Understanding, however, doesn't say anything about motivation. And I haven't seen any evidence that anyone is working on a reasonably complex motivational structure. (They could be, but I haven't seen evidence of it.)
The thing is, motivational structure is the most important thing to get right before you hook it into a reasonably powerful AI program.
I think we've pushed this "anyone can grow up to be president" thing too far.
Ahem. Flying cars.
Have gnu, will travel.
Actually we could adopt the same approach if we gave up on switching speed and went instead for low power and parallel execution. But we'd need to redesign all our algorithms.
That said, even with low power switching electronics are less power efficient than the brain, just not so much so.
I think we've pushed this "anyone can grow up to be president" thing too far.
As I understand it AlphaGo operated via deep learning. That's not only an AI, that's a rather advanced AI. Deep Blue was an expert machine. Different technology.
I think we've pushed this "anyone can grow up to be president" thing too far.
Pretty sure some of us do.
Comprehension, in the context of intelligence: Capable of abstract thought about any subject or input presented. When we get there, we'll have AI. Not before. Everything to date, while often marvelously useful, is just marketing speak on the order or "3d television", which is to say, not.
I've fallen off your lawn, and I can't get up.
The article abstract suggests that a Resistive Processing Unit will run 30,000 times faster than a cluster of CPUs using less power. But nobody runs neural nets on CPUs; they use GPUs.
So then, how does a RPU compare to a GPU?
The human brain runs on about twenty watts. The computational power required to match it is barely imaginable.
Clearly we are a long, long way from the limits imposed by the laws of physics.
No, not really. The 'barely imaginable' computational power required now comes from clumsy, inaccurate, and barely-informed emulation.
When we finally understand it, a faithful execution of the brain's design will have fairly modest hardware requirements. It will be like a graphics card.
There is not comprehension in machines. That very likely requires consciousness, i.e. some major fundamental breakthrough that is not even on the very far horizon as nobody has any idea what it is and as it does not seem to be part of what can be implemented with physical machines (there just is no mechanism for it).
And please do not tell me that consciousness is an "emergent property" of complex machinery. That is pseudo-mystical bullshit. There are no emergent properties in Physics and the whole cannot be more than the sum of its parts and their configuration in Physics.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
At the current state-of-the-art computing machinery has exactly zero capability for "thought" or for creating/understanding abstractions. All it can do is use abstractions it is programmed to use. One reason I see why so many people get this wrong is that they do not have a lot of effective intelligence themselves and are mostly driven by an emotional system that may well be mostly mechanical.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Hahahhaha, no. AI is even farther removed from reaching its stated goals today then it ever was. It looks quite possible today that AI is infeasible in this universe.
If you think this Go-machine is intelligent, then you probably also think that a slide-rule of book of mathematical tables is intelligent: It both can do computations far better than humans can. Yet clearly both are inanimate objects and hence clearly not intelligent at all. The Go-machine is just the same idea scaled up and with some motors added. It can do one special extremely well defined but tiny task extremely well. It has no understanding of what it does though, and outside of this task, it is about as useful as a paperweight. True, you can change the software, then you get another tiny, extremely well defined task it can do, but you lose the original one.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Moore's law has effectively been over about 10 years ago.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Very much this. It cannot to basically everything that the human opponent it beat _can_ do. It can do this really simple game with really simple rules extremely well, but that is it. Here is a comparison I like to use: Take a pocket-calculator or slide-rule or even a book of mathematical tables. All are several orders of magnitude better than humans at some, very specific mathematical operations. Yet nobody sane would claim either of these objects is intelligent.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You fell for the hype. "Deep learning" is not learning at all and carries zero qualities of insight or understanding. It is parameter adjustment to a sample of data. It is something that looks very well on grant applications or marketing material though.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Not nearly as well, obviously. That is why they did not do that far more appropriate comparison. Just like the D-Wave scammers that compare their machine to a simulation of their machine on a single CPU and get ridiculous speed-ups, when in actual reality they are slower when said far cheaper single CPU actually runs an algorithm suitable for it. It is lying with numbers and it has gotten very bad indeed because a lot of people fall for it.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Understanding requires that you construct a model relating multiple streams of input
No, that is not "required". Intelligence is a characteristic of behavior. If a system behaves intelligently, then it is intelligent. Internal mechanism is irrelevant.
Well said. I have noted that too. Explains a lot. Physicalists are a very strange kind of fundamentalist religious, as they always assume their view is obviously true and the only possible one. These people are thinking they have rejected religion, only to replace it with something that has all the characteristics of fundamentalist religion. (Well, no personal God, but that is not strictly required for religion.) They are bad at Science as well (like other fundamentalists), because in Science the question is wide open as in "we have no clue at this time". I find it always funny when they claim consciousness is an "emergent property" of complexity, which is just pseudo-mystical bullshit.
The lack of even a credible theory how strong AI could be implemented in this universe is something of a pretty strong hint though, if you take into account how much effort has gone into that. The only thing known that maybe could implement strong AI (automated deduction) does not scale in this universe to things a smart human being can do.
We will see how this evolves. The idea that humans are mind-body hybrids, with the mind only physical in that it can use certain interfaces, has some things going for it though and it is compatible with a number of different world-models, including the one where the physical universe is a simulation.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
That very likely requires consciousness, i.e. some major fundamental breakthrough that is not even on the very far horizon as nobody has any idea what it is and as it does not seem to be part of what can be implemented with physical machines (there just is no mechanism for it).
Whatever consciousness is, it can be implemented by a physical machine: our brains do it. If you feel that this is impossible, then you need to adjust your notion of what consciousness is.
AIUI raw computation power ceased to be a significant concern a long time ago. The problem now is more a case of producing learning techniques that work well, I'm not sure this device adds anything to that problem?
Computation power and especially computation efficiency still leave huge room for improvement when modeling biological neural networks. The human brain runs on about 20 watts and currently would take clusters of supercomputers to simulate, probably using billions of watts.
We certainly do need improved learning techniques for neural networks, but overall hardware efficiency is still an important research goal as well.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
Really? I haven't seen anything yet that I would classify as non-hype.
Then you haven't been keeping up with advances in image and voice recognition. This does not just involve theoretical research, there are actual products used by consumers benefiting from these technologies.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
Physicalists are a very strange kind of fundamentalist religious, as they always assume their view is obviously true and the only possible one. These people are thinking they have rejected religion, only to replace it with something that has all the characteristics of fundamentalist religion
It's just simple observation. We observe that brains are physical objects and that they have intelligence and consciousness. To deny that this is happening, even though we observe it, that's fundamentalist religious.
To the best of our knowledge, every system in the human brain - including the emotional system - is completely mechanical.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
Please explain what you mean by "insight" and "understanding" and why these are necessary qualities for something to qualify as learning? Because "parameter adjustment to a sample of data" seems to cover an awful lot of cases.
It looks good because it gets results. Customers are generally more interested in whether a learning device acts as if it learns, not whether a philosopher thinks it "really" learns.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
Actually we could adopt the same approach if we gave up on switching speed and went instead for low power and parallel execution. But we'd need to redesign all our algorithms.
The problem is that many algorithms can't be redesigned to work efficiently on a (massively) parallel computer.
Then how do you explain human brain? Nature magic?
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
Very little. The whole concept of a Turing machine isn't even a century old, and modern computers are still far from the raw computing power of human brain. Internet broke through in my lifetime, and ubiquitous computing - smart everything - is still just a promise on the horizon. The proverbial sunrise of the Information Age hasn't yet happened, so it's a bit early to declare the day a hoax.
It is not, however, compatible with the fact that booze exists - that is, the observation that any disturbance in the physical state of the brain causes a disturbance in the function of the mind. And even if it was true, it wouldn't actually prove that such hybrid existence would be required for intelligence.
Quite frankly, this all sounds a lot like the old idea that soul is a little man inside you, made up of some kind of "spirit matter" - let's call it ectoplasm - and doesn't need any kind of internal structure to do its work.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
You don't. You assume everything is physical and from that you can conclude *surprise* that everything is physical. It is an elementary beginner's mistake. Physics, incidentally, makes no such claim.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Nicely clueless. Proves my point.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Why would I need to explain it? Are you of the school of though that a theory is only valid if it explains everything and has no gray areas and leaves no unexplained things? If so, you are exceptionally stupid. This is, incidentally, an idea that is also frequently encountered in fundamentalist religion.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You are making invalid assumptions. You are assuming everything observable at the outer interface is created inside. That is a rather simplistic and unsophisticated model. There is no reason to assume it is true.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
This is how I understood it too.
So far, AI has only been pursued in ways that destroy without room for replacement. When (on net) do they start becoming a force that helps humanity without requiring retraining?
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
Why would I need to explain it?
Because you're asserting that artificial intelligence is infeasible in "this universe", yet natural intelligencies - us - exist. That's equivalent to claiming human beings are supernatural - that is, there's a component to human intelligence which can't be replicated by any engineering, no matter how advanced - which is an extraordinary claim.
No, but I do think extraordinary claims require extraordinary evidence.
So is expecting others to believe your pet theories despite all available evidence being against them.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.