Slashdot Mirror
IBM Researchers Propose Device To Dramatically Speed Up Neural-Net Learning (arxiv.org)
skywire writes: We've all followed the recent story of AlphaGo beating a top Go master. Now IBM researchers Tayfun Gokmen and Yurii Vlasov have described what could be a game changer for machine learning — an array of resistive processing units that would use stochastic techniques to dramatically accelerate the backpropagation algorithm, speeding up neural network training by a factor of 30,000. They argue that such an array would be reliable, low in power use, and buildable with current CMOS fabrication technology.
"Even Google's AlphaGo still needed thousands of chips to achieve its level of intelligence," adds Tom's Hardware. "IBM researchers are now working to power that level of intelligence with a single chip, which means thousands of them put together could lead to even more breakthroughs in AI capabilities in the future."
With this technology, chatbots can become neo-nazi holocaust deniers in less than two hours!
Really? I haven't seen anything yet that I would classify as non-hype.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
There are some major limitations with the design they have gone with for deep learning. You may think that thousands of chips will soon shrink to fit in a phone. (~15 years if moore's law holds). But thermodynamics won't let this happen, you can't flip an arbitrary number of bits for zero energy. There is a minimum amount of energy necessary for a register to perform a simple operation, and the amount needed for a deep learning system of this scale is more than you would want to comfortably power in your pocket.
This is for training. Once the training is done, the model can be used in a cell phone.
Case in point, voice recognition.
The human brain runs on about twenty watts. The computational power required to match it is barely imaginable.
Clearly we are a long, long way from the limits imposed by the laws of physics.
The human brain runs on about twenty watts. The computational power required to match it is barely imaginable.
AlphaGo required megawatt-hours of energy to learn to play Go well enough to beat Lee Se-dol. But how much did Lee Se-dol's brain consume in the ~20 years that he spent learning, not to mention the energy expended by the brains of his opponents (remember that much of AlphaGo's education was from playing against itself)? Supposing Lee Se-dol spent 2000 hours per year on Go for 20 years, that's about 800 kWh, plus some more for the energy expended by his opponents. AlphaGo's education required more energy input than Lee Se-dol's, but it's probably an order of magnitude more, maybe two. Not three or four. Switching from general-purpose to special-purpose hardware will probably get us to the same order of magnitude.
That said, my guess is that you're right that we're still a long way from physics-imposed limitations. My guess is that current technology would already be capable of building something vastly more efficient than a human brain... if only we knew what to build. We're learning.
'cause this is how we get skynet
As I understand it AlphaGo operated via deep learning. That's not only an AI, that's a rather advanced AI. Deep Blue was an expert machine. Different technology.
I think we've pushed this "anyone can grow up to be president" thing too far.
The article abstract suggests that a Resistive Processing Unit will run 30,000 times faster than a cluster of CPUs using less power. But nobody runs neural nets on CPUs; they use GPUs.
So then, how does a RPU compare to a GPU?