Slashdot Mirror
IBM Researchers Propose Device To Dramatically Speed Up Neural-Net Learning (arxiv.org)
skywire writes: We've all followed the recent story of AlphaGo beating a top Go master. Now IBM researchers Tayfun Gokmen and Yurii Vlasov have described what could be a game changer for machine learning — an array of resistive processing units that would use stochastic techniques to dramatically accelerate the backpropagation algorithm, speeding up neural network training by a factor of 30,000. They argue that such an array would be reliable, low in power use, and buildable with current CMOS fabrication technology.
"Even Google's AlphaGo still needed thousands of chips to achieve its level of intelligence," adds Tom's Hardware. "IBM researchers are now working to power that level of intelligence with a single chip, which means thousands of them put together could lead to even more breakthroughs in AI capabilities in the future."
With this technology, chatbots can become neo-nazi holocaust deniers in less than two hours!
AIUI raw computation power ceased to be a significant concern a long time ago. The problem now is more a case of producing learning techniques that work well, I'm not sure this device adds anything to that problem?
Been hearing about the joys of truly good AI for about 50 years. I guess the technology finally caught up with the hype.
Hopefully fusion is next.
Knowing that they can possibly speed it up to this extent? I might have to bother thinking about what may come, if true. I never had a concern about AI, since making a strong one has always been in the realm of fantasy, where we are just scratching at the toes of the giant.
I have always thought that AI techniques lacked elegance, but I never put forth the effort to sort it out and look for a better method, other things I wanted to do. This may be part of the answer to that problem that annoyed me at that time.
I do have one question, for those that want to read and think about it some more.
Would this technique be used to bring it closer to making a human-like mind, or simply a better mind? Those two have always been the goals of AI research. Replicate what we have as a study of the mind or make something better than what we have.
Agreed.
There are some major limitations with the design they have gone with for deep learning. You may think that thousands of chips will soon shrink to fit in a phone. (~15 years if moore's law holds). But thermodynamics won't let this happen, you can't flip an arbitrary number of bits for zero energy. There is a minimum amount of energy necessary for a register to perform a simple operation, and the amount needed for a deep learning system of this scale is more than you would want to comfortably power in your pocket.
Clearly this is another PR lying article from IBM, so that they can grab attention. Just like their previous efforts such as so-called brain-like CPU (TrueNorth) that nobody uses today. I have a friend who tried their latest Power chips and said it is a piece of crap that should directly put IBM out of business. IBM is now only filled with sales people, not a single active and productive engineer or scientist.
... so it was just like everything MicroSoft ever published.
I wish people would get it right. AlphaGo is not AI itsis an expert machine.
'cause this is how we get skynet
The article abstract suggests that a Resistive Processing Unit will run 30,000 times faster than a cluster of CPUs using less power. But nobody runs neural nets on CPUs; they use GPUs.
So then, how does a RPU compare to a GPU?
Not nearly as well, obviously. That is why they did not do that far more appropriate comparison. Just like the D-Wave scammers that compare their machine to a simulation of their machine on a single CPU and get ridiculous speed-ups, when in actual reality they are slower when said far cheaper single CPU actually runs an algorithm suitable for it. It is lying with numbers and it has gotten very bad indeed because a lot of people fall for it.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Nft
So then, how does a RPU compare to a GPU?
GPUs, while massively parallel, still merely run a simulation of a neural network. Meaning, its evaluates some mathematical formulae for each neuron object it keeps in memory, and then those results are fed back into the object stack to do more calculations.
With memristors, you can build an *actual* electronic neural network. This means it's basically a massively parallel analog computer. There are no calculations needed, just interactions of electric signals, which will alter resistive values in the network.
So IF this works in real life, it should be both faster, and consume a lot less energy.
But nobody runs neural nets on CPUs; they use GPUs.
When you say "runs", do you perhaps mean trains? Because many people run neural nets on CPUs, but few would train on CPUs.
The hardware needs for training and running are *very* different.
This is how I understood it too.
So far, AI has only been pursued in ways that destroy without room for replacement. When (on net) do they start becoming a force that helps humanity without requiring retraining?
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.