Slashdot Mirror
The Believers: Behind the Rise of Neural Nets
An anonymous reader writes Deep learning is dominating the news these days, but it's quite possible the field could have died if not for a mysterious call that Geoff Hinton, now at Google, got one night in the 1980s: "You don't know me, but I know you," the mystery man said. "I work for the System Development Corporation. We want to fund long-range speculative research. We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers." The Chronicle of Higher Ed has a readable profile of the minds behind neural nets, from Rosenblatt to Hassabis, told primarily through Hinton's career.
"You don't know me, but I know you," the mystery man said.
We call them "NSA" now.
Skimmed the article, conspiratorial themes aside, it seems like a good general history of neural nets.
:)
To answer what I see as the main question in TFA - Here's the difference "this time around".
I've been interested in AI and automata since the early 80's, sporadically following closely over the years. Life distracted me from this interest for most of the noughties. The first time I watched IBM's Jeopardy stunt with Watson I was blown away, the missus shrugged and said "It's impressive but what's the big deal, it's just looking up the answers, like google with talking, right?" I tried to explain why my jaw was on the floor, but all I got was a blank look and a change of subject.
Far from being overhyped I think the general public simply don't comprehend the significance of these developments. They see it as 'hype' because like my missus they simply don't comprehend the problem and tend to grossly underestimate the difficulty of solving it. IMO the Watson stunt is one of the most significant technological feats I've witnessed since the moon landings, and possibly the start of a new Apollo style arms race based on the same old fears. That doesn't mean I think all the problems in AI have been solved, but machines like Watson are very strong evidence that we have recently cleared a significant hurdle (that few in the general public have even noticed).
To me, this period in AI is very reminiscent of where digital comms were in the early 90's. Most of the bits for the comms revolution existed but rarely talked to each other; pagers, email, mobile phones, computers, printers, fax, GPS, fibre optics, etc. Just a few years later everyone was talking about "convergence", "as foretold" pretty much all of those things and more have now converged into the ubiquitous smart phone. In 1990, virtually nobody on the planet saw the internet coming (including me), I was at Uni, mature age CS/Math student, 88-91. I was perfectly placed in space and time to see it born but didn't notice it.
I first heard about HTML and Mosaic at Uni, one of our CS lectures was very impressed and went on a tangential rant about it one day in a networking lecture. Still, nobody in his hijacked audience I talked to afterwards could figure out why he was so impressed. "What's wrong with zmodem?" was a typical comment that I would have agreed with then.
I think we are more or less at that "1990" point where everyone will soon start talking more and more about "convergence" in AI. The Watson that won Jeopardy in 2011(?) required 20 tons of air-conditioning alone, today an instance of Watson fits on a "pizza box" server and you can try out your own Watson instance for free with a web based developer's API (google it). Their goal is to squeeze Watson into a smart phone.
A couple of things that a Watson style AI may "converge" with aside from phones are, "Big Dog" which has pretty much solved the autonomous movement/balance problem, and face recognition software which has also made big strides in the past few years. What the end result will be when it all converges and evolves, or even when it will converge, I have no idea, but a dystopian SkyNet style future is no longer purely fiction. From a less pessimistic POV, AI could serve as a "check and balance" in a democracy full of bullshitters, a tool to fact check the waffle and make evidence based, transparent, recommendations on public policy free from partisan politics, in other words "speak truth to power", like the public service in a democracy is supposed to be doing now.
Disclaimer: The "missus" is far from dumb, she has a Phd in Business and Marketing, she lectures to several hundred students at a time. I sometimes fail to see why she is interested/impressed by some obscure event in the Business News and politely change the subject
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.
Is it possible that last time you checked was a long time ago? Deep neural networks are again all the rage now (i.e. huge teams working with them at Facebook and Google) because
Check the wikipedia page for "convolutional neural networks" as well as other /. entries:
http://slashdot.org/tag/deeple... , and from yesterday http://tech.slashdot.org/story... .
Compute power is only part of the reason for the recent success of neural nets. Other factors include:
- Performance of neural nets increase with the amount of training data you have, almost without limit. Nowadays big datasets are available on the net (plus we have the compute power to handle them).
- We're now able to train deep (multi-layer) nerural nets using backprop whereas it used to be considered almost impossible. It turns out that initialization is critical, as well as various types of data and weight regularization and normalization.
- A variety of training techniques (SGD + momentum, AdaGrad, Nesterov accelerated gradients, etc, etc) have been developed that both accelerate training (large nets can take weeks/months to train) and remove the need for some manual hyperparameter tuning.
- Garbage-In, Garbage Out. You're success in recognition tasks is only going to be as good as the feature representation available to the higher layers of your algorithms (whether conventional or neural net). Another recent advance has been substituting self-learnt feature representations for laboriously hand-designed ones, and the recent there is now a standard neural net recipe of autoencoders+sparsity for implementing this.
- And a whole bunch of other things...
As Newton said "if I have achieved great things it is by standing on the shoulders of giants".. there are all sorts of surprising successes (e.g. language translation) and architectural advances in neural nets that are bringing the whole field up.
These arn't your father's neural nets.