Slashdot Mirror
The Believers: Behind the Rise of Neural Nets
An anonymous reader writes Deep learning is dominating the news these days, but it's quite possible the field could have died if not for a mysterious call that Geoff Hinton, now at Google, got one night in the 1980s: "You don't know me, but I know you," the mystery man said. "I work for the System Development Corporation. We want to fund long-range speculative research. We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers." The Chronicle of Higher Ed has a readable profile of the minds behind neural nets, from Rosenblatt to Hassabis, told primarily through Hinton's career.
"You don’t know me, but I know you," Smith told him. "I work for the System Development Corporation. We want to fund long-range speculative research. We’re particularly interested in research that either won’t work or, if it does work, won’t work for a long time. And I’ve been reading some of your papers."
Hinton won $350,000 from this mysterious group. He later learned its origins: It was a subsidiary of the nonprofit RAND Corporation that had ended up making millions in profit by writing software for nuclear missile strikes. The government caught them, and said they could either pay up or give the money away—fast. The grant made Hinton a much more palatable hire in academe.
Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.
Is it possible that last time you checked was a long time ago? Deep neural networks are again all the rage now (i.e. huge teams working with them at Facebook and Google) because
Check the wikipedia page for "convolutional neural networks" as well as other /. entries:
http://slashdot.org/tag/deeple... , and from yesterday http://tech.slashdot.org/story... .
Compute power is only part of the reason for the recent success of neural nets. Other factors include:
- Performance of neural nets increase with the amount of training data you have, almost without limit. Nowadays big datasets are available on the net (plus we have the compute power to handle them).
- We're now able to train deep (multi-layer) nerural nets using backprop whereas it used to be considered almost impossible. It turns out that initialization is critical, as well as various types of data and weight regularization and normalization.
- A variety of training techniques (SGD + momentum, AdaGrad, Nesterov accelerated gradients, etc, etc) have been developed that both accelerate training (large nets can take weeks/months to train) and remove the need for some manual hyperparameter tuning.
- Garbage-In, Garbage Out. You're success in recognition tasks is only going to be as good as the feature representation available to the higher layers of your algorithms (whether conventional or neural net). Another recent advance has been substituting self-learnt feature representations for laboriously hand-designed ones, and the recent there is now a standard neural net recipe of autoencoders+sparsity for implementing this.
- And a whole bunch of other things...
As Newton said "if I have achieved great things it is by standing on the shoulders of giants".. there are all sorts of surprising successes (e.g. language translation) and architectural advances in neural nets that are bringing the whole field up.
These arn't your father's neural nets.