The Believers: Behind the Rise of Neural Nets

An anonymous reader writes Deep learning is dominating the news these days, but it's quite possible the field could have died if not for a mysterious call that Geoff Hinton, now at Google, got one night in the 1980s: "You don't know me, but I know you," the mystery man said. "I work for the System Development Corporation. We want to fund long-range speculative research. We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers." The Chronicle of Higher Ed has a readable profile of the minds behind neural nets, from Rosenblatt to Hassabis, told primarily through Hinton's career.

Comments

NSA

[thhamm]

"You don't know me, but I know you," the mystery man said.

We call them "NSA" now.

Re:NSA

[OakDragon]

My guess is Rusty Shackleford.

Re:NSA

[ceoyoyo]

If you read the article, it turned out to be someone from the RAND corporation. So you're modded as funny but.

Re:NSA

[PPH]

Funny, but System Development Corporation (aka RAND) is primarily a supplier for the US Military and other three letter intelligence agencies. There was probably more good research in various fields that was intercepted by the likes of them, stamped 'Top Secret' and lost from public view for decades.

I used to work for an outfit with some serious machine learning, natural language recognition applications. When 9/11 hit, they saw the handwriting on the wall. With the Patriot Act, Homeland Security and the NSA treating every American as an enemy, they understood the utility of such software to these organizations and the negative consequences for its commercial use. They promptly boxed up everything and shipped it to overseas contractors for further development. Out of the reach of the Top Secret stamp.

We want bad ideas!

[narcc]

We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers.

Sounds like a pretty damning indictment.

Re:We want bad ideas!

[Anonymous Coward]

Makes perfect sense. You research what won't work, then you tell your enemies that it will work, and watch them waste their time trying to make it work.

Re:We want bad ideas!

[michelcolman]

And then they actually do make it work and you lose your job.

Re:We want bad ideas!

[Registered Coward v2]

We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers.

Sounds like a pretty damning indictment.

It actually has at least a couple of advantages. The sooner you learn what doesn't work the quicker you avoid sinking vast sums of money into trying to make it work. If an adversary is working on it you can be assured it is a waste of time and money they could spend on something that might actually work. Of course, sometimes people are wrong about what won't work because they give up to soon; which is the downside of asking what won't work.

Or RTFA...

[Anonymous Coward]

"You don’t know me, but I know you," Smith told him. "I work for the System Development Corporation. We want to fund long-range speculative research. We’re particularly interested in research that either won’t work or, if it does work, won’t work for a long time. And I’ve been reading some of your papers."

Hinton won $350,000 from this mysterious group. He later learned its origins: It was a subsidiary of the nonprofit RAND Corporation that had ended up making millions in profit by writing software for nuclear missile strikes. The government caught them, and said they could either pay up or give the money away—fast. The grant made Hinton a much more palatable hire in academe.

Ha

[Anonymous Coward]

No mystery caller was responsible for neural nets taking off. Computers exist to compute as extensions of ourselves, a neural net is the way to extend more of ourselves into the computational system. Saying "neural nets wouldn't exist if x didn't call y in the middle of the night" is a bit like saying "the if statement wouldn't exist if the orignal person to think of the word 'if' didn't exist" - it filled a role so it was a natural advancement and the stranger thing would be it not existing.

Re:Ha

[TapeCutter]

Skimmed the article, conspiratorial themes aside, it seems like a good general history of neural nets.

To answer what I see as the main question in TFA - Here's the difference "this time around".

I've been interested in AI and automata since the early 80's, sporadically following closely over the years. Life distracted me from this interest for most of the noughties. The first time I watched IBM's Jeopardy stunt with Watson I was blown away, the missus shrugged and said "It's impressive but what's the big deal, it's just looking up the answers, like google with talking, right?" I tried to explain why my jaw was on the floor, but all I got was a blank look and a change of subject.

Far from being overhyped I think the general public simply don't comprehend the significance of these developments. They see it as 'hype' because like my missus they simply don't comprehend the problem and tend to grossly underestimate the difficulty of solving it. IMO the Watson stunt is one of the most significant technological feats I've witnessed since the moon landings, and possibly the start of a new Apollo style arms race based on the same old fears. That doesn't mean I think all the problems in AI have been solved, but machines like Watson are very strong evidence that we have recently cleared a significant hurdle (that few in the general public have even noticed).

To me, this period in AI is very reminiscent of where digital comms were in the early 90's. Most of the bits for the comms revolution existed but rarely talked to each other; pagers, email, mobile phones, computers, printers, fax, GPS, fibre optics, etc. Just a few years later everyone was talking about "convergence", "as foretold" pretty much all of those things and more have now converged into the ubiquitous smart phone. In 1990, virtually nobody on the planet saw the internet coming (including me), I was at Uni, mature age CS/Math student, 88-91. I was perfectly placed in space and time to see it born but didn't notice it.

I first heard about HTML and Mosaic at Uni, one of our CS lectures was very impressed and went on a tangential rant about it one day in a networking lecture. Still, nobody in his hijacked audience I talked to afterwards could figure out why he was so impressed. "What's wrong with zmodem?" was a typical comment that I would have agreed with then.

I think we are more or less at that "1990" point where everyone will soon start talking more and more about "convergence" in AI. The Watson that won Jeopardy in 2011(?) required 20 tons of air-conditioning alone, today an instance of Watson fits on a "pizza box" server and you can try out your own Watson instance for free with a web based developer's API (google it). Their goal is to squeeze Watson into a smart phone.

A couple of things that a Watson style AI may "converge" with aside from phones are, "Big Dog" which has pretty much solved the autonomous movement/balance problem, and face recognition software which has also made big strides in the past few years. What the end result will be when it all converges and evolves, or even when it will converge, I have no idea, but a dystopian SkyNet style future is no longer purely fiction. From a less pessimistic POV, AI could serve as a "check and balance" in a democracy full of bullshitters, a tool to fact check the waffle and make evidence based, transparent, recommendations on public policy free from partisan politics, in other words "speak truth to power", like the public service in a democracy is supposed to be doing now.

Disclaimer: The "missus" is far from dumb, she has a Phd in Business and Marketing, she lectures to several hundred students at a time. I sometimes fail to see why she is interested/impressed by some obscure event in the Business News and politely change the subject :)

Re:Ha

[kaiser423]

Yea, I'm with you on people not getting it. I wanted to show people a picture of a shed that I had taken a couple of months back. It was buried under hundreds of photos, so was hard to find. I just punched in "shed" into the Google Photos search for my photo collection and low and behold dozens of pictures of different types of sheds in different angles all showed up in my search results. Typing in "brown shed" filtered it down to brown, and then "light brown shed" gave me just the light brown shed, which is what I was looking for.

I was pretty damn impressed, and pretty much every common day object I searched for, I got pictures back with that in it, very accurately. Screw manually tagging pictures (all the rage a couple of years ago), a computer just goes through and classifies them. Everyone else didn't see it as a big deal -- "so what, they figured out a light brown shed". Without really realizing the sheer amount of computing horsepower and sophistication that went into something like that.

Re:Ha

[jythie]

What does being in the computer industry for 20 years have to do with knowing the history of CS? Very few CS programs have any coursework on history, and it is not a topic that is highly covered in industry. Working in tech might give one a default knowledge of products and technologies developed during their time in industry, but even that is going to be limited to a person's specific subdomain unless they are actually independently interested in learning deeper history.

Re: Ha

[Namarrgon]

it's just a semantic matching algorithm with a really big computer to power it

And it achieved a milestone of human-scale lookup and response that we'd never seen before. It's still an impressive feat, even as a powered-up refinement of existing techniques and just another step on the road.

As for the brain, while the complexity gets huger the deeper we look, there's an excellent chance we simply don't need to build that level of detail ourselves to get useful results. We already have useful AI with much less, and it's getting more impressive every day.

I'm guessing we'll have non-sentient AI that can do human-level tasks and interaction long before we emulate the whole brain completely, just like we have useful and cheap flight without needing to build a whole bird.

Re:Ha

[TapeCutter]

I understand how it works, that's why I was so impressed. What they (and others) have done in total is solve a long standing problem with NN's, their tendency to be single minded, ie: you train it to recognise cats then train it to recognise dogs, you end up with something that recognises dogs and non-dogs but has forgotten what it knew about cats. The hint is in the name "deep learning".

As for a "huge computer" Watson now knows a lot more than the original and runs on a commodity rack mounted server. Agree, prosthetics is where AI will converge with the human mind, again the technological bits and pieces are already in use, but still very much isolated from each other.

If you define AI to be the replication of human intelligence then it will never arrive except via birth and environment. IMO, it's a very narrow definition and not particularly useful since we presumably all posses our own human like intelligence. No matter how you slice it, it was a major milestone when an AI defeated the best humans in an unbounded problem space where humans excel, such as Jeopardy.

I guess it would be cooler before I knew how it worked but I was playing with ANNs on a smaller scale well before Watson came about.

Ditto, I taught myself programming in the early 80's because playing Conway's game of life on graph paper was tedious. Sure, by definition knowledge removes the mystery but to paraphrase Feynman "Knowledge can only add to the awe and beauty of a flower, I don't understand how it can detract"

Do they actually work well now?

[Pinky's Brain]

Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.

Re:Do they actually work well now?

[CanarDuck]

Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.

Is it possible that last time you checked was a long time ago? Deep neural networks are again all the rage now (i.e. huge teams working with them at Facebook and Google) because

1. (1) They have resulted in a significant performance improvement over previously state-of-the-art algorithms in many application tasks,
2. (2) Although they are computation-heavy, they are amenable to massive parallelization (modern computational power is probably the main reason why they have improved singificantly with respect to ANNs of the 80-90s, given that the main architecture itself has not changed a lot, except possibly for the "convolution" trick which effectively introduces hard-coded localization and spatial invariance).

Check the wikipedia page for "convolutional neural networks" as well as other /. entries: http://slashdot.org/tag/deeple... , and from yesterday http://tech.slashdot.org/story... .

Re:Do they actually work well now?

[lorinc]

Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.

Is it possible that last time you checked was a long time ago? Deep neural networks are again all the rage now (i.e. huge teams working with them at Facebook and Google) because

1. (1) They have resulted in a significant performance improvement over previously state-of-the-art algorithms in many application tasks,
2. (2) Although they are computation-heavy, they are amenable to massive parallelization (modern computational power is probably the main reason why they have improved singificantly with respect to ANNs of the 80-90s, given that the main architecture itself has not changed a lot, except possibly for the "convolution" trick which effectively introduces hard-coded localization and spatial invariance).

To be fair, it always seems to me that (1) and (2) are very closely related. CNNs that won recent computer vision benchmarks are the only methods that used so much processing power so far. Not that they're less efficient than other, tough. It's just that I would love to see other methods with that many engineering, tunning, dedicated computational power and how they compare.
Also, not that when it comes to classification, the standard is to throw the last layer and train a linear SVM on the penultimate layer, which also show that CNNs alone are not enough.

Re:Do they actually work well now?

[SpinyNorman]

Nowadays (typically deep, convolutional) neural nets are achieving state of the art (i.e. better than any other technique) results in most perception fields such as image recognition, speech recognition, handwriting recognition. For example, Google/Android speech recognition is now neural net based. Neural networks have recently achieved beyond-human accuracy on a large scale image recognition test (ImageNet - a million images covering thousands of categories including fine-grained ones such a as recognizing various breeds of dog, types of flower, etc).

Re:Do they actually work well now?

[SpinyNorman]

Compute power is only part of the reason for the recent success of neural nets. Other factors include:

- Performance of neural nets increase with the amount of training data you have, almost without limit. Nowadays big datasets are available on the net (plus we have the compute power to handle them).

- We're now able to train deep (multi-layer) nerural nets using backprop whereas it used to be considered almost impossible. It turns out that initialization is critical, as well as various types of data and weight regularization and normalization.

- A variety of training techniques (SGD + momentum, AdaGrad, Nesterov accelerated gradients, etc, etc) have been developed that both accelerate training (large nets can take weeks/months to train) and remove the need for some manual hyperparameter tuning.

- Garbage-In, Garbage Out. You're success in recognition tasks is only going to be as good as the feature representation available to the higher layers of your algorithms (whether conventional or neural net). Another recent advance has been substituting self-learnt feature representations for laboriously hand-designed ones, and the recent there is now a standard neural net recipe of autoencoders+sparsity for implementing this.

- And a whole bunch of other things...

As Newton said "if I have achieved great things it is by standing on the shoulders of giants".. there are all sorts of surprising successes (e.g. language translation) and architectural advances in neural nets that are bringing the whole field up.

These arn't your father's neural nets.

Re:Do they actually work well now?

[ShanghaiBill]

Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ...

This is true, but someone has to write those more efficient algorithms. ANNs learn, and program themselves. Once a ANN has been trained to solve a problem, it can often be trimmed to a minimal implementation, making it more efficient, but no longer trainable.

Re:Do they actually work well now?

[Darinbob]

Neural Networks were the rage when I was in grad school at UCSD, and also genetic algorithms a bit later. A rage across many departments. They did some good work, though there was this attitude that only their favorite methods counted for anything and that more traditional AI was not worth discussing. But you should use all techniques if you can, otherwise it's like trying to build a circuit using only capacitors.

A mysterious call from the future?

[jeffb (2.718)]

Sounds a bit reminiscent of the Eschaton...

Long Range Foundation

[CrimsonAvenger]

This sounds like the LRF from Heinlein's Time for the Stars.

They were required to spend their money researching things whose payback was so far in the future that no-one else would touch it.

And they kept making embarrassing amounts of money as a result of the products of their research. wonder if this lot will do the same?

Both

[Tablizer]

Should they imitate how we imagine the mind to work, as a Cartesian wonderland of logic and abstract thought that could be coded into a programming language? Or should they instead imitate a drastically simplified version of the actual, physical brain, with its web of neurons and axon tails, in the hopes that these networks will enable higher levels of calculation? Itâ(TM)s a dispute that has shaped artificial intelligence for decades.

I suspect to get "true" AI, both of these will have to work together. Neural nets (NN's) will provide hunches and guesses, but the AI will have to model these hunches and guesses in an abstract or semi-realistic way to both test the logic of them, and to be able to communicate with humans about its findings or suggestions.

The AI will be able to "draw" or describe a cartoon-like model of suggestions or events the way a human might in a meeting explaining something about travel, events, human relationships, time-lines, etc. This requires some kind of abstract modelling.

This is pretty much how most human minds work: hunches based on past and/or re-occurring patterns teamed up with abstract modeling at an "object" level to both communicate and test hunches, as created by NN-like pattern matching at a mostly sub-conscience level.

System Development Foundation

[Baldrson]

Its "System Development Foundation" not "System Development Corporation" and Charlie's full name is Charles Sinclair Smith. He's semi-retired now and living the next county over from me in southeast Iowa where we've been collaborating on a couple of projects -- one of which is to photosynthesize all of the CO2 effluent from US fossil fuel power plants (as Charlie got his start co-founding the Energy Information Administration of the DoE under Carter).

Its ironic that in the 80s I was living in La Jolla, which was an epicenter of the neural net revival at UCSD, had taken neural net courses from Robert Hecht-Nielsen and by 1990 had prototyped the highest performance neural network image processing system (as Neural Engines Corporation) -- but I then later worked with Charlie for almost 15 years before discovering he had had played such a key role in the revival of neural nets. Even more ironic is that, circa 2005, I came up with the idea for the Hutter Prize for Lossless Compression of Human Knowledge -- based on Hutter's entirely different, top down mathematics approach to AI -- and Shane Legg, founder of Deep Mind, which is largely identified with deep learning neural nets, actuality studied under Hutter and achieved Deep Mind's famous ability to learn to play video games using Hutter's approach but everyone thinks that capability is uniquely attributable to deep neural net learning alone.

Re:Time to write a neural net worm.

[StonyCreekBare]

Did that! In Chromosome Quest. http://www.chromosomequest.com...