Slashdot Mirror
The Believers: Behind the Rise of Neural Nets
An anonymous reader writes Deep learning is dominating the news these days, but it's quite possible the field could have died if not for a mysterious call that Geoff Hinton, now at Google, got one night in the 1980s: "You don't know me, but I know you," the mystery man said. "I work for the System Development Corporation. We want to fund long-range speculative research. We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers." The Chronicle of Higher Ed has a readable profile of the minds behind neural nets, from Rosenblatt to Hassabis, told primarily through Hinton's career.
"You don't know me, but I know you," the mystery man said.
We call them "NSA" now.
We're particularly interested in research that either won't work or, if it does work, won't work for a long time. And I've been reading some of your papers.
Sounds like a pretty damning indictment.
Required reading for internet skeptics
"You don’t know me, but I know you," Smith told him. "I work for the System Development Corporation. We want to fund long-range speculative research. We’re particularly interested in research that either won’t work or, if it does work, won’t work for a long time. And I’ve been reading some of your papers."
Hinton won $350,000 from this mysterious group. He later learned its origins: It was a subsidiary of the nonprofit RAND Corporation that had ended up making millions in profit by writing software for nuclear missile strikes. The government caught them, and said they could either pay up or give the money away—fast. The grant made Hinton a much more palatable hire in academe.
No mystery caller was responsible for neural nets taking off. Computers exist to compute as extensions of ourselves, a neural net is the way to extend more of ourselves into the computational system. Saying "neural nets wouldn't exist if x didn't call y in the middle of the night" is a bit like saying "the if statement wouldn't exist if the orignal person to think of the word 'if' didn't exist" - it filled a role so it was a natural advancement and the stranger thing would be it not existing.
Last time I looked there was no application of ANNs which couldn't be solved more efficiently by other algorithms ... and the best ANNs used spiking neurons with Hebbian learning which are not amenable to efficient digital implementation.
What did the machine know? How did it learn? A gap broader than any we’ve known has opened between our use of technology and our understanding of it. How did the machine work? As I would discover, no one could say for certain.
The ones who built the machine?
Sounds a bit reminiscent of the Eschaton...
This sounds like the LRF from Heinlein's Time for the Stars.
They were required to spend their money researching things whose payback was so far in the future that no-one else would touch it.
And they kept making embarrassing amounts of money as a result of the products of their research. wonder if this lot will do the same?
"I do not agree with what you say, but I will defend to the death your right to say it"
I suspect to get "true" AI, both of these will have to work together. Neural nets (NN's) will provide hunches and guesses, but the AI will have to model these hunches and guesses in an abstract or semi-realistic way to both test the logic of them, and to be able to communicate with humans about its findings or suggestions.
The AI will be able to "draw" or describe a cartoon-like model of suggestions or events the way a human might in a meeting explaining something about travel, events, human relationships, time-lines, etc. This requires some kind of abstract modelling.
This is pretty much how most human minds work: hunches based on past and/or re-occurring patterns teamed up with abstract modeling at an "object" level to both communicate and test hunches, as created by NN-like pattern matching at a mostly sub-conscience level.
Table-ized A.I.
The developers hand-craft layers and then put them together and do more "tuning", so deep neural nets really aren't good examples of unsupervised learning.
In fact they're much like Watson: hand-crafted to the nth degree, except for Watson you can debug it more easily by tracing code. With deep NN you're tuning parameters for this case or that.
I don't know about you but if machines ever did exhibit AI and was connected to the net I would be obligated to infect them with a worm. I experimented with several neural net algorithms several years ago on an EFICA single board PPC computer. Specifically I configured it to parse logs. The ultimate goal was to build a smart active firewall. When the parser logged "Why do I get shutdown every night ? " to the response.out file I yanked the board off my desk and smashed it with my steel toe boots, made a fire outside and burned it to a crisp. Not making it up.
Its "System Development Foundation" not "System Development Corporation" and Charlie's full name is Charles Sinclair Smith. He's semi-retired now and living the next county over from me in southeast Iowa where we've been collaborating on a couple of projects -- one of which is to photosynthesize all of the CO2 effluent from US fossil fuel power plants (as Charlie got his start co-founding the Energy Information Administration of the DoE under Carter).
Its ironic that in the 80s I was living in La Jolla, which was an epicenter of the neural net revival at UCSD, had taken neural net courses from Robert Hecht-Nielsen and by 1990 had prototyped the highest performance neural network image processing system (as Neural Engines Corporation) -- but I then later worked with Charlie for almost 15 years before discovering he had had played such a key role in the revival of neural nets. Even more ironic is that, circa 2005, I came up with the idea for the Hutter Prize for Lossless Compression of Human Knowledge -- based on Hutter's entirely different, top down mathematics approach to AI -- and Shane Legg, founder of Deep Mind, which is largely identified with deep learning neural nets, actuality studied under Hutter and achieved Deep Mind's famous ability to learn to play video games using Hutter's approach but everyone thinks that capability is uniquely attributable to deep neural net learning alone.
Seastead this.
They get some results, they do, but the results they get are not provably unobtainable through other means, i.e. statistical analysis or regular old GOFAI. No one can say *why* or *how* they solved any particular problem- specifically, there is no identifiable algorithm that they're responsible for discovering or using. They get results through a combination of small tweaks to their guesses and massive iterations of guesses. When the network is "trained' , i.e. producing results, there's not much more than the description of the network (layers, connection architecture, update rules) how they were trained (number of training exemplars, number of iterations and their respective characteristics), and their final weights between nodes that serves as a step by step procedure on how to solve the problem they solved. It ain't much.
What's more, you can't look at one and know if it will successfully solve the next problem or not. It can fail, again for reasons unknown, at any time .
In other words, they aren't algorithms or computer programs in the normal analytic sense of those words and more to the point, they're not subject to analytical methods of understanding.
I have never been impressed, really. It's true that many hard problems have given way to statistical methods of analysis, Google's translation engine being one, but there we have a good grasp of the math behind the stats and we can reason about the program at least as a hypothesis testing machine. With NN, i'ts really all just a black box of unknown quality . Do you trust it? Do you trust it make the right Big Decision in context X? I sure as fuck don't. To the extent that it produces anything useful, it's really just applied statistics through other (and probably better) means
Looked at in that way, it's really a measure of the slight advantage just throwing some shit against the wall has wrt to coming to understand the chemistry behind glues. Hey, look, some of this shit is sticky. Yep. And? And? And now we know what exactly?