Slashdot Mirror
A.I. Advances Through Deep Learning
An anonymous reader sends this excerpt from the NY Times:
"Advances in an artificial intelligence technology that can recognize patterns offer the possibility of machines that perform human activities like seeing, listening and thinking. ... But what is new in recent months is the growing speed and accuracy of deep-learning programs, often called artificial neural networks or just 'neural nets' for their resemblance to the neural connections in the brain. 'There has been a number of stunning new results with deep-learning methods,' said Yann LeCun, a computer scientist at New York University who did pioneering research in handwriting recognition at Bell Laboratories. 'The kind of jump we are seeing in the accuracy of these systems is very rare indeed.' Artificial intelligence researchers are acutely aware of the dangers of being overly optimistic. ... But recent achievements have impressed a wide spectrum of computer experts. In October, for example, a team of graduate students studying with the University of Toronto computer scientist Geoffrey E. Hinton won the top prize in a contest sponsored by Merck to design software to help find molecules that might lead to new drugs. From a data set describing the chemical structure of 15 different molecules, they used deep-learning software to determine which molecule was most likely to be an effective drug agent."
I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...
A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning.
It's hard to tell from the article, but they probably are trying to refer to Deep Belief Networks, which are a more recent and advanced type of Neural Network, which incorporates many layers:
Deep belief nets are probabilistic generative models that are composed of multiple layers of stochastic, latent variables. The latent variables typically have binary values and are often called hidden units or feature detectors. The top two layers have undirected, symmetric connections between them and form an associative memory. The lower layers receive top-down, directed connections from the layer above. The states of the units in the lowest layer represent a data vector.
I wonder how much of these improvements in accuracy are due to fundamental advances
I was wondering the same thing, and just now found this interview on Google. Perhaps someone can fill in the details.
But basically, machine learning is at its heart hill-climbing on a multi-dimensional landscape, with various tricks thrown in to avoid local maxima. Usually, humans detemine the dimensions to search on -- these are called the "features". Well, philosophically, everything is ultimately created by humans because humans built the computers, but the holy grail is to minimize human invovlement -- "unsupervised learning". According to the interview, this one particular team (the one mentioned at the end of the Slashdot summary) actually rode the bicycle with no hands and to demonstrate how strong their neural network was at determining its own features, did not guide it, even though it meant their also-excellent conventional machine learning at the end of the process would be handicapped.
The last time I looked at neural networks was circa 1990, so perhaps someone writing to an audience more technically literate than the New York Times general audience could fill in the details for us on how a neural network can create features.
A lot of vague marketing-speak in this article. "Deep learning"?
Agreed. Why do we need the adjective "deep"? Perhaps it's because a lot of AI jargon uses "learning" when they really just mean "adaptive" (as in, "programmed to respond to novel stimuli in anticipated ways"), whereas normal human "learning" is much more fluid.
The article basically talks about neural networks
Yet another victory for marketing. These things have been around for at least 25-30 years, and the connection to what little we actually have deciphered about how the brain encodes, decodes, and processes information has always been incredibly tenuous. There always seems to be these AI strands of "cognitive science" or "neural modeling," which are often nothing than just somebody's pet algorithm or black box dressed up with words that make it sound like it has some scientific basis in actual neurophysiology or something.
Don't get me wrong -- I'm sure some of the examples in TFA have made great advances, partly due to speed and hardware unthinkable 25-30 years ago. And some of the functionality of the "neural nets" might give significantly better results than previous models.
But I really wish people would lay off the pretend connections to humanity. Why can't we just accept that a machine might just function better with a better program or algorithm or whatever, rather than saying that "our research in cognitive science [i.e., BS philosophy of the mind] has resulted in neural networks [i.e., a mathematical model instantiated into programming constructs] that exhibit deep learning [i.e., work better than the previous crap]."
(Please note: I mean no insult to anyone who works in neuroscience or AI or whatever. But I do question the jargon that seems to make unfounded connections and assumptions that the brain works anything like many algorithmic "models." We may succeed in creating artificial intelligence by developing our own algorithms or we might succeed by imitating the brain, but I don't think we're making progress by pretending that we're imitating the brain when we're really just using marketing jargon for our pet mathematical algorithm.)
yet?
Humans can't even solve those, lol.
We need to open all the documentation for everyone who want to learn and investigate about IA.
Absolutely. It's about time we figured out who really won those caucuses -- and what the heck is up with the ethanol subsidies?
Advances are in ways of learning hidden layers that are slightly more clever than backpropagation. For example, lets say you have an image, apply some transform to it (dct, wavelet, whatever, neural net layer, etc.) and save all the important features, but at say 10x less space. Then do the same to those features. Every time reducing the amount of data by 10x. After a few such layers, lets say you're left with 10 bits worth of information---the ``most important'' (according to your benchmark used) ten bits of the whole image.
The ten bits could be anything, such as `this image is a car' or `this image is a face', or ``this face looks angry', etc.
The trick is applying the benchmark on the hidden layers---e.g. how do you pick out which features are important after applying a transform. For that, you train another (inverse) transform that recovers original data from the features---the one that gets you closest to the original wins (e.g. lets say you feed 1000 bits into a neural net to get 100 bits out, and then via inverse transform turn those 100 bits into the *original* 1000 bits... that would mean that your 100 bits represented all the information in the input 1000 bits---obviously more often than not you won't get a perfect match but something close---repeat for any number of layers you want).
"If anything can go wrong, it will." - Murphy
I'm doing Prof Hinton course on Neural Network on Coursera this semester. It covers the old school stuff plus the latest and greatest. From what I gather from the lecture, training neural networks using lots of layers hasn't been practical in the past and was plauged with numerical and computational difficulties. Nowadays, we have better algorithms and much faster hardware. As a result we now have the ability to use more complex networks for modelling data. However, they need a lot of computational power thrown at them to learn compared to other machine learning algorithms (random forest). The lecture quotes training taking days on a Nvidia GTX 295 GPU to learn the MNIST handwritten dataset. Despite this, the big names are already using this technology for applications like speech recognition (Microsoft, Siri), object recognition (Google Cat video, okay that's not a real application yet).
Why do we need the adjective "deep"?
Because the "deep learning" technologies use artificial neural networks with many more layers than traditionally, making them "deep architectures".
It's widely accepted that the first hidden layer of an ANN serves as a feature detector (possibly sub-symoblic features that you can't put a name to), and each successive layer serves as a detector for higher-order features. Thus the deep architectures can be expected to have some utility for any problem that depends on feature analysis.
Sheesh, evil *and* a jerk. -- Jade
I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning," though, in the broader world... and even if we equate the jargony connotations of "machine learning" with "learning," it still seems a stretch to use "deep" as an adjective directly applied to that... but perhaps it's just me.
They haven't done anything that wasn't already being done by others. They're just doing more of it. Essentially, the approach consist of using Bayesian statistics and a hierarchy of patterns. Prof. Hinton pretty much pioneered the use of Bayesian statistics in artificial intelligence. With a rare notable exception (e.g. Judea Pearl), the entire AI community has jumped on the Bayesian bandwagon, not unlike the way they jumped on the symbolic bandwagon in the latter half the 20th century, only to be proven wrong fifty years later.
The Bayesian model essentially assumes that the world is inherently probabilistic and that the job of an intelligent system is to discover the probabilities. A competing model (see links below), by contrast, assumes that the world is perfectly consistent and that the job of an intelligent system is to capture this perfection.
See The Myth of the Bayesian Brain and The Second Great AI Red Herring Chase if you're interested in an alternative approach to AI.
It looks like you are seeing something that is not there. The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works.
As someone who has read a lot of the founding literature of modern cognitive science and the philosophy of mind in the 1950s through 80s, which was hugely influential in setting up the early approaches to AI (including neural nets), I have to say -- this is where the stuff came from.
And frankly, a lot of applications in more obscure disciplines, such as in AI analysis in the humanities, researchers are still making claims about these models and their relationships to the actual brain. Hell, just a few years ago I heard a leading cognitive scientist claim that he found evidence for a sort of musical "circle of fifths" neural network in an actual circular physical structure of neurons in the brain... a made-up musical model grafted onto a made-up AI brain model, supported by noisy data... I admit this is an extreme example, but it's not unique.
I understand that modern researchers in "pure" AI may want to avoid recognizing the history or the implications of the terminology -- but there's a reason why the Starship Voyager was equipped with "neural gel-packs" that could get anxious and cause a warp-core breach at a temporal anomaly... words like "neural" actually mean something, and these "neural nets" have about as much connection to the biological function of actual neurons as Voyager's bizarre "neural gel-packs." Yet the implicit metaphor made in continuing to use the term should not be underestimated, not just in a general audience NYT article, but in the way fields are subtly shaped by their nomenclature.
In the past few years, a few things happened almost simultaneously:
1. New algorithms were invented for training of what previously was considered nearly impossible to train (biologically inspired recurrent neural networks, large, multilayer networks with tons of parameters, sigmoid belief networks, very large stacked restricted Boltzmann machines, etc).
2. Unlike before, there's now a resurgence of _probabilistic_ neural nets and unsupervised, energy-based models. This means you can have a very large multilayer net (not unlike e.g. visual cortex) figure out the features it needs to use _all on its own_, and then apply discriminative learning on top of those features. This is how Google recognized cats in Youtube videos.
3. Scientists have learned new ways to apply GPUs and large clusters of conventional computers. By "large" here I mean tens of thousands of cores, and week-long training cycles (during which some of the machines will die, without killing the training procedure).
4. These new methods do not require as much data as the old, and have far greater expressive power. Unsurprisingly, they are also, as a rule, far more complex and computationally intensive, especially during training.
As a result of this, HUGE gains were made in such "difficult" areas as object recognition in images, speech recognition, handwritten text (not just digits!) recognition, and in many more. And so far, there's no slowdown in sight. Some of these advances were made in the last month or two, BTW, so we're speaking about very recent events.
That said, a lot of challenges remain. Even today's large nets don't have the expressive power of even a small fraction of the brain, and moreover, the training at "brain" scale would be prohibitively expensive, and it's not even clear if it would work in the end. That said, neural nets (and DBNs) are again an area of very active research right now, with some brilliant minds trying to find answers to the fundamental questions.
If this momentum is maintained, and challenges are overcome, we could see machines getting A LOT smarter than they are today, surpassing human accuracy on a lot more of the tasks. They already do handwritten digit recognition and facial recognition better than humans.
I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning," though, in the broader world... and even if we equate the jargony connotations of "machine learning" with "learning," it still seems a stretch to use "deep" as an adjective directly applied to that... but perhaps it's just me.
I have a bigger issue with "learning" than with "deep", since with very few exceptions ANNs don't learn anything autonomously, but rather are adjusted by an external algorithm to to perform well on a given problem. "Deep training" would make sense for "deep architectures".
Sheesh, evil *and* a jerk. -- Jade
This is a very misleading metric. First, some not-insignificant number of the neurons in the brain are involved in non-cognitive computations. Muscle control, hormone regulation, kinesthesia, vision (not thinking about what is seen, but simply recognizing it), heart rates and other system regulation and so on.
Examples also exist of low-neuron (and synapse) count individuals who retain cognitive (and all other major) function; these examples cannot be explained away by "counting neurons."
We don't know which yet, but given that high neuron count has been ruled out as the single way to accommodate intelligence, we do know that we need to look to other mechanisms for human cognition. Structure, algorithm, other features known or unknown may be responsible for intelligence; and it may be that something entirely disjoint is responsible for the rise of intelligence; but we know it isn't simply high neuron count.
--fyngyrz (anon due to mod points)
Actually, it seems your post is the vague one. "normal human "learning" is much more fluid." What does that mean?
Many machine learning algorithms "learn" exactly the way you'd teach a child. They see examples, you tell them what the object, word, etc. is, and they remember that answer imperfectly. Repetition improves their accuracy and a breadth of examples improves their generality. After not seeing something for a while, they may forget it.
As the other poster pointed out, "deep" describes algorithms that are better able to teach multi-level systems. The changes associated with learning are better propagated to deeper levels, better utilizing all the capacity of the system.
No, it's not just you. There are a lot of people who see the brain as the last bastion of their identity as some kind of special and privileged creature, therefore it must be magical and any attempts to explain how it works are misguided, childish and silly. Whether that's your actual belief or not, that's what your post sounds like. Modern computational neuroscience has actually come a long way. We're even capable of producing chips that can be implanted and replace some parts of the brain. It's not magic.
Here is a good video of a talk given by Dr. Hinton about Restricted Boltzman Machines. It is a very promising technique for deep learning strategies.
'The tyrant will always find pretext for his tyranny.' - Aesop's Fables
"The term "deep" comes from the idea that the algorithm is trying to learn something deeper than previous algorithms. In fact, the usual set of machine learning algorithms are termed shallow learning now. The difference is that deep learning tries to model P(X) whereas shallow learning (SVM, NN, naive Bayes, etc..) try to learn P(X|Y) where X is your input space and Y is the label space. "
Well, more correctly, the deep learning tries to model P(X) as P(X | H) for some set of "hidden" or latent features H which in some ways, is far simpler than the raw data and space of X, and then learns P(Y | H, X) after doing some training for P(X|H).
"The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works."
This isn't true---or the connotation of it isn't true. I don't know how to quantify "majority", but there is substantial interest in computational modeling of actual biology across all levels of biological/chemistry fidelity and attention to engineering and statistical problems.
A glance at the work in the journal _Neural Computation_ shows papers both on entirely theoretical statistical computational models and models much more closely tied to experimental neuroscience results.
Many people want to do both: derive useful methods for solving engineering problems and understand biological systems which have shown such abilities. The field is very difficult and deep.