A.I. Advances Through Deep Learning

An anonymous reader sends this excerpt from the NY Times: "Advances in an artificial intelligence technology that can recognize patterns offer the possibility of machines that perform human activities like seeing, listening and thinking. ... But what is new in recent months is the growing speed and accuracy of deep-learning programs, often called artificial neural networks or just 'neural nets' for their resemblance to the neural connections in the brain. 'There has been a number of stunning new results with deep-learning methods,' said Yann LeCun, a computer scientist at New York University who did pioneering research in handwriting recognition at Bell Laboratories. 'The kind of jump we are seeing in the accuracy of these systems is very rare indeed.' Artificial intelligence researchers are acutely aware of the dangers of being overly optimistic. ... But recent achievements have impressed a wide spectrum of computer experts. In October, for example, a team of graduate students studying with the University of Toronto computer scientist Geoffrey E. Hinton won the top prize in a contest sponsored by Merck to design software to help find molecules that might lead to new drugs. From a data set describing the chemical structure of 15 different molecules, they used deep-learning software to determine which molecule was most likely to be an effective drug agent."

Comments

It's the dawn of the roboapocalypse

[Anonymous Coward]

Take cover while you can!

Re:It's the dawn of the roboapocalypse

[Anonymous Coward]

Jeff Hawkins will be the first against the wall.

Go die you sellout to all that is evil.

Sources of improvements?

[drooling-dog]

I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...

Re:Sources of improvements?

[PlusFiveTroll]

from TFA

" Modern artificial neural networks are composed of an array of software components, divided into inputs, hidden layers and outputs. The arrays can be “trained” by repeated exposures to recognize patterns like images or sounds.

These techniques, aided by the growing speed and power of modern computers, have led to rapid improvements in speech recognition, drug discovery and computer vision. "

Sounds like both.

Re:Sources of improvements?

[xtal]

Computers have gotten very cheap. Pretty much any prof that wants to pursue something now can build enough hardware to do so with a relatively small amount of money. Neural networks ran into a big wall twenty years ago because the tools weren't there yet.

Once people start having some successes, more funds will be made available, more advances will be made, justifyiing even more funding.. and then we'll turn control of the military over to SkyNet. :)

Re:Sources of improvements?

[Anonymous Coward]

Don't forget that it's not impossible to build a specially designed processor to do a particular task; such as the digital orrery. Such devices created to do nothing but neural net simulations would be more efficient than using a general purpose computer. It would be linked to such to provide a convenient interface but do most of the heavy lifting itself.

Re:Sources of improvements?

[iggymanz]

no, that first sentence pretty much sums up digital neural nets over two decades ago. So more likely the over two orders magnitude processing power per chip improvement since then, with addressable memory over three orders magnitude bigger....

Re:Sources of improvements?

[tirerim]

from TFA

" Modern artificial neural networks are composed of an array of software components, divided into inputs, hidden layers and outputs. The arrays can be “trained” by repeated exposures to recognize patterns like images or sounds.

These techniques, aided by the growing speed and power of modern computers, have led to rapid improvements in speech recognition, drug discovery and computer vision. "

Sounds like both.

Well, that doesn't say anything; that just described every neural network for the past couple of decades, except for the "rapid improvement" part. I haven't read TFA, so I don't know if there's more detail, but just describing the basics of how neural networks operate isn't an explanation for why they're suddenly improving.

Re:Sources of improvements?

[Anonymous Coward]

Glad they were able to make it work so quick, but drug discovery has been done like this for over a decade. I worked at an "Infomesa" startup that was doing this in Santa Fe in 2000.

Re:Sources of improvements?

[Prof.Phreak]

The ``new'' (e.g. last decade or so) advances are in training hidden layers of neural networks. Kinda like peeling an onion, each layer getting progressively coarser representation of the problem. e.g. if you have 1000000 inputs, and after a few layers, only have 100 hidden nodes, those 100 nodes are in essence representing all the ``important'' (some benchmark you choose) information of those 1000000 inputs.

Re:Sources of improvements?

[PlusFiveTroll]

Article didn't say, but if I had to make a guess, this is where I would start.

http://www.neurdon.com/2010/10/27/biologically-realistic-neural-models-on-gpu/
"The maximal speedup of GPU implementation over dual CPU implementation was 41-fold for the network size of 15000 neurons."

This was done on cards 7 years old now. The massive increase of power in GPUs in the past few years along with more features and better programing languages for them means the performance increase could possibly be many hundreds of times faster. An entire cluster of servers gets crunched down in to one card, multiple cards in one server, and build a cluster of those and you can quickly see that amount of computing power available to neural networks is much much larger now. I'm not even sure how to compare the GT6800 to a modern GTX680 because of their huge differences, but the 6800 did 54 FLOPs and the 680 does 3090.4. A 57x increase. CPU's how far back to we have to go where CPUs are 57 times slower. If everything scales the same in the papers calculations it would mean over a 2000x performance increase on a single computer with 1 GPU. In 7 years.

Re:Sources of improvements?

[Black Parrot]

I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...

I'm sure all of that helped, but the key ingredient is training mechanisms. Traditionally networks with multiple layers did not train very well, because the standard training mechanism "backpropagates" an error estimate, and it gets very diffuse as at goes backwards. So most of the training happened in the last layer or two.

This changed in 2006 with Hinton's invention of the Restricted Boltzman Machine, and someone else's insight that you can train one layer at a time using auto-associative methods.

"Deep Learning" / "Deep Architectures" has been around since then, so this article doesn't seem like much news. (However, it may be that someone is just now getting the kind of results that they've been expecting for years. Haven't read up on it very much.)

These methods may be giving ANN a third lease on life. Minsky & Papiert almost killed them off with their book on perceptrons in 1969[*], then Support Vector Machines nearly killed them again in the 1990s.

They keep coming back from the grave, presumably because of their phenomenal computational power and function-approximation capabilities.[**]

[*] FWIW, M&P's book shouldn't have done anything, since it was already known that networks of perceptrons don't have the limitations of a single perceptron.

[**] Siegelmann and Sontag put out a couple of papers, in the 1990s I think, showing that (a) you can construct a Turing Machine with an ANN that uses rational numbers for the weights, and (b) using real numbers (real, not floating-point) would give a trans-Turing capability.

Re:Sources of improvements?

[phantomfive]

I think this quote says it all:

Referring to the rapid deep-learning advances made possible by greater computing power, and especially the rise of graphics processors, he added: “The point about this approach is that it scales beautifully. Basically you just need to keep making it bigger and faster, and it will get better. There’s no looking back now.”

I'm sure they've come up with a few incremental advances, but it looks primarily like they've just taken advantage of hardware improvements. You can see from the numbers in the article the results are about what you'd expect from improved hardware (as opposed to actually solving the problem):

[some guy] programmed a cluster of 16,000 computers to train itself to automatically recognize images in a library of 14 million pictures of 20,000 different objects. Although the accuracy rate was low — 15.8 percent — the system did 70 percent better than the most advanced previous one.

Re:Sources of improvements?

[PlusFiveTroll]

Why build a special processor when ATI and Nvidia already do. Probably at a much lower cost per calculation then a custom machine.

Re:Sources of improvements?

[ShanghaiBill]

Why build a special processor when ATI and Nvidia already do. Probably at a much lower cost per calculation then a custom machine.

A GPU can run a neural net much more efficiently than a general purpose CPU, but specialized hardware designed just for NNs could be another order of magnitude more efficient. Of course GPUs are more cost effective because they are mass market items, but if NN applications take off it is likely that everyone will want one running on their cellphone, and then customized NN hardware will be mass market too.

Re:Sources of improvements?

[phantomfive]

using real numbers (real, not floating-point) would give a trans-Turing capability.

What on earth is trans-Turing capability?

Re:Sources of improvements?

[Black Parrot]

using real numbers (real, not floating-point) would give a trans-Turing capability.

What on earth is trans-Turing capability?

Can compute things that a TM can't.

I think the paper was controversial when it first came out, but I'm not aware that anyone has ever refuted their proof.

Re:Sources of improvements?

[timeOday]

You can see from the numbers in the article the results are about what you'd expect from improved hardware (as opposed to actually solving the problem)

"As opposed to actually solving the problem"? You brain has about 86 billion neurons and around 100 trillion synapses. It accounts for 2% of body weight and 20% of energy consumed. Do you think these numbers would be large if they didn't need do be?

I think the emphasis in computer science on focusing so exclusively on polynomial-time algorithms has really stunted it. Maybe most of the essential tasks for staying alive and reproducing don't happen to have efficient solutions, but the constants of proportionality are small enough to brute-force with several trillion neurons.

Re:Sources of improvements?

[Tagged_84]

IBM recently announced success in simulating 2 billion of their custom designed synaptic cores, 1 trillion synapses apparently. Here's the pdf report

Re:Sources of improvements?

[PhamNguyen]

I work in this area. It is mainly the latter, that is bigger data sets and faster hardware. At first, people thought (based on fairly reasonable technical arguments) that deep networks could not be trained with backpropagation (which is the way gradient descent is implemented on neural networks). Now it turns out that with enough data, they can.

On the other hand there have been some theoretical advances by Hinton and others where networks can be trained on unsupervised data (e.g. the Google cats thing).

Re:Sources of improvements?

[HalfFlat]

[...] using real numbers (real, not floating-point) would give a trans-Turing capability.

Given that almost every real number encodes an uncountable number of bits of information, I guess this isn't especially surprising in retrospect. The result though should make us suspicious of the assumption that the physical constants and properties in our physical theories can indeed take any real number value.

Re:Sources of improvements?

[smallfries]

The problem comes when you try larger inputs. Regardless of constant factors if you are playing with O(2^n) algorithms then n will not increase above about 30. If you start looking at really weird stuff (optimal circuit design and layout) then the core algorithms are O(2^2^n) and then if you are really lucky n will reach 5. Back in the 80s it only went to 4, buts thats Moore's law for you.

Re:Sources of improvements?

[snarkh]

> (b) using real numbers (real, not floating-point) would give a trans-Turing capability.

Not sure what it means -- a Turing machine is not even capable of storing a single (arbitrary) real number.

Re:Sources of improvements?

[Anonymous Coward]

It's everything together: more data, better computer performance and better algorithms.

One idea you can use is to take your training set, add random noise to it and then train again, add different random noise and train again and so on. You can also exploit symmetries of the task to generate extra data - for image recognition of things that are still recognizable when mirrorred and/or rotated, you can mirror and rotate the input to generate extra data. You can also zoom in and out on the input pictures to teach the network to ignore scale.

A more recent advance is to randomly disable nodes in the network while training. The effect of that is to defeat over-training and also it improves performance because it makes the rest of the network more resilient to errors - there will be several redundant ways that the network recognizes something, which means it is now more able to recognize that thing when you use it without disabling any nodes.

Another thing you can do is to do unsupervised learning with neural nets. Normally you have to know what the correct output is for each input in order to train a neural net. So if you want the neural net to learn to recognize images, then you have to give it a lot of pictures annotated with what is in those pictures. However, it is much easier to get terabytes of images than it is to annotate those images. Same thing with speech recognition. So what you do is that you train the network on the unannotated data. It will have no idea what anything is, but what you train it to do is to get an idea of what images or speech looks like in general. After that you can train on a much smaller annotated training set and now the network will perform better because it already knows what to look for in pictures in general. More precisely, imagine running the network in reverse, so outputs become inputs and vice versa - now apply random input to get the network to generate an image. In this way you create a probability distribution over all images - how likely are they to turn up in this process? For unsupervised learning, you train the network to give a larger probability to pictures in the training set than to random pictures or to random noise. The outcome is that you can make good use of a huge set of unannotated data as long as it is accompanied by a much smaller set of annotated data. For example it is now helpful to trawl the net for random pictures without knowing what they are pictures of, while before that was not helpful at all.

Then there are advances in algorithms for setting up a plausible initial set of weights for the network and ideas for how to wire the network up. There are also algorithms that allow training networks on a GPU which is much faster.

This is just what I'm aware of without having read any papers or books on the subject and without using any of this stuff for anything, so I'm sure there is a lot more than just this.

Re:Sources of improvements?

[K. S. Kyosuke]

I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...

There are limits to what you can achieve with that. I was once surprised to discover how often I actually mishear words (when watching, e.g., episodes of US TV series) and no amount of repeating helps me. After thinking about it for a while, it became apparent to me that I actually interpolate based on the context. This, however, requires understanding what the particular speech is about. The same goes for reading badly printed or (more often) badly scanned text - quite often I reconstruct the word based on actual understanding of the discourse around the gap. I don't think the provisions you've posited here can contribute to that in any way.

Re:Sources of improvements?

[Anonymous Coward]

The way they are trained is very different, and it's this change that improves the performance. It's more than just making them faster, a fast idiot is still an idiot.

Re:Sources of improvements?

[Anonymous Coward]

Robo-Bush for Prezident!

Re:Sources of improvements?

[maxwell demon]

Given that almost every real number encodes an uncountable number of bits of information, I guess this isn't especially surprising in retrospect. The result though should make us suspicious of the assumption that the physical constants and properties in our physical theories can indeed take any real number value.

The number of bits needed to represent an arbitrary real number exactly is infinite, but not uncountable.

Re:Sources of improvements?

[Anonymous Coward]

These methods may be giving ANN a third lease on life. Minsky & Papiert almost killed them off with their book on perceptrons in 1969[*], then Support Vector Machines nearly killed them again in the 1990s.

Aren't support vector machines provably more powerful than ANNs?

Re:Sources of improvements?

[aaaaaaargh!]

He meant that an ANN with real numbers is a hypercomputer, which is true.

The problem is that like most conceivable hypercomputers neural networks with real numbers would violate natural laws, e.g. the laws of thermodynamics.

Re:Sources of improvements?

[TheTurtlesMoves]

In reality or in the physical. It get quantum at some point. So even with zero noise any real parameter has finite bits for "perfect" representation. Then there is the noise issue. Real system don't match perfect math.

Re:Sources of improvements?

[TheTurtlesMoves]

How so? The math of thermodynamics uses real numbers and does not need any "tricks" to make it work.

Re:Sources of improvements?

[Anonymous Coward]

FYI, the 80487 (1987) had about 1 mflop/s, the GTX-690 has about 5 tflop/s, which is a factor of 5'000'000x faster, or 6.5 orders of magnitude.

Re:Sources of improvements?

[HalfFlat]

Indeed you are right.

Re:Sources of improvements?

[snarkh]

Well, real numbers are inherently very problematic from the computational point of view.

Re:Sources of improvements?

[snarkh]

>Aren't support vector machines provably more powerful than ANNs?

In a sense yes. Both (non-linear) SVM's and Neural Nets are universal approximators. However, SVM's can be shown to converge to the ground truth given sufficiently many observations. No such result exists for neural networks. zz

Re:Sources of improvements?

[Anonymous Coward]

A garden snail has about 20,000 neurons, a cat has 1 billion neurons, a human has 86 billion neurons.

http://www.guardian.co.uk/science/blog/2012/feb/28/how-many-neurons-human-brain

Re:Sources of improvements?

[mikael]

Everything ran into a big wall 20 years ago. There were 680x0, DEC Alpha and SPARC systems, but they were either $10,000 workstations (with no disk drive, server or monitor for the price) or there were embedded systems requiring a rack chassis development kit (manuals cost extra).

Image processing on a PC CPU (= 80386) had to be implemented as a script of image processing command line functions as it wasn't even possible to reliably allocate more than one 64K block. You would load the image in line by line, apply a DFFT, write out the image line by line, flip the image across the major diagonal, then repeat the process. Every image processing function would have to be implemented in this way. General purpose servers were much faster.

Alternative was to use primeval graphics processing boards which had some exotic combination of DSP's and CPU's (Intel i860 CPU, TI32020 DSP, TMS340x0 chip). Some graphics boards at the time actually had a network stack/socket built in so that images could be downloaded straight into video memory and bypass the CPU.

Now any department can buy a cloud server with Terabytes of storage, a couple of PC's with GTX680's, HD webcams, and download free image and video processing software.

Re:Sources of improvements?

[mikael]

I used to do some transcription work to make a bit of spare cash. At the beginning of the tape, I really wouldn't understand the accent, not recognising some words, but after going through the tape once and replaying it, I would immediately recognise the words. It's almost as if there were a set of mask images for every word, and these didn't quite fit at first, but after 20-30 minutes they were scaled, rotated, and transformed in some way until they made a better match. Each word would also have a limited set of other words that would come after it, so that also narrowed down the set of possibilities.

Re:Sources of improvements?

[illaqueate]

Yes, we often interpolate from knowing what is being discussed. We can have algorithms to stand in to some extent but there is a limitation when the inference we make is from a representation of things out there in the world and knowledge about how those things work. We can sometimes get a sense of a conversation from very lossy understanding of what is being said.

Re:Sources of improvements?

[Black Parrot]

[...] using real numbers (real, not floating-point) would give a trans-Turing capability.

Given that almost every real number encodes an uncountable number of bits of information, I guess this isn't especially surprising in retrospect. The result though should make us suspicious of the assumption that the physical constants and properties in our physical theories can indeed take any real number value.

My intuition is that the difference between the TM's finite set of discrete symbols and the infinite/continuous nature of real numbers is exactly the reason.

I'm not aware of any theory of continuous-state computing along the lines of the Chomsky hierarchy, but maybe there's one out there.

Re:Sources of improvements?

[Black Parrot]

How so? The math of thermodynamics uses real numbers and does not need any "tricks" to make it work.

I think there is a theoretical minimal entropy production for any computation, so there's a limit to the amount of computation you could do if you used the entire observable universe.

Of course, you can't have the infinite tape required by a TM either.

Re:Sources of improvements?

[ceoyoyo]

No, "deep learning" refers mostly to new training algorithms. More computer power helps of course, but the problem previously was that your training became less efficient the bigger your system got. If that doesn't happen, you can scale things up indefinitely.

Re:Sources of improvements?

[timeOday]

When you talk about O() you're talking about the worst case for finding an exact solution. Brains don't find exact solutions to anything.

Re:Sources of improvements?

[Anonymous Coward]

Do you think these numbers would be large if they didn't need do be?

We can do better than speculate; we have examples of low-neuron count intelligence. So high neuron count is known not to be a requirement.

fyngyrz -- anon due to mod points

Re:Sources of improvements?

[Rockoon]

You are confusing notation with representation. Just because we truncated all those zeros on the left and right of the number in our notation is irrelevant. The infinite number of 0's to the left and to the right are encoded, implicitly, in the notation that we use.

Re:Sources of improvements?

[Rockoon]

This BBC video on the McGurk Effect will knock your socks off. What you 'see' effects what you 'hear.'

Re:Sources of improvements?

[iggymanz]

we already had the analog / bio version, Cheney's and mega-corporate's meat puppet. But now we've upgraded to Barack the Empty Suit, another mega-corporate bitch.

Re:Sources of improvements?

[Mr. Slippery]

a Turing machine is not even capable of storing a single (arbitrary) real number.

Sure it is. A TM has an infinite tape. It would take infinitely long to read in or out, of course...

Re:Sources of improvements?

[phantomfive]

I think there are actually a lot of AI researchers who are happy with approximate answers (the guy in the article was ecstatic getting 15% right), so it's probably a deeper problem than that.

Re:Sources of improvements?

[Anonymous Coward]

In other news

  A research lab has achieved a breakthrought by fully modelling Bush brain using a damgad celery and a single transistor
We had to use a damaged celery as a healty one proved way to bright to produce an accurate model of the expresident, the lead team researched said

Re:Sources of improvements?

[phantomfive]

FWIW, those synapses are vast simplifications of real synapses. Whether those simplifications matter, nobody knows (but my guess is yes).

Re:Sources of improvements?

[K. S. Kyosuke]

I still have my socks on. This has never worked on me. In noisy environments, I can study your mouth with a microscope and I will still have problems hearing you correctly. I had once someone on a noisy tram repeat to me the same sentence five or six times and then I gave up. That just happens to me every now and then, visual cues or not. Well, everyone's brain is different, I guess.

Re:Sources of improvements?

[Tablizer]

Maybe it's a naive question, but why not use evolutionary algorithms to evolve better neural network engines (NNE) rather than manually try different kinds of NNE's?

Come up with an ASCII notation to describe NNE's, and use mutation and cross-over (sex) to create variations to be re-tested for success rates. Testing has to be automated, and one has to be careful that the evolution fitness test is not just fitting a specific NNE test set. This may be the hard part because evolutionary algorithms love to "cheat" by finding un-anticipated shortcuts.

Re:Sources of improvements?

[raftpeople]

Not sure what you mean by "engine", but neural networks are sometimes evolved instead of trained.

Re:Sources of improvements?

[ColdWetDog]

You might check your server to see if we took all the magic smoke out of it....

Re:Sources of improvements?

[LeDopore]

The big difference is that biology isn't concerned with finding the optimal solution to problems; any very good solution (optimal or not) will let you live to see another day. A lot of math and computer science is dedicated to finding ironclad proofs that under every circumstance, a particular algorithm will deliver he optimal solution. While that's great when it's feasible, sometimes it's OK to go with something that works well even if it isn't optimal.

The set of good heuristics is a strict superset of the set of provably good heuristics. Nature can discover the former, but academics (largely) get paid only for the latter.

Re:Sources of improvements?

[Anonymous Coward]

One of the main differences between this and the earlier nonlinear classifiers is that many layers are trained essentially with "unsupervised" methods (not using the training target of the correct answer for any application) which applies heuristic biases to find "interesting" internal representations.

These architectures are likely to do better on tasks where humans have high performance and machines typically had very low performance using statistical models.

They are not likely to offer a huge advance on tasks where humans can't just "eyeball it" and know the right answer---more traditional statistical learning where the task is to balance quantitatively many competing inputs is not likely be radically improved because there the problem is the balance between generalization and accuracy, in other words regularization. Regular multi-layer perceptrons, support vector machines, boosted ensembles, and just plain generalized linear regression methods are likely to remain the most competitive. Deep nets won't do worse, but the overhead in training and scoring will not make them preferred in practice.

Re:Sources of improvements?

[Tagged_84]

Oh yes and that's repeated several times in the paper, the 530 billion neurons simulated are not comparable to our meagre 30-100 billion. Should have noted that though! thanks.

Re:Sources of improvements?

[mbkennel]

It's a naive question. People have been combining evolutionary methods for architecture selection with more traditional gradient/function value optimization since, well at least the late 1970's. It will still evolve only in the space that the human set it up to evolve.

The breakthroughs did not occur because of an automated computer. They occurred through an large-scale evolutionary algorithm known as Smart-Fraction-Of-Human-Civilization-Thinking-And-Working-and-Writing-For-Decades. We needed new ideas and persistence to test them thoroughly.

What made it possible in our society is long-term government funding of research.

Re:Sources of improvements?

[mbkennel]

"I'm sure all of that helped, but the key ingredient is training mechanisms. Traditionally networks with multiple layers did not train very well, because the standard training mechanism "backpropagates" an error estimate, and it gets very diffuse as at goes backwards. So most of the training happened in the last layer or two."

This problem can be easily remedied by scaling up the gradient terms for earlier and earlier layers. Doing so doesn't solve the deep network problem.
As others have mentioned the breakthroughs were combining highly parallelizable unsupervised representation methods with traditional supervised learning.

Re:Sources of improvements?

[Anonymous Coward]

actually, that quote says nothing about larger or better training sets. however, quality of a neural net is dependent upon the quality of the training set and the size of the net...both of which would benefit from improved computing power and speed.

Re:Sources of improvements?

[Anonymous Coward]

I took a look; 7 hits on that article yesterday, 12 today; nothing unusual in terms of traffic on any other page. So no, don't think whatever you experienced had anything to do with my server.

fyngyrz -- anon due to mod points

Re:Sources of improvements?

[schlachter]

Although if you're talking about training neural nets, then you don't get to chose what's important for learning at the hidden layer of 100 nodes. The network will chose the salient features that allow it to best differentiate between positive and negative examples of whatever it's trying to link/classify/learn.

That's the downside to neural nets. They're basically black boxes. You don't get to chose which features they learn.

Re:Sources of improvements?

[schlachter]

They iPhone runs a neural net for its audio processing to do noise cancellation for outgoing voice. The future is now.

Re:Sources of improvements?

[giuda]

No, he is talking about the worst case for finding a solution that might or might not be exact. O() notation is about algorithms, and the solution can be "good enough" (like floating point math), which is different than exact.

Re:Sources of improvements?

[Raenex]

No, he is talking about the worst case for finding a solution that might or might not be exact.

The specific example he gave, "(optimal circuit design and layout) then the core algorithms are O(2^2^n)", was for exact solutions. In the real world, this problem is tackled with inexact solutions using heuristics.

O() notation is about algorithms, and the solution can be "good enough" (like floating point math), which is different than exact.

Yes, that's true in principle, but it gets a bit more complicated when talking about searching with some element of randomness, as is usually the case with AI-type problems.

Re:Sources of improvements?

[TheTurtlesMoves]

what are you talking about?

Re:Sources of improvements?

[TheTurtlesMoves]

But that has nothing to do with the fact that there can/could be a physical process that can "compute" things a Turing machine cannot.

Deep learning?

[olegalexandrov]

A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning. Neural networks were hyped for a long time, perhaps because of the catchy name.

Re:Deep learning?

[Anonymous Coward]

While you're right that "deep learning" is mostly excellent marketing by Hinton, there is some substance behind that marketing. For a long time AI folks had more or less abandoned neural architecture inspired algorithms because they did not perform well and there were some no-go results proven about classes of functions which were not learnable with the architectures of the time. Over the last 5-6 years there has been substantial progress made on finding tractable ways of training deeper architectures (more difficult because of the large parameter space). These algorithms are now starting to be competitive with other state of the art learning algorithms, with reason to believe there may be further progress to be made.

Robot Apocalypse it's not, but it is definitely an exciting area of machine learning right now.

Re:Deep learning?

[Mr. Mikey]

A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning. Neural networks were hyped for a long time, perhaps because of the catchy name.

You could have answered your own questions with a quick search, rather than assume that that which you are ignorant about is mere "marketing-speak."

deeplearning.net

Deep learning (Wikipedia)

Unsupervised Feature Learning and Deep Learning

Re:Deep learning?

[AthanasiusKircher]

A lot of vague marketing-speak in this article. "Deep learning"?

Agreed. Why do we need the adjective "deep"? Perhaps it's because a lot of AI jargon uses "learning" when they really just mean "adaptive" (as in, "programmed to respond to novel stimuli in anticipated ways"), whereas normal human "learning" is much more fluid.

The article basically talks about neural networks

Yet another victory for marketing. These things have been around for at least 25-30 years, and the connection to what little we actually have deciphered about how the brain encodes, decodes, and processes information has always been incredibly tenuous. There always seems to be these AI strands of "cognitive science" or "neural modeling," which are often nothing than just somebody's pet algorithm or black box dressed up with words that make it sound like it has some scientific basis in actual neurophysiology or something.

Don't get me wrong -- I'm sure some of the examples in TFA have made great advances, partly due to speed and hardware unthinkable 25-30 years ago. And some of the functionality of the "neural nets" might give significantly better results than previous models.

But I really wish people would lay off the pretend connections to humanity. Why can't we just accept that a machine might just function better with a better program or algorithm or whatever, rather than saying that "our research in cognitive science [i.e., BS philosophy of the mind] has resulted in neural networks [i.e., a mathematical model instantiated into programming constructs] that exhibit deep learning [i.e., work better than the previous crap]."

(Please note: I mean no insult to anyone who works in neuroscience or AI or whatever. But I do question the jargon that seems to make unfounded connections and assumptions that the brain works anything like many algorithmic "models." We may succeed in creating artificial intelligence by developing our own algorithms or we might succeed by imitating the brain, but I don't think we're making progress by pretending that we're imitating the brain when we're really just using marketing jargon for our pet mathematical algorithm.)

Re:Deep learning?

[Anonymous Coward]

You're wrong. These networks have a unique structure and a unique method for learning neuron weights. The nets can be shown to be equivalent to a Bayesian network and the learning technique does a remarkable job of bypassing local minima when learning the parameters of the target probability distribution.

Re:Deep learning?

[Prof.Phreak]

Advances are in ways of learning hidden layers that are slightly more clever than backpropagation. For example, lets say you have an image, apply some transform to it (dct, wavelet, whatever, neural net layer, etc.) and save all the important features, but at say 10x less space. Then do the same to those features. Every time reducing the amount of data by 10x. After a few such layers, lets say you're left with 10 bits worth of information---the ``most important'' (according to your benchmark used) ten bits of the whole image.

The ten bits could be anything, such as `this image is a car' or `this image is a face', or ``this face looks angry', etc.

The trick is applying the benchmark on the hidden layers---e.g. how do you pick out which features are important after applying a transform. For that, you train another (inverse) transform that recovers original data from the features---the one that gets you closest to the original wins (e.g. lets say you feed 1000 bits into a neural net to get 100 bits out, and then via inverse transform turn those 100 bits into the *original* 1000 bits... that would mean that your 100 bits represented all the information in the input 1000 bits---obviously more often than not you won't get a perfect match but something close---repeat for any number of layers you want).

Re:Deep learning?

[Black Parrot]

Why do we need the adjective "deep"?

Because the "deep learning" technologies use artificial neural networks with many more layers than traditionally, making them "deep architectures".

It's widely accepted that the first hidden layer of an ANN serves as a feature detector (possibly sub-symoblic features that you can't put a name to), and each successive layer serves as a detector for higher-order features. Thus the deep architectures can be expected to have some utility for any problem that depends on feature analysis.

Re:Deep learning?

[Anonymous Coward]

It looks like you are seeing something that is not there. The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works. Some of the terminology might be borrowed from things related to the brain due to past inspirations, but researchers could care less if the algorithms actually model what goes on in the brain or not, because that is not the point. Much of the jargon does refer to specific things and isn't just a marketing layer on top of the actual math, and it wouldn't be the first time a math related field has used terminology based on very loose analogies, or even complete lack of analogy (e.g., don't assume work on happy numbers has anything to do with modeling psychology).

Re:Deep learning?

[AthanasiusKircher]

I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning," though, in the broader world... and even if we equate the jargony connotations of "machine learning" with "learning," it still seems a stretch to use "deep" as an adjective directly applied to that... but perhaps it's just me.

Re:Deep learning?

[AthanasiusKircher]

It looks like you are seeing something that is not there. The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works.

As someone who has read a lot of the founding literature of modern cognitive science and the philosophy of mind in the 1950s through 80s, which was hugely influential in setting up the early approaches to AI (including neural nets), I have to say -- this is where the stuff came from.

And frankly, a lot of applications in more obscure disciplines, such as in AI analysis in the humanities, researchers are still making claims about these models and their relationships to the actual brain. Hell, just a few years ago I heard a leading cognitive scientist claim that he found evidence for a sort of musical "circle of fifths" neural network in an actual circular physical structure of neurons in the brain... a made-up musical model grafted onto a made-up AI brain model, supported by noisy data... I admit this is an extreme example, but it's not unique.

I understand that modern researchers in "pure" AI may want to avoid recognizing the history or the implications of the terminology -- but there's a reason why the Starship Voyager was equipped with "neural gel-packs" that could get anxious and cause a warp-core breach at a temporal anomaly... words like "neural" actually mean something, and these "neural nets" have about as much connection to the biological function of actual neurons as Voyager's bizarre "neural gel-packs." Yet the implicit metaphor made in continuing to use the term should not be underestimated, not just in a general audience NYT article, but in the way fields are subtly shaped by their nomenclature.

Re:Deep learning?

[Black Parrot]

I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning," though, in the broader world... and even if we equate the jargony connotations of "machine learning" with "learning," it still seems a stretch to use "deep" as an adjective directly applied to that... but perhaps it's just me.

I have a bigger issue with "learning" than with "deep", since with very few exceptions ANNs don't learn anything autonomously, but rather are adjusted by an external algorithm to to perform well on a given problem. "Deep training" would make sense for "deep architectures".

Re:Deep learning?

[Anonymous Coward]

Come on this is just the age old scientific naming. Everyone thinks their discovery is the last ever, the newest, shiniest one and will be the solution to all the world's problem, and they're named accordingly.

Treat it like electricty, right, where electricity flows from negative voltage to positive voltage. Brilliant naming there (perfectly explainable given the perspective of the time it was discovered though). Or terms like "the modern age", "the new age" and, my favorite "the newest age" (generally considered to be in our past, naturally). Or the name "atom" (greek for "indivisible"), brilliant naming there. Some people are also getting bit ahead of themselves like the color "strange" in quantum mechanics (yes, really). You can add the colors which goes like this : charm + anti-strange = strange D meson. Elementary, right ?

Re:Deep learning?

[Anonymous Coward]

I don't see how that is avoiding to recognize the history. One can both acknowledge that something was inspired by something and no longer has any connection to it. And while "neural" means something specific in biology, it can mean something specific but different in computer science. That is the nature of jargon sometimes. Simulated annealing has gone a long ways beyond its roots in thermodynamics, and hill climbing algorithms don't seem to have much to do with actual hills any more... This comes up in some many examples in so many fields, many people move on and just have to live with reminding outsiders to the field that the meanings have diverged, as opposed to inventing new words for things that already developed a well established meaning one way or another.

Re:Deep learning?

[AthanasiusKircher]

One can both acknowledge that something was inspired by something and no longer has any connection to it. And while "neural" means something specific in biology, it can mean something specific but different in computer science. That is the nature of jargon sometimes.

I completely get your point, and if it were just one or two words ("neural" or whatever), I might agree. But the influence in this case is pervasive, and it has shaped and continues to shape the way we talk about the field. New nomenclature often continues to extend the mind metaphors, when there is no necessary reason to. Why call it "deep learning" when "multilevel" or "multilayered" might better describe the process? Etc. That was the point of my original post. And frankly, the nomenclature seems to continue to generate a lot of confusion among scholars interested in cognitive science, if my pretty thorough familiarity with cognitive models applied to problems in the professional literature of the humanities is any indication.

Re:Deep learning?

[maxs-pooper-scooper]

The term "deep" comes from the idea that the algorithm is trying to learn something deeper than previous algorithms. In fact, the usual set of machine learning algorithms are termed shallow learning now. The difference is that deep learning tries to model P(X) whereas shallow learning (SVM, NN, naive Bayes, etc..) try to learn P(X|Y) where X is your input space and Y is the label space.

In deep learning, these neural networks are not your usual NNs. Deep learning isn't just taking advantage of hardware scaling for more nodes and layers, rather it uses convolutional NNs which are slightly different.

Another difference is that deep learning is trying to learn an efficient representation for the inputs, i.e. automatic feature generation. This is not to say it trying to become an automatic unsupervised learning technique, but instead a supervised learning approach that takes care of the most time intensive and critical process (and typically unappreciated and overlooked) of any machine learning process -- feature extraction/generation.

Re:Deep learning?

[Anonymous Coward]

People often mistake neural networks for the multilayer perceptron(MLP) algorithm trained using backpropagation, which is what has been around for decades. Neural networks should really be seen as a class of algorithm instead. Things like restricted boltzmann machines and deep belief networks are very different from the MLP in terms of the theories they were based on as well as capabilities.

Re:Deep learning?

[ceoyoyo]

Actually, it seems your post is the vague one. "normal human "learning" is much more fluid." What does that mean?

Learning: (dictionary.com)
1. knowledge acquired by systematic study in any field of scholarly application.
2. the act or process of acquiring knowledge or skill.
3. Psychology. the modification of behavior through practice, training, or experience.

Many machine learning algorithms "learn" exactly the way you'd teach a child. They see examples, you tell them what the object, word, etc. is, and they remember that answer imperfectly. Repetition improves their accuracy and a breadth of examples improves their generality. After not seeing something for a while, they may forget it.

As the other poster pointed out, "deep" describes algorithms that are better able to teach multi-level systems. The changes associated with learning are better propagated to deeper levels, better utilizing all the capacity of the system.

No, it's not just you. There are a lot of people who see the brain as the last bastion of their identity as some kind of special and privileged creature, therefore it must be magical and any attempts to explain how it works are misguided, childish and silly. Whether that's your actual belief or not, that's what your post sounds like. Modern computational neuroscience has actually come a long way. We're even capable of producing chips that can be implanted and replace some parts of the brain. It's not magic.

Re:Deep learning?

[Anonymous Coward]

New nomenclature often continues to extend the mind metaphors, when there is no necessary reason to.

Isn't this on par with many other aspects of jargon? People like to think in terms of analogies, and will continue to names things that way even if rather rough You should see some of the stuff physicists come up continuing the analogies to spin and color for particle states that have nothing to do with wavelength or angular momentum.

Why call it "deep learning" when "multilevel" or "multilayered" might better describe the process?

Because it is a specific kind of multilayer learning. There are previous versions that sucked, and this was a new approach once upon a time. It is a specific kind of multilayer learning and needs its own name.

And frankly, the nomenclature seems to continue to generate a lot of confusion among scholars interested in cognitive science,

Welcome to what it is like to do research in quantum mechanics related fields. Even in cases with distinctive terminology, some people make up their own analogies and run with it, science fiction will re-purpose terminology, and others delve into word salad.

Re:Deep learning?

[Rockoon]

Why call it "deep learning" when "multilevel" or "multilayered" might better describe the process?

..because the 'might' in this case is 'wrong.' Multi-layer Networks was all the rage as early as 1966 (perhaps quite a bit earlier.) The only resemblance Deep Learning has to Multi-Layer Networks is a somewhat similar organizational topology. The methodology is quite different. "Deep' in this case is comparative to other learning methods. You train one layer, then the next, and so on.. never going back. The learning propagates straight down to the depths. Its a very radical advance in that other methods arent even close to the rate at which learning can be propagated downwards, that they are in fact 'shallow' in comparison.

Re:Deep learning?

[swillden]

I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning,"

It's for the same reason as "deep architecture", essentially. Early ANN training methods did not effectively propagate training through many layers of nodes, so the learning was "shallow". New methods allow training each layer at a time, so training can be effectively performed on deep layers. Deep training/learning.

I see your point if you try to interpret the "deep" as implying some sort of more profound learning of the domain. It's not. It's just deep in the sense that deep layers can be effectively trained (i.e. learn). Of course, deep learning may allow the deep nodes to discover very non-obvious meta-features, so the learning may be deep in the sense of profundity. But that's not required for the name to be applicable.

Re:Deep learning?

[Anonymous Coward]

They are "learning" the values of parameters in the model.

Re:Deep learning?

[mbkennel]

"The term "deep" comes from the idea that the algorithm is trying to learn something deeper than previous algorithms. In fact, the usual set of machine learning algorithms are termed shallow learning now. The difference is that deep learning tries to model P(X) whereas shallow learning (SVM, NN, naive Bayes, etc..) try to learn P(X|Y) where X is your input space and Y is the label space. "

Well, more correctly, the deep learning tries to model P(X) as P(X | H) for some set of "hidden" or latent features H which in some ways, is far simpler than the raw data and space of X, and then learns P(Y | H, X) after doing some training for P(X|H).

Re:Deep learning?

[mbkennel]

"The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works."

This isn't true---or the connotation of it isn't true. I don't know how to quantify "majority", but there is substantial interest in computational modeling of actual biology across all levels of biological/chemistry fidelity and attention to engineering and statistical problems.

A glance at the work in the journal _Neural Computation_ shows papers both on entirely theoretical statistical computational models and models much more closely tied to experimental neuroscience results.

Many people want to do both: derive useful methods for solving engineering problems and understand biological systems which have shown such abilities. The field is very difficult and deep.

Re:Deep learning?

[blue trane]

"It's widely accepted that the first hidden layer of an ANN serves as a feature detector (possibly sub-symoblic features that you can't put a name to), and each successive layer serves as a detector for higher-order features."

Why can't you put a name to them?

More info plz

[Anonymous Coward]

Without the rate of success it's hard to see why the Merck contest is an impressive example, since "rand()%15", which presumably is the same for an untrained neural net, will win it sometimes too and is not very interesting.
That being said the other examples in tfa are better.

Open knowledge

[Anonymous Coward]

We need to open all the documentation for everyone who want to learn and investigate about IA.

Re:Open knowledge

[AthanasiusKircher]

We need to open all the documentation for everyone who want to learn and investigate about IA.

Absolutely. It's about time we figured out who really won those caucuses -- and what the heck is up with the ethanol subsidies?

Deep Belief Networks

[Guppy]

A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning.

It's hard to tell from the article, but they probably are trying to refer to Deep Belief Networks, which are a more recent and advanced type of Neural Network, which incorporates many layers:

Deep belief nets are probabilistic generative models that are composed of multiple layers of stochastic, latent variables. The latent variables typically have binary values and are often called hidden units or feature detectors. The top two layers have undirected, symmetric connections between them and form an associative memory. The lower layers receive top-down, directed connections from the layer above. The states of the units in the lowest layer represent a data vector.

Automatic creation of features

[michaelmalak]

I wonder how much of these improvements in accuracy are due to fundamental advances

I was wondering the same thing, and just now found this interview on Google. Perhaps someone can fill in the details.

But basically, machine learning is at its heart hill-climbing on a multi-dimensional landscape, with various tricks thrown in to avoid local maxima. Usually, humans detemine the dimensions to search on -- these are called the "features". Well, philosophically, everything is ultimately created by humans because humans built the computers, but the holy grail is to minimize human invovlement -- "unsupervised learning". According to the interview, this one particular team (the one mentioned at the end of the Slashdot summary) actually rode the bicycle with no hands and to demonstrate how strong their neural network was at determining its own features, did not guide it, even though it meant their also-excellent conventional machine learning at the end of the process would be handicapped.

The last time I looked at neural networks was circa 1990, so perhaps someone writing to an audience more technically literate than the New York Times general audience could fill in the details for us on how a neural network can create features.

Re:Automatic creation of features

[Daniel Dvorkin]

the holy grail is to minimize human invovlement -- "unsupervised learning"

Unsupervised learning is valuable, but calling it a "holy grail" is going a little too far. Supervised, unsupervised, and semi-supervised learning are all active areas of research.

Re:Automatic creation of features

[Anonymous Coward]

One does not preclude another. The current "cool thing" is to learn features first using an energy based unuspervised model, and then use those features in supervised, discriminative classifier.

Re:Automatic creation of features

[mbkennel]

There is a new thing. It has long been known that "deep networks" could theoretically represent more sophisticated features and concepts, and there were obvious biological examples of this working successfully.

The artificial neural network methods of 1990, as you say hill-climbing on a multi-dimensional landscape, turned out not to work particularly successfully on deep networks, or more correctly, provide little additional benefit vs shallow networks. After this time, resarch in statistical learning moved from just these parametric models to more clearly statistical methods with some clever tricks, e.g. support vector machines and boosted ensembles of simple learners. These appeared to have some advantages in training over traditional neural networks, because they could be transformed into more deterministic optimization sub problems, compared to the neural networks. SVM's in particular could be transformed into quadratic optimization which had deterministic solutions (i.e. convex optimization instead of the very rough and fractal error surface of MLPs/networks).

However, it turned out that some of these methods did not scale well to really large problem sizes, e.g. training and scoring SVM's on millions to billions of data points instead of the 1000-50000 of typical academic "benchmark" datasets doesn't work well in convex optimization. The time necessary to train in the convenient dual space which has this property can be quadratic or worse in the number of points. So what is state of the art in large scale SVM training? Uh, stochastic gradient descent just like those yucky neural networks.

Now, back to the new generation of neural networks. The typical trick now for the newer generation of neural networks (and yes Geoff Hinton and his lab is the leader in this revival) is that most of the training does NOT use the supervised methods (matching to the target). Much of the initial phase of trainings involve unsupervised methods which are statistical methods which attempt to find "interesting" structure in the input data--for some arguable form of "interesting"---thereby doing "dimensionality reduction".

Then at the end, there is traditional supervised optimization to 'clean things up'. Of course now the trick is matching the right biases in the unsupervised layers for the task at hand, and that is likely still trial and error. The papers show the successes. But the point is that their successes are so spectacular occasionally that the general approach appears to be pretty valuable.

It's quite possible this is exactly what evolution has done---evolve different unsupervised priors/algorithms by evolving wetware---which happen to turn out to be useful for the types of statistical patterns occurring in the various forms of sensory inputs.

As far as I can tell, the original connectionist manifesto is still correct. This is the only plausible approach I see towards artificial intelligence, as opposed to being just machine learning (which is in some ways a superset, but also a subset in ambition).

Re:Automatic creation of features

[Anonymous Coward]

Basically, you input a bunch of features that you hope to be relevant into the net. Then each layer, broadly speaking, sequentially minimizes the error of an identity function for the layer below it under some constraints. f(X, W, B)=X+error where f(X,W)=transpose(g(X))*W+B where g(X) might be tanh(X) or sigmoid(X) or whatever. Examples of constraints might be that the current layer has fewer neurons than the layer below it, that you withhold some inputs (set them to 0), you enforce sparsity in the current layers neural activity (this one is really popular), or that you train only a random subset of the current layers neurons for each training example.

The effects of this can be seen from many perspectives. One parable is of course lossy compression, or a higly nonlinear principal componen analysis. Another is that each layer learns a more abstract view of the data, e.g: the first layer learns lines, the second shapes, the third ears mouts and noses, the fourth faces, the fifth layer learns to recognize specific people. Either way, there are tons more untagged example data then tagged data for supervised learning, and the neural weights you learn during unsupervised learning are a great starting point to performed supervised learning on. For classification task just slap as many neurons as you have classes on top of the unsupervised net and perform softmax with error back-propagation on your tagged data.

Or something like that

Can their handwriting recognition solve captchas

[blue trane]

yet?

Can You Imagine a Beowulf Cluster of These?

[jjh37997]

Can You Imagine a Beowulf Cluster of These?

Re:Can You Imagine a Beowulf Cluster of These?

[maxwell demon]

Maybe. Can it learn to run Linux?

Re:Can You Imagine a Beowulf Cluster of These?

[Anonymous Coward]

Yes, but it has great difficulty making any sense of Unity or Gnome3. The first time it tried, it ran up a virtual tab on Amazon amounting to trillions of dollars.

Re:Can You Imagine a Beowulf Cluster of These?

[AthanasiusKircher]

And then tried to get out of its virtual debt by mining bitcoins.

Re:Can You Imagine a Beowulf Cluster of These?

[Anonymous Coward]

Wrong question. Can it learn to write Linux?

Re:Can their handwriting recognition solve captcha

[slashmydots]

Humans can't even solve those, lol.

Neural Network for Machine Learning on Coursera

[Anonymous Coward]

I'm doing Prof Hinton course on Neural Network on Coursera this semester. It covers the old school stuff plus the latest and greatest. From what I gather from the lecture, training neural networks using lots of layers hasn't been practical in the past and was plauged with numerical and computational difficulties. Nowadays, we have better algorithms and much faster hardware. As a result we now have the ability to use more complex networks for modelling data. However, they need a lot of computational power thrown at them to learn compared to other machine learning algorithms (random forest). The lecture quotes training taking days on a Nvidia GTX 295 GPU to learn the MNIST handwritten dataset. Despite this, the big names are already using this technology for applications like speech recognition (Microsoft, Siri), object recognition (Google Cat video, okay that's not a real application yet).

Re:Neural Network for Machine Learning on Coursera

[PlusFiveTroll]

The hardware since the 295 days is around least 3 times as fast too. It seems just about every publication on neural networks has had something about GPUs in the last few years.

http://www.neuroinformatics2011.org/abstracts/speeding-25-fold-neural-network-simulations-with-gpu-processing

Re:Neural Network for Machine Learning on Coursera

[IamTheRealMike]

Actually, Google has already launched neural network based speech recognition. The cat demo was for fun, the underlying technology is already applied to real problems though. I can tell you now based on practical experience as a user that the accuracy boost from it has been amazing. The dictation feature in Android went from being "amusing toy" to "actually useful" almost overnight.

Re:Neural Network for Machine Learning on Coursera

[Anonymous Coward]

Minor correction after looking at the lecture slide again. It took a few days using a Nvidia GTX 285 (not 295) GPU to train 2 million 32x32 color images on a network with approximately 67,000,000 parameters, not the handwritten database.

Re:Neural Network for Machine Learning on Coursera

[Sulphur]

The hardware since the 295 days is around least 3 times as fast too. It seems just about every publication on neural networks has had something about GPUs in the last few years.

http://www.neuroinformatics2011.org/abstracts/speeding-25-fold-neural-network-simulations-with-gpu-processing

From the article : Furthermore, to increase the number of calculated time steps increases exponentially the computation time with the CPU while the computation time increases only linearly with the Graphic Processor Unit.

Eh?

Re:Neural Network for Machine Learning on Coursera

[Sulphur]

From the article : Furthermore, to increase the number of calculated time steps increases exponentially the computation time with the CPU while the computation time increases only linearly with the Graphic Processor Unit.

Eh?

P=NP?

Just more of the same

[qbitslayer]

They haven't done anything that wasn't already being done by others. They're just doing more of it. Essentially, the approach consist of using Bayesian statistics and a hierarchy of patterns. Prof. Hinton pretty much pioneered the use of Bayesian statistics in artificial intelligence. With a rare notable exception (e.g. Judea Pearl), the entire AI community has jumped on the Bayesian bandwagon, not unlike the way they jumped on the symbolic bandwagon in the latter half the 20th century, only to be proven wrong fifty years later.

The Bayesian model essentially assumes that the world is inherently probabilistic and that the job of an intelligent system is to discover the probabilities. A competing model (see links below), by contrast, assumes that the world is perfectly consistent and that the job of an intelligent system is to capture this perfection.

See The Myth of the Bayesian Brain and The Second Great AI Red Herring Chase if you're interested in an alternative approach to AI.

Re:Just more of the same

[martin-boundary]

Sorry, but those blogposts aren't very convincing. Do you have *actual* arguments comparing Bayesian to these hypothetical alternatives, or should we just take the claims on trust?

Re:Just more of the same

[Daniel Dvorkin]

Do you have *actual* arguments comparing Bayesian to these hypothetical alternatives, or should we just take the claims on trust?

It's the "Rebel Science" guy. He's a nutcase. So no, he's not going to have any actual arguments, just a bunch of pseudoscientific babble.

Re:Just more of the same

[Rockoon]

In the case of evolutionary optimization algorithms, they jumped onto the Bayesian bandwagon in 1999 but they jumped off it only one year later.. onto the much larger Shannon bandwagon.

Re:Just more of the same

[qbitslayer]

Do you have *actual* arguments comparing Bayesian to these hypothetical alternatives

The argument is simple. As Judea Pearl (an early proponent of Bayesian statistics for AI who has since changed his mind) explained, humans are not probability thinkers; they are cause/effect thinkers. If you drop a ball, you know it's going to hit the ground. You don't think that there is a probability that it might not. If you read the word Bayesian in this sentence, you know for certain that you did. There is nothing probabilistic about it. Sure we handle probabilistic sensory signals but we build a perfect model of the world in our cortical memories. We simply compare incoming sensory inputs to our perfect internal model and decide which patterns in memory best fit the sensory evidence. This truth will be forcefully demonstrated in the not distant future. Wait for it.

The Bayesian brain is a myth, a rather dumb one in retrospect.

Re:Just more of the same

[raftpeople]

Amazingly (I knew this poster "smelled" like rebel science), you aren't completely wrong here. We do create a model and we predict based on that model, but the basis of the prediction is pattern matching/detection from previous experience. A pattern match isn't guaranteed (hence the connection to probabilities), but it's the best guess based on experience.

Re:Just more of the same

[qbitslayer]

Amazingly (I knew this poster "smelled" like rebel science), you aren't completely wrong here.

Sorry, wrong. I am 100% right on this issue. All the Bayesian bandwagoneers are out to lunch. The brain does not build a probabilistic model of the world. It builds a perfect deterministic model. Wait for the demo.

Re:Just more of the same

[Anonymous Coward]

"You don't think that there is a probability that it might not. If you read the word Bayesian in this sentence, you know for certain that you did. There is nothing probabilistic about it."

This is assuming that one's own introspection using 'feelings' and not experimental data about the technical operation of cognition yields accurate and useful results for resolving this question. I question that.

Bayesian systems work with likelihoods which are mathematical objects which conveniently have properties which are the same as probability. There isn't necessarily a probabilistic assumption in the sense of hypothetical many trials.

Bayesian systems often result in large log likelihood differences which are awfully good approximations for "certain" just as the probabilities for simians taking flight from my rear orfice in the next hour is less than 10^-120.

Bayesian and other soft quantitative computational systems may be able to handle some problems better than humans where the balance of evidence is not obvious. In these cases a humans' intuition may be "I'm not sure", and a properly calibrated quantitative model could be superior. For instance, humans know for certain that an unemployed, bankrupt person in prison without an income should not get a loan for a trillion dollars. For vast majority of intermediate cases in the real world, quantitative models built, potentially with Bayesian and probabilistic methods, are likely to be superior than humans' gut feeling.
 

Re:Just more of the same

[CTachyon]

[...] If you read the word Bayesian in this sentence, you know for certain that you did. There is nothing probabilistic about it. [...]

Not quite. You've never had the experience of remembering having done something, then having someone contradict you, then asking around and finding out that your memory is faulty? If you were certain of your memory, no finite amount of evidence would ever convince you that you were mistaken. Your example instead demonstrates that we pick the most probable (most "familiar") explanation without conscious consideration of alternatives, and we only backtrack to alternatives when the first explanation is sufficiently falsified to demote it from the best explanation.

That's not to say that this has any bearing on Judea Pearl's research into causal networks. Causal networks complement a probabilistic approach, as each causal node operates on purely Bayesian principles; the only difference is the added operation of graph surgery to represent counterfactuals. It's certainly true that the naïve extension of Bayesian probability to a decision theory (Evidential Decision Theory) is silly -- it results in "Speeding on the way to work is correlated with being late to work, therefore if I don't speed I can't be late!", and it's also true that causal graphs naïvely extend to a decision theory (Causal Decision Theory) that fixes the most egregious silliness. But Bayesian probability is still a key piece of CDT, and even CDT doesn't fix everything (look up Newcomb's paradox).

hardware

[globaljustin]

It's the latter...one could assiduously identify common research buzzwords

From a neuroscience perspective, it's about transmission of signals continuously in a highly complex network...a **hardware limit**

The idea that there will be a 'fundamental advance' that allows for 'artificial intelligence' is really just hype.

All we can ever make is better things to follow our instructions.

Re:hardware

[Black Parrot]

All we can ever make is better things to follow our instructions.

What is the basis for that claim?

In 50 years when we can simulate a brain to any arbitrary level of detail, or build a wet-brain one neuron at a time, why wouldn't it be able to do what naturally occurring intelligence can?

Is there some Special Ingredient that cannot be simulated, even in principle? Or that cannot be understood well enough to try?

Re:hardware

[Anonymous Coward]

http://www.gizmag.com/ibm-supercomputer-simulates-a-human-sized-brain/25093/

It's both

[Anonymous Coward]

In the past few years, a few things happened almost simultaneously:

1. New algorithms were invented for training of what previously was considered nearly impossible to train (biologically inspired recurrent neural networks, large, multilayer networks with tons of parameters, sigmoid belief networks, very large stacked restricted Boltzmann machines, etc).
2. Unlike before, there's now a resurgence of _probabilistic_ neural nets and unsupervised, energy-based models. This means you can have a very large multilayer net (not unlike e.g. visual cortex) figure out the features it needs to use _all on its own_, and then apply discriminative learning on top of those features. This is how Google recognized cats in Youtube videos.
3. Scientists have learned new ways to apply GPUs and large clusters of conventional computers. By "large" here I mean tens of thousands of cores, and week-long training cycles (during which some of the machines will die, without killing the training procedure).
4. These new methods do not require as much data as the old, and have far greater expressive power. Unsurprisingly, they are also, as a rule, far more complex and computationally intensive, especially during training.

As a result of this, HUGE gains were made in such "difficult" areas as object recognition in images, speech recognition, handwritten text (not just digits!) recognition, and in many more. And so far, there's no slowdown in sight. Some of these advances were made in the last month or two, BTW, so we're speaking about very recent events.

That said, a lot of challenges remain. Even today's large nets don't have the expressive power of even a small fraction of the brain, and moreover, the training at "brain" scale would be prohibitively expensive, and it's not even clear if it would work in the end. That said, neural nets (and DBNs) are again an area of very active research right now, with some brilliant minds trying to find answers to the fundamental questions.

If this momentum is maintained, and challenges are overcome, we could see machines getting A LOT smarter than they are today, surpassing human accuracy on a lot more of the tasks. They already do handwritten digit recognition and facial recognition better than humans.

Why?

[Anonymous Coward]

Why do we want to obsolete ourselves with AI?

Re:Why?

[Anonymous Coward]

Why do we want to obsolete ourselves with AI?

It would be a lot of fun surveying the AI derived discoveries of the moment. We'll also need AI to sift through and prototype the most promising results. Don't worry though, cognitive enhancements are on the horizon too.

deep shit

[globaljustin]

Why do we need the adjective "deep"?

Because the "deep learning" technologies use artificial neural networks with many more layers than traditionally, making them "deep architectures".

So, you admit 'deep' is a marketing buzzword...thank you. It's *obviously* not a technical term.

It is a discrete, ordinal description of a quantity...that's ALL the word 'deep' in this context means...which means it's a non-technical word...and non-technical words used to make non-existent distinctions in order to gain attention...

well that's a marketing word...

Re:deep shit

[smallfries]

When people write a paper for publication they have to differentiate their approach from previous approaches. You seem to have latched onto deep as an imprecise description of the number of layers. It is not. It is an accurate distinction in comparison to previous approaches. Because previous approaches were limited to about (not exactly) two layers it makes the definition of the label a little fuzzy, but the partition into shallow / deep approaches is crisp.

Re:deep shit

[AthanasiusKircher]

Then how about something like "multilevel" or "multilayer adaptive networks of transfer functions" or something like that (I'm sure someone can improve the precision of that description)... rather than the vague and imprecise "deep learning neural networks", which makes implicit and inaccurate connections to brain processes for no good scientific reason (other than to fool people into giving grant money).

Re:deep shit

[Rockoon]

Then how about something like "multilevel" or "multilayer adaptive networks

Taken.

How about this. First there was the Genetic Algorithm (GA), and then came many variants with different names.. Messy GA, Linkage Learning GA, and so on and on.

Then along comes the Compact Genetic Algorithm, which is actually a variant of an Estimation of Distribution Algorithm (EDA) rather than a variant of a GA, but it was so named because it propagates the same information at the same rate as a simple GA with its crossover parameter set to 0.5 (called Uniform Crossover) so thats the provenance of the GA term, but the new algorithm uses significantly less space than any other GA, hence the provenance of the 'Compact' term.

Now here is the great stuff...

The Extended Compact Genetic Algorithm comes along, and it doesnt use less space at all (it uses slightly more space than a bog standard GA!), and its still actually an EDA instead of a GA, but make no mistake.. this is currently the Gold Standard of evolutionary optimization algorithms.

The reason this happens is because people doing the research are not married to methodology. They are married to the field of optimization. They care more about the rate of information propagation and the amount of storage space required rather than they do the technical details of how it happens. The Compact Genetic Algorithm propagates the same information at precisely the same rate as a Simple GA with Uniform Crossover, and uses less space.. and that information is more important than the methodology.

Re:deep shit

[mbkennel]

"So, you admit 'deep' is a marketing buzzword...thank you. It's *obviously* not a technical term."

Technical people in the field, when they hear "deep belief networks" have an excellent idea about the class of computational methods in that class. So yes, "deep" is part of a reasonably precise technical phrase, and the word "deep" in itself does have a connotation in the technical literature: certainly more than one hidden layer, and probably more than two hidden layers.

People in the field also know that typical earlier training approaches relying only on supervised learning did not show any consistent advantage to going beyond two hidden layers, whereas the newer class of methods do show some clear advantages in some problems.

If you want to attack something, it is the "belief" part of the phrase, and not "deep", as this is less clearly defined.

Summary: it's not vapid marketing-speak just invented, any more than "nuclear magnetic resonance".

Re:Sources of improvements? Mod parent up plz

[kanweg]

Thanks.
Back in the early nineties I bought a neural network program to play with. I couldn't get it to learn anything (except for the XOR etc. examples) even when it was so easy (range of boiling points of hydrocarbons depending on the number of carbon atoms. Predict the boiling point of the next one). So when I read about advances in computing power I knew that wasn't the reason. Your remark on back propagation could be the explanation because that was what this network did.

Bert

Old News

[Dr_Ish]

While there have been advances since the 1980s, as best I can tell most of this report is yet more A.I. vaporware. It is easy to put out a press release. It is much harder to do the science to back it up. How did this even get posted on the/. front page? If this stuff was true, I'd be happy, as most of my career has been working with so-called 'neural nets'. However, they are not neural, that is just a terminological ploy to get grants (anyone ever heard of the credit assignment problem with bp?) Also, there have been some compelling proofs that most neural networks are just statistical machines. So, move on. Nothing to see here folks, etc.

Common sense

[Anonymous Coward]

Another win for common sense. They only figured out to use entity relationships for learning?

Need some good drugs to believe it?

[3seas]

... wake up people..... its the fucking drug industry looking for any excuse it can to sell you aanother one of their drugs...

And pot remains, for the most part, illegal.....

I think we already have achieved artificial intelligence... in humans...

Referencing

[Tempest451]

Computers are great at storing and retrieving data, but what they lack is the ability to reference the data in a meaningful way. An AI can recognize an Eagle, a white star, and red and white stripes, but can't readily see the commonality of those objects to the American Flag. Everything about how humans see the world is pattern recognition, but it is the way we reference those patterns that express our intelligence.

Neural networks have their limitations

[Hentes]

While neural networks do amazingly well for a certain type of problems, they do have their limitations. Neural networks are good for designing reflex machines, that react to their current environment. They aren't efficient when they have to learn on the field or plan ahead.

It had to be asked

[cellocgw]

From a data set describing the chemical structure of 15 different molecules, they used deep-learning software to determine which molecule was most likely to be an effective drug agent."

So the AI is going to turn some molecules into an FBI undercover snitch? That's some serious DNA-FU there!

Yes, but... no.

[Anonymous Coward]

This is a very misleading metric. First, some not-insignificant number of the neurons in the brain are involved in non-cognitive computations. Muscle control, hormone regulation, kinesthesia, vision (not thinking about what is seen, but simply recognizing it), heart rates and other system regulation and so on.

Examples also exist of low-neuron (and synapse) count individuals who retain cognitive (and all other major) function; these examples cannot be explained away by "counting neurons."

We don't know which yet, but given that high neuron count has been ruled out as the single way to accommodate intelligence, we do know that we need to look to other mechanisms for human cognition. Structure, algorithm, other features known or unknown may be responsible for intelligence; and it may be that something entirely disjoint is responsible for the rise of intelligence; but we know it isn't simply high neuron count.

--fyngyrz (anon due to mod points)

Re:Yes, but... no.

[Mr. Slippery]

Examples also exist of low-neuron (and synapse) count individuals who retain cognitive (and all other major) function; these examples cannot be explained away by "counting neurons."

The example you cite shows images of a compressed brain. It says nothings about "counting neurons"; the person in question could have roughly the same number of neurons as you or I compressed into a smaller space.

Also the guy is said to have an IQ of 75. That's "borderline intellectual functioning", and it's incorrect to say that cognitive function has been retained, it's clearly degraded.

Of course it takes more than a high number of neurons and a high degree of interconnection to perform processing; training the network is vital.

Re:Yes, but... no.

[Anonymous Coward]

The example you cite shows images of a compressed brain. It says nothings about "counting neurons"; the person in question could have roughly the same number of neurons as you or I compressed into a smaller space.

No, but nice try. Not only will the same number of neurons not fit or function in such a small space, but neither will the blood supply for them, and neither is it plausible that even a fraction of the normal number of interconnections are present.

Also the guy is said to have an IQ of 75.

Yep. Got any software with an IQ of 75? or any number you might get from testing a walking, talking human? No? Well, then. :)

Of course it takes more than a high number of neurons and a high degree of interconnection to perform processing; training the network is vital.

Oh, good grief. I've said this before, and I'll say it again: We, meaning all of us, have no idea how intelligence functions or arises, or even how many approaches there might be that could serve as a successful platform for either or both, or even if they're separable. Therefore, to glibly claim that this or that is "a" or "the" vital component... we're entirely on the wrong side of the learning curve for such a claim to be anything but purest hubris.

When we've successfully crafted an AI, then you can tell us "of course" it takes whatever it actually does take. Until then, find some humility and try not to let your assumptions take over your ability to reason.

fyngyrz -- anon due to mod points

Re:Can their handwriting recognition solve captcha

[ceoyoyo]

There have been several stories about captchas being broken, to the point where secure ones today have to be barely decipherable by humans. That suggests the character recognition algorithms are performing very similarly to humans.

It's not about speed.

[Anonymous Coward]

Every image processing function would have to be implemented in this way.

Only separable IP functions can be implemented this way.

Back to present day: The only thing a GPU gives us is speed. Everything else, we could already do, and furthermore, only in the context of smaller memory, which can negatively interact with a GPU speed advantage. Speed is great, of course, but as someone else put it above, "a fast moron is still a moron." Intelligence is not about speed in any way. How useful it is certainly will be, but if you get an intelligent answer in a century, or in a fraction of a second, like the moron, it's still what it is — intelligent.

We don't know — yet — what intelligence is, and so we don't know what the lower limits are for hardware that implements it. That "big wall" you refer to could just as easily represent a drought of ideas in the right areas as it could most other limits; even memory could have been made very large if someone really wanted to. There's nothing magical about adding address bits to a custom computer design. Today's machines may be vastly overpowered for the minimums required for the task — how can we know until we've identified what the task is, and then worked on optimization for a while?

fyngyrz -- anon due to mod points

Re:It's not about speed.

[xtal]

There is no shortage of ideas. There has been a shortage of ways to test them in the real world.

Re:It's not about speed.

[Anonymous Coward]

There is no shortage of ideas.

I didn't say there was. I suggested it was possible there was "a drought of ideas in the right areas", which is something entirely different from a shortage of ideas in general. Ideas we do have. Most of these ideas have not worked out. Some of them, like Minsky's shallow idiocy about neural networks, have been downright harmful. Others may have not been taken far enough. And it is entirely plausible that the "right" ideas or even the general region of "right" ideas has yet to be broached. We'll know when we're on the other side of the problem; when AI has been achieved. Until then, it's all speculation.

fyngyrz -- anon due to mod points

Restricted Boltzman Machine

[Fnord666]

Here is a good video of a talk given by Dr. Hinton about Restricted Boltzman Machines. It is a very promising technique for deep learning strategies.

Re:Can their handwriting recognition solve captcha

[swillden]

That's the point. blue trane was hoping for an automated captcha-solving assistant so he wouldn't be frustrated by them.

Re:Can their handwriting recognition solve captcha

[phantomfive]

I've really wondered about that though, I've seen the stories, but I've never seen the evidence. Were they really broken? Or was it just a claim that was never verified?

multilevel shit

[globaljustin]

I like the suggested nomenclature changes...I think this is an area where any everyday techie or 'nerd' can make the world better for him/herself and everyone *and* make their job easier

make the words we use make sense! It helps **us** signal value to people outside of our in-group...

haha...if I was redesigning computing I'd start with the 'help' tab ;)

fuzzy shit

[globaljustin]

Because previous approaches were limited to about (not exactly) two layers it makes the definition of the label a little fuzzy, but the partition into shallow / deep approaches is crisp.

I'm glad you mentioned 'fuzzy'...encountering that word partially formed my strong opinions about non-technical language...

See, I started out loving physics, especially astrophysics. An 8 year old in the library science section with a pile of books trying to figure out the sky and learning about the lives of other scientists.

If you look at the history of physics, the idea of rigor is obviously very important. When I encountered things like *Heisenberg Uncertainty* and the consequences of Einstien's theories on time and gravity...then Black holes....the Fournier Transform...etc etc

Well, it pissed me off! How dare science be uncertain!!!!

I was **so mad** that the Bohr Model wasn't the definitive model...I **hated** that science...SCIENCE...had to resort to stupid uncertain, unreliable concepts like *fuzzy math*

When I understood that science is a dance with uncertainty, and that no ammount of experimental rigor can create 100% truth...

Well, I got fuzzy...

But I still resist getting 'fuzzy' as the easy way out for a lazy researcher...and that's why I think Computing still has so many hitches...

Re:fuzzy shit

[smallfries]

If you think fuzzy is bad look at anything called quazi-something in maths. It may as well read not-actually-anything-at-all-like-something. That one is my pet peeve

Four reasons why this is still a bad idea:

[Anonymous Coward]

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
4. iRobot and similar fiction.

Re:Can their handwriting recognition solve captcha

[ceoyoyo]

Here's one you can try out yourself: http://code.google.com/p/captchacker/

The captcha's now are harder than they used to be but I have no doubt that if you run a few hundred through a breaker you'd get a few hits. Not quite human level, but impressively close considering where we were five years ago. Someone with some serious computer power to put behind it could probably do significantly better.

AI got a bad name because of the promises it made in the 60s and 80s, and there are lots of mystics who are critical of any AI, but practical things that have come out of AI research are in use every day by Google, Apple, Microsoft and millions of regular people.

Imagine what one of those 60s AI researchers (or even one from the 80s) would think if they saw the translator app I've got on my phone.

Re:Can their handwriting recognition solve captcha

[phantomfive]

cool, thanks

Old news...

[Anonymous Coward]

I have been talking about Geoffrey Hinton and Jeff Hawkins for 5 years now.

by definition

[schlachter]

captchas are by definition/design unsolvable.

as computers learn to crack current gen captchas; captchas will be updated to be more complex.