Slashdot Mirror
A Common Logic To Seeing Cats and the Cosmos
An anonymous reader sends this excerpt from Quanta Magazine:
"Using the latest deep-learning protocols, computer models consisting of networks of artificial neurons are becoming increasingly adept at image, speech and pattern recognition — core technologies in robotic personal assistants, complex data analysis and self-driving cars. But for all their progress training computers to pick out salient features from other, irrelevant bits of data, researchers have never fully understood why the algorithms or biological learning work.
Now, two physicists have shown that one form of deep learning works exactly like one of the most important and ubiquitous mathematical techniques in physics, a procedure for calculating the large-scale behavior of physical systems such as elementary particles, fluids and the cosmos. The new work, completed by Pankaj Mehta of Boston University and David Schwab of Northwestern University, demonstrates that a statistical technique called "renormalization," which allows physicists to accurately describe systems without knowing the exact state of all their component parts, also enables the artificial neural networks to categorize data as, say, "a cat" regardless of its color, size or posture in a given video.
"They actually wrote down on paper, with exact proofs, something that people only dreamed existed," said Ilya Nemenman, a biophysicist at Emory University.
Now, two physicists have shown that one form of deep learning works exactly like one of the most important and ubiquitous mathematical techniques in physics, a procedure for calculating the large-scale behavior of physical systems such as elementary particles, fluids and the cosmos. The new work, completed by Pankaj Mehta of Boston University and David Schwab of Northwestern University, demonstrates that a statistical technique called "renormalization," which allows physicists to accurately describe systems without knowing the exact state of all their component parts, also enables the artificial neural networks to categorize data as, say, "a cat" regardless of its color, size or posture in a given video.
"They actually wrote down on paper, with exact proofs, something that people only dreamed existed," said Ilya Nemenman, a biophysicist at Emory University.
offers an interesting look upon what generalizes, and what does not generalize, when you "zoom out" from a system built up of neighbouring spins, replacing groups of neighbouring spins by single-spin blocks. The interesting link with CS is the fact that the arxiv paper considers binary spins. Thinking this through, the paper might indeed offer some explanation for large-scale behaviour ( read: macroscopic ) as composed of small-scale ( read: microscopic ) interactions. Quite interesting, indeed.
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Cats ARE from the Cosmos.
They are superior beings from a far far far away galaxy.
Can anyone prove they evolved on Earth? Does the Bible say God created them?
No and No!
See.
I think you might be on to something.
So who invented buttered toast? And if you strap it to the cosmos and drop it, what happens?
That's OK. It looks like we can open a can of kitty food and distract it.
Faster! Faster! Faster would be better!
If I understand the article correctly, what the paper says is that when the machines rise, we should wear false ears and whiskers as camoflage. Thank God for science!
Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
That we might make an artificial intelligence greater than human intelligence and it will sit around watching lolcats.
"If any question why we died, Tell them because our fathers lied."
Let's hope this approach works better than the current state of the art.
If it weren't for deadlines, nothing would be late.
This article is way overblown. This is not the kind of paper that is likely to attract significant attention in the deep learning community. And the person who they got to say it was important, Ilya Nemenman, is not someone I have heard of.
Move along. Nothing to see here.
The
I also think they're underestimating cats. But If they're connecting deep learning systems to telescopes (which is not explicitly stated), when a cat is positively identified, perhaps somewhere millions of light-years away and hundreds of thousands of light-years across, I guess we'll be sorry.
The Admin and the Engineer
Could you comment on some of the claims in the abstract?
1. Deep learning is a broad set of techniques that uses multiple layers of representation...
Agreed- that's what "deep" implies.
Is multi-scale analysis a primary component of 'deep learning'?
This may be true in vision, but not in general (e.g. in linguistics tasks and in speech, there is usually not a natural notion of scale).
2. "relatively little is understood theoretically about why these techniques are so successful at feature learning and compression.
True... deep learning methods are not very easy to analyze (personally I am skeptical that there is much point in trying very hard to analyze them).
"We construct an exact mapping from the variational renormalization group..." Is this not new, not correct, or is this simply not of much use to deep learning?
I think the closest is to say it's not of much use. I didn't read the paper super carefully (and I'm not a physicist so am not familiar with the renormalization group), but I imagine the analogy is not very close at all and only applies in specific cases, e.g. in convolutional nets or something like that.
The renormalization group theory is so general and powerful, it's had profound impacts on many areas of theoretical and mathematical physics. Do you think this can't or won't impact the field of deep learning? If deep learning has multi-scale analysis at its heart, it appears on the surface that RG should be a good treatment. Have there been attempts to use RG for deep learning aside from the present work?
If the connection is real, it would seem to suggest that perhaps deep learning may have something to offer physics, if it really is "employing a generalized RG-like scheme." Do you have any comment on this?
I haven't read the paper in detail but I just don't think it's plausible that there is a very interesting connection as they are such different things.
To pick a random example, imagine you are a botanist and someone told you there is a connection between hydroelectric dams and oranges. Even if there is a connection, it's probably not something that is going to help you very much, and you probably wouldn't be so excited to read the paper explaining the purported connection.
The
As somebody who has worked on artificial neural networks in the past, and holds a physics degree, I don't think that this assessment is wrong.
I think at this point this is more a curiosity. Interesting in it's own right, but not something that I would expect to yield new and improved algorithms.