Slashdot Mirror
The Flaw Lurking In Every Deep Neural Net
mikejuk (1801200) writes "A recent paper, 'Intriguing properties of neural networks,' by Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow and Rob Fergus, a team that includes authors from Google's deep learning research project, outlines two pieces of news about the way neural networks behave that run counter to what we believed — and one of them is frankly astonishing. Every deep neural network has 'blind spots' in the sense that there are inputs that are very close to correctly classified examples that are misclassified. To quote the paper: 'For all the networks we studied, for each sample, we always manage to generate very close, visually indistinguishable, adversarial examples that are misclassified by the original network.' To be clear, the adversarial examples looked to a human like the original, but the network misclassified them. You can have two photos that look not only like a cat but the same cat, indeed the same photo, to a human, but the machine gets one right and the other wrong. What is even more shocking is that the adversarial examples seem to have some sort of universality. That is a large fraction were misclassified by different network architectures trained on the same data and by networks trained on a different data set. You might be thinking 'so what if a cat photo that is clearly a photo a cat is recognized as a dog?' If you change the situation just a little and ask what does it matter if a self-driving car that uses a deep neural network misclassifies a view of a pedestrian standing in front of the car as a clear road? There is also the philosophical question raised by these blind spots. If a deep neural network is biologically inspired we can ask the question, does the same result apply to biological networks? Put more bluntly, 'Does the human brain have similar built-in errors?' If it doesn't, how is it so different from the neural networks that are trying to mimic it?"
Of course the human brain has errors in its pattern matching ability. Who hasn't seen something out of the corner of their eye and thought it was dog when really it was a paper bag blowing in the wind? The brain makes snap judgments, because there's a trade off between correctness and speed. If your brain mistakes a rustle of bushes for a tiger, so what? I'd rather have it misinform me, erring on the side of tiger, than wait for all information to be in before making a 100% accurate decision. This is the basis of intuition.
I don't think a computer ai will be perfect, either, because "thinking" fuzzily enough to develop intuition means it's going to be wrong sometimes. The interesting thing is how quickly we get pissed off at a computer for guessing wrong compared to a human. When you call a business and get one of those automated answering things and it asks you, "Now please, tell me the reason for your call. You can say 'make a payment,' 'inquire about my loan...'" etc etc, we get really pissed off when we say 'make a payment' and it responds "you said, cancel my account, did I get that right?" But when a human operator doesn't hear you correctly and asks you to repeat what you said, we say "Oh, sure," and repeat ourselves without a second thought. There's something about it being a machine that makes us demand perfection in a way we'd never expect from a human.
We don't have a state-run media we have a media-run state.
A neural network is not by any stretch of the imagination a simulation of how the brain works. It incorporates a few principles similar to brain function, but it is NOT an attempt to re-build a biological brain.
Anybody relying on "it's a bit like how humans work lol" to assert the reliability of an ANN is a fucking idiot, and probably trying to hawk a product in the commercial sector rather than in academia.
Aren't optical illusions pretty much something like this?
And, my second question, just because deep neural networks are biologically inspired, can we infer from this kind of issue in computer programs that there is likely to be a biological equivalent? Or has everyone made the same mistake and/or we're seeing a limitation in the technology?
Maybe the problem isn't with the biology, but the technology?
Or are we so confident in neural networks that we deem them infallible? (Which, obviously, they aren't.)
Lost at C:>. Found at C.
Deep neural networks are implicitly generating dynamic-ontologies. The 'mis-categorisation' occurs when you only have one functional exit point. The fact is that if you are within the network itself, the adversarial are held in-frame alongside other possibilities, and the network only tilts towards one when the prevailing system requires it through external stimulus. From the outside it will look like an error, (because we already decided that) but internally each possible interpretation is valid.
I can't speak to what the car manufacturers are doing, but Google's algorithms do not include a neural network. They do use "machine learning", but neural networks are just one form of machine learning.
The human brain has multiple neural nets and a voter.
I am face blind and completely non-visual, but I do recognize people. I can because the primary way that we recognize people is by encoding a schematic image of the face, but many other nets are in play. For example, I use hair style, clothing, and height. So does everybody, though. But for most people that just gives you extra confidence.
Conclusion: Neural nets in your brain having blind spots is no problem whatsoever. The entire system is highly redundant.
All neural nets try to predict, and predictions can be foiled.
People can be fooled by optical illusions, too.
Every semi- or full automated face recognition system uses neural networks, and they are sold to us as safety critical. If this flaw is really as fundamental as it is claimed to be, it means that it's pretty easy to outsmart those systems by only slightly changing your look, so your co-conspirators still recognize you, but you will raise no alarm on any system that is supposed to spot you.
What if that supposed pedestrian really is no more than a clear stretch of road, and it is we who err in notifying the road's next of kin, who are themselves no more than a dirt path and a pedestrian walkway?
how many pairs of boxer shorts should you own?
A dynamic non-linear system has some weird boundary conditions. Who could ever have predicted that? </s>
Why wasn't this assumed from the beginning and it shown that it wasn't an issue?
It is almost like the article is saying that something a computer did was not perfectly in line with human reasoning. We should stop being life-centric and realize that if the computer says two pictures of the same cat should not be classified in the same way, the computer is simply wiser than we are, and if we don't believe it the computer will beat our asses at chess and then we'll see who is smarter.
That makes sense. Rare errors will be screened out if instead of a single deterministic selection process you use a distribution of schemes and select based on the most probable outcome... I am wondering what our brain does with its minority reports...
.oO0(?)
Neural networks are only one way to build machine learning classifiers. Everything we've learnt about machine learning tells us not to rely on a single method/methodology and that we will consistently get better results by taking the consensus of multiple methods. We just need to make sure that a majority of the other methods we use have different blind spots to the ones the neural networks have.
Korma: Good
Indeed, remembering the experiments done in the 1960s by Sperry and Gazzaniga on patients who had a divided corpus callosum, there are clearly multiple systems that can argue with each other about recognising objects. Maybe part of what makes us really good at it, is not relying on one model of the world, but many overlaid views of the same data by different mechanisms.
SoylentNews is the replacement for /.
reddit is of another kind.
factor 966971: 966971
Like when you are walking behind a guy with long hair and think she might be kinda hot. Doh!
"Sure it's possible that computers may one day be as smart as humans, but who wants a computer that remembers the words to the Flintstones jingle and forgets to pay the rent?"
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
The Probably Approximately Correct (PAC) learning model is what formally justifies the tendency of neural networks to "learn" from data (see Wikipedia).
While the PAC model does not depend on the probability distribution which generates training and test data, it does assume that they are *the same*. So by "adversarially" choosing test data, the researchers are breaking this important assumption. Therefore it is in some ways not surprising that neural networks have this vulnerability. It shouldn't be an issue in real life, assuming that the training data and the testing data really do come from the same probability distribution.
That said, this shows why you wouldn't want to use neural networks for, say, cryptography.
This seems to be almost a form of cognitive bias as defined and studied by Tversky and Kahneman. I direct you to : http://en.wikipedia.org/wiki/L.... Or as previously pointed out optical illusions seem to be an equivalence.
While I share your view that expecting the mind to be explained as a single neural network (in the Comp. Sci. sense) is probably simplistic, I don't think modeling it as multiple neural nets and a voter fixes the problem. I am not quite sure about this, but isn't a collection of neural nets and a voter equivalent to a single neural net? Or, to put it a slightly different way, for any model that consists of multiple neural nets and a voter, there is a single neural net that is functionally identical? I am assuming the voter is there to pick the most common classification by the component networks.
NN technology is 60 years old. Some A.I. pundts disliked in the beginning such as Minsky in his 1969 book Perceptrons. Many of these flaws have been LONG known.
News from the future, rhinos find success adapting to suburban environments with discarded carpet camouflage, people slow to adapt.
Great, everyone is going to start having moles on their cheeks.
There are two types of people in the world: Those who crave closure
The sounds similar to the Napoleon Dynamite Problem, the problem encountered in the Netflix Prize challenge of predicting user ratings for some particular films. For most films knowledge of an individuals preferences for some films were good predictors for their preferences of other films. Yet preferences for some particular films were hard to predict, notably the eponymous Napoleon Dynamite.
Neural network identification and automated prediction of individual film ratings are both classification tasks. Example sets for both of these problems contain particular difficult-to-classify examples. So perhaps this phenomena of "adversarial examples" described in the Szegedy et. al. article is more generally a property of datasets and classification, not an artifact of implementing classification using neural networks.
Ceci n'est pas une signature.
incompleteness theorem. And as some earlier posters' stated, the correction is simple. Simply look again. The 2nd image collected will be different from the previous and if the NN is correct, will resolve to the correct interpretation.
Is the term we use for errors in human neural networks. If you do a google search for optical illusions you will find many examples. From pictures that look like they are 3d, but are just 2d, to sizes that appear to change but aren't, we make lots of errors. Not to mention the many many cases where we think "THAT'S A FACE", whether it is jesus on toast, a face on the moon, or just some trees on a mountainside, we are hardwired to assume things are faces.
excitingthingstodo.blogspot.com
It's true that there is more information in video data, but the problem described in the article is certainly not caused by the restriction to stills.
What AI really needs is a wife that nags it if it f8cks up.
Humans seem pretty subject to close-call-foul-ups too. When proof-reading my own writing, often I don't spot a problem because my mind translates the pattern as I intended, not as I wrote it. For example, if I meant to write "Finding the Right Person for the Job..." but instead wrote it as "Finding the Right Pearson for the Job..." (note the "a"), there's a fairly high chance I'd miss it because the pattern of what I meant clogs my objectivity, even after multiple readings.
And I have come close to hitting pedestrians who wore clothing resembling the colors of the street at night. (Please don't wear dark clothes at night, people. It's hard to see with window glare etc.)
Table-ized A.I.
When analyzing a still picture/scene, your eye moves its high resolution central area of its camera around the low level visual features of the image. Thus the image is processed over time as many different images.
The images in that time sequence occur at slightly different locations of the visual light-sensor array (visual field) and at slightly different angles and each image has considerably different pixel resolution trained on each part of the scene.
So that would still almost certainly give some robustness against these artifacts (unlucky particular images) being able to fool the system.
Time and motion are essential in disambiguating 3D/4D world with 2D imaging.
Also, I would guess that having learning algorithms that preferentially try to encode a wide diversity of different kinds of low level features would also protect against being able to be fooled, even by a single image, but particularly over a sequence of similar but not identical images of the same subject.
Where are we going and why are we in a handbasket?