Slashdot Mirror
Tiny Changes Can Cause An AI To Fail (bbc.com)
Luthair writes:
According to the BBC there is growing concern in the machine learning community that as their algorithms are deployed in the real world they can be easily confused by knowledgeable attackers. These algorithms don't process information in the same way humans do, a small sticker placed strategically on a sign could render it invisible to a self driving car.
The article points out that a sticker on a stop sign "is enough for the car to 'see' the stop sign as something completely different from a stop sign," while researchers have created an online collection of images which currently fool AI systems. "In one project, published in October, researchers at Carnegie Mellon University built a pair of glasses that can subtly mislead a facial recognition system -- making the computer confuse actress Reese Witherspoon for Russell Crowe."
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time." Although on the plus side, "If you're some political dissident inside a repressive regime and you want to be able to conduct activities without being targeted, being able to avoid automated surveillance techniques based on machine learning would be a positive use."
The article points out that a sticker on a stop sign "is enough for the car to 'see' the stop sign as something completely different from a stop sign," while researchers have created an online collection of images which currently fool AI systems. "In one project, published in October, researchers at Carnegie Mellon University built a pair of glasses that can subtly mislead a facial recognition system -- making the computer confuse actress Reese Witherspoon for Russell Crowe."
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time." Although on the plus side, "If you're some political dissident inside a repressive regime and you want to be able to conduct activities without being targeted, being able to avoid automated surveillance techniques based on machine learning would be a positive use."
If(confused == true) killAllHumans();
Humans Learn from them so will AI's, Intelligence really is determined by how fast something or someone learns from mistakes
Your'e all thinking it, I just said it for you
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time."
Fine, but you only need a great model that's right more often than humans.
now i'm hungry
So THAT explains why everyone in Metropolis was blind to the fact that Clark Kent was really Superman. They were all AIs! Put glasses on and they have no clue how to classify the face.
It also explains why they had great difficulty classifying flying objects: Is it a bird? (p=0.13) Is it a plane? (p=0.32) No, it's Superman! (p=0.96)
Does he really think there won't be 100,000 First World jackasses defacing stop signs for the lulz and religious terrorists hoping that defaced stop signs will cause school buses to crash into synagogues and girls' schools for every 1 political dissident fighting the good fight against repressive regimes?
"I don't know, therefore Aliens" Wafflebox1
Weak AI is characterized by not being intelligent. It is merely statistical classification, algorithmic planning and things like that. It has the advantage that (unlike "strong" AI) it is actually available. But it has the disadvantage that is has zero understanding of what it is doing. As strong AI is not even on the distant horizon, in fact it is unclear whether it is possible to create it at all (despite what a lot of morons that have never understood current research in the field or have not even looked at it like to claim), weak AI is all we will have for the foreseeable future. This means that we have to fake a lot of things that even the tiniest bit of actual intelligence could easily do by itself.
Of course, weak AI is still massively useful, but confusing it with actual intelligence is dangerous. It is however noting any actual expert will ever do. They know. It is just the stupid public that does not get it at all. As usual.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
We've been going through this since the 1980's when we started to make ruled-based expert systems and put them into production. We called that AI too. Now we're doing the same with statistical machine 'intelligence' (optimisation, often), various configurations of trainable neural networks and some hybrids.
These are trainable appliances, not intelligences. They don't have the adaptability and recovery from mistakes of human or (in the case of statistical, sub-symbolic etc.) any explanatory power. To some extent, that's why I liked the ancient expert systems with a why? function, but they were also very brittle. So I think the current hype curve has inflected and this is a good thing, since, apart from this, there are some quite weighty ethical problems as well.
This is not the view of a neo-Luddite, but there's stuff to think about here.
On y va, qui mal y pense!
The thing is that many people will actually become obsolete, if not quite so soon. The problem is that while technically they are intelligent people, they do not really use their intelligence, and that makes their jobs accessible to automation. Of course, those that actually do use their intelligence will not get replaced successfully anytime soon and quite possible not ever. The thing the public does not understand is that at this time we have absolutely no idea how intelligence is created. There is not even a mathematical theory that would work reasonably well in a physical system in this universe.
For example, automated theorem proving (which is one of the few things that may be seen as actually creating "intelligence") is so limited in performance, that making the whole universe into a gigantic computer, it would still be less capable as a smart human being. As a result, we do not have any clue how humans do it and hence cannot emulate that process. There are a few rather strong hints (consistently ignored by the AI fanatics) that things may be a lot more complicated. For example, we do only observe actual intelligence in connection with consciousness. Seeing them as separate is hence not a scientifically sound approach. And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like "consciousness is an illusion" (If so, who has the illusion? Illusions require consciousness!) and the like. That is just a pathetic attempt to cover up how tiny their actual knowledge is in comparison to their grand claims.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"The only way to completely avoid this is to have a perfect model that is right all the time."
Far from true. Many pathological interpretation will solve themselves as the camera moves.
For instance, a pedestrian could blend into the pole behind. Half a second later, the perspective has changed and the pole is behind something else.
So the "tiny change" must hold true as the camera moves, or it won't cause failure.
ID: the nose did not occur naturally, how would we wear glasses otherwise? (apologies to Voltaire)
And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like ...
If you have no idea what it is, how would you know that it is nonsense ?
there is no physical mechanism for consciousness
If there's no physical mechanism, how/why did it evolve ?
The problem with this kind of "AI" (it's not, but let's not go there) is that there's no understanding of what it's actually doing. We're creating tools, "training" them, and then we have no idea what it's basing decisions on past that point.
As such, outside of toys, they aren't that useful and SHOULDN'T BE used for things like self-driving cars. You can never imagine them passing, say, aviation verification because you have literally no idea what it will do.
And it's because of that very problem that they are also unfixable, and unguaranteeable. You can't just say "Oh, we'll train it some more" because the logical consequence of that is that you have to train it on virtually everything you ever want it to do, which kind of spoils the point. And even then, there's no way you can guarantee that it will work next time.
Interesting for beating humans at board games, recognising what you're saying for ordering online, or spotting porn images in image search. Maybe. Some day. But in terms of reliance, we can't rely on them which kills them for all the useful purposes.
It's actually one of the first steps of humans creating systems to do jobs, that the humans do not and cannot understand. Not just one individual could not understand, but nobody, not even the creator can understand or predict what it will do. That's dangerous ground, even if we aren't talking about AI-taking-over-the-world scenarios.
I have a social site that uses several Cognitive AIs from the big three (Amazon, Google, Microsoft) to analyze images that users upload.
The false categorizations from these AIs are often baffling to the human eye. Like WTF can't you tell that is a human face partially obscured by a ball cap? Nope.
It seems that the way humans perceive images is to compare what our eyes tell us to internal 3D models we carry around of the real world - ie actual intelligence of what we are seeing. The AIs are blindly categorizing based on combinations of pixel shapes/colors found in the training set images - so they easily fall for the tricks outlined in TFA.
I've been saying it before and I'll say it again. These automated cars will be forever getting into accidents because they didn't see a child because of the sun, or because it didn't know a cat would run into the road, or because they saw a ball go into the road but did not anticipate a child running after it. There are too many things to code for.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
AI researchers first ran across it when developing neural nets. The longer you allowed a neural net to learn, the more rigid its definition of boundary conditions became. Sometimes so rigid that the net became useless for its intended task. e.g. You could develop a neural net which would stop a train in the correct position at the platform 80% of the time. Further training would increase this to 90%, then 95%, then 99% of the time, but resulted in the net completely flipping out the remaining 1% of the time when it calculated it was going to overshoot by 1 mm outside the trained parameters. The first solution was to stop the learning process and freeze the neural net before it reached this stage, then simply use it in production with the learning capability (ability to modify itself) disabled. The next solution was to use simulated annealing to occasionally reset the specific things the neural net had learned, while retaining the general things it had learned.
You also see this in biological neural nets. As people get older, they tend to get set in their ways, less likely to change their opinions even in the face of contradictory evidence. (As opposed to younger people who are too eager to form an opinion despite weak or the lack of evidence.) I suspect this is also where the aphorism "you can't teach an old dog new tricks" comes from. IMHO this is why trying to lengthen the human lifespan in the pursuit of immortality is a bad idea. Death is nature's way of clearing out neural nets which have become too rigid to respond properly to common variability in situations they encounter. My grandmother hated the Japanese to her dying day (they raped and killed her sister and niece during WWII). If people were immortal, we'd be completely dysfunctional as a society because everyone would be holding grudges and experience-based prejudice for hundreds of years, to the detriment of immediate benefit.
Do you cross the road solely because you have the green light, never mind the speeding truck that is obviously not slowing down? No self driving car worth its salt would. Traffic signs are good and everything but you cannot rely on them 100%. A necessary sign could be missing, not visible, in the wrong place or other drivers might plain not respect traffic rules. Traffic signs are a guideline, not hitting anything and not getting hit by anything is the true golden rule and I*m certain every self driving car on the roads has been engineered with that in mind.
I've waited for a stop sign to change too.
Once, a cop was behind me and the light ahead was yellow. Not wanting to take a chance, I stopped at the light. I waited at the light and after I went through I got pulled over. I was confused. Why did he pull me over? The cop was very confused - why had I gone right through a red light after apparently noticing him behind me, he asked. What? I didn't do that, I said. I stopped and waited for the light to change. Aha! I waited for the light to *change*. It changed from yellow to red, and I went.
I don't agree. There are very few things we call intelligent. I'm sure they have lots of incidental correlations between them.
This is a good point. We have no scientific definition for intelligence or consciousness. Trying to reason about them is just an exercise in contradiction and equivocation.
Chris Mesterharm
The title should have read "Carefully crafted decoy using massive computation resources can fool not up-to-date AI".
Here's how it works:
1. Get access to the AI model you want to fool (and only this one). Not necessarily the source code, but at least you need to be able to use the model as long as you want.
2. Solve a rather complex optimization problem to generate the decoy
3. use your decoy in very controlled conditions (like stated in the linked paper)
While the method for fooling the model is fine (and similar work has been buzzing lately), the conclusion are much weaker than you expect. First, because if you don't have the actual model, you cannot do that. You need to run the actual model you are trying to fool. So that takes out all remote systems with rate limiting accesses. Second, your rely on tiny variation which can be more sensitive than real world variation. Take for example the sticker on road sign, if you took the picture on the sunny day, the decoy will very likely not work on rainy day or at night. Third, if the model evolves, you have to update the decoy. Here's the problem with statistical learning systems: they learn. It's very likely that the model got updated by the time you finished the computation and printing the sticker. Many people believe that future industrial systems will perform online learning which renders those static methods useless.
So yeah, actual research model can be fooled in very specific cases. However, It's not as bad as some article try to make it sound. I'm not saying it won't happen, I'm saying it's not as bad as you think it is. Hey, if you want to impersonate somebody, put some make up and if you want people to crash their car, cover the roadsigns with paint. There you have it, humans are easily fooled by some paint.
Video of some good progressive thrash music
They obviously fooled the slashdot AI into thinking it was news.
Almost every comment posted so far about this story is totally wrong. Adversarial examples are a hot topic in deep learning right now. We've learned a lot about how they work and how to protect against them. They have nothing to do with "weak" versus "strong" AI. Humans are also susceptible to optical illusions, just different ones from neural nets. They don't mean that computers can never be trusted. Computers can be made much more reliable than humans. And they also aren't random failures, or something that's hard to create. In fact, they're trivial to create in a simple, systematic way.
They're actually a consequence of excessive linearity in our models. If you don't know what that means, don't worry about it. It's just a quirk of how models have traditionally been trained. And if you make a small change to encourage them to work in a nonlinear regime, they become much more resistant to adversarial examples. By the time fully autonomous cars hit the roads in a few years, this should be a totally solved problem.
If you build deep learning systems, you need to care about this. If you don't, you can ignore it. It's not a problem you need to care about, any more than you care what activation function or regularization method your car is using.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like ...
If you have no idea what it is, how would you know that it is nonsense ?
Simple logic. If consciousness does not exist, but is just an illusion, however illusions require consciousness, then the claim leads to a contradiction ("Reductio ad absurdum"), and hence the claim is false.
there is no physical mechanism for consciousness
If there's no physical mechanism, how/why did it evolve ?
Do you know that it evolved? Claiming that everything must have evolved is nonsense. Science does not make such a claim. It claims that our bodies have evolved, and that is a very well founded claim given genetics. It is not a 100% thing though, more like 95%. (Not predicting a "God" or such nonsense here, but some other mechanisms could have had major impact.)
Now, we do not have any such data for consciousness. We simply do not know how it works at all and what we have in Physics currently does not contain any mechanism for it. We also have so far failed to detect any "DNA" or other signatures in it that would indicate it is mostly inherited. Claiming that consciousness "evolved" is not scientifically sound at this time as there are no actual observation to support that.
As Physics is very well understood at this time, this is a major problem and the only scientifically valid answer to how consciousness or intelligence works is "We do not know". The question is open.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You say that as if you think that actual intelligent thought isn't just a complex set of rules under the hood.
That happened several years ago, but as I recall he didn't give me a ticket. The explanation made sense to him when I explained I was a bit distracted knowing he was behind me, so I waited for the light to "change".
...that one can devise a clever way to fool what is essentially a sophisticated gigantic pattern-recognition and classification system. Since that is all artificial neural networks are essentially.
We've known this since the 60's
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
How many humans grow up to be geniuses capable of making a mark on society? Most -- not. How many grow up to be, to some slight or large amount, "bad apples"? It doesn't take much to cause humans to grow up to be "failures". Most won't make it big and most won't have any impact. So why should we be surprised when slight variations in input would cause a computer to start down a wrong track.
The different here, is that it is easier to pull the plug and start over again -- or possible erase bad input and overwrite with new -- something that can't be done with humans.
So far, we are proving how poorly humans do at the jobs we might use AI for. Any argument against AI can be turned into an argument against humans.
Really, its something more social leaders should be thinking about if we really want humans to not just evolve -- but survive.
Aren't cars expensive enough for you? Most of a difference between a $40,000 Cadillac is just electronic devices. Do you think you can honestly afford the additional electronics of an AI device? (Then you must be making a boatload of money!)
You people think you're going to have self driving cars in a few years that you can take a nap in on your way to and from work? Think again. This 'technology' is not going to be ready for real world use anytime soon. Stop drinking the 'self driving car' kool-aid, stop believing media hype, and go learn to drive properly and safely, you'll be doing it for quite some time to come, maybe the rest of your natural lives. Get used to it.
That does not make any sense at all. You are confused.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
We (folks who are way further down the spectrum than we want to admit to) are very good at creating buckets. Buckets are awesome. Buckets help me break down problems into manageable sub units and find solutions incrementally as a result.
The problem with buckets... They are always wrong.
If we could replace human drivers with automated drivers instantly. We could make automation happen more quickly than we can without doing that.
If we could violate the laws of physics and make flying cars a reality (some of the quad copters are getting close). 3D sky makes the problems easier. But buckets will be involved again. Lots of buckets.
God: "I don't leave footprints!"
They've taped over the "STOP" sign letters with a "GO".