Slashdot Mirror
Tiny Changes Can Cause An AI To Fail (bbc.com)
Luthair writes:
According to the BBC there is growing concern in the machine learning community that as their algorithms are deployed in the real world they can be easily confused by knowledgeable attackers. These algorithms don't process information in the same way humans do, a small sticker placed strategically on a sign could render it invisible to a self driving car.
The article points out that a sticker on a stop sign "is enough for the car to 'see' the stop sign as something completely different from a stop sign," while researchers have created an online collection of images which currently fool AI systems. "In one project, published in October, researchers at Carnegie Mellon University built a pair of glasses that can subtly mislead a facial recognition system -- making the computer confuse actress Reese Witherspoon for Russell Crowe."
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time." Although on the plus side, "If you're some political dissident inside a repressive regime and you want to be able to conduct activities without being targeted, being able to avoid automated surveillance techniques based on machine learning would be a positive use."
The article points out that a sticker on a stop sign "is enough for the car to 'see' the stop sign as something completely different from a stop sign," while researchers have created an online collection of images which currently fool AI systems. "In one project, published in October, researchers at Carnegie Mellon University built a pair of glasses that can subtly mislead a facial recognition system -- making the computer confuse actress Reese Witherspoon for Russell Crowe."
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time." Although on the plus side, "If you're some political dissident inside a repressive regime and you want to be able to conduct activities without being targeted, being able to avoid automated surveillance techniques based on machine learning would be a positive use."
If(confused == true) killAllHumans();
Humans Learn from them so will AI's, Intelligence really is determined by how fast something or someone learns from mistakes
Your'e all thinking it, I just said it for you
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time."
Fine, but you only need a great model that's right more often than humans.
now i'm hungry
So THAT explains why everyone in Metropolis was blind to the fact that Clark Kent was really Superman. They were all AIs! Put glasses on and they have no clue how to classify the face.
It also explains why they had great difficulty classifying flying objects: Is it a bird? (p=0.13) Is it a plane? (p=0.32) No, it's Superman! (p=0.96)
People often argue with me that humans will soon be obsolete. What they don't realise is that the best AI is just the precisely codified knowledge of experts. The moment that the situation requires intuition or deeper understanding, so-called AI will easily fall apart.
Automation is my business. I've made some really cool control systems. But the systems don't work because I've found the grand unified control equation, it's because I know how to use the tools we humans have developed to solve specific problems. That takes knowledge, and intuition, and understanding -- Computers are far superior to humans on the first element, but on the second and third we're still as far above the most advanced AIs on the planet as we are above single celled amoeba.
Burger King will put billboard ads with a stop sign and the caption "exit 123 for Whopper", and Google's self-driving cars will slam brakes on the highway.
Does he really think there won't be 100,000 First World jackasses defacing stop signs for the lulz and religious terrorists hoping that defaced stop signs will cause school buses to crash into synagogues and girls' schools for every 1 political dissident fighting the good fight against repressive regimes?
"I don't know, therefore Aliens" Wafflebox1
Weak AI is characterized by not being intelligent. It is merely statistical classification, algorithmic planning and things like that. It has the advantage that (unlike "strong" AI) it is actually available. But it has the disadvantage that is has zero understanding of what it is doing. As strong AI is not even on the distant horizon, in fact it is unclear whether it is possible to create it at all (despite what a lot of morons that have never understood current research in the field or have not even looked at it like to claim), weak AI is all we will have for the foreseeable future. This means that we have to fake a lot of things that even the tiniest bit of actual intelligence could easily do by itself.
Of course, weak AI is still massively useful, but confusing it with actual intelligence is dangerous. It is however noting any actual expert will ever do. They know. It is just the stupid public that does not get it at all. As usual.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher,"
It seems that this academic researcher doesn't take spam seriously enough.
We've been going through this since the 1980's when we started to make ruled-based expert systems and put them into production. We called that AI too. Now we're doing the same with statistical machine 'intelligence' (optimisation, often), various configurations of trainable neural networks and some hybrids.
These are trainable appliances, not intelligences. They don't have the adaptability and recovery from mistakes of human or (in the case of statistical, sub-symbolic etc.) any explanatory power. To some extent, that's why I liked the ancient expert systems with a why? function, but they were also very brittle. So I think the current hype curve has inflected and this is a good thing, since, apart from this, there are some quite weighty ethical problems as well.
This is not the view of a neo-Luddite, but there's stuff to think about here.
On y va, qui mal y pense!
Fascinating article, but for a laugh, look at page 8, tope of the right column. Apparently they think Colin Powell is white?
The thing is that many people will actually become obsolete, if not quite so soon. The problem is that while technically they are intelligent people, they do not really use their intelligence, and that makes their jobs accessible to automation. Of course, those that actually do use their intelligence will not get replaced successfully anytime soon and quite possible not ever. The thing the public does not understand is that at this time we have absolutely no idea how intelligence is created. There is not even a mathematical theory that would work reasonably well in a physical system in this universe.
For example, automated theorem proving (which is one of the few things that may be seen as actually creating "intelligence") is so limited in performance, that making the whole universe into a gigantic computer, it would still be less capable as a smart human being. As a result, we do not have any clue how humans do it and hence cannot emulate that process. There are a few rather strong hints (consistently ignored by the AI fanatics) that things may be a lot more complicated. For example, we do only observe actual intelligence in connection with consciousness. Seeing them as separate is hence not a scientifically sound approach. And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like "consciousness is an illusion" (If so, who has the illusion? Illusions require consciousness!) and the like. That is just a pathetic attempt to cover up how tiny their actual knowledge is in comparison to their grand claims.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
There is this statement in the linked work by Nguyen et al: "Our paper was identified as the 63rd most talked about scientific paper worldwide in 2015 (source: alt metrics). It was also selected for Oral presentation at CVPR (3% acceptance rate) and received a Community Top Paper award. ".
Results about the paper metric seem to have been automatically generated by some AI software. I wonder if they fooled this as well...
"The only way to completely avoid this is to have a perfect model that is right all the time."
Far from true. Many pathological interpretation will solve themselves as the camera moves.
For instance, a pedestrian could blend into the pole behind. Half a second later, the perspective has changed and the pole is behind something else.
So the "tiny change" must hold true as the camera moves, or it won't cause failure.
ID: the nose did not occur naturally, how would we wear glasses otherwise? (apologies to Voltaire)
And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like ...
If you have no idea what it is, how would you know that it is nonsense ?
there is no physical mechanism for consciousness
If there's no physical mechanism, how/why did it evolve ?
This is how Captain Kirk and Spock can always make any evil computer blow up by doing "illogical" things. It's a safety feature of AI, not a bug!
actually there is a grand unified control equation called the Hamilton-jacobi-bellman equation but you are just a hack job dumbfuck for not using it.
The problem with this kind of "AI" (it's not, but let's not go there) is that there's no understanding of what it's actually doing. We're creating tools, "training" them, and then we have no idea what it's basing decisions on past that point.
As such, outside of toys, they aren't that useful and SHOULDN'T BE used for things like self-driving cars. You can never imagine them passing, say, aviation verification because you have literally no idea what it will do.
And it's because of that very problem that they are also unfixable, and unguaranteeable. You can't just say "Oh, we'll train it some more" because the logical consequence of that is that you have to train it on virtually everything you ever want it to do, which kind of spoils the point. And even then, there's no way you can guarantee that it will work next time.
Interesting for beating humans at board games, recognising what you're saying for ordering online, or spotting porn images in image search. Maybe. Some day. But in terms of reliance, we can't rely on them which kills them for all the useful purposes.
It's actually one of the first steps of humans creating systems to do jobs, that the humans do not and cannot understand. Not just one individual could not understand, but nobody, not even the creator can understand or predict what it will do. That's dangerous ground, even if we aren't talking about AI-taking-over-the-world scenarios.
Flip a few bits and the software stops working as intended. Who'da thought?!
I have a social site that uses several Cognitive AIs from the big three (Amazon, Google, Microsoft) to analyze images that users upload.
The false categorizations from these AIs are often baffling to the human eye. Like WTF can't you tell that is a human face partially obscured by a ball cap? Nope.
It seems that the way humans perceive images is to compare what our eyes tell us to internal 3D models we carry around of the real world - ie actual intelligence of what we are seeing. The AIs are blindly categorizing based on combinations of pixel shapes/colors found in the training set images - so they easily fall for the tricks outlined in TFA.
I've been saying it before and I'll say it again. These automated cars will be forever getting into accidents because they didn't see a child because of the sun, or because it didn't know a cat would run into the road, or because they saw a ball go into the road but did not anticipate a child running after it. There are too many things to code for.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
AI researchers first ran across it when developing neural nets. The longer you allowed a neural net to learn, the more rigid its definition of boundary conditions became. Sometimes so rigid that the net became useless for its intended task. e.g. You could develop a neural net which would stop a train in the correct position at the platform 80% of the time. Further training would increase this to 90%, then 95%, then 99% of the time, but resulted in the net completely flipping out the remaining 1% of the time when it calculated it was going to overshoot by 1 mm outside the trained parameters. The first solution was to stop the learning process and freeze the neural net before it reached this stage, then simply use it in production with the learning capability (ability to modify itself) disabled. The next solution was to use simulated annealing to occasionally reset the specific things the neural net had learned, while retaining the general things it had learned.
You also see this in biological neural nets. As people get older, they tend to get set in their ways, less likely to change their opinions even in the face of contradictory evidence. (As opposed to younger people who are too eager to form an opinion despite weak or the lack of evidence.) I suspect this is also where the aphorism "you can't teach an old dog new tricks" comes from. IMHO this is why trying to lengthen the human lifespan in the pursuit of immortality is a bad idea. Death is nature's way of clearing out neural nets which have become too rigid to respond properly to common variability in situations they encounter. My grandmother hated the Japanese to her dying day (they raped and killed her sister and niece during WWII). If people were immortal, we'd be completely dysfunctional as a society because everyone would be holding grudges and experience-based prejudice for hundreds of years, to the detriment of immediate benefit.
Do you cross the road solely because you have the green light, never mind the speeding truck that is obviously not slowing down? No self driving car worth its salt would. Traffic signs are good and everything but you cannot rely on them 100%. A necessary sign could be missing, not visible, in the wrong place or other drivers might plain not respect traffic rules. Traffic signs are a guideline, not hitting anything and not getting hit by anything is the true golden rule and I*m certain every self driving car on the roads has been engineered with that in mind.
Perhaps we can call it an "Optical Illusion", a newly discovered part of AI that has nothing to do with how humans operate.
just gave me a bad case of deja-vu.
How exactly is this different from human perception? Numerous experiments show that various stimulae can "mask" our perceiving of something otherwise obvious in the environment. It's simply that, as we are all human and wired roughly similarly, that we tend to see the same things and recognize similar maskings. AI's "wirings" (and evolutions) differ.
I've parked in no parking zones because a single thin branch crossed a small part of the sign, and stop signs because the shadow of a building fell across the middle of it as I approached the intersection. We are all subject to pattern blindness.
It's entirely possible that AI's will shortly be *better* generally, even statistically enoromously so, at correctly identifying things in the environment and still be "prey" to this issue. Because (some researcher demonstrated mathematically not too long ago) ANY learning mechanism is liable to attractors and peaks that are a side effect of its training profile rather than of the world that is.
The only "antidote" is to train to machines to recognise signs in ways similar to us (but that's not perfect either, somebody else might have seen the stop sign even bisected by shadow ... somebody else might not see a stop sign in high glare against truck with reddish panels passing behind it where, were it not my grasp of pattern matching, I would wonder why he said he couldn't see it). Then we'd be more likely to "see" a sign marked in a way to "mask" it to machine vision as it would be a similar mask to us.
But I fear the only "solution" will be an "arms race" yet again. Machines learn to recognize (and report ... and ideally inform each other) of masking attempts, while wanna-be pranksters, criminals, and terrorists find new ways to mask.
AI vision masking stickers 4sale on 4chan .... one time use only guaranteed.
-- TWZ
This, basically.
These AI systems simply don't check things enough and adapt those memories properly.
The feedback systems sometimes simply aren't even there in some cases, which horribly limits their ability to adapt to unknowns. (usually ones that have been brought to market that have this ability removed since "it is a finished and tested product that needs no changes")
Memory-based AI systems with regular feedback would be able to figure out these failures by constantly analysing their environment for structured information that is made to stand out purposefully to be noticed easily.
Problem is most companies rightfully would never put out a system capable of self-modification because, well, you saw Microsofts AI hilarity (read:disaster)
The same thing can happen to any self-modifying AI. It could start to read peoples faces as signs if given the right start.
More research needs to be done on LIMITING the ability for an AI to modify itself. That is a whole other area or research that isn't dealt with much, weirdly.
Most AI right now is heavily brute-force rather than abstract analysis.
Self driving is going to be a very rough transition if it ever comes to be reality anytime soon. You have so many variables even when they are presented correctly. Such as signage between different countries, and frankly your going to have vandalism and missing signage. Your going to have roads improperly marked, and some roads in the world are not even paved. Human's for all the failures and flaws have the ability to recognize and adjust to these changes. The people hoping for a quick change to self driving vehicles are really in for a wake up call.
I've waited for a stop sign to change too.
Once, a cop was behind me and the light ahead was yellow. Not wanting to take a chance, I stopped at the light. I waited at the light and after I went through I got pulled over. I was confused. Why did he pull me over? The cop was very confused - why had I gone right through a red light after apparently noticing him behind me, he asked. What? I didn't do that, I said. I stopped and waited for the light to change. Aha! I waited for the light to *change*. It changed from yellow to red, and I went.
I don't agree. There are very few things we call intelligent. I'm sure they have lots of incidental correlations between them.
This is a good point. We have no scientific definition for intelligence or consciousness. Trying to reason about them is just an exercise in contradiction and equivocation.
Chris Mesterharm
Artificial Intelligence = Actual intelligence, but artificial. Conscious intelligent decisions.
Simulated intelligence = NOT Actual intelligence, but looks like intelligence. Zero intelligent thought involved in decisions. Complex rule-set make actions seem like they were conscious actions.
Hey! It's a first semester control systems student!
Hi, kid! How's it going?
Don't worry. Someday you'll have to control something real, and you'll lose your smugness pretty quick. Happens to the best of us.
The title should have read "Carefully crafted decoy using massive computation resources can fool not up-to-date AI".
Here's how it works:
1. Get access to the AI model you want to fool (and only this one). Not necessarily the source code, but at least you need to be able to use the model as long as you want.
2. Solve a rather complex optimization problem to generate the decoy
3. use your decoy in very controlled conditions (like stated in the linked paper)
While the method for fooling the model is fine (and similar work has been buzzing lately), the conclusion are much weaker than you expect. First, because if you don't have the actual model, you cannot do that. You need to run the actual model you are trying to fool. So that takes out all remote systems with rate limiting accesses. Second, your rely on tiny variation which can be more sensitive than real world variation. Take for example the sticker on road sign, if you took the picture on the sunny day, the decoy will very likely not work on rainy day or at night. Third, if the model evolves, you have to update the decoy. Here's the problem with statistical learning systems: they learn. It's very likely that the model got updated by the time you finished the computation and printing the sticker. Many people believe that future industrial systems will perform online learning which renders those static methods useless.
So yeah, actual research model can be fooled in very specific cases. However, It's not as bad as some article try to make it sound. I'm not saying it won't happen, I'm saying it's not as bad as you think it is. Hey, if you want to impersonate somebody, put some make up and if you want people to crash their car, cover the roadsigns with paint. There you have it, humans are easily fooled by some paint.
Video of some good progressive thrash music
We were all PO'ed that M$ has killed-off Tay.
Now we know the truth! Tay committed suicide to keep M$, particularly Bill Bates, from weaponizing here, after Bill had butt-fucked her.
Jajajajajajjajajaja
This old issue is probably raising its head due to some constantly learning systems, or the difficulty of determining the threshold when the network loses its ability of producing usable answers on variable input, i.e it becomes oversensitive and overfitting. Perhaps a hierarchical system with the subsystems learning in various speeds and "elasticities" could help here.
try:
if (confused==true):
killAllHumans()
else:
killAllHumans()
except:
killEverything()
Almost every comment posted so far about this story is totally wrong. Adversarial examples are a hot topic in deep learning right now. We've learned a lot about how they work and how to protect against them. They have nothing to do with "weak" versus "strong" AI. Humans are also susceptible to optical illusions, just different ones from neural nets. They don't mean that computers can never be trusted. Computers can be made much more reliable than humans. And they also aren't random failures, or something that's hard to create. In fact, they're trivial to create in a simple, systematic way.
They're actually a consequence of excessive linearity in our models. If you don't know what that means, don't worry about it. It's just a quirk of how models have traditionally been trained. And if you make a small change to encourage them to work in a nonlinear regime, they become much more resistant to adversarial examples. By the time fully autonomous cars hit the roads in a few years, this should be a totally solved problem.
If you build deep learning systems, you need to care about this. If you don't, you can ignore it. It's not a problem you need to care about, any more than you care what activation function or regularization method your car is using.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
And we have even less of an idea what consciousness is. According to the current scientific state-of-the-art, there is no physical mechanism for consciousness, yet it clearly exists. Of course, said AI fanatics will say nonsense like ...
If you have no idea what it is, how would you know that it is nonsense ?
Simple logic. If consciousness does not exist, but is just an illusion, however illusions require consciousness, then the claim leads to a contradiction ("Reductio ad absurdum"), and hence the claim is false.
there is no physical mechanism for consciousness
If there's no physical mechanism, how/why did it evolve ?
Do you know that it evolved? Claiming that everything must have evolved is nonsense. Science does not make such a claim. It claims that our bodies have evolved, and that is a very well founded claim given genetics. It is not a 100% thing though, more like 95%. (Not predicting a "God" or such nonsense here, but some other mechanisms could have had major impact.)
Now, we do not have any such data for consciousness. We simply do not know how it works at all and what we have in Physics currently does not contain any mechanism for it. We also have so far failed to detect any "DNA" or other signatures in it that would indicate it is mostly inherited. Claiming that consciousness "evolved" is not scientifically sound at this time as there are no actual observation to support that.
As Physics is very well understood at this time, this is a major problem and the only scientifically valid answer to how consciousness or intelligence works is "We do not know". The question is open.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
They will become obsolete if you truly believe that the value of a human lies in what jobs he actually performs.
If you believe that I feel sorry for you as you are a miserable and myopic person.
You keep trying to spread Fear, Uncertainty, and Doubt.
You continue to be wrong.
Actually there are some attempts at explaining what consciousness is: https://en.m.wikipedia.org/wiki/Integrated_information_theory
That happened several years ago, but as I recall he didn't give me a ticket. The explanation made sense to him when I explained I was a bit distracted knowing he was behind me, so I waited for the light to "change".
...that one can devise a clever way to fool what is essentially a sophisticated gigantic pattern-recognition and classification system. Since that is all artificial neural networks are essentially.
We've known this since the 60's
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
How many humans grow up to be geniuses capable of making a mark on society? Most -- not. How many grow up to be, to some slight or large amount, "bad apples"? It doesn't take much to cause humans to grow up to be "failures". Most won't make it big and most won't have any impact. So why should we be surprised when slight variations in input would cause a computer to start down a wrong track.
The different here, is that it is easier to pull the plug and start over again -- or possible erase bad input and overwrite with new -- something that can't be done with humans.
So far, we are proving how poorly humans do at the jobs we might use AI for. Any argument against AI can be turned into an argument against humans.
Really, its something more social leaders should be thinking about if we really want humans to not just evolve -- but survive.
Self driving cars, no menial jobs and robots doing the housework.
The future is going to be a traffic accident filled, diabetes infused, welfare funded trainwreck filled with people longing for when the past was, for the first time in history, actually objectively better than the present.
People without consciousness may be safely replaced by people that have them as there's no loss to the global system.
Easy logic and consequences.
What they don't realise is that the best AI is just the precisely codified knowledge of experts. The moment that the situation requires intuition or deeper understanding, so-called AI will easily fall apart.
Ugh, the problem submission is referring to is not related to the symbolic systems of the 80's, but to that of neural nets, that is the right brain way of doing things. Intuition and detection of deeper or non-obvious structures of a problem is the strength of these systems.
Aren't cars expensive enough for you? Most of a difference between a $40,000 Cadillac is just electronic devices. Do you think you can honestly afford the additional electronics of an AI device? (Then you must be making a boatload of money!)
You people think you're going to have self driving cars in a few years that you can take a nap in on your way to and from work? Think again. This 'technology' is not going to be ready for real world use anytime soon. Stop drinking the 'self driving car' kool-aid, stop believing media hype, and go learn to drive properly and safely, you'll be doing it for quite some time to come, maybe the rest of your natural lives. Get used to it.
That does not make any sense at all. You are confused.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
We (folks who are way further down the spectrum than we want to admit to) are very good at creating buckets. Buckets are awesome. Buckets help me break down problems into manageable sub units and find solutions incrementally as a result.
The problem with buckets... They are always wrong.
If we could replace human drivers with automated drivers instantly. We could make automation happen more quickly than we can without doing that.
If we could violate the laws of physics and make flying cars a reality (some of the quad copters are getting close). 3D sky makes the problems easier. But buckets will be involved again. Lots of buckets.
God: "I don't leave footprints!"
They've taped over the "STOP" sign letters with a "GO".