Tiny Changes Can Cause An AI To Fail (bbc.com)

Luthair writes: According to the BBC there is growing concern in the machine learning community that as their algorithms are deployed in the real world they can be easily confused by knowledgeable attackers. These algorithms don't process information in the same way humans do, a small sticker placed strategically on a sign could render it invisible to a self driving car.
The article points out that a sticker on a stop sign "is enough for the car to 'see' the stop sign as something completely different from a stop sign," while researchers have created an online collection of images which currently fool AI systems. "In one project, published in October, researchers at Carnegie Mellon University built a pair of glasses that can subtly mislead a facial recognition system -- making the computer confuse actress Reese Witherspoon for Russell Crowe."

One computer academic says that unlike a spam-blocker, "if you're relying on the vision system in a self-driving car to know where to go and not crash into anything, then the stakes are much higher," adding ominously that "The only way to completely avoid this is to have a perfect model that is right all the time." Although on the plus side, "If you're some political dissident inside a repressive regime and you want to be able to conduct activities without being targeted, being able to avoid automated surveillance techniques based on machine learning would be a positive use."

Comments

Re:Oh no

[Anonymous Coward]

else killAllHumans();

Re:Speed Bump

[SlaveToTheGrind]

Fine, but you only need a great model that's right more often than humans.

I don't know that I've ever heard of a human driver who ran a stop sign thinking it was a banana.

Remember, this is "weak" AI

[gweihir]

Weak AI is characterized by not being intelligent. It is merely statistical classification, algorithmic planning and things like that. It has the advantage that (unlike "strong" AI) it is actually available. But it has the disadvantage that is has zero understanding of what it is doing. As strong AI is not even on the distant horizon, in fact it is unclear whether it is possible to create it at all (despite what a lot of morons that have never understood current research in the field or have not even looked at it like to claim), weak AI is all we will have for the foreseeable future. This means that we have to fake a lot of things that even the tiniest bit of actual intelligence could easily do by itself.

Of course, weak AI is still massively useful, but confusing it with actual intelligence is dangerous. It is however noting any actual expert will ever do. They know. It is just the stupid public that does not get it at all. As usual.

Re:Remember, this is "weak" AI

[religionofpeas]

Dunno about you, but this is not how my mind works. I honor or ignore stop signs depending on the present conditions (e.g. empty roads at night, etc). How's that weak?

The recognition of the stop sign is weak. You don't go: "hey, that's a red octagonal sign with the word "STOP" in it, I think that's a stop sign", you just automatically recognize is. In your case, the recognition is then tied to a more complex decision mechanism.

Re:Remember, this is "weak" AI

[drinkypoo]

The recognition of the stop sign is weak. You don't go: "hey, that's a red octagonal sign with the word

I don't consciously do that, no. But somewhere in my brain, various structures which have formed in my brain for recognizing stop signs are going into action. It's not any more or less automatic than what the computer is doing. The difference isn't whether something is happening automatically (by definition, what the computer is doing is automation) but that in my head, it's both analog and parallel. Lots of different parts of my brain are trying to recognize that stop sign at the same time, and whichever ones shout loudest are going to be heard. Some part of my brain might think it's a giant strawberry, but even if that comes through with the impression that it's actually a stop sign, I'm still going to hit the brake pedal because that's what I'm up to right now. (Besides, I brake for giant strawberries.)

Current AI isn't...

[hughbar]

We've been going through this since the 1980's when we started to make ruled-based expert systems and put them into production. We called that AI too. Now we're doing the same with statistical machine 'intelligence' (optimisation, often), various configurations of trainable neural networks and some hybrids.

These are trainable appliances, not intelligences. They don't have the adaptability and recovery from mistakes of human or (in the case of statistical, sub-symbolic etc.) any explanatory power. To some extent, that's why I liked the ancient expert systems with a why? function, but they were also very brittle. So I think the current hype curve has inflected and this is a good thing, since, apart from this, there are some quite weighty ethical problems as well.

This is not the view of a neo-Luddite, but there's stuff to think about here.

Re:Speed Bump

[gweihir]

That is nonsense. AIs have never surpassed human performance (of course, you always need to compare to a human expert) and there is no rational reason to expect that they ever will. Incidentally, said "great" model is currently completely out of reach, even for relatively simple things like driving a car (which almost all humans can learn to do, i.e. it does not require much). The best we will get is a model that solves a lot of standard situations from a catalog and appeals to human help in the rest. That is pretty useful and will make things like self-driving cars a reality, but some things that smart human beings can do will likely remain out of reach for a long time and quite possibly forever.

Re:Mistakes

[gweihir]

"Weak" AI (and that is what we are talking about here) cannot "learn from mistakes". That skill is reserved for actual intelligence and "strong" AI. Strong AI has the little problem that it does not exist as it is currently completely unknown how it could be created, despite about half a century of intense research.

"AI"

[ledow]

The problem with this kind of "AI" (it's not, but let's not go there) is that there's no understanding of what it's actually doing. We're creating tools, "training" them, and then we have no idea what it's basing decisions on past that point.

As such, outside of toys, they aren't that useful and SHOULDN'T BE used for things like self-driving cars. You can never imagine them passing, say, aviation verification because you have literally no idea what it will do.

And it's because of that very problem that they are also unfixable, and unguaranteeable. You can't just say "Oh, we'll train it some more" because the logical consequence of that is that you have to train it on virtually everything you ever want it to do, which kind of spoils the point. And even then, there's no way you can guarantee that it will work next time.

Interesting for beating humans at board games, recognising what you're saying for ordering online, or spotting porn images in image search. Maybe. Some day. But in terms of reliance, we can't rely on them which kills them for all the useful purposes.

It's actually one of the first steps of humans creating systems to do jobs, that the humans do not and cannot understand. Not just one individual could not understand, but nobody, not even the creator can understand or predict what it will do. That's dangerous ground, even if we aren't talking about AI-taking-over-the-world scenarios.

Re:"AI"

[fluffernutter]

I think most people putting a lot of money into AI don't really understand the difference between programming a response to a stop sign and understanding what a stop sign is. If a stop sign is bent over from a previous accident and covered in snow, you will still stop if you truly understand what that object is. If you have programmed a stop sign, the vehicle is lucky to sail on through because stop signs aren't white objects on a pole close to the ground. How do you program for even every physical condition a stop sign may find itself in?

Re:"AI"

[mesterha]

How do you program for even every physical condition a stop sign may find itself in?

This assume the AI even needs to see the stop sign. A driverless car has many advantages over a human. It can have a database of the locations of all stop signs. It have telemetry information from other nearby cars. It can have 360 degree sensors that include cameras and lidar. It doesn't get tired or drunk. It can receive updates based on "mistakes" made by other driverless cars.

Even if there are problems with some of the information, the system can still perform an action based on the total information that is safe for the people in the situation. For example, even if doesn't see a new stop sign, it might still have enough information to see that there is another car entering the intersection.

Of course, it will make mistakes, but it just has to make significantly fewer mistakes than humans. Honestly, given the pace of progress, that doesn't seem too hard.

Re:"AI"

[ledow]

Sigh.

If you have to record every stop sign for the cars, you don't have AI. You have an expert system, at best, running on a database.

Also, you could just make the roads like railways. It would be infinitely safer to have driverless cars on driverless-only, fixed route highways. No pedestrians, no unexpected actions, no junctions to deal with, no need to "interpret" anything. Nobody has serious objection to automation. What we object to is pretending that these things are thinking or learning when they're following orders. And rather than try to make them "human" and learn how to read our roads, it would be much easier to just paint a giant magnetic-paint line on the road to indicate the middle of the lane and have done with it, rather than try to train a computer vision module to recognise white lines from a perspective-skewed camera image obscured by everything from rain to debris to garbage to other cars.

And we're assuming that we can teach machines to "learn" at all, which isn't true. If that was true, the best machines would be kicking our arses at everything by now, just from sheer amount of data input, speed of data input and processing capability. In actuality, they are winning - with our help - at a handful of board games when we spend years crafting a machine to do just that. That's it. That's the limit of "AI" and deep learning.

Hell, even Google's own search indexes have to be manually moderated and pruned still, because otherwise they get scammed in seconds - by other automated systems and a bunch of low-paid link-spammers on the Internet.

SIGH

[fluffernutter]

I've been saying it before and I'll say it again. These automated cars will be forever getting into accidents because they didn't see a child because of the sun, or because it didn't know a cat would run into the road, or because they saw a ball go into the road but did not anticipate a child running after it. There are too many things to code for.

Re:SIGH

[wonkey_monkey]

I've been saying it before and I'll say it again.

And you'll presumably keep saying it until it suddenly isn't true, when you'll have to stop.

It doesn't matter much if auto-cars do get in accidents as long as they get in fewer accidents than humans do, as a result of the scenarios you've outlined and more. One day they will be smart enough to consider that a child might appear when a ball does, but for now they can just stop or slow down when they see the ball (which is an obstruction in the road).

They used to think computers would never beat humans at chess. Then it was Jeopardy. Then it was Go. One of the few certainties in life is that the "it can't be done!" crowd are invariably proven wrong, sooner or later.

This has been known for over 30 years

[Solandri]

AI researchers first ran across it when developing neural nets. The longer you allowed a neural net to learn, the more rigid its definition of boundary conditions became. Sometimes so rigid that the net became useless for its intended task. e.g. You could develop a neural net which would stop a train in the correct position at the platform 80% of the time. Further training would increase this to 90%, then 95%, then 99% of the time, but resulted in the net completely flipping out the remaining 1% of the time when it calculated it was going to overshoot by 1 mm outside the trained parameters. The first solution was to stop the learning process and freeze the neural net before it reached this stage, then simply use it in production with the learning capability (ability to modify itself) disabled. The next solution was to use simulated annealing to occasionally reset the specific things the neural net had learned, while retaining the general things it had learned.

You also see this in biological neural nets. As people get older, they tend to get set in their ways, less likely to change their opinions even in the face of contradictory evidence. (As opposed to younger people who are too eager to form an opinion despite weak or the lack of evidence.) I suspect this is also where the aphorism "you can't teach an old dog new tricks" comes from. IMHO this is why trying to lengthen the human lifespan in the pursuit of immortality is a bad idea. Death is nature's way of clearing out neural nets which have become too rigid to respond properly to common variability in situations they encounter. My grandmother hated the Japanese to her dying day (they raped and killed her sister and niece during WWII). If people were immortal, we'd be completely dysfunctional as a society because everyone would be holding grudges and experience-based prejudice for hundreds of years, to the detriment of immediate benefit.

Re:Mistakes

[Dutch Gun]

If tiny changes cause these "weak AI" algorithms to fail, then they've been trained badly, or else aren't sophisticated enough algorithms at their core. That, or they don't have enough context. For instance, a stop sign should be recognizable almost purely based on the fact that it's a uniquely shaped sign (octagonal) in the US, at least, along with its proximity and relative position to an intersection. An AI looking at a photo has none of this contextual information, and so has a severe disadvantage.

More importantly, no car manufacturer will be relying solely on vision systems to make navigation systems, which is a huge advantage they'll have over human drivers. Suggesting otherwise seems disingenuous at best on the part of these "computer academics" quoted in the summary. These vehicles will be relying on a *range* of sensors to detect what's going on around them, and I'd argue that vision may well be the least important among them.

Wrong title, not a real problem

[lorinc]

The title should have read "Carefully crafted decoy using massive computation resources can fool not up-to-date AI".

Here's how it works:
1. Get access to the AI model you want to fool (and only this one). Not necessarily the source code, but at least you need to be able to use the model as long as you want.
2. Solve a rather complex optimization problem to generate the decoy
3. use your decoy in very controlled conditions (like stated in the linked paper)

While the method for fooling the model is fine (and similar work has been buzzing lately), the conclusion are much weaker than you expect. First, because if you don't have the actual model, you cannot do that. You need to run the actual model you are trying to fool. So that takes out all remote systems with rate limiting accesses. Second, your rely on tiny variation which can be more sensitive than real world variation. Take for example the sticker on road sign, if you took the picture on the sunny day, the decoy will very likely not work on rainy day or at night. Third, if the model evolves, you have to update the decoy. Here's the problem with statistical learning systems: they learn. It's very likely that the model got updated by the time you finished the computation and printing the sticker. Many people believe that future industrial systems will perform online learning which renders those static methods useless.

So yeah, actual research model can be fooled in very specific cases. However, It's not as bad as some article try to make it sound. I'm not saying it won't happen, I'm saying it's not as bad as you think it is. Hey, if you want to impersonate somebody, put some make up and if you want people to crash their car, cover the roadsigns with paint. There you have it, humans are easily fooled by some paint.

Not how it works

[SoftwareArtist]

Almost every comment posted so far about this story is totally wrong. Adversarial examples are a hot topic in deep learning right now. We've learned a lot about how they work and how to protect against them. They have nothing to do with "weak" versus "strong" AI. Humans are also susceptible to optical illusions, just different ones from neural nets. They don't mean that computers can never be trusted. Computers can be made much more reliable than humans. And they also aren't random failures, or something that's hard to create. In fact, they're trivial to create in a simple, systematic way.

They're actually a consequence of excessive linearity in our models. If you don't know what that means, don't worry about it. It's just a quirk of how models have traditionally been trained. And if you make a small change to encourage them to work in a nonlinear regime, they become much more resistant to adversarial examples. By the time fully autonomous cars hit the roads in a few years, this should be a totally solved problem.

If you build deep learning systems, you need to care about this. If you don't, you can ignore it. It's not a problem you need to care about, any more than you care what activation function or regularization method your car is using.

Re:Speed Bump

[gweihir]

I have heard about it, but unlike you I actually understand what it means. It only surpasses humans in its "Big Data" aspects, not in the actual AI parts. These are so bad that the expert "beaten" thought he would have no trouble finding a strategy to beat it, and that after he had seen it play only a few times. AlphaGo had the full history of the expert's playing style, the expert had nothing the other way round before.

In short, this was a stunt. It does not show what most people think it shows. No AI expert got really excited about this either.