Slashdot Mirror
You Can Trick Self-Driving Cars By Defacing Street Signs (bleepingcomputer.com)
An anonymous reader quotes a report from Bleeping Computer: A team of eight researchers has discovered that by altering street signs, an adversary could confuse self-driving cars and cause their machine-learning systems to misclassify signs and take wrong decisions, potentially putting the lives of passengers in danger. The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or (2) attach smaller stickers on a legitimate sign in order to fool the self-driving car into thinking it's looking at another type of street sign. While scenario (1) will trick even human observers and there's little chance of stopping it, scenario (2) looks like an ordinary street sign defacement and will likely affect only self-driving vehicles. Experiments showed that simple stickers posted on top of a Stop sign fooled a self-driving car's machine learning system into misclassifying it as a Speed Limit 45 sign from 67% to 100% of all cases. Similarly, gray graffiti stickers on a Right Turn sign tricked the self-driving car into thinking it was looking at a Stop sign. Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs. In addition, car vendors should also take into account contextual information for their machine learning systems. For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
A better title would be, "Researchers fool Google's TensorFlow library in laboratory tests".
As it turns out, they did NOT test this against actual self-driving vehicle image recognition, but a generic deep neural network library. This seemed obvious, as there are still no commercially available fully autonomous vehicles, but I skimmed the paper to confirm it.
There was another issue I noticed as well. They resized all their training images down to 32x32 pixels. I admit I'm no expert in neural networks, but this seems like it would greatly favor the ability to fool classification algorithms. Maybe someone more knowledgeable can correct me if I'm off base here. Still, my suspicion seems to be confirmed by this little gem:
"Our final classifier accuracy was 91% on the test dataset."
So, their baseline algorithm only worked properly slightly better than 9/10 times. Should we believe that this represents the state of the art that will be applied in actual self-driving vehicles? It seems like the researchers didn't even have a highly robust classifier from the start.
I believe the merits of the paper lie in demonstrating this as a theoretical concern, but this should in no way be construed to represent a definitive threat against actual vehicle systems. You can't necessarily blame the researchers for the crappy headline, of course, as the title is "Robust Physical-World Attacks on Machine Learning Models". But I wouldn't necessarily rate this as the most robust research I've ever seen either.
Irony: Agile development has too much intertia to be abandoned now.
My wife and I drove from Seattle to Anchorage back in the late 1980s - her sister had gotten married, and we went up to meet her husband and his family. Not long after we crossed from Canada into Alaska, we started noticing that pretty much every road sign had been shot multiple times. It got worse, the further into Alaska we travelled. Along the stretch of highway that heads down the peninsula towards Anchorage, many of the signs had so many bullet holes that they were unreadable.
After meeting my (now ex-) brother-in-law and his friends, I ceased to be surprised at the state of the road signs - instead, I wondered why none of them had thought of destroying the signposts using automatic weapons.
#DeleteChrome
Deface a sign enough and it fails inspection as a sign. Now the intersection has no stop sign as far as the computer's concerned.
Nonsense. SDCs are not designed with a single point of failure. When approaching an intersection they do all of the following:
1. Look for a sign or light. ... and also says it requires a stop.
2. Access map data, which shows it is an intersection
3. Access historical data for the intersection that shows other SDCs recently stopped there.
4. Look at the road markings and tire markings that indicate cross traffic.
If these data contradict each other, the SDC will do the safe thing and stop. It will also report the missing and defaced sign.
A human is more likely to drive through the intersection than an SDC.
The actual paper is here: https://arxiv.org/abs/1707.08945v3
They did NOT "trick" any SDCs, nor did they even try. They just defeated an algorithm that they assumed is similar to what SDCs use for #1 in the list above.