Slashdot Mirror
You Can Trick Self-Driving Cars By Defacing Street Signs (bleepingcomputer.com)
An anonymous reader quotes a report from Bleeping Computer: A team of eight researchers has discovered that by altering street signs, an adversary could confuse self-driving cars and cause their machine-learning systems to misclassify signs and take wrong decisions, potentially putting the lives of passengers in danger. The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or (2) attach smaller stickers on a legitimate sign in order to fool the self-driving car into thinking it's looking at another type of street sign. While scenario (1) will trick even human observers and there's little chance of stopping it, scenario (2) looks like an ordinary street sign defacement and will likely affect only self-driving vehicles. Experiments showed that simple stickers posted on top of a Stop sign fooled a self-driving car's machine learning system into misclassifying it as a Speed Limit 45 sign from 67% to 100% of all cases. Similarly, gray graffiti stickers on a Right Turn sign tricked the self-driving car into thinking it was looking at a Stop sign. Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs. In addition, car vendors should also take into account contextual information for their machine learning systems. For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
My point exactly. So-called 'machine learning' doesn't actually think. Your dog has better cognitive capability. This 'technology' is being rushed way too quickly to market.
Here in Texas, they just shoot the traffic sign and skip the middleman. Because freedom has to be irrigated by the blood of patriotic drivers in self-driving cars. Or something. I don't remember the exact quote, but it's in the Second Amendment or the Bible, I'm pretty sure.
https://s-media-cache-ak0.pini...
You are welcome on my lawn.
"...there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."
What about here? (Cross Island Parkway, New York USA, Exit 31)
Stop signs often do appear on highway entry ramps, especially where they are short. This is true in construction areas, as well as on some older entrance ramps around New York City.
Technically this is a 50 MPH (~80 km/h) Parkway and not an Interstate, but rather than randomly searching the area this was the first that came to mind.
What horrifically terrible machine learning algorithm sees a red octagon and thinks it's a black and white rectangular speed limit sign? How is the visual machine learning matrix so bad that a triangular yellow sign would be registered as a stop sign?
Do they not train the machine learning algorithms with color images? Considering you can rely on 1-2 seconds of latency for a sign there is no reason to use the same sort of low latency machine learning algorithms used for pedestrian identification or road lines.
For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Except during road construction when a signman holds up a "stop" sign and the self-driving car says "You're not fooling me! There are no stop signs on freeways, and even your 15mph speed limit sign is fake, my database says the speed limit here is 75mph. See ya!"
You can trick humans by defacing street signs... So... What else is new? This is a "no-duh!"
TFS makes this point.. Deface a sign enough and it fails inspection as a sign. Now the intersection has no stop sign as far as the computer's concerned.
Things like this are exactly the kind of corner cases their 'AI' will never be able to deal with, at least not with current solutions.
Instead of a car making horrific errors in judgment, why not have it safely pull over and say, "I'm lost, please ask for directions."
Better yet, set it up so the female voice pulls over and asks for help and the male voice just keeps going until it thinks it reached the destination.
That switch is a pulsing light that triggers the traffic signal to change the cycle in the same manner as a pedestrian push button or in ground sensor loop might. It just forces the priority of the change so that the normal green-to-yellow-to-red change starts now instead of a bit later. They are not exactly difficult to fake out. Putting them on every car on the road would be a terrible idea, or not. As long as I am in my manual operated vehicle, having one of those would be quite enjoyable at times. "Hey look, all the 'autonomous' cars pulled over for no apparent reason! Everybody should run down to the dealer and have the software diagnosed."
But what if the stop light draws first?
George Lucas will make new ones that don't.
It must have been something you assimilated. . . .
Seriously, these machines should be using signs to augment mapping info.
In addition, the feds should come up with a SINGULAR approach on how to put up secured temporary local notifications.
Perhaps a digital form of NOTAMs.
I prefer the "u" in honour as it seems to be missing these days.
Have a very attractive lady(s) walk on the side of the road. I guarantee there will eventually be a smashup. Most men are suckers that way. I've had multiple close calls due to such "distractions". Plus, it's not illegal to arrange such, unlike sign tampering.
Hmmm, let's see if bot-cars are distracted by R2D2 in lingerie.
Table-ized A.I.
A better title would be, "Researchers fool Google's TensorFlow library in laboratory tests".
As it turns out, they did NOT test this against actual self-driving vehicle image recognition, but a generic deep neural network library. This seemed obvious, as there are still no commercially available fully autonomous vehicles, but I skimmed the paper to confirm it.
There was another issue I noticed as well. They resized all their training images down to 32x32 pixels. I admit I'm no expert in neural networks, but this seems like it would greatly favor the ability to fool classification algorithms. Maybe someone more knowledgeable can correct me if I'm off base here. Still, my suspicion seems to be confirmed by this little gem:
"Our final classifier accuracy was 91% on the test dataset."
So, their baseline algorithm only worked properly slightly better than 9/10 times. Should we believe that this represents the state of the art that will be applied in actual self-driving vehicles? It seems like the researchers didn't even have a highly robust classifier from the start.
I believe the merits of the paper lie in demonstrating this as a theoretical concern, but this should in no way be construed to represent a definitive threat against actual vehicle systems. You can't necessarily blame the researchers for the crappy headline, of course, as the title is "Robust Physical-World Attacks on Machine Learning Models". But I wouldn't necessarily rate this as the most robust research I've ever seen either.
Irony: Agile development has too much intertia to be abandoned now.
Now, not so often. Once Google deploys self-driving software, every time a connected car with their software goes down your street.
I don't agree that the system will evolve to give up the databases. Rather, the databases will become real-time and include much more than signs. There will be warnings about icy spots that are derived from earlier drivers hitting them, puddles, new potholes, a home that frequently has kids running into the street, etc. Every little thing you can imagine will be communicated. But the system won't rely on it, it's just one input of many.
My wife and I drove from Seattle to Anchorage back in the late 1980s - her sister had gotten married, and we went up to meet her husband and his family. Not long after we crossed from Canada into Alaska, we started noticing that pretty much every road sign had been shot multiple times. It got worse, the further into Alaska we travelled. Along the stretch of highway that heads down the peninsula towards Anchorage, many of the signs had so many bullet holes that they were unreadable.
After meeting my (now ex-) brother-in-law and his friends, I ceased to be surprised at the state of the road signs - instead, I wondered why none of them had thought of destroying the signposts using automatic weapons.
#DeleteChrome
Deface a sign enough and it fails inspection as a sign. Now the intersection has no stop sign as far as the computer's concerned.
Nonsense. SDCs are not designed with a single point of failure. When approaching an intersection they do all of the following:
1. Look for a sign or light. ... and also says it requires a stop.
2. Access map data, which shows it is an intersection
3. Access historical data for the intersection that shows other SDCs recently stopped there.
4. Look at the road markings and tire markings that indicate cross traffic.
If these data contradict each other, the SDC will do the safe thing and stop. It will also report the missing and defaced sign.
A human is more likely to drive through the intersection than an SDC.
The actual paper is here: https://arxiv.org/abs/1707.08945v3
They did NOT "trick" any SDCs, nor did they even try. They just defeated an algorithm that they assumed is similar to what SDCs use for #1 in the list above.
This 'technology' is being rushed way too quickly to market.
I'd like to agree with you, particularly with respect to the semi-autonomous systems presently deployed. I argued for years that having a system that worked most of the time but expected the user to take over when necessary was extremely dangerous. But the thing is that human drivers are extremely dangerous. Tesla has very compelling data showing that, as half-baked as their system is, it's actually better than the human drivers that it's replacing. The same will be even more true of the first fully-autonomous vehicles.
The systems don't have to be perfect, they just have to be better, and the bar is not very high.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.