Slashdot Mirror
You Can Trick Self-Driving Cars By Defacing Street Signs (bleepingcomputer.com)
An anonymous reader quotes a report from Bleeping Computer: A team of eight researchers has discovered that by altering street signs, an adversary could confuse self-driving cars and cause their machine-learning systems to misclassify signs and take wrong decisions, potentially putting the lives of passengers in danger. The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or (2) attach smaller stickers on a legitimate sign in order to fool the self-driving car into thinking it's looking at another type of street sign. While scenario (1) will trick even human observers and there's little chance of stopping it, scenario (2) looks like an ordinary street sign defacement and will likely affect only self-driving vehicles. Experiments showed that simple stickers posted on top of a Stop sign fooled a self-driving car's machine learning system into misclassifying it as a Speed Limit 45 sign from 67% to 100% of all cases. Similarly, gray graffiti stickers on a Right Turn sign tricked the self-driving car into thinking it was looking at a Stop sign. Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs. In addition, car vendors should also take into account contextual information for their machine learning systems. For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Bwahahaa!
It should be the easiest thing in the world to classify street signs using an *algorithm*. They are a specific size, specific shape, specific color, and have writing on them. More than that, the writing is limited to a set of a few dozen variations. Given so many different ways to identify and cross-check identification, it should be nearly impossible to misclassify.
This just proves their "machine learning" is total shit.
You set up snipers in strategic locations across town to cover every and all traffic sign; and you shoot the fucker who dares get even close to it.
I tend to rant.
https://slashdot.org/submissio...
Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
But, the edge cases will become increasingly troublesome as they move from prototype into widespread use
Road signs are commonly missing, rotated, shot, stolen or defaced
I love the idea of autonomous vehicles. I wrote autonomous vehicle software for a major auto manufacturer. This shit is hard
Make a circle of them and the self-driving car stops moving.
If you time it right, you can do it right in front of an oncoming truck.
-- Tigger warning: This post may contain tiggers! --
It just occurred to me today: will self-driving cars be smart enough to pull over for cops and fire trucks? If so, does that mean all you have to do to get them out of your way is flash some lights for a bit?
Self-driving cars will only be reading signs during a transitional period. Google can easily generate, and probably has generated, a database of street sign locations extracted from StreetView data.
There will be services that track all signs along with GPS coordinates and which are updated by planning authorities.
Eventually self-driving vehicles will only rely on visual input for corroboration on permanent signage and to identify temporary signage. As with everything else in the self-driving world this will be more reliable than the current system.
"...there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."
What about here? (Cross Island Parkway, New York USA, Exit 31)
Stop signs often do appear on highway entry ramps, especially where they are short. This is true in construction areas, as well as on some older entrance ramps around New York City.
Technically this is a 50 MPH (~80 km/h) Parkway and not an Interstate, but rather than randomly searching the area this was the first that came to mind.
What horrifically terrible machine learning algorithm sees a red octagon and thinks it's a black and white rectangular speed limit sign? How is the visual machine learning matrix so bad that a triangular yellow sign would be registered as a stop sign?
Do they not train the machine learning algorithms with color images? Considering you can rely on 1-2 seconds of latency for a sign there is no reason to use the same sort of low latency machine learning algorithms used for pedestrian identification or road lines.
Snow accumulates on street signs. Add 30 mph wind that's common here in the upper Midwest and these automated systems are a failure before they leave the garage
Road signs are commonly missing,
I feel like a missing stop sign is a problem regardless if your brain is squishy or silicon. In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
You can deface human drivers by tricking street signs.
#DeleteFacebook
For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Except when you do, like when there's construction or accidents, and a guy stands there with a stop sign.
I nearly ran through a stop sign last winter... ...because it was covered with blowing snow. The octagonal shape was barely visible, but it definitely wasn't red. At night it may have been altogether different.
If we can't get signs with stickers right, then what chance do we have against snow?
I can see it now: a company puts up a billboard with a red octagon containing their brand of motor oil, and the car gets thirsty.
It has begun!
In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.
For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Except during road construction when a signman holds up a "stop" sign and the self-driving car says "You're not fooling me! There are no stop signs on freeways, and even your 15mph speed limit sign is fake, my database says the speed limit here is 75mph. See ya!"
But not as easily. That was the claim in the article, anyway.
You can trick humans by defacing street signs... So... What else is new? This is a "no-duh!"
In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.
It's not just "yield to the right", it's yield to oncoming traffic, yield to the car that gets there first, and then (maybe) yield to the right.
Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.
Instead of a car making horrific errors in judgment, why not have it safely pull over and say, "I'm lost, please ask for directions."
Better yet, set it up so the female voice pulls over and asks for help and the male voice just keeps going until it thinks it reached the destination.
I like how this is written like it is a surprise. Did people really think that autonomous vehicles actually thought about the signs?
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
And they will the do that 55 on I-294 when all others are doing 75+.
my 1st thought years ago was pranking cars by jumping out in front of them. Crazy to risk it; however, when it becomes predictably safe...
Next thought was some radio nerds experimenting with broadcasting signals towards cars.
Democracy Now! - uncensored, anti-establishment news
Seriously, these machines should be using signs to augment mapping info.
In addition, the feds should come up with a SINGULAR approach on how to put up secured temporary local notifications.
Perhaps a digital form of NOTAMs.
I prefer the "u" in honour as it seems to be missing these days.
Why are self-driving cars reading signs in the first place? Seriously, don't we have all of this information available digitally? It makes no sense for them to even be attempting to read the signs. If the car needs to travel into an area where we don't have digital information available, it should require manual control. This is just silly.
It's a LOT harder to trick a human than it is to trick a computer.
You can trick human drivers by defacing street signs
Yea, not always. Try to change the signs on a road that I drive every day and you won't trick me. I'm not likely to even look at the things. Robots? Totally different story. They cannot think or reason and have to be programed to look at ALL signs, defaced, false or not. The guy flying the bird next to the sign isn't going to register as anything but a pedestrian to avoid hitting.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs
Spend tons of money covering signs with sticker-proof material and you are again defeated by spray paint and stencils. Or by magnetic graffiti! This is not the most efficient way of thinking to remedy this problem.
Road signs are commonly missing, rotated, shot, stolen or defaced
Or, like around here, just plain wrong because it costs money to change them and the government doesn't have the cash.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
You got that right... Black Ice is NOT much fun to drive on and very hard to see... I've driven on it in the past and lived to tell the tale. It was no fun waiting for the car to slow down from 55MPH without using the brakes, hoping it stayed on the two lane road...
I don't think automation would deal kindly with that....
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
It's not just "yield to the right", it's yield to oncoming traffic
No, it's not. Oncoming traffic won't cross your path unless they turn left, in which case they have you on their right, and must yield.
yield to the car that gets there first
At least imprecise. If a car has entered the intersection and cannot reasonably be expected to stop before entering your projected path, you have to yield to it, but for a different reason - you're not allowed to cause an accident by intent or negligence. But that doesn't mean the other driver hasn't broken the rules by not yielding to you.
Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.
Many countries have mainly unmarked intersections, and have drivers follow the yield-to-right rules, and it works fine. Americans not being able to handle unmarked intersections appears to be an American phenomenon.
Have a very attractive lady(s) walk on the side of the road. I guarantee there will eventually be a smashup. Most men are suckers that way. I've had multiple close calls due to such "distractions". Plus, it's not illegal to arrange such, unlike sign tampering.
Hmmm, let's see if bot-cars are distracted by R2D2 in lingerie.
Table-ized A.I.
A better title would be, "Researchers fool Google's TensorFlow library in laboratory tests".
As it turns out, they did NOT test this against actual self-driving vehicle image recognition, but a generic deep neural network library. This seemed obvious, as there are still no commercially available fully autonomous vehicles, but I skimmed the paper to confirm it.
There was another issue I noticed as well. They resized all their training images down to 32x32 pixels. I admit I'm no expert in neural networks, but this seems like it would greatly favor the ability to fool classification algorithms. Maybe someone more knowledgeable can correct me if I'm off base here. Still, my suspicion seems to be confirmed by this little gem:
"Our final classifier accuracy was 91% on the test dataset."
So, their baseline algorithm only worked properly slightly better than 9/10 times. Should we believe that this represents the state of the art that will be applied in actual self-driving vehicles? It seems like the researchers didn't even have a highly robust classifier from the start.
I believe the merits of the paper lie in demonstrating this as a theoretical concern, but this should in no way be construed to represent a definitive threat against actual vehicle systems. You can't necessarily blame the researchers for the crappy headline, of course, as the title is "Robust Physical-World Attacks on Machine Learning Models". But I wouldn't necessarily rate this as the most robust research I've ever seen either.
Irony: Agile development has too much intertia to be abandoned now.
Yeah, the first step is to get the human to look at the sign in the first place.
If I have been able to see further than others, it is because I bought a pair of binoculars.
AI is stupid.
News at 11.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Signs vary widely between countries.
Here in New Zealand a stop sign is alway accompanied by a yellow line and the word "STOP" painted on the road at the intersection. Give Way signs are either unmarked or have white lines with a triangle on the road.
I assume that means if the sign is damaged, you always know the difference between a stop sign controlled intersection and a regular give way intersection.
Eh... computers will learn the tricks, future generations of machines tend to become immune to the tricks the first generation fell for, heck it is very quick and easy to educate an entire generation of systems to the specific trick that the first generation fell for. Meanwhile there are still new generation humans, vulnerable to the "nigerian prince" exploit.
Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
For one thing, there's a need for temporary signs.
And the sign has to physically exist for everything that isn't a self-driving car.
This.
You have to be really carefully how you design this. The self-driving car that refuses to see a stop sign on an interstate is going to absolutely love construction zones.
Real lawyers write in C++
Every new Tesla car (including Model 3) has the full "Hardware 2" platform for self-driving, and even when it's not being used for self-driving it's on and watching the world. Tesla has said that it is already using "fleet learning" to map out roads. This blog post is talking about how radar has problems but is still useful for self-driving, and they are working around the problems:
https://www.tesla.com/blog/upgrading-autopilot-seeing-world-radar
In a world with fleet learning this hack will be of very limited effectiveness. The first cars to reach the hacked signs will learn about them and then other cars will know. In the early days of self-driving cars the car can make the human take over and the fleet can learn what the human did.
Sooner or later I imagine there will be an interoperative standard for fleet learning, where all the cars will cooperate instead of only Tesla cars learning from other Tesla cars and so on. All cars would share learning over the Internet. This then suggests an attack where false learning data is injected into the system!
Once the world has "Level 5 self-driving" cars built with no steering wheel or other human controls, this sort of attack could be a bit of a problem and will need to be solved. One idea: if there is an interoperative standard then the Department of Transportation would publish learning data about temporary stop signs or whatever. A new stop sign appearing right where the learning data said it would would be trusted a lot.
I don't think this will be a huge issue though. Self-driving cars will already have to deal with the unexpected, such as a pedestrian jumping out into the road. If you want to get a self-driving car to stop suddenly, just throw a realistic dummy out into the road when it's coming.
lf(1): it's like ls(1) but sorts filenames by extension, tersely
"For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."
I can think of at least two places on I-15 which have a stop sign directly on the interstate, and one on I-40.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Give me a stencil and some paint, and I can trick YOU by defacing street signs.
The only difference here is that idiots don't need the stencils.
To quote a famous idiot, FAKE NEWS.
excitingthingstodo.blogspot.com
On the kind of mapped terrain where self-driving cars currently mix with manual traffic, most of the information on traffic signs can be coded into the cars' database, such as speed limits on each stretch of road and the location of no-passing zones and crosswalks. Self-drivers must be able to recognize sudden and temporary control changes, such as for construction, weather damage, and police operations. If someone tries to spoof signs in one of these areas or do something like cover up a Stop sign with a picture of the scenery behind it, it will be just as immediately apparent to human drivers, and be just as likely to cause accidents (scenario 1 in the summary).
When self-drivers replace manual traffic, road signs will eventually disappear. Special short-term alterations to mapped traffic flow will be triggered by radio beacons on construction barricades, traffic cones, police cars, and crime scene boundaries.
I have done some experimenting with neural nets, including traffic sign detection and was not surprised when I read this...
However I feel a large part of this vulnerability comes from an awful lot of the neural networks being trained mostly considering shapes, not color. If you factor in color at all, none of that tape nonsense is going to confuse a stop sign for a street sign.
Also I feel like this attack is probably based on well-known public traffic sign recognizers and would not work on hardier commercial systems of today, much less tomorrow - I didn't see they even tested it with a Tesla which I think recognizes such things (perhaps it's just street lights, can't remember).
"There is more worth loving than we have strength to love." - Brian Jay Stanley
The question wasn't is it harder, the posited issue was can you. Stick to the topic.
and who will pay that data bill? much less roaming fees.
And what about LAG feeding in dated info?
Many companies use a multi layered neural network to classify street signs. As it highly depends on the learning data how such system classifies, it is relevant to note which implementation was tested.
Furthermore, being able to also classify signs with stickers just requires more learning data.
So you mean if they changed all the signs to a lower speed you will go a higher speed?, or if they closed down a road and altered a sign from going forward-right to just forward and you would still turn right because it has always been possible?
4-way stops is US/Canada only phenomenon, and it's really inefficient and stupid. Most sane countries use roundabouts instead
i mean, what is stopping anybody of making fake signs and putting them up and misleading self driving cars.
you could lure people to a certain place by making fake 'turn here' signs, or make them stop suddenly with a fake stop sign.
who could tell the difference (human or AI)? might as well be some temporary sign because of road works up ahead, etc.
On a long enough timeline, the survival rate for everyone drops to zero.
The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or [...]While scenario (1) will trick even human observers and there's little chance of stopping it
Nope, it won't in the case of the stop sign. It's octogonal, the only street sign that has this shape. So it will be recognizable whether clear, covered in a weird poster, or in snow. Btw, possibility of snow cover was the reason why the street sign design committee decided to give it this shape. The importance of the stop sign is so that it must not be confused with something else, even if for whatever reason it becomes unreadable.
If someone alters a sign, it might trick a robot. Then again, it might trick a human too. So what's the difference?
Easy fix... some type of transponder on the sign that is cryptographically signed and is GPS locked to a small area, so if the sign is moved beyond a few feet, it gets ignored by the vehicle. That way, a 30 changed to an 80 will be ignored. Not 100% secure, as someone can hack the private key, but it will stop mischief like this.
The Federal Government decided we needed new fonts for street signs.
Again.
deleting the extra space after periods so i can stay relevant, yeah.
This basically means that tere needs to be a federal DB of all signage in the USA. Along with requirements that any removals, additions and modifications to such signs be made in real time. Then toss on a manual auditing system to inspect areas for correctness. You could even have cars report any conflicting data to authorities.
This is fixable but it needs to be done governmentally.
Now for hands free driving, please advance the development of robot sex workers.
If YOU change the signs, I'm not paying attention to them.. All your examples are GOVERNMENT changing the signs... That I would notice because I'd likely see the road crews out in the official government vehicles and read the news story about the regulatory changes in the press releases.
But my point is, I know the route, false signs are not likely to mislead me. A computer isn't so thoughtful about this kind of thing because it cannot consider the wider context...
For instance, I know where a bridge is under construction on my daily commute route. Last week, I was driving over the old bridge, this week the new one, though the road and signs still are up as before on the old road, only a route change has been made with some orange barrels. Next week that old bridge may be gone. I know the context of the construction effort, so I know that I will need to pay closer attention at various locations because I'm expecting the route to change. A computer doesn't have this contextual information nor does it have the reason required to think though the implications of it. I do. So if you remove those barrels, I'm not going to blindly drive the old route now, a computer likely would.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Self driving cars will undoubtedly have many problems - that's not the question. The question is: Will they have more problems than humans? If you deface a sign enough - then a human can't recognize it either. The car, however, can be equipped with a database of where the signs are - it can compare the picture it sees with the database and with the pictures other cars have seen at that same location.
A car has MUCH more information than a person.
I would also bet that it could use the fact that signs are retro-reflective and return more energy from LIDAR than a sticker or spray paint can.
There are MANY ways to make this tiny problem "go away" for cars - but none to make it "go away" for humans.
www.sjbaker.org
That way cars can use it better than having to read the actual traffic signal. Better yet, have cars report to police if there is some inconsistency to the police. That way after several reports they will know the signal has a problem.
The thing is in this case or other cases is that people can change the sign to look like government signs.
And as you say, "not likely to mislead me", meaning it might.
Not likely to mislead me, but very likely will be taken as gospel by a computer. (which is my point) Automation is more subject to this risk than I am.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Except in the onramp case, all of these occur on the freeway itself.
Then there's Interstate 5, which, in addition to having stop signs at either end of it because DUH, the two busiest border crossings in the world at opposite ends of it, it also has traffic lights in the Portland area thanks to the drawbridge, since Clark County is so spiteful it can't be arsed to accept light rail to Oregon.
Furries make the internet go.
Bad data in, bad data out. This is what you get when you train your neural network with only "perfect" examples. It has no context for any variation whatsoever.