Slashdot Mirror
You Can Trick Self-Driving Cars By Defacing Street Signs (bleepingcomputer.com)
An anonymous reader quotes a report from Bleeping Computer: A team of eight researchers has discovered that by altering street signs, an adversary could confuse self-driving cars and cause their machine-learning systems to misclassify signs and take wrong decisions, potentially putting the lives of passengers in danger. The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or (2) attach smaller stickers on a legitimate sign in order to fool the self-driving car into thinking it's looking at another type of street sign. While scenario (1) will trick even human observers and there's little chance of stopping it, scenario (2) looks like an ordinary street sign defacement and will likely affect only self-driving vehicles. Experiments showed that simple stickers posted on top of a Stop sign fooled a self-driving car's machine learning system into misclassifying it as a Speed Limit 45 sign from 67% to 100% of all cases. Similarly, gray graffiti stickers on a Right Turn sign tricked the self-driving car into thinking it was looking at a Stop sign. Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs. In addition, car vendors should also take into account contextual information for their machine learning systems. For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Bwahahaa!
It should be the easiest thing in the world to classify street signs using an *algorithm*. They are a specific size, specific shape, specific color, and have writing on them. More than that, the writing is limited to a set of a few dozen variations. Given so many different ways to identify and cross-check identification, it should be nearly impossible to misclassify.
This just proves their "machine learning" is total shit.
You set up snipers in strategic locations across town to cover every and all traffic sign; and you shoot the fucker who dares get even close to it.
I tend to rant.
https://slashdot.org/submissio...
Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
You can trick human drivers by defacing street signs
But, the edge cases will become increasingly troublesome as they move from prototype into widespread use
Road signs are commonly missing, rotated, shot, stolen or defaced
I love the idea of autonomous vehicles. I wrote autonomous vehicle software for a major auto manufacturer. This shit is hard
Make a circle of them and the self-driving car stops moving.
If you time it right, you can do it right in front of an oncoming truck.
-- Tigger warning: This post may contain tiggers! --
It just occurred to me today: will self-driving cars be smart enough to pull over for cops and fire trucks? If so, does that mean all you have to do to get them out of your way is flash some lights for a bit?
Unless you live where I do and they put one up for the construction that's currently going on. Every rule has exceptions and being able to adapt past common sense rules like this one is why self driving cars won't be fit for the road for probably another 10-20 years IMO.
Self-driving cars will only be reading signs during a transitional period. Google can easily generate, and probably has generated, a database of street sign locations extracted from StreetView data.
There will be services that track all signs along with GPS coordinates and which are updated by planning authorities.
Eventually self-driving vehicles will only rely on visual input for corroboration on permanent signage and to identify temporary signage. As with everything else in the self-driving world this will be more reliable than the current system.
"...there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."
What about here? (Cross Island Parkway, New York USA, Exit 31)
Stop signs often do appear on highway entry ramps, especially where they are short. This is true in construction areas, as well as on some older entrance ramps around New York City.
Technically this is a 50 MPH (~80 km/h) Parkway and not an Interstate, but rather than randomly searching the area this was the first that came to mind.
What horrifically terrible machine learning algorithm sees a red octagon and thinks it's a black and white rectangular speed limit sign? How is the visual machine learning matrix so bad that a triangular yellow sign would be registered as a stop sign?
Do they not train the machine learning algorithms with color images? Considering you can rely on 1-2 seconds of latency for a sign there is no reason to use the same sort of low latency machine learning algorithms used for pedestrian identification or road lines.
Snow accumulates on street signs. Add 30 mph wind that's common here in the upper Midwest and these automated systems are a failure before they leave the garage
Road signs are commonly missing,
I feel like a missing stop sign is a problem regardless if your brain is squishy or silicon. In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
Forget stickers and graffiti, just add some winter weather.
You can deface human drivers by tricking street signs.
#DeleteFacebook
For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Except when you do, like when there's construction or accidents, and a guy stands there with a stop sign.
I nearly ran through a stop sign last winter... ...because it was covered with blowing snow. The octagonal shape was barely visible, but it definitely wasn't red. At night it may have been altogether different.
If we can't get signs with stickers right, then what chance do we have against snow?
I can see it now: a company puts up a billboard with a red octagon containing their brand of motor oil, and the car gets thirsty.
It has begun!
In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.
For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
Except during road construction when a signman holds up a "stop" sign and the self-driving car says "You're not fooling me! There are no stop signs on freeways, and even your 15mph speed limit sign is fake, my database says the speed limit here is 75mph. See ya!"
You can trick humans by defacing street signs... So... What else is new? This is a "no-duh!"
In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.
If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.
It's not just "yield to the right", it's yield to oncoming traffic, yield to the car that gets there first, and then (maybe) yield to the right.
Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.
Instead of a car making horrific errors in judgment, why not have it safely pull over and say, "I'm lost, please ask for directions."
Better yet, set it up so the female voice pulls over and asks for help and the male voice just keeps going until it thinks it reached the destination.
I like how this is written like it is a surprise. Did people really think that autonomous vehicles actually thought about the signs?
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
And they will the do that 55 on I-294 when all others are doing 75+.
my 1st thought years ago was pranking cars by jumping out in front of them. Crazy to risk it; however, when it becomes predictably safe...
Next thought was some radio nerds experimenting with broadcasting signals towards cars.
Democracy Now! - uncensored, anti-establishment news
Seriously, these machines should be using signs to augment mapping info.
In addition, the feds should come up with a SINGULAR approach on how to put up secured temporary local notifications.
Perhaps a digital form of NOTAMs.
I prefer the "u" in honour as it seems to be missing these days.
But, the edge cases will become increasingly troublesome as they move from prototype into widespread use
Road signs are commonly missing, rotated, shot, stolen or defaced
I love the idea of autonomous vehicles. I wrote autonomous vehicle software for a major auto manufacturer. This shit is hard
It is really hard to imagine a future of autonomous vehicles without the supporting infrastructure of digital markers in the road, but that is a really cool gig you had there. Is it in the phoenix area by chance?
Why are self-driving cars reading signs in the first place? Seriously, don't we have all of this information available digitally? It makes no sense for them to even be attempting to read the signs. If the car needs to travel into an area where we don't have digital information available, it should require manual control. This is just silly.
It's a LOT harder to trick a human than it is to trick a computer.
Cameras alone aren't effective in all situations.
The state needs to control the signage, and keep it updated, digitally, for FSD vehicles.
My friend started a company to solve this problem, and I invested in it. Posting as AC for obvious reasons.
Check out SinWaves at http://www.sinwaves.com.
Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs
Spend tons of money covering signs with sticker-proof material and you are again defeated by spray paint and stencils. Or by magnetic graffiti! This is not the most efficient way of thinking to remedy this problem.
Road signs are commonly missing, rotated, shot, stolen or defaced
Or, like around here, just plain wrong because it costs money to change them and the government doesn't have the cash.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
It's not just "yield to the right", it's yield to oncoming traffic
No, it's not. Oncoming traffic won't cross your path unless they turn left, in which case they have you on their right, and must yield.
yield to the car that gets there first
At least imprecise. If a car has entered the intersection and cannot reasonably be expected to stop before entering your projected path, you have to yield to it, but for a different reason - you're not allowed to cause an accident by intent or negligence. But that doesn't mean the other driver hasn't broken the rules by not yielding to you.
Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.
Many countries have mainly unmarked intersections, and have drivers follow the yield-to-right rules, and it works fine. Americans not being able to handle unmarked intersections appears to be an American phenomenon.
Have a very attractive lady(s) walk on the side of the road. I guarantee there will eventually be a smashup. Most men are suckers that way. I've had multiple close calls due to such "distractions". Plus, it's not illegal to arrange such, unlike sign tampering.
Hmmm, let's see if bot-cars are distracted by R2D2 in lingerie.
Table-ized A.I.
A better title would be, "Researchers fool Google's TensorFlow library in laboratory tests".
As it turns out, they did NOT test this against actual self-driving vehicle image recognition, but a generic deep neural network library. This seemed obvious, as there are still no commercially available fully autonomous vehicles, but I skimmed the paper to confirm it.
There was another issue I noticed as well. They resized all their training images down to 32x32 pixels. I admit I'm no expert in neural networks, but this seems like it would greatly favor the ability to fool classification algorithms. Maybe someone more knowledgeable can correct me if I'm off base here. Still, my suspicion seems to be confirmed by this little gem:
"Our final classifier accuracy was 91% on the test dataset."
So, their baseline algorithm only worked properly slightly better than 9/10 times. Should we believe that this represents the state of the art that will be applied in actual self-driving vehicles? It seems like the researchers didn't even have a highly robust classifier from the start.
I believe the merits of the paper lie in demonstrating this as a theoretical concern, but this should in no way be construed to represent a definitive threat against actual vehicle systems. You can't necessarily blame the researchers for the crappy headline, of course, as the title is "Robust Physical-World Attacks on Machine Learning Models". But I wouldn't necessarily rate this as the most robust research I've ever seen either.
Irony: Agile development has too much intertia to be abandoned now.
Yeah, the first step is to get the human to look at the sign in the first place.
If I have been able to see further than others, it is because I bought a pair of binoculars.
AI is stupid.
News at 11.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Have you met any actual people lately?
Are you aware of the fake news epidemic?
It's exceedingly easy to trick a human.
Signs vary widely between countries.
Here in New Zealand a stop sign is alway accompanied by a yellow line and the word "STOP" painted on the road at the intersection. Give Way signs are either unmarked or have white lines with a triangle on the road.
I assume that means if the sign is damaged, you always know the difference between a stop sign controlled intersection and a regular give way intersection.
Eh... computers will learn the tricks, future generations of machines tend to become immune to the tricks the first generation fell for, heck it is very quick and easy to educate an entire generation of systems to the specific trick that the first generation fell for. Meanwhile there are still new generation humans, vulnerable to the "nigerian prince" exploit.
Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
For one thing, there's a need for temporary signs.
And the sign has to physically exist for everything that isn't a self-driving car.
This.
You have to be really carefully how you design this. The self-driving car that refuses to see a stop sign on an interstate is going to absolutely love construction zones.
Real lawyers write in C++
Every new Tesla car (including Model 3) has the full "Hardware 2" platform for self-driving, and even when it's not being used for self-driving it's on and watching the world. Tesla has said that it is already using "fleet learning" to map out roads. This blog post is talking about how radar has problems but is still useful for self-driving, and they are working around the problems:
https://www.tesla.com/blog/upgrading-autopilot-seeing-world-radar
In a world with fleet learning this hack will be of very limited effectiveness. The first cars to reach the hacked signs will learn about them and then other cars will know. In the early days of self-driving cars the car can make the human take over and the fleet can learn what the human did.
Sooner or later I imagine there will be an interoperative standard for fleet learning, where all the cars will cooperate instead of only Tesla cars learning from other Tesla cars and so on. All cars would share learning over the Internet. This then suggests an attack where false learning data is injected into the system!
Once the world has "Level 5 self-driving" cars built with no steering wheel or other human controls, this sort of attack could be a bit of a problem and will need to be solved. One idea: if there is an interoperative standard then the Department of Transportation would publish learning data about temporary stop signs or whatever. A new stop sign appearing right where the learning data said it would would be trusted a lot.
I don't think this will be a huge issue though. Self-driving cars will already have to deal with the unexpected, such as a pedestrian jumping out into the road. If you want to get a self-driving car to stop suddenly, just throw a realistic dummy out into the road when it's coming.
lf(1): it's like ls(1) but sorts filenames by extension, tersely
"For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."
I can think of at least two places on I-15 which have a stop sign directly on the interstate, and one on I-40.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Give me a stencil and some paint, and I can trick YOU by defacing street signs.
The only difference here is that idiots don't need the stencils.
To quote a famous idiot, FAKE NEWS.
excitingthingstodo.blogspot.com
On the kind of mapped terrain where self-driving cars currently mix with manual traffic, most of the information on traffic signs can be coded into the cars' database, such as speed limits on each stretch of road and the location of no-passing zones and crosswalks. Self-drivers must be able to recognize sudden and temporary control changes, such as for construction, weather damage, and police operations. If someone tries to spoof signs in one of these areas or do something like cover up a Stop sign with a picture of the scenery behind it, it will be just as immediately apparent to human drivers, and be just as likely to cause accidents (scenario 1 in the summary).
When self-drivers replace manual traffic, road signs will eventually disappear. Special short-term alterations to mapped traffic flow will be triggered by radio beacons on construction barricades, traffic cones, police cars, and crime scene boundaries.
Just like the commercials which said "Alexa, order me a [thing]" we will see shameless billboard ads and perhaps even graffiti intended to trick navigation systems.
Seriously, some advertisers will do anything to drive traffic. Imagine a billboard ad including a mock street sign in the graphics that says, "CHICAGO NEXT RIGHT" knowing that 1% of the traffic will get diverted, half of which will then pull in to your truck stop just to figure out what happened.
What about when someone defaces the sign that says "BRIDGE OUT" not quite enough to alarm humans, but enough to defeat character recognition.
These things will definitely happen, and not at all because I foresee them, but merely because they CAN happen, and you know Murphy's Law.
Trump fooled 40% of America. Even my computer wasn't stupid enough to fall for his baloney.
I have done some experimenting with neural nets, including traffic sign detection and was not surprised when I read this...
However I feel a large part of this vulnerability comes from an awful lot of the neural networks being trained mostly considering shapes, not color. If you factor in color at all, none of that tape nonsense is going to confuse a stop sign for a street sign.
Also I feel like this attack is probably based on well-known public traffic sign recognizers and would not work on hardier commercial systems of today, much less tomorrow - I didn't see they even tested it with a Tesla which I think recognizes such things (perhaps it's just street lights, can't remember).
"There is more worth loving than we have strength to love." - Brian Jay Stanley
The question wasn't is it harder, the posited issue was can you. Stick to the topic.
and who will pay that data bill? much less roaming fees.
And what about LAG feeding in dated info?
Snow? Dirt? Heavy rain?
Many companies use a multi layered neural network to classify street signs. As it highly depends on the learning data how such system classifies, it is relevant to note which implementation was tested.
Furthermore, being able to also classify signs with stickers just requires more learning data.
Since the car has to stop for a construction worker holding a STOP-sign, I guess this opens up a nice opportunity for highway robbery. Perhaps the car could automatically lock the car doors when stopping?
Human: Some idiot knocked over the stop sign and put a band sticker over our street name.
Self driving car: I don't know why I just slammed on the brakes and swerved. I'm being put in manual mode now...
Better self driving car: Data says there should be a stop here. This is an intersection with a major street. I will stop. Dave's street is indicated at these GPS coordinates.
Best self-driving car: The police officer is swerving back and forth in front of us, but not flashing lights or sounding sirens. I have never seen this before. This is my first time in California. There must be a purpose. I will slow down and not pass the police car.
That last one actually happened to me. It's called a "traffic break" and CHP will do it sometimes because of debris or other things going on ahead. I'd like to see a self-driving algo that can handle things like that.
4-way stops is US/Canada only phenomenon, and it's really inefficient and stupid. Most sane countries use roundabouts instead
I thought about how it might become a thing, once self driving cars are prevalent, to trick self driving cars into reacting a certain way to your actions. For example, as a pedestrian pretending to run into the street but stopping at the last second, as a driver braking a particular way or feinting into a lane, or in this case, defacing or modifying a sign to confuse the computer. These actions might result in the self driving car moving over a lane or giving you some sort of "advantage" on the road. Or it might just result in a laugh because all of the cars react a certain way to specific actions. Kind of hacking/influencing the behavior of the self driving cars using your own behavior.
I do still think this might happen on occasion but ultimately I don't think it will last long. Eventually almost everything that confuses a self driving car will confuse at least some human drivers. There is nothing preventing people from pulling these same stunts on human drivers, but as far as I know, it does not happen very often (maybe road rage is an exception). Maybe the prevalence of self driving cars will start to lessen people trying to fuck with other drivers. Why would you bother trying to prove a point to a self driving car?
i mean, what is stopping anybody of making fake signs and putting them up and misleading self driving cars.
you could lure people to a certain place by making fake 'turn here' signs, or make them stop suddenly with a fake stop sign.
who could tell the difference (human or AI)? might as well be some temporary sign because of road works up ahead, etc.
On a long enough timeline, the survival rate for everyone drops to zero.
make a worldwide database of streets and signs and make that available on the internet... oh wait dont we already have google maps... just add the real sign to this and no matter what people do on the street the database will have the correct answer...
No, it's not. Oncoming traffic won't cross your path unless they turn left, in which case they have you on their right, and must yield.
Your way of thinking of it doesn't take the case where you are about to turn left into consideration.
It's a LOT harder to trick a human than it is to trick a computer.
Not really. As teenagers, we would just remove, move, or swap signs around. It fooled a lot of humans.
Try this: find an intersection with a traffic signal. Place a stop sign on each road. Drivers won't know whether to obey the signal or the sign.
Randomly placed ONE WAY and DO NOT ENTER signs are especially confusing to humans.
The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or [...]While scenario (1) will trick even human observers and there's little chance of stopping it
Nope, it won't in the case of the stop sign. It's octogonal, the only street sign that has this shape. So it will be recognizable whether clear, covered in a weird poster, or in snow. Btw, possibility of snow cover was the reason why the street sign design committee decided to give it this shape. The importance of the stop sign is so that it must not be confused with something else, even if for whatever reason it becomes unreadable.
If someone alters a sign, it might trick a robot. Then again, it might trick a human too. So what's the difference?
Easy fix... some type of transponder on the sign that is cryptographically signed and is GPS locked to a small area, so if the sign is moved beyond a few feet, it gets ignored by the vehicle. That way, a 30 changed to an 80 will be ignored. Not 100% secure, as someone can hack the private key, but it will stop mischief like this.
The Federal Government decided we needed new fonts for street signs.
Again.
deleting the extra space after periods so i can stay relevant, yeah.
This basically means that tere needs to be a federal DB of all signage in the USA. Along with requirements that any removals, additions and modifications to such signs be made in real time. Then toss on a manual auditing system to inspect areas for correctness. You could even have cars report any conflicting data to authorities.
This is fixable but it needs to be done governmentally.
Now for hands free driving, please advance the development of robot sex workers.
Self driving cars will undoubtedly have many problems - that's not the question. The question is: Will they have more problems than humans? If you deface a sign enough - then a human can't recognize it either. The car, however, can be equipped with a database of where the signs are - it can compare the picture it sees with the database and with the pictures other cars have seen at that same location.
A car has MUCH more information than a person.
I would also bet that it could use the fact that signs are retro-reflective and return more energy from LIDAR than a sticker or spray paint can.
There are MANY ways to make this tiny problem "go away" for cars - but none to make it "go away" for humans.
www.sjbaker.org
That way cars can use it better than having to read the actual traffic signal. Better yet, have cars report to police if there is some inconsistency to the police. That way after several reports they will know the signal has a problem.
Yes, replacing every street sign in the nation in order to accommodate your shitty computer vision sounds like an appropriate solution. (Are the self-driving car manufacturers volunteering to pay for this?)
Except in the onramp case, all of these occur on the freeway itself.
Then there's Interstate 5, which, in addition to having stop signs at either end of it because DUH, the two busiest border crossings in the world at opposite ends of it, it also has traffic lights in the Portland area thanks to the drawbridge, since Clark County is so spiteful it can't be arsed to accept light rail to Oregon.
Furries make the internet go.
Bad data in, bad data out. This is what you get when you train your neural network with only "perfect" examples. It has no context for any variation whatsoever.