Slashdot Mirror
Researchers Trick Tesla Autopilot Into Steering Into Oncoming Traffic (arstechnica.com)
An anonymous reader quotes a report from Ars Technica: Researchers have devised a simple attack that might cause a Tesla to automatically steer into oncoming traffic under certain conditions. The proof-of-concept exploit works not by hacking into the car's onboard computing system. Instead, it works by using small, inconspicuous stickers that trick the Enhanced Autopilot of a Model S 75 into detecting and then following a change in the current lane. Researchers from Tencent's Keen Security Lab recently reverse-engineered several of Tesla's automated processes to see how they reacted when environmental variables changed. One of the most striking discoveries was a way to cause Autopilot to steer into oncoming traffic. The attack worked by carefully affixing three stickers to the road. The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
You'd have to be able to predict the future to know exactly where to place the stickers where the assassination victim would head-on the truck coming the other way
So, optical illusions fool a driver. They just fund a kind that fools a digital driver. Film at 11
Because machines "think" very differently from people, the optical illusions will be very different. No surprise there,
Next we'll get a headline that if you put a number sticker over speed limit signs, human drivers can be tricked into driving at the wrong speed - even though very clearly the stickers have the wrong UV patterns and react to LIDAR clearly in an altered way.
'Autopilot is not autopilot and should not be relied on.' Self-driving cars will go down in history being as silly as military 'UFOs' from the 1950s.
They, in fact, did not "steer a Tesla into oncoming traffic", but instead made the software "think" there was a lane line where there was none. The car did go the wrong way (or would have if they'd let it), but there was no traffic. They even said, if there had been cars there, the Tesla likely would have noticed them and not blithely crashed head on.
Everything you know is wrong, Just forget the words and sing along.
"The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
While I agree that it shouldn't be a realistic concern, people are stupid. And there have been a few cases where drivers have relied on autopilot too much and have caused wrecks. I don't think this is Tesla's fault really, people don't understand the different levels of automation although they could probably do a better job of explaining what autopilot does and does not do. The name is appropriate, like real autopilot in an aircraft, the driver has to pay attention and be ready to take control at any time.
Human drivers too would be affected if someone adds fake lane marking. I remember a prankster was arrested for rearranging the traffic cones in a construction zone to create two colliding lanes. There is so much more mayhem that can happen, people might remove stop signs, drop stuff from overpasses, or scatter nails or tacks on the high way....
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I found that it's super easy to make human drivers crash with a simple $5 laser.
It's amazing how many of our systems only work with the underlying assumption that we're not actively trying to murder each other at any given moment.
---- El diablo esta en mis pantalones! Mire, mire!
All ready? Patch!
It's like Telsa pwnage x 10000e2.0000000000000
Were these engineers contracted out to Boeing to design their MCAS system for the 737max?
Seriously, the design pattern of a life critical system that makes decisions based on one set of or type of sensor is asinine. Boeing should have had the MCAS's AoA indicator cross checked with velocity, GPS, and engine data. Tesla should have the wiper controls visual sensor crosschecked with a humidifier, and the lane sensor crosschecked with a LIDAR. Isn't this just basic stuff here. I don't consider myself a genius, but this seems fairly obvious.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
Computers will never be people. I don't think we WANT them to become that smart. Imagine the moral questions on that.
Corporatism != Free Market
These and other funky glitches are reasons why I wouldn't really want to fully depend on the Tesla system. Google Car on the other hand uses a much larger complement of sensors and a 3D space mapping LIDAR to avoid these issues, unless you're going to go as far as placing a styrofoam lifesized car or panel onto the road which would almost fool real-life drivers as well. Google believes in the concept of making sure the system fully works instead of taking dangerous compromises.
You're not in control, but you have to be constantly ready to take control. You don't have insight into its mental processes so you never know what it's about to do, but you have to be constantly ready to react to what it just did.
And people find driving with Autopilot to be less stressful than driving without it? I guess I'm different from most people.
The difference being a human that sees lane markers leading into active oncoming traffic will decide there are shenigans and not follow.
I guarantee you I can find examples of humans would would be fooled. There are a LOT of humans that are quite easy to mislead and all humans can be mislead sometimes. The only difference is that the tactics that fool a human will usually be different than those that fool a machine but make no mistake that both can be fooled. There are plenty of examples of people very dutifully following the instructions from their GPS into trouble despite it being painfully obvious that the GPS instructions were faulty in some way.
Your notion that people are harder to fool is not entirely supported by the facts.
This is not 'a machine can be fooled like a human', it's a reminder that the machine is still a *lot* dumber than a human.
That depends very much on the human in question. I will be happy to introduce you to some humans I know who should not be permitted to drive on public roads. I'm pretty sure you know some like this as well. Not all humans are "smarter" than machines for driving purposes even today. Your average human almost certainly is a better driver than the current state of the art machine but some machines have already surpassed some humans and they are getting better all the time while human drivers aren't.
Work zones with lines all over the place may trigger this??
Things like this lead me to believe we are many decades away from effective self-driving cars. Just look at the Boeing issues recently, which among other things demonstrated the need for easy human override. I don't see much value in a self-steering car where the driver has to keep watching and correct the autopilot on a moment's notice. That's not a realistic expectation.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
Cause of death of the new leader who promised to root out corruption: Automatic driving malfunction.
Case closed! Nothing more to see! That strange wireless device in the wreckage...quietly vanished!
If you read the PDF, you will see that the "researchers" went to great length to hack into the CANBus on the Tesla and deliberately feed it false information to get it to see something that wasn't real.
As anyone with two brain cells understands, something that is programmed to react deterministically to data input can be made to do whatever one wants by feeding it the right data.
There really and truly is nothing to see here as this "trickery" required a somewhat monumental engineering effort to pull off.
The sooner self-driving cars are outlawed the better. Deploy it.
A plastic bag over a stop-sign should work too and it would get the non-Tesla drivers as well.
Also continuing the middle line into the abyss and hiding the original line that goes around.
Putting a fake stop-sign on the middle of the highway should be fun too.
No need to be a 'researcher' for stuff like that.
"...not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually..."
Oh yeah, I'm sure the majority of people can be trusted to remain ready at all times to take over the system called "Autopilot"... Most would assume they can just fall asleep and the thing will magically drive itself. People are dumb, crash reports at 11.
Yes, I'm sure you can, idiots will always exist, but I bet you can't find anything that will systemically cause human drivers to.
Want to bet? All it takes is a badly labeled street sign in the right place and most humans will be fooled at least once. Naturally the failure modes for humans are different from machines but we have 40,000 deaths from auto accidents each year that prove that humans aren't especially safe or reliable drivers.
Furthermore you are aware that software can be upgraded, right? Every machine can learn from the errors of every other machine. You can make a machine less idiotic - humans not so much. When Tesla or some other auto maker finds a bug it is quite possible to push the fix out to every car and completely eliminate that error mode going forward. Humans do nearly so easily learn from the mistakes other drivers. Some in fact are quite obstinate about not changing. See the resistance by many to wearing seat belts.
Would stuff like autopilot be considered less controversial?
I'd guess it would get promoted by the company as something other than quite such an autonomous self-driving platform.
Volvo (and I'm sure others, I've only been exposed to Volvo's system personally) has what amounts to a nearly self-driving system -- distance sensing cruise, lane centering, you very nearly don't need to "drive" to drive, yet there's not nearly the constant promotion/hostility to their system and other similar ones.
Even my lowly Subaru has really good optical distance sensing cruise and lane keeping (but not centering) assist and lane departure warnings.
My old Volvo's radar-based distance sensing cruise actually proved quite useful in a couple of bad snowstorms -- it could see the car in front of me better than I could.
True, but what idiot keeps autopilot engaged in a work zone? Oh, right. Idiot to he named in future news report. #cantfixstupid
In my state we have a "no cell phone use in work zones" law, and giant signs before all work zones notifying drivers. I'd bet that the same will happen with self-driving cars as they become more popular. And like the no-phones law, most people will obey it, some won't and will get fined, some won't and will get into accidents and fined, and some will get into accidents and injure someone else and get fined and potentially also get jail time.
Velociraptor = Distiraptor / Timeraptor
Yes.... But machines are supposed to be BETTER.
They may be someday. They already are better than some human drivers. There are some people who really should not be allowed to drive and many people drive impaired/distracted with some regularity. Currently your hypothetical average human driver is probably still better than even the best machine driver but the machine's are getting better and human drivers are not. Eventually it seems probable that machine drivers will be safer than most (or all) human drivers. Exactly when that happens is unclear but within my lifetime seems reasonable.
Before self-driving cars are ready, they must be able to avoid jumping into the same lane as active oncoming traffic while traveling down a road or highway, even if the road markings are confusing or in error.
Umm, you do know that humans routinely are involved in head on collisions (about 10% of all fatal accidents are head on) and people voluntarily jump into the oncoming traffic lanes all the time for various reasons. Humans are also routinely confused by bad road markings - even good drivers. You seem to be under the delusion that humans do a good job avoiding these problems when the data clearly shows that humans are really quite bad at it to the tune of about 40,000 deaths per year in the US alone.
self-driving with no controls? or taxis? will do what in work zones then?
Computers suck at processing analog inputs. It wouldn't be hard to spoof a car with minimal effort especially something a primitive autonomous vehicle. Not that other autonomous vehicles will fair any better. Assuming they ever appear on the roads without drivers (not for a long time), people will make sport of griefing them - gum on sensors, traffic cones on their roofs, boxes laid in front of them, graffiti tags etc. Even without the griefing it won't be surprising if they become so frequently stuck, blocking traffic that suddenly they won't see like a good idea at all.
I don't think that giving money like the author suggest will make him happy and will fix everything. I think its more of a social, family, real life issue problems than anything else here.
This is not artificial intelligence. Humanity would benefit if you drank several CCs of strychnine!
More likely is they will have an official "'NO AUTOPILOT ZONE" sign with distinctive markings and some RFID that signals the vehicle, which in turn signals that driver that AP is being disengaged. The sign probably isn't even needed.
From an engineering perspective, that's much more robust than leaving it to chance and hoping either the driver obeys or the software detects it, and much easier than teaching the car to "determine a work zone area in widely-varying, arbitrary contexts"
Can't the same thing be said about teenagers stealing a stop sign? If you actually put stickers in the road to steer a car into oncoming traffic, you are a murderer.
The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
Hah! Jokes on you, Tencent researchers! I live in Australia.