Slashdot Mirror
Researchers Trick Tesla Autopilot Into Steering Into Oncoming Traffic (arstechnica.com)
An anonymous reader quotes a report from Ars Technica: Researchers have devised a simple attack that might cause a Tesla to automatically steer into oncoming traffic under certain conditions. The proof-of-concept exploit works not by hacking into the car's onboard computing system. Instead, it works by using small, inconspicuous stickers that trick the Enhanced Autopilot of a Model S 75 into detecting and then following a change in the current lane. Researchers from Tencent's Keen Security Lab recently reverse-engineered several of Tesla's automated processes to see how they reacted when environmental variables changed. One of the most striking discoveries was a way to cause Autopilot to steer into oncoming traffic. The attack worked by carefully affixing three stickers to the road. The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
So, optical illusions fool a driver. They just fund a kind that fools a digital driver. Film at 11
Because machines "think" very differently from people, the optical illusions will be very different. No surprise there,
Next we'll get a headline that if you put a number sticker over speed limit signs, human drivers can be tricked into driving at the wrong speed - even though very clearly the stickers have the wrong UV patterns and react to LIDAR clearly in an altered way.
I found that it's super easy to make human drivers crash with a simple $5 laser.
It's amazing how many of our systems only work with the underlying assumption that we're not actively trying to murder each other at any given moment.
---- El diablo esta en mis pantalones! Mire, mire!
Real airline pilots have tons of training, understand the limits of the systems, and are literally PAID over six figures to do the terribly boring job of monitoring the system. Tesla owners could have zero training, are certainly not privy to the actual system limitations, and are shown tons of marketing indicating the main benefit of autopilot is the ability to NOT pay attention.
But I can see how they are "basically the same thing".
They even said, if there had been cars there, the Tesla likely would have noticed them and not blithely crashed head on.
And if the AoA sensor was reading wrong, the pilot likely would have taken control and not let the plane crash. Those "likely" sure are dangerous.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
You're not in control, but you have to be constantly ready to take control. You don't have insight into its mental processes so you never know what it's about to do, but you have to be constantly ready to react to what it just did.
And people find driving with Autopilot to be less stressful than driving without it? I guess I'm different from most people.
The difference being a human that sees lane markers leading into active oncoming traffic will decide there are shenigans and not follow.
I guarantee you I can find examples of humans would would be fooled. There are a LOT of humans that are quite easy to mislead and all humans can be mislead sometimes. The only difference is that the tactics that fool a human will usually be different than those that fool a machine but make no mistake that both can be fooled. There are plenty of examples of people very dutifully following the instructions from their GPS into trouble despite it being painfully obvious that the GPS instructions were faulty in some way.
Your notion that people are harder to fool is not entirely supported by the facts.
This is not 'a machine can be fooled like a human', it's a reminder that the machine is still a *lot* dumber than a human.
That depends very much on the human in question. I will be happy to introduce you to some humans I know who should not be permitted to drive on public roads. I'm pretty sure you know some like this as well. Not all humans are "smarter" than machines for driving purposes even today. Your average human almost certainly is a better driver than the current state of the art machine but some machines have already surpassed some humans and they are getting better all the time while human drivers aren't.
The sooner self-driving cars are outlawed the better. Deploy it.
"...not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually..."
Oh yeah, I'm sure the majority of people can be trusted to remain ready at all times to take over the system called "Autopilot"... Most would assume they can just fall asleep and the thing will magically drive itself. People are dumb, crash reports at 11.