Slashdot Mirror
Researchers Trick Tesla Autopilot Into Steering Into Oncoming Traffic (arstechnica.com)
An anonymous reader quotes a report from Ars Technica: Researchers have devised a simple attack that might cause a Tesla to automatically steer into oncoming traffic under certain conditions. The proof-of-concept exploit works not by hacking into the car's onboard computing system. Instead, it works by using small, inconspicuous stickers that trick the Enhanced Autopilot of a Model S 75 into detecting and then following a change in the current lane. Researchers from Tencent's Keen Security Lab recently reverse-engineered several of Tesla's automated processes to see how they reacted when environmental variables changed. One of the most striking discoveries was a way to cause Autopilot to steer into oncoming traffic. The attack worked by carefully affixing three stickers to the road. The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
They, in fact, did not "steer a Tesla into oncoming traffic", but instead made the software "think" there was a lane line where there was none. The car did go the wrong way (or would have if they'd let it), but there was no traffic. They even said, if there had been cars there, the Tesla likely would have noticed them and not blithely crashed head on.
Everything you know is wrong, Just forget the words and sing along.
While "autopilot" is engaged, you do have visibility to "what the car sees" on the screen. That tells you what obstacles it sees as well as where it thinks the vehicle lanes are. If they don't seem to make sense to what you see, then it's time to take over.
Like the "autopilot" in planes, when the cruise control take over, it reduces cognitive load because the driver doesn't need to pay attention to as many things. That translates into less stress and the ability to pay attention for longer.
If the driver does other things instead, that's really the driver's fault. Though Tesla's marketing isn't really helping on that front, either.
The difference being a human that sees lane markers leading into active oncoming traffic will decide there are shenigans and not follow.
Complete bollocks. Care to set up a situation like that and see how many drivers follow the dots blindly?
Unfortunately this situation occurs quite frequently at road construction sites where new lanes are overlaid over existing lanes. The old and new sets of lane markings make lane localization difficult at times even for humans to know where the true lane lies. Often in these cases, the human will follow the preceding and surrounding traffic in an attempt to avoid collisions, even if the true lane appears to be otherwise.
I'm somewhat amazed that you're almost 100% wrong. Both this and the Boeing have a human override, the Tesla far more so than the plane, for good reason. The plane that crashed had the same issue resolved the day before by following the proper steps to address it. Yes, there was a problem with the plane, but it was solvable by the pilots. The failure the next day was threefold, and not just a technical issue. The malfunctioning instrument wasn't fixed, the pilots weren't notified of the issue or the actions of the pilots the day before, and the pilots didn't go through the checklist of procedures to fix the issue.
Passenger plane pilots are not expected to grab the yoke and make a hard left turn and do a barrel roll. They're expected to calmly walk through their checklists to diagnose and fix issues with the plane. The pilots who did saved the plane and all the people on it, while the pilots who didn't lost the plane and everyone on it. The failure was both technical and training.
To this article, I really don't know what you're on about.
In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us."
So because a minor issue was fixed a year or two ago, you don't think we'll have effective self-driving cars in the near future? To me, that suggests the opposite.
Technological issues get discovered and fixed, and we move on. The alternate is not advancing as a species. You also ignore the good that the technology likely has done and will do in the future, to focus entirely on the harm it briefly did. That's a really short-sighted position to take.
Velociraptor = Distiraptor / Timeraptor