Slashdot Mirror
Researchers Trick Tesla Autopilot Into Steering Into Oncoming Traffic (arstechnica.com)
An anonymous reader quotes a report from Ars Technica: Researchers have devised a simple attack that might cause a Tesla to automatically steer into oncoming traffic under certain conditions. The proof-of-concept exploit works not by hacking into the car's onboard computing system. Instead, it works by using small, inconspicuous stickers that trick the Enhanced Autopilot of a Model S 75 into detecting and then following a change in the current lane. Researchers from Tencent's Keen Security Lab recently reverse-engineered several of Tesla's automated processes to see how they reacted when environmental variables changed. One of the most striking discoveries was a way to cause Autopilot to steer into oncoming traffic. The attack worked by carefully affixing three stickers to the road. The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
The researchers noted that Autopilot uses a variety of measures to prevent incorrect detections. The measures include the position of road shoulders, lane histories, and the size and distance of various object. [A section of the researchers' 37-page report] showed how researchers could tamper with a Tesla's autowiper system to activate wipers on when rain wasn't falling. Unlike traditional autowiper systems -- which use optical sensors to detect moisture -- Tesla's system uses a suite of cameras that feeds data into an artificial intelligence network to determine when wipers should be turned on. The researchers found that -- in much the way it's easy for small changes in an image to throw off artificial intelligence-based image recognition (for instance, changes that cause an AI system to mistake a panda for a gibbon) -- it wasn't hard to trick Tesla's autowiper feature into thinking rain was falling even when it was not. So far, the researchers have only been able to fool autowiper when they feed images directly into the system. Eventually, they said, it may be possible for attackers to display an "adversarial image" that's displayed on road signs or other cars that do the same thing. In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us." They added: "The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
So, optical illusions fool a driver. They just fund a kind that fools a digital driver. Film at 11
Because machines "think" very differently from people, the optical illusions will be very different. No surprise there,
Next we'll get a headline that if you put a number sticker over speed limit signs, human drivers can be tricked into driving at the wrong speed - even though very clearly the stickers have the wrong UV patterns and react to LIDAR clearly in an altered way.
They, in fact, did not "steer a Tesla into oncoming traffic", but instead made the software "think" there was a lane line where there was none. The car did go the wrong way (or would have if they'd let it), but there was no traffic. They even said, if there had been cars there, the Tesla likely would have noticed them and not blithely crashed head on.
Everything you know is wrong, Just forget the words and sing along.
"The rest of the findings are all based on scenarios in which the physical environment around the vehicle is artificially altered to make the automatic windshield wipers or Autopilot system behave differently, which is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually operate the windshield wiper settings at all times."
While I agree that it shouldn't be a realistic concern, people are stupid. And there have been a few cases where drivers have relied on autopilot too much and have caused wrecks. I don't think this is Tesla's fault really, people don't understand the different levels of automation although they could probably do a better job of explaining what autopilot does and does not do. The name is appropriate, like real autopilot in an aircraft, the driver has to pay attention and be ready to take control at any time.
Clearly you've never lived anywhere with traffic or freeways.
Human drivers too would be affected if someone adds fake lane marking. I remember a prankster was arrested for rearranging the traffic cones in a construction zone to create two colliding lanes. There is so much more mayhem that can happen, people might remove stop signs, drop stuff from overpasses, or scatter nails or tacks on the high way....
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I found that it's super easy to make human drivers crash with a simple $5 laser.
It's amazing how many of our systems only work with the underlying assumption that we're not actively trying to murder each other at any given moment.
---- El diablo esta en mis pantalones! Mire, mire!
Were these engineers contracted out to Boeing to design their MCAS system for the 737max?
Seriously, the design pattern of a life critical system that makes decisions based on one set of or type of sensor is asinine. Boeing should have had the MCAS's AoA indicator cross checked with velocity, GPS, and engine data. Tesla should have the wiper controls visual sensor crosschecked with a humidifier, and the lane sensor crosschecked with a LIDAR. Isn't this just basic stuff here. I don't consider myself a genius, but this seems fairly obvious.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
Computers will never be people. I don't think we WANT them to become that smart. Imagine the moral questions on that.
Corporatism != Free Market
Autosteering via rails will never be reliable. Scientists have discovered yet another way to derail a train. Railroads will go down in history as silly as mechanical chess players.
These and other funky glitches are reasons why I wouldn't really want to fully depend on the Tesla system. Google Car on the other hand uses a much larger complement of sensors and a 3D space mapping LIDAR to avoid these issues, unless you're going to go as far as placing a styrofoam lifesized car or panel onto the road which would almost fool real-life drivers as well. Google believes in the concept of making sure the system fully works instead of taking dangerous compromises.
You're not in control, but you have to be constantly ready to take control. You don't have insight into its mental processes so you never know what it's about to do, but you have to be constantly ready to react to what it just did.
And people find driving with Autopilot to be less stressful than driving without it? I guess I'm different from most people.
The difference being a human that sees lane markers leading into active oncoming traffic will decide there are shenigans and not follow.
I guarantee you I can find examples of humans would would be fooled. There are a LOT of humans that are quite easy to mislead and all humans can be mislead sometimes. The only difference is that the tactics that fool a human will usually be different than those that fool a machine but make no mistake that both can be fooled. There are plenty of examples of people very dutifully following the instructions from their GPS into trouble despite it being painfully obvious that the GPS instructions were faulty in some way.
Your notion that people are harder to fool is not entirely supported by the facts.
This is not 'a machine can be fooled like a human', it's a reminder that the machine is still a *lot* dumber than a human.
That depends very much on the human in question. I will be happy to introduce you to some humans I know who should not be permitted to drive on public roads. I'm pretty sure you know some like this as well. Not all humans are "smarter" than machines for driving purposes even today. Your average human almost certainly is a better driver than the current state of the art machine but some machines have already surpassed some humans and they are getting better all the time while human drivers aren't.
You'd have to be able to predict the future to know exactly where to place the stickers where the assassination victim would head-on the truck coming the other way
After observing someone for a little while, you can start to predict their movements. Most people operate on a general routine and schedule. You can know their route to work, and approximately when they will get there. Your confederate could be driving the truck. I'm sure there are less elaborate ways to do someone in, however.
"What the American public doesn't know is what makes them the American public." -Ray Zalinsky (Tommy Boy)
Work zones with lines all over the place may trigger this??
If you read the PDF, you will see that the "researchers" went to great length to hack into the CANBus on the Tesla and deliberately feed it false information to get it to see something that wasn't real.
As anyone with two brain cells understands, something that is programmed to react deterministically to data input can be made to do whatever one wants by feeding it the right data.
There really and truly is nothing to see here as this "trickery" required a somewhat monumental engineering effort to pull off.
The military finally admitted to the UFOs. Do you get any news at all?
LOL.. Technically maybe, as in "We don't (at this point) know what that was." but not in the sense that aliens have landed. Could have been swamp gas, weather balloons, optical illusions or even a bad acid trip, we don't know, but nobody has any evidence that aliens landed.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
A plastic bag over a stop-sign should work too and it would get the non-Tesla drivers as well.
Also continuing the middle line into the abyss and hiding the original line that goes around.
Putting a fake stop-sign on the middle of the highway should be fun too.
No need to be a 'researcher' for stuff like that.
Just don't place the stickers on both sides or they will still miss each other.
"...not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so and can manually..."
Oh yeah, I'm sure the majority of people can be trusted to remain ready at all times to take over the system called "Autopilot"... Most would assume they can just fall asleep and the thing will magically drive itself. People are dumb, crash reports at 11.
Yes, I'm sure you can, idiots will always exist, but I bet you can't find anything that will systemically cause human drivers to.
Want to bet? All it takes is a badly labeled street sign in the right place and most humans will be fooled at least once. Naturally the failure modes for humans are different from machines but we have 40,000 deaths from auto accidents each year that prove that humans aren't especially safe or reliable drivers.
Furthermore you are aware that software can be upgraded, right? Every machine can learn from the errors of every other machine. You can make a machine less idiotic - humans not so much. When Tesla or some other auto maker finds a bug it is quite possible to push the fix out to every car and completely eliminate that error mode going forward. Humans do nearly so easily learn from the mistakes other drivers. Some in fact are quite obstinate about not changing. See the resistance by many to wearing seat belts.
Would stuff like autopilot be considered less controversial?
I'd guess it would get promoted by the company as something other than quite such an autonomous self-driving platform.
Volvo (and I'm sure others, I've only been exposed to Volvo's system personally) has what amounts to a nearly self-driving system -- distance sensing cruise, lane centering, you very nearly don't need to "drive" to drive, yet there's not nearly the constant promotion/hostility to their system and other similar ones.
Even my lowly Subaru has really good optical distance sensing cruise and lane keeping (but not centering) assist and lane departure warnings.
My old Volvo's radar-based distance sensing cruise actually proved quite useful in a couple of bad snowstorms -- it could see the car in front of me better than I could.
You know very well the remaining unexplained cases were all experimental aircraft. I didn't say anything about aliens.
Or you just place them in an area where almost everyone drives a Tesla like around Google or Facebook HQ.
As long as its the people driving them its fine.
Wanna buy a shirt?
https://www.redbubble.com/people/stealthfinger/shop?asc=u
In my state we have a "no cell phone use in work zones" law, and giant signs before all work zones notifying drivers. I'd bet that the same will happen with self-driving cars as they become more popular. And like the no-phones law, most people will obey it, some won't and will get fined, some won't and will get into accidents and fined, and some will get into accidents and injure someone else and get fined and potentially also get jail time.
Velociraptor = Distiraptor / Timeraptor
Heading into a concrete wall is also deadly at high enough speed. And that is much easier to do, since walls don't move.
https://en.wikipedia.org/wiki/...
Yes.... But machines are supposed to be BETTER.
They may be someday. They already are better than some human drivers. There are some people who really should not be allowed to drive and many people drive impaired/distracted with some regularity. Currently your hypothetical average human driver is probably still better than even the best machine driver but the machine's are getting better and human drivers are not. Eventually it seems probable that machine drivers will be safer than most (or all) human drivers. Exactly when that happens is unclear but within my lifetime seems reasonable.
Before self-driving cars are ready, they must be able to avoid jumping into the same lane as active oncoming traffic while traveling down a road or highway, even if the road markings are confusing or in error.
Umm, you do know that humans routinely are involved in head on collisions (about 10% of all fatal accidents are head on) and people voluntarily jump into the oncoming traffic lanes all the time for various reasons. Humans are also routinely confused by bad road markings - even good drivers. You seem to be under the delusion that humans do a good job avoiding these problems when the data clearly shows that humans are really quite bad at it to the tune of about 40,000 deaths per year in the US alone.
self-driving with no controls? or taxis? will do what in work zones then?
I'm somewhat amazed that you're almost 100% wrong. Both this and the Boeing have a human override, the Tesla far more so than the plane, for good reason. The plane that crashed had the same issue resolved the day before by following the proper steps to address it. Yes, there was a problem with the plane, but it was solvable by the pilots. The failure the next day was threefold, and not just a technical issue. The malfunctioning instrument wasn't fixed, the pilots weren't notified of the issue or the actions of the pilots the day before, and the pilots didn't go through the checklist of procedures to fix the issue.
Passenger plane pilots are not expected to grab the yoke and make a hard left turn and do a barrel roll. They're expected to calmly walk through their checklists to diagnose and fix issues with the plane. The pilots who did saved the plane and all the people on it, while the pilots who didn't lost the plane and everyone on it. The failure was both technical and training.
To this article, I really don't know what you're on about.
In a statement, Tesla officials said that the vulnerabilities addressed in the report have been fixed via security update in 2017, "followed by another comprehensive security update in 2018, both of which we released before this group reported this research to us."
So because a minor issue was fixed a year or two ago, you don't think we'll have effective self-driving cars in the near future? To me, that suggests the opposite.
Technological issues get discovered and fixed, and we move on. The alternate is not advancing as a species. You also ignore the good that the technology likely has done and will do in the future, to focus entirely on the harm it briefly did. That's a really short-sighted position to take.
Velociraptor = Distiraptor / Timeraptor
This wouldn't work on a freeway since they are generally divided by K-Rail which are too heavy to move without a crane.
Lane markings are standard and set in a way for a very specific reason. There is a ton of regulations around how they work.
Plus, even if you got the Tesla onto oncoming traffic, who's to say it wouldn't see the approaching car and ABS itself to a dead stop, which it is supposed to do. Hopefully, the other driver does the same!
Computers suck at processing analog inputs. It wouldn't be hard to spoof a car with minimal effort especially something a primitive autonomous vehicle. Not that other autonomous vehicles will fair any better. Assuming they ever appear on the roads without drivers (not for a long time), people will make sport of griefing them - gum on sensors, traffic cones on their roofs, boxes laid in front of them, graffiti tags etc. Even without the griefing it won't be surprising if they become so frequently stuck, blocking traffic that suddenly they won't see like a good idea at all.
I don't think that giving money like the author suggest will make him happy and will fix everything. I think its more of a social, family, real life issue problems than anything else here.
I'll grant you I'm not the best communicator. In the Boeing example, I used the word "easy" which gives me a little wiggle room. According to the reports, the MCAS system applies a lot of mechanical pressure to the stick in order to help the pilot counteract a stall. manually overcoming that pressure is something of a challenge. Sure, there was a cut out procedure if the pilots realized in time that the MCAS was behind the issue. That's a problematic "if" as the recent disasters demonstrated. How much time did they spend trying to pull the stick back before realizing there was a different sort of issue? Did they have that time in that situation? What if there was some other system at fault and the pilots wasted time cutting out the MCAS? As for self driving cars, the issue in the article is just one of many already found, and no doubt more lie in the future. My point is, as long as we have to have a human trying to stay alert on the wheel (which was Tesla's response) there is no point he may as well just drive the car. An actual sufficiently trustworthy self-driving setup for cars is a long way away as findings like this illustrate. I doubt people would accept an autonomous car which is only slightly safer than a human driver, it would just take one accident with some kids and the things will get pulled.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
it seems like it would work really well along hwy 1 here in CA. plus I don't think someone needs to be a hitman to try this. It just needs to be someone looking for a little anarchy who isn't thinking about consequences. Kids throw stuff off overpasses all the time just to see some shit happen.
https://www.nytimes.com/1994/0...
My mom's car was hit by a D battery thrown off an overpass once. She and her passenger saw the kids toss it. I don't think it was assassins.
I was once driving hwy 45 through northern wisconsin when some kids in the woods rolled a car wheel down a hill at me. again, probably not assassins.
The fact that one group of pilots did it right and the other didn't shows that they had the time. The question is if they had the training. That's looking like a part of the issue. Having a single point of failure and selling instrument failure lights as an upgrade seem to be the largest issue.
But you're still ignoring the fact that the MCAS likely kept a bunch of planes in the sky before an instrument failure brought a couple down. Pointing to it as some horrible idea that humans should never have done is silly if you're counting both the positives and the negatives of the system. Regardless, it's fixed now. And the first fix for this Tesla non-issue was two years ago.
I doubt people would accept an autonomous car which is only slightly safer than a human driver, it would just take one accident with some kids and the things will get pulled.
You know tesla has already sold more than 250,000 cars, right? Every day tens of thousands of people are turning on Autopilot and letting the car drive while they try to stay alert on the wheel.
I don't get why you think monitoring the car requires the same effort as driving. By all accounts it does not. It allows you to offload a ton of mental effort to the car, and your job boils down to, "is the car doing what it's supposed to?". If the answer is yes, you just sit there and enjoy the scenery.
Velociraptor = Distiraptor / Timeraptor
So instead of driving I now have to constantly monitor a computer to make sure it's doing what it's supposed to do?
No thanks, I'd rather have control at all times. That way if it fucks up, it's on me.
Yes, AC, instead of checking your blind spots, your speed, your lane position, adjusting your cruise control for slower cars in front of you, braking for emergencies, calculating when you need to recharge and finding your way to the most convenient and free charger, turning on your windshield wipers when it rains, adjusting the temperature of your batteries to make them as efficient as possible and a whole lot of other things, you instead have to keep a hand on the wheel and check the nav screen every few seconds.
That sounds like an onerous task, and totally not worth buying a tesla for. You definitely should keep driving your '84 stick-shift chevy. I'm sure you've got far more control of that, and it's nowhere near as mentally taxing to drive for a couple of hours.
FFS I don't even have a tesla and I can see the benefits. I get that there are drawbacks, but to claim that the self-driving mode is anything other than a major labor savor is crazy. I'd take that in my current car in a heartbeat.
Velociraptor = Distiraptor / Timeraptor
Also, this can be used without targeting. Just put some stickers on the road with frequent Tesla traffic, grab some popcorn and watch the mayhem. Bonus points for causing two Teslas to crash into each other.
The stickers were nearly invisible to drivers, but machine-learning algorithms used by by the Autopilot detected them as a line that indicated the lane was shifting to the left. As a result, Autopilot steered in that direction.
Hah! Jokes on you, Tencent researchers! I live in Australia.
So the car steers into a concrete barrier and you think that's better, somehow? Sure it may save the people in the oncoming lane but driver sill dead. Also, I doubt even two Tesla S's coming at each other at highway speed would stop in time. That would be something I would have to see to believe.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.