Slashdot Mirror
Researchers Discover How To Fool Tesla's Autopilot System (cnet.com)
An anonymous reader writes from a report via CNET: Researchers from the University of South Carolina, Zhejiang University and Qihoo 360 have discovered how to fool Tesla's Autopilot sensors, according to a report from Wired. The researchers were able to trick the system into thinking an object didn't exist when it did, and that an object existed when in fact it did not. Therefore, possible security concerns arise as Autopilot could drive incorrectly, potentially putting passengers and others in danger. CNET reports: "Two pieces of radio equipment were used to convince Tesla's radar sensor that a cart was not placed directly in front of it. One of those pieces, a signal generator from Keysight Technologies, costs about $90,000. The group also tricked the car's short-range parking sensors into malfunctioning using about $40 worth of equipment. Wired points out that this was, thankfully, a rather difficult feat. Most of the technological tomfoolery was done on a stationary car. Some of the required equipment was expensive, and it didn't always work. But it brings up an important point -- even though Autopilot is quite capable, there's still no substitute for an attentive human driver, ready to take control at a moment's notice."
Because it's so hard to make humans see or not see things.
If you spent the same resources to fool a human driver, how hard would that be?
Of course you can trick any sensor invented by man some way or other. That's nothing new. We even know tons of ways to trick the sensors made by god/nature aka our eyes as well. Shine a bright light into them for $10 or maybe $100 and the driver will be forced to drive blind. Or you can have a $0 natural snow storm and the driver will also be on literally very dangerous ground: zero visibility and icy roads.
The point is not that either can be fooled, the point is, is the mechanical sensor better or at least equal to mark I eyeball? Is the program doing the automatic driving at least as good as an average driver? as good as the best driver possible?
Researchers have discovered a way to disable the human "autopilot" system using just $10 worth of equipment. By shining a flashlight in their eyes, they are able to totally disable the primary optical sensors, so preventing the navigation system from avoiding objects in the car's path. While this was only tested in a lab environment, it is feared that Russian agents and ISIS terrorists could use a similar technique to cause mass casualties.
Don't drive behind a truck full of signal generators on a bumpy road.
So for $90,000 you can set up a trap for a Tesla car running with autopilot, and possibly cause a crash and kill the driver. Lots of people can do that a lot cheaper using a gun. So what's the problem?
Seriously, who autoenables autonomous driving but a fool.
Now, if you'll excuse me, I'll go back to using my 60 mpg stick vehicle that doesn't bleed security info as it was made in 1989.
-- Tigger warning: This post may contain tiggers! --
Yes, someone going through great effort can cause a crash. I've know cases where people stoodn on overpasses and threw down bricks to cause crashes. Nobody published papers on the "brick loophole" in car security. In most of the examples, it'd have been easier to just cut the brake lines. But we have to target the sensors to get media attention, for a non-story.
Learn to love Alaska
There are plenty of optical illusions for people, too.
Unless this was done on purpose/malevolently (and that could be prosecuted regardless), this seems to me far different from things like hacking into the car's computer itself.
So it's easy to do an DOS attack to auto drive cars just wait for the days when you can drop an $40 box and shut down a major road
For that much money (and even with an order of magnitude less), there are probably a ton of ways to fool humans. What exactly are they accomplishing here?
Researchers discover that for $100 they can dig a hole, cover it with a thin layer of asphault and potentially kill a driver.
In other news, researchers also discover that $3 hedge clippers can cut a brake line endangering drivers.
Researchers discover that $10 high powered flash light carefully timed at a blind corner can confuse a human driver's imaging sensors.
The last sentence sums up what Elon Musk has been saying about AutoPilot:
"even though Autopilot is quite capable, there's still no substitute for an attentive human driver, ready to take control at a moment's notice."
The technology is not called "self driving" - it is called autopilot. Similar to plane where course and speed are maintained. Tesla reminds users to keep hands on the wheel and remain attentive.
No news here. Couple that with the cost of the hack, and there is not much to report. I could fool a real driver with mirrors and some Acme landscape canvas.
So not something you would ever encounter.
We aren't worrying about people using high end equipment to make a car crash.
We are worrying about a passive object that is shiny and reflective in just the right way to make it invisible.
Or an active object the size of a small house that because of the light colored paint appears invisible on the horizon.
Those are legitimate concerns at this point.
ZOMG system crashes when exposed to light at 572nm flashing at 13.37HZ while the left rear window control is pressed in the down direction while the car is driving upside down in reverse on july 22 2017 at 2:32.021PM in the rain while being struck by lightning.
Is not however anything anyone is or should be concerned about at this point.
Minimum threshold fixed. Thanks!
It seems to me that the best choice is if both the human driver is paying attention and the autopilot is on.
And how in the HELL do you figure that when the autopilot is going to make split-second sub-human decisions to react and manipulate a vehicle to dodge an imaginary object that the human can clearly see does not exist? By the time the human reacts, the autopilot may have already put them and others around them in considerable danger (like SLAMMING on the brakes to avoid an imaginary object on a freeway).
Second best is the autopilot by itself (at least it can pay attention all of the time)
And as these sensor hacks get fine-tuned, they will be able to fool the autopilot all of the time. No, this isn't the answer either.
, and last is the human (who can both be fooled and be inattentive).
Inattentive? Yes. Fooled by an imaginary invisible object? Highly unlikely.
This entire proof of concept was to demonstrate that machine sensors are still not that intelligent, and can be fooled worse than a 5-year old.
That it takes $90,000 worth of equipment and then always doesn't work right is pretty darn impressive to me. Where I live a good 30 percent of drivers are too old to be behind the wheel and another 10 percent are functional alcoholics. Share the road with south Florida drivers long enough and you'll be begging for autopilot.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
Obviously, you want a self-driving car to have the best possible auto-pilot technology you can put into it. But purposeful attempts to trick it into not detecting objects, or into thinking objects are there that aren't really there? That means nothing, IMO. What matters is that it does a reliable job of these things in real-world situations where nobody is TRYING to fool the system.
Human drivers see things all the time and misinterpret them. (There's that popular photo going around social media where someone painted a tunnel on the side of a concrete wall and a car tried to drive off the road, into it, for example. And certainly, people have reported mirages ahead of them on roads for decades.)
Did they paint a tunnel in a wall?
Warning! Keep Out of Eyes! Wash Out with Water! Don't Drink Soap! Dilute! Dilute!
And that is the reference here. Automatic drive systems do not need to be perfect to be a good replacement. They just need to be better than the average driver. They will start to safe lives when they are better than bad drivers though.
Of course, everybody believes themselves to be good drivers, but the simple statistical reality is that most are in the range from somewhat above average to really bad.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
The Tesla autopilot feature does NOT (we're randomly using all caps here, right?) randomly jerk you around the road in split second maneuvers.
Then tell me something; if the entire purpose of a car controlled by a machine is to be able to react better and faster than a human, why in the hell would it be designed to literally react slower than one?
Paying "attention" better than a human becomes rather fucking worthless if it reacts worse than a 80-year old drunk behind the wheel. Split-second decisions are exactly what is often needed to save lives, as humans have demonstrated for decades now. Autonomous solutions are justified for the specific purpose of enhancing that reaction time beyond human capability, not making it worse.
And my example had to do with hard braking in a high-speed and highly congested scenario, which is hardly "random" movements, but can and likely will put others in danger.
Give me a 35 YEAR (ha, all caps. This is fun. I see why you do this) old and I'll fool him for a lot less money than 80 grand. If that's what they were trying to demonstrate, they failed miserably.
Short of pumping a 35-year old full of hallucinogens, I'd like to see how you're going to fool an adult experienced driver with an invisible object.
0.o "Autopilot" literally means "self piloting".
Seriously Tesla/Musk apologists words mean things - and autopilot does not mean what you keep claiming it does. It doesn't mean "an assistant which still requires constant human monitoring and supervision", it means "an automatic system that replaces human operators". From the first line of the Wikipedia entry on autopilots entry - An autopilot is a system used to control the trajectory of a vehicle without constant 'hands-on' control by a human operator being required . (Emphasis mine.) And if you go searching dictionary definitions, you'll find endless repitions of the same themes. Etc.. etc... *That* is what the general public (correctly) believes an autopilot does - automatically operates the aircraft.
Tesla is using the word in a misleading fashion. And if this were Toyota or GM, the tech hipsters would be breaking out the torches and pitchforks.
*Dons Nomex suit* Flame away at the truth, I've got karma to to burn.
Couldn't you just use black tape to cover the road markings and white tape to make new ones, to veer the car off the road and over a cliff?
Don't need bricks; get a scantly-clad attractive female* to walk down the side of a busy road. I can think of several times where such distractions almost got me into a tangle. And it's legal, unlike bricks.
I'd be happy to assist in such research by inspecting the applicants for free.
* Or scantly clad males might work also, who knows. Didn't mean to be discriminatory. (Insert obligatory Commander Taco joke here.)
Table-ized A.I.
Sure. Accidents happen because of distracted driving all the time. Accidents happen because the driver didn't see something in the road all the time. Accidents happen when people swerve to avoid potholes and instead hit something else. Simply put, accidents happen. And it happens to adults, not 5 year olds. Get your head out of your ass.
The test demonstrated that it was possible to fool the car into thinking an imaginary object was in front of it. Now take a 5-year old and stand them in a room and try and convince them that a large imaginary wooden crate is right next to them. Needless to say you would be hard-pressed to convince that toddler, much less an experienced driver. And yet a highly complex autonomous system was fooled, and clearly demonstrates a weakness that humans would likely not suffer from.
My example was in context and relevant to this discussion. Yes, accidents do happen, but someone purposely fucking with autonomous sensors is no accident. If you were not suffering from cranial-rectal inversion, this would be obvious.
For that kinda scratch, I'm pretty sure we could rig something up that fools a human driver too.
This signature is false.
Basically.
I was thinking more of painting the road lines into a rock wall, then painting a picture of a tunnel on said wall.
(Then waiting for a roadrunner to come by)
After all, Coyote v. Acme was this country's longest running product liability suit. Though the plaintiff, a partially disabled male Canis latrans, meticulously documented his problems with Acme's mail order line of bird-trapping hardware in a long series of filmstrips and videos, his evidence was leaked to the public, exposing Coyote to generations of ridicule. Most recently the Ninth Circuit sent the case back to lower courts, ruling that Coyote had no standing to invoke the Americans With Disabilities Act, since the ADA applies only to humans.
Wow. Talk about reaching. I'm no tesla fan boy, but saying you can trick a radar system with RF equipment is like saying you can talk down a building with dynamite. Of course you can! In fact there exists RF test sets specifically for simulating radar targets for easier system validation.
People are susceptible to optical illusions. So are machines when you understand the assumptions made. People and machines don't have to make the same assumptions but each is fallible in it's own way. As they say nothing is perfect. No one and no thing is perfect. But is it good enough? Or which is better?
... by a painted tunnel.
That that is is that that that that is not is not.
You can also blind a human driver by shining a sufficiently bright light in the eyes. This fact has not caused anyone to say, "even though human drivers are quite capable, there's still no substitute for a good horse."
That that is is that that that that is not is not.
>"there's still no substitute for an attentive human driver, ready to take control at a moment's notice."
That would be utterly useless. Try "at a fraction of a second's notice". And that simply won't happen once one is in any type of autopilot mode.
> After all, Coyote v. Acme was this country's longest running product liability suit.
I don't see where he'd have standing to sue under the ADA in any case, since Wile. E. Coyote _won_ his lawsuit for manufacturing defects in 1990.
http://www.newyorker.com/magaz...
Yay, signal sensing equipment can be fooled by sending it fake signals!
Oh, and rain is wet.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
You can always DoS a human driver too. Laser pointers, or even perfectly natural feminine breasts could distract a human driver.
Patents Drive Free Software as Hurricanes Drive Construction Industry
News at eleven. Specialists using radio equipment is able to simulate objects on a radar, a technology also based on radio waves. Also, the world ends in about one hour.
I'm sure there are much better ways to "fool" it with much more low-tech items.
How does it distinguish the density of an object it detects? Does a paper bag blowing across the road get marked as something it can drive over or will it take evasive action? Where does it draw the line between running over the paper bag and steering into the car in the next lane to have what it might think is a "softer" crash? And how does it tell the difference between a bag and a child, or even a pigeon?
Isn't most of its autopilot visual rather than actual sensing? So how does it determine whether or not the bag is solid (a paper-coloured rock in the road), or not?
There are just too many scenarios for a blanket rule, and - as in the blasphemous I-Robot movie - the wrong decisions will be made.
Once you start including deliberate interference, these things just get dangerous.
As a driver would never notice their foot going straight to the floorboard when they had to apply the brakes to shift the car out of park.
one could argue that you could easily blind a human pilot with a floodlight, he will also not know if there was an object in front of him or not ..
Sure, hell, I can fool you into thinking a deck of cards changed color. have a look.
I hallucinate things on the road all the time when the sun is in my eyes. Peripheral vision, fast-moving scan (because I don't stare locked straight ahead), and dark spots caused by the sun being right-fucking-there will give enough data for my brain to imagine that smudge is that car skidding in front of me, and so render the car there, when in fact it hasn't moved. I have compensation systems to IGNORE THE EVIDENCE IN FRONT OF MY EYES because what's happening doesn't make logical sense.
Support my political activism on Patreon.
Actually to play devil's advocate here...a smaller cut would give you similar feeling on the brake pedal for light braking and the fail catastrophically (or at least not provide adequate breaking) under emergency braking...or when you finally emptied all the brake fluid.
Point being, there are malicious failure methods available that work just fine on plain old humans. Simple, cheap, easy ones. To say autopilot is less safe than a human drive because other, complicated and technical, things can fool it is misleading and disingenuous. Autopilot, especially in it's more advanced iterations that are coming, is highly likely to be a much, much safer driver than people in the vast majority of situations.
Will some circumstances fool autopilot and cause an accident? Yep. Could some of those have been avoided by a human driver? Yep. However, when you compare the number of times a computerized driver doesn't something wrong vs. a human I've got my money on the computer. They don't play pokemon go while driving, txt, get drunk, etc.
You can get rich if you own a politician, but you have to be rich to buy one in the first place.
A crack that allows you to control a large number of vehicles IS important and needs to be solved
OTOH, a hack in which you modify a single car to interfere with another single car's ability to drive is absolutely WORTHLESS. Why?
Assume that somebody is going to go after another person because they are rich or something. Then said person likely is driving themselves OR has a chauffer.
Let assume that this is used to go after a kid or a neighbor, etc. IOW, you just want to punish somebody. Now, you have to spend 100K to do so.
How many of you will do so? None.
This is your typical CHinese BS in which these ppl are trying to make themselves look good by making wild ass claims.
With that said, the real culprit on these was NOT Telsa, but Mobileye. And in this case, Tesla is RUNNING away from them. IOW, they know that they are too expensive for what they deliver.
I prefer the "u" in honour as it seems to be missing these days.
Split-second decisions are exactly what is often needed to save lives, as humans have demonstrated for decades now.
Demonstrated the ability to kill tens of thousands of people every year because of the human inability to consistently make split second decisions? I think is what you mean. We have a well known and well studied problem that is almost completely unsolvable without autonomous driving. Finding issues is fine, but there are a hundred people dying every day who's families can't wait for widespread adoption of autonomous driving.
This technology will save lives and we need it yesterday.
Yes, I'm well aware of the problem we're trying to solve here, but an obituary that reads "killed by autonomous bug #285A" isn't going to make any family member feel better vs one that reads "killed by a drunk driver". In fact, it's likely going to make them even more angry because their loved one that was sober, alert, and attentive behind the wheel was killed by their own damn car through no fault of their own.
Yes, the thousands of people dying every year is something that we do need to address yesterday. Are we there yet with autonomous tech? Not even close.
And for all the humans who might not make that split-second decision fast enough, I'm certain there are plenty of examples of humans who did, and saved lives.
Seem to be aimed at the current Tesla autopilot is the end all evolution as far as autonomous driving is concerned? I personally would believe that this is just a starting point . It is not the end. This stuff will get better and better as time goes on . That is why as a human driver you have to be ready to take over . If someone has a better idea please tell me where my thinking is wrong
Paywalled, but whatever pittance Coyote may have won in the lower courts after being rejected by the Ninth Circuit was Pyrrhic. When Sanyo Portable Hole K. K. of Akihabara, Tokyo, became Roadrunner's exclusive supplier of these highly effective countermeasures to Coyote's use of Acme's flawed devices, Coyote's medical bills from running into unexpected canyon walls made it impossible to continue work. He now resides in an assisted care home in Tusayan, AZ, and is permanently barred by the Parks Service from ever revisiting the adjacent Grand Canyon.
Driving people have blind spots, so illusion is not required.