Driver Killed In a Tesla Crash Using Autopilot Ignored At Least 7 Safety Warnings (usatoday.com)

An anonymous reader quotes a report from USA Today: U.S. investigators said a driver who was killed while using Tesla's partially self-driving car ignored repeated warnings to put his hands on the wheel. In a 538-page report providing new details of the May 2016 crash that killed Ohio resident Joshua Brown in a highway crash in Florida, the National Transportation Safety Board described the scene of the grisly incident and the minutes leading up to it. The agency, which opened an investigation to explore the possibility that Tesla's Autopilot system was faulty, said it had drawn "no conclusions about how or why the crash occurred." The NTSB report appears to deliver no conflicting information. The agency said the driver was traveling at 74 miles per hour, above the 65 mph limit on the road, when he collided with the truck. The driver used the vehicle's self-driving system for 37.5 minutes of the 41 minutes of his trip, according to NTSB. During the time the self-driving system was activated, he had his hands on the wheel for a total of only about half a minute, investigators concluded. NTSB said the driver received seven visual warnings on the instrument panel, which blared "Hold Steering Wheel," followed by six audible warnings.

Comments

Simple question

[Anonymous Coward]

Why would the car continue to operate for 37.5 minutes of the trip if the driver didn't have his hands on the steering wheel? If that's a requirement, why didn't the car just pull over and shut off? It seems like Tesla failed to implement some common sense safety protocols here.

Re:Simple question

[hawguy]

Why would the car continue to operate for 37.5 minutes of the trip if the driver didn't have his hands on the steering wheel? If that's a requirement, why didn't the car just pull over and shut off? It seems like Tesla failed to implement some common sense safety protocols here.

Because they trusted that the owner of an $80,000 car had at least some minimal intelligence and even if the driver had blind trust in the car, that when the car says "put your hands on the wheel and pay attention", that the driver would listen.

Yet this driver has demonstrated that people are about as dumb as you think they can be, so now they've implemented a 3 strikes policy that disabled autopilot after 3 reminders.

Re:Simple question

[wonkey_monkey]

If that's a requirement, why didn't the car just pull over and shut off?

Because the car isn't smart enough to do that. It can keep you between the lines on the road; it can't take you out of the lanes and park you up. That's actually a harder thing to do.

Re:Simple question

[110010001000]

No, the point is should the car have stopped, or kept driving after seven warnings and 37 minutes? It is supposed to be a smart car.

Re:Simple question

[quantaman]

Why would the car continue to operate for 37.5 minutes of the trip if the driver didn't have his hands on the steering wheel? If that's a requirement, why didn't the car just pull over and shut off? It seems like Tesla failed to implement some common sense safety protocols here.

Because they trusted that the owner of an $80,000 car had at least some minimal intelligence and even if the driver had blind trust in the car, that when the car says "put your hands on the wheel and pay attention", that the driver would listen.

Yet this driver has demonstrated that people are about as dumb as you think they can be, so now they've implemented a 3 strikes policy that disabled autopilot after 3 reminders.

For the first few days people will be extremely cautious letting the autopilot do anything.

For the first few weeks they'll give it more leeway, but be very attuned to any warnings it gives.

After a few months, if they haven't had any real scares, they'll assume the auto-pilot knows what it's doing and generally ignore warnings.

Some people will be more cautious, but as a software developer this is exactly what I expect to happen with a significant portion of people. Everyone knows the right thing to do, we should backup our data rigorously, always use good unique password, follow the proper procedures, etc. But that's not how people work. If it's not part of a routine, and it's not given an immediate payoff, then people won't do it.

Give people a car that can self-drive in some situations and they will inevitably let it self-drive in every situation they can.

Re:Simple question

[OrangeTide]

Because they trusted that the owner of an $80,000 car had at least some minimal intelligence and even if the driver had blind trust in the car, that when the car says "put your hands on the wheel and pay attention", that the driver would listen.

What if I go unconscious in my $80,000 car? Perhaps I have a heart attack, stroke, or simply had an unexplained fainting spell. So the car just beeps at me uselessly until it drives into something? Might as well use a traditional car, set it to cruise at 75 mph and take a nap because the end result will be the same.

Re:Simple question

[The Evil Atheist]

Driving while tired is also a decision. If you make that decision, you are also dumb.

Re:Simple question

[The Evil Atheist]

So why is it Tesla's fault it can't handle something no other car can handle?

Re:Simple question

[Jhon]

How do they know he was watching it? It might have been PLAYING... but how do they know he wasn't asleep?

Re:Simple question

[Jhon]

"Driving while tired is also a decision. If you make that decision, you are also dumb."

Tired or exhausted? Tired with autopilot might end up being as bad as driving while exhausted to the point of falling asleep behind the wheel without autopilot.

Re:Simple question

[ArchieBunker]

So you want the car to come to a dead stop on a highway? Oh yeah nothing bad will ever happen.

Re:Simple question

[sexconker]

Because it named its feature "Autopilot", which means it's supposed to be able to pilot itself.

Cue the idiots who will come in and say airline autopilot is incredibly limited and that people should similarly expect Tesla's Autopilot features to be extremely limited as well. The term autopilot has a meaning. It doesn't matter if you think you should apply airline autopilot features (which are much, much more involved than what Tesla offers) to what people should expect from a feature named "Autopilot". What matters is what a reasonable person expects a feature with that name and associated advertising to be able to handle.

Of course, the fact is it's not much more than cruise control, lane assist, and collision prevention that's increasingly present, as standard, on cars made for plebs.

Re:Simple question

[ElizabethGreene]

Tesla updated the Autopilot feature to have a five-mile-over-the-limit maximum after this crash.

The US is not like Europe vis-a-vis speed limits. Driving at or under the speed limit here is an uncommon and sometimes dangerous behavior. That's really stupid, but that's the way it has been ever since they set the interstate speed limits nationally at 55 mph.

(Yes, I know that speed limits are higher than 55mph now, but that's when it started.)

Re:Simple question

[war4peace]

Car price is irrelevant.
You can go unconscious in a Bugatti Veyron - which would immediately turn you dead.
At least, with a Tesla, you have a much higher chance of actually arriving at your destination than if going unconscious in another car.

Re:Simple question

[mark-t]

I would say so, yes.. the driver could be incapacitated for some reason... What if a driver had a heart attack and is unconscious, for example? If the driver ignores repeated warnings to put their hands on the wheel, the most sensible thing to do is to pull the car over as soon as it is prudent to do so, and call 911 once the car is stopped, with a declaration that the driver of the vehicle appears to be incapacitated and unresponsive to audio or visual cues to control the vehicle..

Re:Simple question

[mark-t]

No... it should pull over and stop as soon as it is possible to safely do so. What if there were a medical emergency in the vehicle that is causing the driver to be unresponsive?

Of course, if you expect drivers that are able to maintain consciousness while having a heart attack to just keep driving instead of pulling over and coming to a stop as soon as they can, then I'm not sure what world you are living in.

Re: Simple question

[mark-t]

I don't believe that he was in this case, but I think that you raise a valid point in general... the proper thing for an autopilot to do when a driver has repeatedly ignored warnings to assume control of the vehicle is to pull over and stop as soon as it is possible to safely do so, and probably even call 911.

Re:Simple question

[Mr D from 63]

The driver was watching a Harry Potter movie when he crashed, according to earlier published reports.

Turns out that was just rumor. The investigation concluded there was no movie being watched.

Re:Simple question

[Mr D from 63]

For the first few days people will be extremely cautious letting the autopilot do anything.

For the first few weeks they'll give it more leeway, but be very attuned to any warnings it gives.

After a few months, if they haven't had any real scares, they'll assume the auto-pilot knows what it's doing and generally ignore warnings.

Some people will be more cautious, but as a software developer this is exactly what I expect to happen with a significant portion of people. Everyone knows the right thing to do, we should backup our data rigorously, always use good unique password, follow the proper procedures, etc. But that's not how people work. If it's not part of a routine, and it's not given an immediate payoff, then people won't do it.

Give people a car that can self-drive in some situations and they will inevitably let it self-drive in every situation they can.

Its human nature. Bad driving habits reinforced over time. A certain percentage of people will grow unsafely confident and increase their risk. Just like texting while driving .

Re:Simple question

[HiThere]

Well, one doesn't need to have anticipated this precise incident to have anticipated it. Preventing it, however, is a different matter. The classic way to say this it "You can't make something idiot-proof, because idiots are so ingenious.".

Re:Simple question

[Strider-]

Why would the car continue to operate for 37.5 minutes of the trip if the driver didn't have his hands on the steering wheel? If that's a requirement, why didn't the car just pull over and shut off? It seems like Tesla failed to implement some common sense safety protocols here.

The real problem is that Tesla's "Autopilot" is the worst possible solution to the problem. One of the realities of the human brain is that shifting your attention is hard... To put it in computational terms, the context switch is expensive. Even when the car is doing the driving, it theoretically requires the driver to be auditing it, and be paying attention to their surroundings in case things go sideways. Concentrating on something that you're not actively participating in can be quite difficult, as we already know. If the driver is actually, well, driving, they are already hopefully paying attention to the road and their surroundings.

Until we get self-driving cars that can run without even having a steering wheel, it is a bad idea to have the computer half in control. The better solution is to work the other way. The driver does most of the driving, and the computer only takes over in emergency/collision type situations (avoiding that box on the highway, the idiot merging into your blind spot, etc...)

Re:Simple question

[Anonymous Coward]

...what's your point?

All of the above are stupid.

If you are too tired to drive safely, you should not be driving, with or without fucking "autopilot".

Re: Simple question

[Type44Q]

Do they involve, putting commas in incorrect places?

Re:Simple question

[BronsCon]

And when you engage autopilot in the left lane of a 5 lane freeway? What, it's supposed to signal and get over 4 lanes before pulling to the side of the road (where the shoulder may be closed or nonexistent alongside a concrete wall or rock embankment and stopping may be forbidden by law) and stopping?

Re:Simple question

[jeffmeden]

Other cars (like the Honda lane assist/adaptive cruise) warn you and then FORCE you to not be a fucking idiot, by turning the system off before you get the stupid idea to completely rely on it. All that this Tesla system does differently than that is... give you a nice thick layer of overconfidence until the moment that the system completely fails you and careens you into a semi. Tesla deserves every ounce of bad press from this, they designed a system that failed in just the right way as to carry a man to his death. Plenty of other car makers saw the problem with this and thats why they "lag behind" Tesla. Tesla got way too fucking close to the sun and their customer got burned. They need to pay the repercussions.

Re:Simple question

[0100010001010011]

Because it named its feature "Autopilot", which means it's supposed to be able to pilot itself.

I'll make you a wager. You get on any airplane and turn on auto pilot and walk away. Come talk to me after the plane lands itself when it's low on fuel and collect a million dollars.

Re:Simple question

[Theaetetus]

If that's a requirement, why didn't the car just pull over and shut off?

Because the car isn't smart enough to do that. It can keep you between the lines on the road; it can't take you out of the lanes and park you up. That's actually a harder thing to do.

Why didn't the car put its hazards on and let go of the accelerator until it slowed to a crawl while staying between the lines on the road? Yes, there's a possibility he could get rear-ended by some yahoo behind him who wasn't paying attention, but that's better than plowing into a truck in front of him.

Basically, the failure mode for autopilot doesn't have to be "pull over and park safely".

Re:Simple question

[BronsCon]

It's not autonomous driving, it doesn't navigate, it's glorified lane assist with the ability to maintain speed with the flow of traffic and a bit of accident avoidance logic mixed in. Far from autonomous.

Other than that, we're on the same page.

Re:Simple question

[Ol Olsoc]

So rich people don't have medical issues while driving that might prevent them from responding to verbal cues? This is the dumbest thing I've read all day. Do wealthy people who have heart attacks in a non-Tesla car just slow down and stop. No they don't. I've seen it twice now. Two guys - Heart attack - and they both went careening off the road. One pinball off the barriers, and the other went about 50 feet up a hill until he hit a tree and rolled over.

Re:Simple question

[Anonymous Coward]

If you engage any sort of cruise control in the far left lane of a freeway, you probably intend to camp out in the lane and obstruct traffic so your car should detect this and helpfully explode.

Re:Simple question

[lgw]

For all your skepticism, Volvo has hundreds of cars like this on the road, accumulating test mileage. They plan to launch for the public in 2020. Their cars very much will pull over safely if something seems wrong, or they won't sell them.

Tesla has glorified cruise control. They're just now selling cars with the right sensors to be self-driving, with the software to follow Real Soon Now(tm), probably after Volvo IMO.

Re: Simple question

[lgw]

We are quite close. Just ignore the Tesla hype and look to serious companies that will make their big marketing splash after they have working product. Volvo plans self-driving cars (of the sort the pull over safely when something's wrong) by 2020. Be interesting to see if they make it, but they're gathering test mileage already.

Re:Simple question

[religionofpeas]

If it's "Far from autonomous", they shouldn't be calling it autopilot.

Seems like an appropriate name, actually. A plane on autopilot also just keeps course and altitude, but doesn't fly around storms, or start an automatic landing procedure when fuel is running low.

Re: Simple question

[TheFakeTimCook]

...not to mention using incorrect verb-tenses.

Two Things

[Mr D from 63]

What we know from this incident;

1) The driver was responsible for the accident because he didn't maintain control
2) Tesla Autopilot was not good enough on its own to prevent the car from driving into the truck.

Re: Two Things

[guruevi]

There are various assumptions to make before pulling over the side of the road which may or may not be safer than simply having the car continue. You have to make sure there is an unobstructed emergency strip and you're not just careening the vehicle down a cliff. If your car makes the decision to go on a shoulder and something happens (or the shoulder doesn't exist), at that point the liability shifts because you've gone from passive "cruise control with intelligent lane following" to active intervention in a situation.

Re:Two Things

[SuperKendall]

* Why didn't the car slow down if you have your hands off the wheel for more then 5+ minutes?

Because that would kill more people that what it does now.

Why didn't the car's sensor detect the impending crash?

Because white truck, and completely across the road is something the system probably thought of as something like fog or clouds. There are not supposed to be big white boxes totally across the road... which is also why it warns you after to long to at least hold the wheel.

Why didn't the car pull over the side of the road after 15 minutes of hands free driving?

Because judging where it is safe to pull off the road is actually a much harder problem than driving in a lane.

Basically though I'm fine with the system as it is, this is no different that someone crashing a car because they have a really powerful engine they did not use responsibly either. You can only do so much to protect an idiot.

Re:Two Things

[wonkey_monkey]

* Why didn't the car slow down if you have your hands off the wheel for more then 5+ minutes?

Slow down how much? And if nothing changes, then what? Come to a stop in the middle of the road?

* Why didn't the car pull over the side of the road after 15 minutes of hands free driving?

Because that's a much harder thing to do than simply keeping a car between standardised lane markings and away from relatively slow-moving traffic (not that it fully succeeded in the latter in this case).

Re:Two Things

[blindseer]

I wish I could remember enough details to do a proper search for a proper citation but I recall seeing on TV something about an airplane crash because the pilots ignored one too many warnings. The story goes something like this....

A couple of pilots were flying a commercial jet with only one other crew member aboard. I'm not sure why the plane was empty but it gave the pilots some freedom to pull a stunt for bragging rights. This happened in the 1960s I believe, it was fairly early in the history of commercial flight anyway. The stunt they wanted to pull was hit some altitude that few pilots could achieve at the time and with no passengers on board it made the plane light enough to do so, at least in theory. Also to do this they had to push the engines a bit since flight time and fuel were limited.

As I recall the first warning was pretty minor, something like safe climb rate being exceeded. That alarm was disabled and they continued. Then they got an engine overheat warning, that was also disabled. Then the second engine started to overheat, again the warning was disabled. The next warning, again as I recall, was something they definitely should not have ignored but they did. Not having yet reached the altitude they set out to they had pushed the engines to the point of failure. Both engines died, they were unable to restart either, and a plane that was perfectly functional when it took off was now a twisted wreck. All three on board died.

Why didn't the plane force the throttles back? Why weren't the warnings more "stern"? Why didn't the plane detect the impending engine failure and descend to a safe altitude and speed? I don't know if current aircraft would or could do such a thing. The failure was in the pilots' desire to pull a stunt and they paid the price for it.

The Tesla driver pushed the limits like those hotdog pilots and suffered the same fate. I put no blame on Tesla, the fault is solely on the driver. The idiot driver should take all the blame here, if only because he exceeded the posted speed limit.

If someone knows the which plant crash I referred to and could tell me what it was I'd appreciate letting me know.

Re: Two Things

[blindseer]

https://en.wikipedia.org/wiki/...

That sounds about right, thanks. I was off by 4 decades, 1 crew member, and I forgot that the fourth warning the pilots overrode was a stall speed alarm.

Re:Two Things

[Trogre]

Slow down how much? And if nothing changes, then what? Come to a stop in the middle of the road?

Of course, yes.

If you ever find yourself in a situation where you are no longer able to safely drive your vehicle for any reason your first course of action is to stop it from moving, and of course activate your hazard lights.

If the AI was not able to pull over then the correct action would be for it to just stop.

Re: Two Things

[ckatko]

Yes! How are slashdotters missing this?

There's a huge difference between a TOOL ("stay in the the lanes" ala essentially a next-gen cruise control), and an autonomous machine capable of making DECISIONS that override the driver.

A Tesla is NOT a self-driving car. It's not autonomous. It's a CAR that has a next-gen cruise control. That's it.

There's no difference between falling asleep with the cruise control on in your Lexus and eventually driving off the road, and falling asleep with the Telsa next-gen cruise control and eventually driving off the road. The DRIVER is supposed to be watching and in control at all times. The TOOL is simply taking care of the redundant activities of pushing the pedal to maintain speed and adjusting the wheel to maintain direction. The tool is not a navigator. The tool is not a brain legally permitted to drive on the US highway system. The tool is a freakin' tool! The real brain (the driver) is supposed to watch the tools and ensure they act accordingly.

Insufficient warnings

[dlleigh]

Of course he was going to ignore a warning that said, "Hold steering wheel."

Instead, the car should have said:

"What the hell are you doing with your hands off the wheel, you idiot???! Are you trying to crash? Do you want to die? Do you want to make your kids orphans?"

The warnings could get increasingly forceful as the car complains that its own safety is being jeopardized.

"I don't want to go to a body shop. They use hammers! Kill yourself if you want, but leave me out of it."

The accident was therefore Tesla's fault.

Re:but... /s

[KGIII]

Nothing. Not a damned thing. Someone this stupid was going to take themselves out of the gene pool, sooner or later. I'm sometime baffled that we've managed to keep ourselves from going extinct.

Re:but... /s

[Guspaz]

- Semi trailers could have been mandated to have crash bars under their trailer such that a vehicle will react more like a head-on impact instead of having its top sliced off
- Auto pilot could have bene improved to be able to handle this scenario (this was implemented by Tesla after the crash)
- The driver could have actually paid attention and maintained control of the vehicle
- The vehicle could have been more aggressive in refusing to continue on autopilot without a driver response, such as slowly decelerating to a stop if the warnings are ignored (this has been improved since the crash, but nowhere near this aggressively)

Ignored 1 warning repeated, not 7 different ones

[gurps_npc]

With current technology, people are trained to ignore meaningless errors.

Proper error handling for anything important require you to take action, especially if you repeat the error.

That is, if they want people to pay attention to a "keep hands on wheels" warning, the speed should drop significantly. Not as if the brake was applied, but instead as if the foot was taken off the gas (even if they tried to floor it.). Oh, and the brake light should flash to let people behind know you are slowing, even though no brake is applied.

partially self-driving cars shouldn't exist.

[s1d3track3D]

Tesla's partially self-driving car

A partial self driving car is like sorta being pregnant, the car is either self driving or not, any grey area = not.

This seems like Tesla getting the public to do QA for them untill they have a fully self driving car, it's clear the public does not know what "partially" means...

Re:partially self-driving cars shouldn't exist.

[chrisautrey]

the car is either self driving or not

Tesla did not do themselves any favors with the name. 'Enhanced cruise control' or something similar as a name would have gone a long way.

Re:partially self-driving cars shouldn't exist.

[Hentes]

I take it you prefer pump braking to ABS, then. Automation is not magic, completely self-driving vehicles are not going to appear from nothing in a day. Instead, manufacturers keep automating the mundane parts of driving, letting the human focus on the parts that actually need his attention.

Re:partially self-driving cars shouldn't exist.

[Tom]

I've driven a bunch of cars with adaptive cruise control, automatic lane keeping and such things. Top-of-the line cars, current models, like the new BMW 5.

None of them would be able to drive anywhere near the 30+ minutes of the Tesla by themselves. What Tesla has there is much more than enhanced cruise control.

Re:Were the sensors working?

[chrisautrey]

you'd still be at fault because one of your hands wasn't on the wheel.

So, by your argument, if I was ever in an accident it would automatically be my fault because I drive a stick shift that requires you to drive with one hand regularly?

While many safety organizations agree that 2 handed driving is the safest way, there is no law that says you have to.

Sensor problem

[DrYak]

If the car is not smart enough to see a truck by itself,

This may also not have been a problem of smart, but a problem of sensors not registering the truck in this peculiar situation.
(e.g.: unlike other cars, Teslas only have forward facing radars and 2D video camera, no LIDARs, nor 3D-pair of stereo cameras.)

it may not be smart enough to find a safe space to pull over and shut itself off.

It's definitely not as much a problem of smart as it is of sensors :
whereas some other cars feature read-facing video cameras under the side mirrors (that can see if there's a car in the next lane much further back),
Teslas are among the car that exclusively rely on their ultra-sound sonars (basically souped up parking assistance with a bit improved range) which are much shorter-range. They can reliably tell you when a car is in the next lane or if there's space right next to the Tesla, they can't see if there's a car coming from behind in the lane.

With its current set of sensors, Tesla definitely can't change lane without risking to cut someone of.
That's part of the reasons why the driver needs to signal his request by using the blinker (driver checks blind sport and behind on the lane, and only then pushes the blinker lever).

sensors

[DrYak]

- add a forward facing LIDAR and or 3d-pair of stereo camera (instead of 2d), to supplement the current radar and 2d camera.
This would help the car disambiguate between a truck just in front, and a highway sign much further in the same direction - both could look similar to the current set of sensors)

- add backward facing cameras under the rear-view mirror :
this would help the car change lane safely.
Means that it could cover the scenario "user doesn't touch the wheel for the past 30min despite alarms" => "user has probably passed out / is unconscious / etc." => "telsa should automatically pull to the side" => "means must be able to change lanes until side is reached"
(The car should also automatically call emergency services, and should blink and honk to attract attention)

Too soon

[duke_cheetah2003]

I think this AI assisted driving should be taken out of the cars. This technology is not ready for the real world. It's just too tempting to turn this gadget on and doze off or some other totally stupid behavior, thinking the car can deal with driving on it's own.

Having this incomplete technology in service could hamper efforts to convince government entities that self-driving cars can and will be safe, when an immature technology is turning out to be not so safe. And trust me, regulators are looking at this and saying to themselves, "If this can happen, this technology is not safe."

Trust me, I want a self-driving car like yesterday, but the technology needs to mature more, more testing, in more situations needs to be done before this is ready for the end-user who's going to take a nap while his/her car drives itself.

Tesla is definitely at fault

[140Mandak262Jamuna]

It is basic safety design. Implemented even in steam engines. Trains are a good analog of autopilot because the locomotive drivers don't have to steer. They only control the speed manually. The biggest safety issue in locomotives is boredom. Eyes glaze over looking at the endless track, they ignore signals, their reaction time becomes very slow. Deadman's Treadle was introduced to make sure the loco drivers tread it once in so many minutes, else the train stops. In modern diesels they constantly issue alert and if the driver does not react it stops the train. Japan's bullet trains have their own methods to keep driver alertness. They always point to whatever they should be looking at, and look at it and say it aloud.

Given the history of how to handle inattentive drivers on machines that require very infrequent action, they should have designed the auto pilot with random reaction testing alerts and challenges.

Re:That's the thing about the Tesla drivers don't

[hyades1]

You think only Tesla rats you out?

LOL

http://www.ctvnews.ca/sci-tech/driver-beware-your-new-car-may-be-spying-on-you-1.2296165

There's a link in the story to the actual PDF file of the report by British Columbia's Privacy Commissioner.