Uber's Self-Driving Car Saw Pedestrian 6 Seconds Before Fatal Strike, Says Report (tucson.com)

An anonymous reader quotes a report from Arizona Daily Star: The autonomous Uber SUV that struck and killed an Arizona pedestrian in March spotted the woman about six seconds before hitting her, but did not stop because the system used to automatically apply brakes in potentially dangerous situations had been disabled, according to federal investigators. In a preliminary report on the crash, the National Transportation Safety Board said Thursday that emergency braking is not enabled while Uber's cars are under computer control, "to reduce the potential for erratic vehicle behavior." Instead, Uber relies on a human backup driver to intervene. The system, however, is not designed to alert the driver. The report comes a day after Uber announced it will be ending it's self-driving vehicle testing in Arizona. The full NTSB report is available here.

Comments

Wait, what now?

[Narcocide]

This can't be right. They are saying that Uber's self-driving car rig is neither designed to stop for nor alert the driver about pedestrians obstructing the path of the vehicle. It's just designed to... log them?!

What part about this is considered "self-driving" then, exactly?

Re:Wait, what now?

[Anonymous Coward]

I really liked that comment from the last story: "Oops, we left it in murder mode."

Just sums up the whole situation so well.

Re:Wait, what now?

[bill_mcgonigle]

This can't be right. They are saying that Uber's self-driving car rig is neither designed to stop for nor alert the driver about pedestrians obstructing the path of the vehicle. It's just designed to... log them?!

Apparently the way they had it was that the computer would drive and the driver would stop it from driving, if needed. That doesn't seem like an obviously ridiculous arrangement, even if having the computer ping the driver would have been better.

Except the driver they hired to do the stopping thing was texting instead of watching the road, from what the video looked like. So nobody was on the "stopping job". Results as expected.

Re:Wait, what now?

[pnutjam]

Still waiting on the Uber apologists to show up and tell us how much safer these systems will be and how these accidents will only happen once ever. Like developers never reintroduce errors.

Re:Wait, what now?

[Anonymous Coward]

Except the driver they hired to do the stopping thing was texting instead of watching the road, from what the video looked like.

If you RTFA (I know, I know), you'll see that the backup driver said she was monitoring the self-driving interface, not fiddling with a phone.

Re:Wait, what now?

[MDMurphy]

A very poorly written article which resulted in a poorly written summary.

Buried in the article: "Uber also disabled the Volvo's factory-equipped automatic emergency braking system when the vehicle is in autonomous mode, the report said."

The Volvo XC90 comes with a feature they call "City Safety". https://www.media.volvocars.co...

This is an auto-braking system with sensors. Uber's autonomous system has its own braking and sensors. It's understandable from a system perspective that they don't have two separate, independent, systems deciding when to apply the brakes operational at the same time.

The poorly written article makes it sound like Uber's system either didn't have a feature for braking for obstacles or that it was disabled. This is not accurate. It does appear that Uber's system failed to either detect the pedestrian or to brake when detected.

It's probably also true that when testing they don't use the factory cruise control to maintain speed on the highway. There are likely other standard functions not used when the autonomous equipment is under test.

Re:Wait, what now?

[Kjella]

The poorly written article makes it sound like Uber's system either didn't have a feature for braking for obstacles or that it was disabled. This is not accurate.

No, you're the one spreading misinformation. To quote the NTSB report directly:

According to Uber emergency braking maneuvers are not enabled while the vehicle is under computer control to reduce the potential for erratic vehicle behavior. The vehicle operator is relied on to intervene and take action. The system is not designed to alert the operator.

Basically they had a system that calculated it would crash, but did nothing and warned no one. In other words, the same as having no system at all.

Re:Wait, what now?

[Rei]

What did you expect from a company like Uber, exactly? Their entire business model was built around the concept of "laws don't apply to us, because, internet." This is the same company that beyond running an unregistered taxi service (cheaply disguised as "ride sharing", as if people just happened to be going in the same direction), was tracking critical journalists with "God Mode" and hiring private investigators against them to find things to blackmail them with, ran an active campaign to track law enforcement officers in order to evade them, ran a campaign against competitors like Lyft involving the massive use of fake ride requests, and literally dozens and dozens more types of general scumbaggery.

Anyone shocked that this same company shut off all pedestrian safety controls in order to get their "self driving cars" on the market sooner? Bueller?

Re: Wait, what now?

[Anonymous Coward]

HAL9000 was human error, too.
So many people who watched that movie walked away from it with the wrong moral in their heads. It wasn't a tale of technology gone wrong, it was a tale about what happens when someone has no choice but to comply with all orders fully and completely, and you don't think things through.

So yeah, this is a HAL9000 scenario. The AI was not allowed to slow down, stop, change lanes, or notify the driver. There were no available actions it could take to avoid the collision, because Uber did not allow it any actions.

Re:Wait, what now?

[fluffernutter]

Stop being an apologist. A human driver still doesn't speed past them at full speed under the assumption they will not enter the road. They slow down in order to reduce the damage *in case* they walk into the road. We are told that CPUs can react much quicker than we can, 6 seconds was plenty to react somehow.

Re:Wait, what now?

[Green Mountain Bot]

Then again the pedestrian not wearing dark clothes or having reflectors on the bike while walking in front of a car at night would probably have helped more.

How would that have helped? The car detected her and didn't brake, and the "driver" wasn't looking and so couldn't brake.

Re:Wait, what now?

[vtcodger]

"They are saying that Uber's self-driving car rig is neither designed to stop for nor alert the driver about pedestrians obstructing the path of the vehicle."

What they seem to have done is taken a vehicle with substantial safety equipment, left the sensors operating, but turned off the accident avoidance features. Then they added a distracting task (monitor the system) for the driver. Then, SURPRISE!!!, something went lethally wrong.

We'll have to wait until folks with time and full information perform an analysis. But it sure looks at first glance like this was/is a questionably well designed test program.

Re:Wait, what now?

[slew]

Apparently the way they had it was that the computer would drive and the driver would stop it from driving, if needed. That doesn't seem like an obviously ridiculous arrangement, even if having the computer ping the driver would have been better.

I suspect that the alerting the driver wasn't gonna be any better. This is all speculation, but if Uber turned off the AEB (automatic emergency braking) system in self driving mode because it would have actuated the brakes too often making the driving erratic, simply notifing the driver about the same potential collisions would eventually result in alert fatigue in the driver.

Then the driver would start ignoring the alerts for basically the same reason that they turned off the AEB system originally.

If you've ever taught someone to drive who was overly cautious and braked all the time you know how that goes. The flaw is that simply seeing the pedestrian isn't enough to make the braking decision and the driver has to learn to anticipate the actions in the environment to drive successfully.

The driving instruction not only has to anticipate the environment, but also the driver to know when to intervene (grab steering wheel or apply the brake). Most self driving systems don't give these driving observers (not instructors) enough information and training to anticipate what the system is going to do to make them effective at intervening.

This is the problem of allowing people to walk before they can crawl... Sometimes you can't bypass stages of learning. If the system's AEB system activates too much, you kind of have to let it do that until it can learn not to do that.

Re:Wait, what now?

[careysub]

My new Prius nags me if it thinks my hands aren't on the wheel - I discovered while creeping along in a straight line in nearly stopped traffic with my hands resting lightly on the wheel. Apparently it senses torque inputs (even very slight ones) from the driver to determine whether someone is holding it, and in the very slow traffic the fact that I hadn't needed to apply a steering input for awhile was flagged as "not holding the wheel" by the software.

Why didn't the Uber car have some similar sort of mechanism to detect a non-responsive "driver"? You know, checking to see if he was doing the job he was paid to do? Businesses are usually on that like Trump on a second scoop.

Re:Wait, what now?

[uncqual]

http://dilbert.com/strip/2018-01-30

Re:Wait, what now?

[Jeremi]

Why didn't the Uber car have some similar sort of mechanism to detect a non-responsive "driver"?

Uber has always been a "let's just try something now and worry about the what-ifs later" kind of company. That they jumped into the self-driving car business without adequately thinking things through doesn't surprise me at all. I imagine we'll be witnessing a few more fiascos, if/when they make good on their plan to start flying people around in oversized drones.

Re: Wait, what now?

[c6gunner]

How would that have helped? The car detected her and didn't brake,

As he pointed out, they're talking about two different systems. It's hard to parse out, but I think what they're trying to say is that the built in factory system detected her but couldn't break because it was disabled, but Uber's own system didn't detect her.

If that's the case then yeah, maybe reflectors and such would have helped the Uber system detect her and stop. On the other hand if the factory system detected her and theirs didn't then it also means that their system is pretty shit.

Re:Wait, what now?

[Calydor]

Why do you assume that just because the car could notice something in the dark that human eyes could?

The whole point of self-driving cars is that they have better 'vision' than we do, that they (should) see things sooner and clearer.

At what point is it said that the unfortunate pedestrian was visible to the naked eye for six seconds rather than visible to radar, lidar, laser scan or whatever it is Uber cars use for area awareness?

Re:Wait, what now?

[houghi]

Self driving cars will be a lot safer. This was a self driving car as much as Uber is not a taxi company.

Re:$100 million?

[HornWumpus]

They paid them off within days.

But she was a crackhead who wasn't in touch with her family. They likely got off real cheap.

Time for Regulatory Control

[djbckr]

It's time for regulatory control - like how the FAA regulates the skies, the autonomous car industry needs to have sensible regulations. Right now it just seems like a bunch of cowboys in the wild west are trying to one-up each other.
Get this: The car "saw" the person 6 seconds before striking the person. The emergency system was disabled. The emergency system was not set up to alert the driver. So many things wrong here that would have been avoided had there been sensible regulations written by sensible engineers.

Re:Time for Regulatory Control

[h4ck7h3p14n37]

These vehicles are already regulated.

...it was Nevada that first adopted autonomous legislation back in 2011. Since then, a total of 30 states, plus the District of Colombia, have put some sort of self-driving regulation on the books. Of that group, 25 states—Alabama, Arkansas, California, Colorado, Connecticut, Florida, Georgia, Idaho, Illinois, Indiana, Louisiana, Maine, Michigan, Nevada, New York, North Carolina, North Dakota, Pennsylvania, South Carolina, Tennessee, Texas, Utah, Virginia, Vermont—and Washington, D.C. have adopted official legislation regarding the vehicles. The remaining 6 states—Arizona, Delaware, Massachusetts, Minnesota, Washington State, and Wisconsin—have dictated autonomous car guidelines through executive orders.

The State of State-by-State Autonomous Car Regulation

Re:Right of Way laws are political, not logical.

[mysidia]

most of whom are probably pedestrians on account of the cost of owning a car.

EVERYONE is a pedestrian. And we have a society with specified rule of law carving out certain precautions DRIVERS of dangerous vehicles must take to help pedestrians stay alive. One of those being the operator of a vehicle is legally responsible to account if they kill a pedestrian that they had 6 seconds warning about

Re: $100 million?

[saloomy]

A hero implies courage. She didn't volunteer to test it. More like a martyr if you ask me.

Re: $100 million?

[HornWumpus]

We're _all_ test rats.

Re: $100 million?

[Jarwulf]

A martyr implies actively defending something either through sacrifice through active defense or acceptance of punishment without backing down for a cause.

Uber Solves Los Angeles Homeless problem

[Zorro]

The New Uber Terminator Car will eliminate any unwanted pedestrians.

Ask us how!

Split attention

[steveha]

There was a human in the Uber car. Theoretically she was there to provide a human element that backed up the self-driving. But she was not giving her full attention to safety.

[The Uber safety driver] told investigators she had been "monitoring the self-driving system interface," which is displayed on an iPad mounted on the vehicle's center console.

  https://www.theverge.com/2018/5/24/17388696/uber-self-driving-crash-ntsb-report

I think Uber should change their procedure. Have two humans in the car: one to provide safety backup to the driving, and one to manage the iPad app. That will cost more than just having one human in the car, but would have saved a life in this case.

Note that the NTSB says the pedestrian was crossing unsafely, which contributed to the incident. Again quoting from the above-linked article in The Verge:

The report frames [the pedestrian's] actions in the moments before the crash in a fairly negative light. Investigators note she was crossing the street outside the crosswalk, wearing dark clothing, and, according to a post-crash toxicology report, had methamphetamine and marijuana in her system. The NTSB also notes that the median on Mill Avenue where [the pedestrian] was crossing the street was not illuminated by lighting and featured signage warning pedestrians not to cross there.

I don't see what the drugs in her system had to do with anything; I think this would have happened the same way even if she was completely sober. But she was doing an unsafe thing when she died.

Re:Split attention

[im_thatoneguy]

If she was sober she might have looked both ways before crossing a street or noted that an oncoming vehicle wasn't slowing to avoid her. If I was crossing the street at night sober and a car wasn't stopping for me I would stop and let it pass and then proceed once it was safe.

Re:Split attention

[djinn6]

She did not cross at a cross walk, nor was she on a bike path.

To quote the NTSB report itself:

In this area, northbound Mill Avenue is separated from southbound Mill Avenue by a center median containing trees, shrubs, and brick landscaping in the shape of an X. Four signs at the edges of the brick median, facing toward the roadway, warn pedestrians to use the crosswalk. The nearest crosswalk is at the intersection of Mill Avenue and Curry Road, about 360 feet north of where the crash occurred.

The accident could've been easily avoided, if either she or the driver been paying any attention at all. But really, what do you expect when a phone-addicted ex-con encounters a homeless meth-addict while "driving" a vehicle with experimental software from a "ride-sharing" company?

Car Development

[Only Time Will Tell]

It goes to show that development of the next generation of self-driving or autonomous cars has to be undertaken by companies heavily driven by pedestrian and passenger safety. It seems like an engineering failure was partly to blame here when the car was aware of the pedestrian, but neither alerted the driver nor attempted to stop or redirect its path. A tech company masquerading as a car company strikes me as the exact conditions that could lead to this failure.

No, you are wrong.

[thesupraman]

And I suspect intentionally so.

It has been clearly stated, BY UBER that they both disabled the Volvo emergency braking AND their own emergency stop/avoidance code.
To imply it was just the Volvo system is trying to avoid the fact that Uber had intentionally removed their own code that would make emergency stops.

They had intentionally made a system that had NO automated method of making an emergency stop, and no method of warning the secondary driver that one was needed (which would be stupid anyway,. they are supposed to be testing a system for future use without such a driver).

They should be hammered for this - it would be the equivalent of removing the emergency brakes from a buildings lifts, or the airbags from a car, without telling anyone.

Uber skipped Failure Mode Effects Anslysis

[DanDD]

A bit of system engineering discipline, FMEA , might have prevented this. It helps when innovators are trained to manage complexity, capability, and risk.

Or common sense, but such things can be on short supply on large projects run by program managers who have an 'MBA' mentality: https://www.inc.com/nathan-fur...

None

[rsilvergun]

they originally had 2 people driving the car, one to watch the road and one to watch the numbers and make notes about the car's performance. As a cost cutting measure they dropped it down to 1 person doing both jobs. The woman behind the wheel was busy making notes on the screen when she should have been watching the road.

Uber wants data and they don't care how they get it. You can't get data on risky events if you're too cautious. kinda like how they used to vivisect criminals.

On the positive side...

[hyades1]

At least when the machines are actively trying to kill us, we can be fairly sure that once in a while they'll screw up and leave their victim alive.

Re:Breathtakingly irresponsible

[gweihir]

It is probably a case where a lot of pressure to perform was applied to the engineers by some clueless MBA bean-counter and, in addition, nobody apparently had the experience and training to realize how woefully inadequate a "safety driver" is in such a set-up actually is.

Not According to the FA

[rsilvergun]

According to the FA she wasn't texting, she was minding the screen in the car, per Uber's instructions, to take notes on interesting events for later research. Uber used to have two people, one to watch the road and one to take the notes and they did away with the second one to cut costs.

Re:Not According to the FA

[djinn6]

There's no evidence besides her own report of the situation. Look at how her facial expressions changes in the video. It's initially neutral, then a smirk, then neutral as she looks at the road, then amused as she looks down, and then finally horrified as she looks up for the last time.

She was, without doubt, looking at something much more interesting than a list of vehicle status updates.

Re: $100 million?

[c6gunner]

She has the legal right to walk anywhere, against any light, under any conditions, and even leap out in front of a car in traffic.

ItÃ(TM)s the law.

Kellogg's called and they want you to return your law degree. It was a mixup; you were supposed to receive a decoder ring.

Re: $100 million?

[c6gunner]

So we are agreed that she was neither a hero nor a martyr, but rather a random nobody high on weed and meth, randomly walking into traffic?

GDPR

[MrL0G1C]

"

Weâ(TM)re sorry. This site is temporarily unavailable.

We recognise you are attempting to access this website from a country belonging to the European Economic Area (EEA) including the EU which enforces the General Data Protection Regulation (GDPR) and therefore cannot grant you access at this time.

For any issues, contact us. gdpr@townnews.com

(403 error.)

"

Re:6?!!

[ebvwfbw]

Plenty of time for HER to not be in the way on a highway.