Uber 'Neglected' Simulation Testing For Its Autonomous Vehicles, Says Report (engadget.com)

According to a report from The Information, Uber allegedly "neglected" simulation testing for its autonomous vehicles. "The publication's sources claim that there was a dearth of investment in the simulation software, and lots of incompatible code between the autonomous vehicle software and simulation software Uber is developing internally," reports Engadget. "However, the sources said there isn't a direct link between the lack of investment and the fatal accident involving one of Uber's autonomous taxis and a pedestrian." From the report: It's worth noting that the Unreal Engine-powered simulation software is still relatively new. The Information writes that the suite wasn't developed until after self-driving project lead Anthony Levandowski was fired mid-2017. To add insult to injury, initially, there were also differences in pay between simulation engineers and other engineers in the department. The end goal was to release a self-driving car in Arizona this year, codenamed "Roadrunner," to compete with Waymo's offering just outside of Phoenix.

Comments

Dearth (derp) vader

[nnet]

Uber non.

Re:Dearth (derp) vader

[sexconker]

Imagine being so salty about the election that you shit out this tripe on a daily basis.

Re:Dearth (derp) vader

[Anonymous Coward]

Yeah I type it every time Conker-wannabe, you're not only such a genius that you think that but your life is so good that you comment so! :D But at least you (probably?) won't die a traitor in Federal Prison like Dunny.

But it worked fine in Kerbal Space Program

[Jeremi]

It's worth noting that the Unreal Engine-powered simulation software was relatively new

Hmm, turns out the real world is different than the Unreal world. :/

Re:But it worked fine in Kerbal Space Program

[philmarcracken]

KSP engine is unity.

Re:But it worked fine in Kerbal Space Program

[Anonymous Coward]

You don't very often launch portable nuclear weapons upon your fellow road users in the real world, even though some clearly would require it for the Redemption.

Backseat Engineering

[Anonymous Coward]

Who is to say that they "neglected" simulation? I'm no Uber fanboy. Engineers working on the main show make more than simulation and test engineers at a great many companies. In addition, in my experience, simulation, especially of something complex and cutting edge like this (rather than say, FEA), is often of little value, because it is so hard to really model the world. Right or wrong, paying less to simulation engineers was not negligence. These wild conclusions seem to stem not from the facts, but from "The Information's" desire to create a sensational headline. Maybe Uber WAS negligent. This article does not convincingly make this case.

Re:Backseat Engineering

[Luthair]

You don't need to model the real world. You drive the car around with sensors on and record the input, then in simulation you feed the input from all the weird scenarios you recorded into your system.

The behaviour of Uber, Tesla and others is quite frankly reprehensible. The testing of these systems need should be at a higher bar than anything NASA or aeronautical firms given the widespread deployment, reduced control and much wider variety of possibility of harm.

Re: Backseat Engineering

[Anonymous Coward]

No, no, musk is our hero and you must worship him. He can do no wrong. Deliberately designing life safety critical code is not the muskian way, and is wrong.

I say that not knowing Teslas design processes, but having seen SpaceX's pencil whipped space rating artifacts.

Re:Backseat Engineering

[Anonymous Coward]

I'm GP, and I work in the auto industry. Admittedly, I don't know that much about simulation, but I know a bit about how rigorously the software that runs the vital control systems is tested. The answer is not very. Much less than a jumbo jet or a Nasa spacecraft. MUCH less. And, cars, for the most part, don't have many catastrophic software bugs. There is an opportunity cost of delaying automated vehicles until they are close to perfect (a large one). I am in favor of the wild, wild west style with which we're approaching automated vehicles. I think Uber is taking it too far, but Tesla is pretty close to being reasonable. You can't expect them to test cars like a boondoggle government spacecraft or you'll end up with a $100,000 Model 3, or, worse, we'll set automated driving back a decade or two, costing countless lives in the process.

Re:Backseat Engineering

[Pinky's Brain]

In the case of Tesla I also lay blame with the regulators. At least for Uber and Google the regulators thought they had professional test drivers (which in Uber's case wasn't true, they had a professional diagnostic screen watcher ... which did her job, which was not paying attention to the road as the car killed someone).

In Tesla's case we know the system as designed insures there will often be no one paying attention behind the wheel, yet the system as designed requires the driver to pay attention. Tesla's system is unsafe by design. Regulators should have never allowed this nor keep allowing it. It will continue to kill people and it's only by accident it keeps being the occupants of the Tesla instead of innocents.

Re: Backseat Engineering

[fluffernutter]

Also, how is the guy supposed to sell stuff? Selling stuff for profit is an American right!

Re:Backseat Engineering

[fluffernutter]

There is no assurance that they will ever get to a point where they will save a significant number of lives. Until we have hard evidence of that, I would prefer they stop creating their own deaths. In the case of Autopilot, people are lulled into a false sense of safety until they make a mistake that kills or injures themselves or others.

Re:Backseat Engineering

[HornWumpus]

Weird scenarios are very uncommon.

Unless you captured a _buttload_ of sensor data, you'd overtrain for the few you caught.

A major car manufacturer will need to capture years worth of 'entire production line' worth of autonomous car grade data. Then curate the accidents and near accidents into a trainable set.

Then your still left with an AI that can't infer intent, at best, immediate course.

Re:Backseat Engineering

[murdocj]

I don't know about this. I see weird stuff every day... people opening car doors into traffic, people wandering out into traffic, cars wandering into my lane. I'd have to think if you recorded a month's worth of data you'd have seen a LOT of stuff. Not to mention you could even generate your own weird scenarios. The fact that the Uber car didn't respond to a major obstacle in front of it seems like there wasn't a lot of testing of any form done.

Re:Backseat Engineering

[fluffernutter]

They have likely had an opportunity to record the types of occurrences you are talking about already, and it hasn't gotten them very far. I'm not pretending I know what percentage of possible world scenarios they need in order to be 'safe enough'. I just know that they need a lot more than they are getting, and that it is a problem that they don't seem to know what that level is.

Re:Backseat Engineering

[MobyDisk]

That approach works but is limited in several ways: You can only playback scenarios that have actually happened, and it doesn't work if the hardware & sensor arrangement is still changing. It is also slow since it is more of an integration test than a unit test. I've worked with this approach before and it is nice once everything is locked-down, but it is not great during the R&D phase.

Re:Backseat Engineering

[Anonymous Coward]

None of those examples are weird. Google's example of a person in a motorised wheel chair chasing ducks in circles on the road, now that's weird.

Re:Backseat Engineering

[Luthair]

If you change your sensor arrangement you start from scratch.

Re:Backseat Engineering

[Luthair]

There is an opportunity cost of delaying automated vehicles until they are close to perfect (a large one).

What exactly? Your short term stock options?

we'll set automated driving back a decade or two, costing countless lives in the process.

Ah, this specious argument again. Self-driving cars have yet to show this to be accurate, Waymo is apparently the best of the bunch yet requires human intervention on an average of 5000 miles - given that the average American drives 13k miles a year the best automation would crash a little under 3 times a year. While Tesla touts its autopilot notice how it never discloses how frequently it disengages, how often the driver must pro-actively prevent an accident, nor how often it does something dangerous without resulting in an accident.

Re:Backseat Engineering

[Dutch Gun]

Maybe Uber WAS negligent. This article does not convincingly make this case.

Perhaps. But we do have the evidence that their car mowed down a pedestrian it should have easily detected.

"A person walking in front of a moving car" isn't exactly some incredibly rare edge case that no one could have thought to test for.

Re:Backseat Engineering

[Dare nMc]

> The fact that the Uber car didn't respond to a major obstacle in front of it seems like there wasn't a lot of testing of any form done.

Just shows a lack of regression testing. I guarantee it stopped for people walking in front of it before placing the car into street testing. They changed object identification, almost certainly without repeating those tests. This is why you want simulated testing, when you change any software, you can re-run hundreds of hours of tests in minutes, without wasting any fuel or having tons of hardware. You then run a few important scenarios on real hardware before pushing out to the real world.

Uber clearly missed regression testing that with the death. Tesla clearly missed the real hardware regression testing when their software changes added 20 feet to their stopping distance.

Re:Backseat Engineering

[Anonymous Coward]

You drive the car around with sensors on and record the input, then in simulation you feed the input from all the weird scenarios you recorded into your system.

How would that work? Obviously the input is not going to include outcomes of various decisions, only the "before" data. Then what?

Re: Backseat Engineering

[Anonymous Coward]

a person in a motorised wheel chair chasing ducks in circles on the road, now that's weird.

Hey, I don't tell you how to live your life!

Re:Backseat Engineering

[Paradise Pete]

Weird scenarios are very uncommon.

Here's a good start...

Re: Backseat Engineering

[houghi]

Why should it be higher? I could understand as high, but why would ot be higher and that raises the question why it could be lower for NASA than for cars.

Re:Backseat Engineering

[thegarbz]

The testing of these systems need should be at a higher bar than anything NASA or aeronautical firms given the widespread deployment

Why? NASA considers loss of life unacceptable, yet we're quite happy to accept over 5 million accidents causing over 30000 fatalities every year from cars. Your demands are completely unreasonable in the face of the risk that has already been accepted by the population as well as the policy makers.

Re:Backseat Engineering

[Anonymous Coward]

Valley internet companies are the wrongest places on earth to be developing life-safety critical systems.

They are shielded from any consequences of bad design by the EULA you click when you sign up for their services, and their corporate culture values rapid deployment over everything, with users acting as alpha testers for all changes that make it through the regression test suite.

"Move fast and break things" is not the right culture for developing a guidance system for 1 ton killing machines.

Tesla is probably the only company with the right culture, since Musk makes cars and rockets which both have to work right, but he seems to overpromise and push too hard on the car side, plus he's got those saboteurs running around. Google doesn't seem to be that bad on quality despite being an internet software company, and they have a gaggle of top level AI people, so maybe they're the best ones for the job.

But Uber? I wouldn't trust them to build a bicycle. They should just wait for waymo and license the tech like everybody else.

Re: Backseat Engineering

[Anonymous Coward]

I work in the self driving car industry and safety or regulation is considered a priority if it does not interfere with short term deliverables. The goal is to make the demos look impressive enough so that VCs will want to invest hundreds of Millions. A lot of the AI is âenhancedâ(TM) by invisible remote human drivers, so that braking and turning looks smooth and smart. I definitely would be very worried if we started having our cars drive near playgrounds or schools, they just would not understand that a small kid is actually a human. The greed is strong in this business, and I donâ(TM)t think Iâ(TM)ll be part of it much longer. Iâ(TM)m kind of pissed off at CA DMV for not making the permits more expensive so that they can fund regular safety practices inspections to ensure that the companies are abiding by the letter and spirit of the regulation.
Iâ(TM)m really hoping that charges of involuntary manslaughter will be filed against the business decision makers at Uber but also any and all engineers that disabled the safety features without any form of protest.

Re: Backseat Engineering

[Anonymous Coward]

GP again. I invest primarily in index funds. I own no Tesla stock (unless through a fund).

Re: Backseat Engineering

[Anonymous Coward]

Higher, due to the potential to kill more people.

Re:Backseat Engineering

[Anonymous Coward]

> in simulation you feed the input from all the weird scenarios you recorded

And in simulation, you can run lots of variations of these weird scenarios, so that the system learns to handle the weirdness in a broader way. Or at least, that's how Waymo have been doing it.

Re:Backseat Engineering

[Anonymous Coward]

Beyond reprehensible. The company should be given the "death" penalty for this and even the engineers should serve some time. The responsible managers should serve more time, and the CEO should get 20-life. That would send a "chill" down the tech community that new and shiny when it deals with human life should be extensively tested, you know like the medical industry does. I know as a EE I never got my PE, but CE's tend to have to sign off on stuff that can cause death. Maybe if someone had to "sign off" and was culpable criminally when it failed it might light a fire to you know TEST.

Re:Backseat Engineering

[MobyDisk]

3/4 of the way through the project: "If we move the front-left imager 6 inches to the left and adjust the angle 5% up we should fix scenario thusandsuch..." That means you need all new test data, which in this case might require running over several old ladies a few times to get the test data you need. ;-)

Re:Backseat Engineering

[HornWumpus]

Uber _started_ setting up virtual world testing of its AI _after_ this accident. They are clowns.

Tesla appears to be a little better. But how do you debug an neural net 'AI'? All you can do it change the training dataset and test the results to death. Your test/training (hopefully different/built independently) set will be incomplete and you will kill more people, you're just hoping it will be less than would have died on their own.

So, the Microsoft ...

[CaptainDork]

... model.

Re:So, the Microsoft ...

[Anonymous Coward]

Microsoft models are little soft at all the right places.

beta testing doesn't belong on the road

[Anonymous Coward]

all these self-driving companies are literally beta testing this with people's lives

it's one thing to rush your software out, try to patch later and make up for hardware flaws with workarounds

it's another thing when painted lines and stationary objects are completely missed or purposely ignored by the software

I know everyone wants their fully automatic cars right now but they needs YEARS of independent testing and debugging before this code is allowed on the road

The word you're looking for

[rsilvergun]

is 'skipped', not 'neglected'.

So how did they train their AI?

[Anonymous Coward]

Huh. You would have imagined that modelling some roads, street furniture, other cars and pedestrians, then running massive training and validation datasets based on synthesised sensor readings, would have been how they initially trained their autonomous AI's, before moving out into the real world...

Did they do something different, somehow? Why?

What about the safety driver/face detection?

[Anonymous Coward]

I am glad Uber's involuntary manslaughter is getting fully investigated and all the complexities involved discussed. However, this is one much more simplistic question that I can never find the answer to:

Was Uber performing automated monitoring of the safety driver?

There has been discussion on the AI ignoring what the sensors detected to avoid over-reaction to false-positive events. It makes sense the AI would be sometimes tuned incorrectly provided there was a safety driver *paying attention* to address that situation.

This article points out the degree to which simulation software available today wasn't available before. Again, it makes sense to only use the resources currently available at the time if it is augmented by a safety driver that is *paying attention* to address gaps in the pre-road testing.

Even if unreal engine didn't have a simulation application available until recently, there has been face detection software available for a long time. I have a camera from a decade ago that does auto-focus based on face detection. The ability to monitor the face of the safety driver and automate triggering a response to the driver not performing the role they need to be has been available for a long time. What is Uber's excuse for the car never reacting to the safety driver looking down for extended periods of time while the Uber car performs termination of a person's life?

At the end of the day, if Uber is going to refuse to answer the what/how/why regarding the degree to which they had automated monitoring of the safety driver, then we need to respond accordingly. CEO Dara Khosrowshahi's response that it is "some incredibly sad news" is not good enough. So far after waiting nearly three months the information regarding Uber's method of making sure the safety driver was actually performing as a safety driver has been zilch. This is not "sad news" as much as it is *HORRIFYING* news. I personally do not trust Uber CEO Dara Khosrowshahi at all or the company he runs.

Uber has shown itself to be the Russian roulette of self driving car programs. We shouldn't be forced to ask ourselves how long will it be before we have another tweet from @uber_comms about their hearts going out to another Uber victim's family. Instead of extending their hearts they should get a CEO that makes holding safety drivers accountable via an automated system a top priority of field testing with a zero tolerance policy in performing field testing while the safety driver is not doing their job.

No XIL testing?

[0100010001010011]

How in the world did this make it to production without SIL/MIL/HIL testing?

dSpace sells HIL benches specifically to test ADAS

Re: No XIL testing?

[Anonymous Coward]

Came here to say exactly the same thing. How the hell was uberâ(TM)s autopilot software allowed on the road without SIL/HIL testing?

catch-22

[fluffernutter]

I sense there is a catch-22 here. If they were able to write a simulator capable of using some sort of chaos theory to produce any possible situation that may occur in he real world, wouldn't they know how to write an AI engine that could anticipate any possible situation in a self driving car?

The new way

[Anonymous Coward]

It's the new way of software development.
Fail fast, fail often.
and
I don't want it perfect, I want it Wednesday.

Sequeezing toothpaste back into the tube.

[aberglas]

The hardest part is to take input data and understand what it means. Games engines actually do a pretty good job of creating that data. It is not substitute for real testing, but it is an essential part. Because you can test much more. And because you can test crash scenarios that would be too dangerous otherwise.

If generating an image from a model is like squeezing toothpaste out of a tube, understanding the generated image is like squeezing it back in.

And a good simulator will introduce some noise etc.

Just like Facebook

[cleavet]

I think Uber has adopted the "Move fast and break things" development model...