Slashdot Mirror
Google Wants To Test Driverless Cars In a Simulation
An anonymous reader writes Google has been testing its autonomous vehicles on U.S. roads for a while now. In fact, they're required to, by law. "California's regulations stipulate autonomous vehicles must be tested under "controlled conditions" that mimic real-world driving as closely as possible. Usually, that has meant a private test track or temporarily closed public road." It's easy enough to test a few prototypes, but whenever autonomous cars start being produced by manufacturers, it'll become a lot more complicated. Now, Google is lobbying to change that law to allow testing via computer simulation. Safety director Ron Medford said, "Computer simulations are actually more valuable, as they allow manufacturers to test their software under far more conditions and stresses than could possibly be achieved on a test track." Google spokeswoman Katelin Jabbari said, "In a few hours, we can test thousands upon thousands of scenarios which in terms of driving all over again might take decades." Shee adds that simulator data can also easily provide information on how human behavior creeps into driving. "It's not just about the physics of avoiding a crash. It's also about the emotional expectation of passengers and other drivers." For example, when one of Google's computer-controlled cars is cut off, the software brakes harder than it needs to, because this makes the passengers feel safer. Critics say relying heavily on simulation data is flawed because it doesn't take into account how other cars react to the computer's driving.
until it's your life.
Test in the fscking simulation and then test on the street. Win-win.
the physics of a simulation aren't going to match-up to the real world quite right.
The problem with simulator testing is that you can't test scenarios that you didn't think of. This is particularly important to find problems arising from multiple simultaneous situations. For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.
Real life is far more creative than any scenario designer.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
As head of the robotics program at MIT Dr. Rodney Brooks famously said, "Robot simulations are doomed to succeed."
Require that all autonomous vehicles deliver themselves to your driveway. From the factory. If they make it there, they are deemed safe for use.
Why is it so hard for the test track to be outside the place where they're manufactured? When a car is done it would drive through a "course" by itself, since it is autonomous... or at least they want us to be believe it is. Can see the job description now: Driver required for autonomous vehicle test course, must be able to maintain the stability of the vehicle with automated cars on the road that may occasionally divert from their intended path. Comes with great benefits.
but seriously, Tests could be simulated with the same kind of course a drivers ed student supposedly goes through, and there's no reason these things shouldn't be able to drive themselves to an offsite storage facility, even if it is controlled roads or roads with caution signs. Simulations don't account for faults in the design or manufacturing.
I can grasp how they might program a car to drive between lines, avoid other cars, and obey traffic lights. I've always been curious how well they do with ice, emergency vehicles approaching, and drunk drivers. I'm guessing better than us, but I would still love to see it...
may help test software but not the full hardware and how well it works in all kinds of weather / settings. Also what about road conditions / slight lines? odd traffic light layouts / intersections? Just useing google street view as the input likely will not get the full lay out from each lane / all times of day / all cycles.
... lack of randomness.
Will they simulate a 3 year old tossing a sandwich out the window into oncoming traffic?
It little behooves the best of us to comment on the rest of us.
Are we really having a public, political, emotional discussion about the relative merits of ATE vs Validation testing? Come on, Slashdot, you're a bunch of engineers, right? Does the CA state legislature have ANYTHING of value to add to your FMEA? What about your production planning? Test plan? V&V protocols?
It's the height of hubris for outsiders (especially lawyers in the state legislature) to come in and dictate low-level engineering details. A responsible legislature (and public) would acknowledge that they have NOTHING of value to add to the discussion.
The only appropriate regulation is "make it X safe." Don't tell us engineers to get there, and we won't tell you lawyers how to snort coke of a hooker's tits.
Dance like you're hurt, Love like you need money, and work when somebody's watching.
-Scott Adams
Let's see if...
Google writes the software for the car
Google writes (or pays someone else to write) the simulator
Google runs the test
Google reports the results
Seems like with simulations we would be somehow implicitly trusting google that their simulator sufficiently models reality vs only modeling what the self driving software expected...
Although simulation has its place to improve testability during training and development, how does this test against reality? The reason to test against reality is generally to cover the stuff that you *didn't* expect. It's generally quite easy to fool yourself (and others) that something is good enough if you remove this link back to reality...
It's easier to collect data on the drivers' shopping habits.
Turn the simulation a "car game" with good car simulation (allow to skid or dump a car) and then add the software as a gamer.
You can even move a real car with the same movements on a flat big zone (ej. desert) with people inside the car but with screens instead windows seen the simulation to monitorize the reactions of people.
A mix of virtual reality, augmented reality, real acceleration/speed perception and crazy human drivers and pedestrian f***ing the simulation pushing the software to drive into the limits without real danger (because the obstacles only exists in the simularion, although directly controlled by humans in real time).
Other tests could be record real cars for a lot of hours and later test the software with this video to check if really detects the same that human has seen.
Sure, but the article isn't taking about simulations vs real life. It's talking about simulations vs contrived but legally required tests on manufacturer test tracks. Both are limited by imagination but simulations are more thorough, at least according to Google
Judging by the malicious activity at my mail server, and all manner of malicious hacking attempts of my computers, I cannot wait to see what happens when driver-less cars become the norm.
Will pedestrians simply step off the sidewalk at any place at any time, safe in the knowledge that the driver-less cars will all stop?
Will car-jackers feel liberated by the free availability of cars that stop on command - just by walking out in front of them.
What will the public liability insurance rates be for driver-less cars, given that they wont be programmed to run pedestrians over, but will therefore be stoppable by any pedestrian waving their arms.
Will running red lights therefore also get easier and safer?
Can someone give me the obligatory car analogy?
Can I miss Google spokeswoman Katelin Jabbari?
Who logs in to gdm? Not I, said the duck.
Simulations and real tests aren't mutually exclusive. They have different strengths, and you should do both.
They're doing this so they can rig it. Human brains are amazing at adaptating, object indentification, and fast logical reasoning. Computers are horrible at that. Some humans are pretty awful drivers. So the computer would have to be better than a below average intelligence human. They're not even close. In a simulation, you just program the logic to react to only the logic that you programmed into the simulation. In other words, cheat.
I work in the UAV industry, doing safety protocols. The hubris is on the side of the engineers. For a very simple system about a billion simulation runs in a constructive model, a million in a virtual model (flying against real pilots) and a thousand engagements to validate the modesl will be marginally adequate. The engineers want to claim it's good with a million constructive simulation runs and dozens of live runs, and they're fucking wrong. This is why we have oversight. I'm much more confident in the CA legislation than in the google engineer polyannishness, because driving is a whole hell of a lot more complicated to model than flying.
Just about every day I see a driver perform some maneuver that defies rational explanation. I have little faith that the writers of the simulation will be creative enough to model the levels of real craziness out there.
Found a bug in physics.c, those cars we mass produced last year will spontaneously explode after 367 days of exposure to an atmosphere containing oxygen, or when white lines are painted rather than vinyl, or when attempting a corner of a prime number of degrees when speeding on a cambered road.
Why wasn't this spotted sooner?
Because we hadn't expected to need chemistry or non-Euclidian geometry in a physics engine.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I can't wait for the giant ensuing lawsuits. Someone is going to die, and ensuing legal battle will finally take the media's eye off of, what, squirrel!!
Are these cars going to have a big 'beta' sticker on them the way the simple "type a word here and we will try to find a relevant web page" text box did for ten years or so?
In a simulation you can have people do all kinds of crazy shit you wouldn't see in thousands of on-the-road miles. You can simulate malfunctioning equipment that you wouldn't get without years of wear and tear. You can test modifications to the AI without real-world consequences. You can test the human-ai interaction on average drivers without liability problems. I could totally see a simulation being superior to reality for testing purposes.
But you could also have a broken simulation, which could make the whole thing near worthless.
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
Steering around a problem is, on average, safer than applying the brakes. Frequently, pulling onto the shoulder and THEN applying the brakes in order to come to a stop next to the car you would have rear-ended is the best course of action.
In the scenario, visibility is reduced and the pavement is slick with rain. "Maintain heading and come to a halt " in those conditions practically guarantees you'll get rear-ended. The car behind you has their vision obscured by rain, can't stop quickly on the slick syrface, and is most likely following too close (most US drivers follow too closely) .
Just make a cheap drone thing that is unlikely to hurt anyone or anything if it fubars. Think PVC, small electric motor(s), no passenger, just a space frame and some type of low cost motive force..
If they aren't already testing a LOT with simulations, and a LOT with closed track, and LOT with real roads under controlled real-world-like conditions, they're complete morons, and I'd be surprised if they're complete morons so I suspect that's not the case.
That having been said, trying to push for eliminating real-road testing requirements provides some evidence that they might actually be morons after all.
You CANNOT test properly with simulations and closed track only. The requirement is a good one and I hope the other states that don't yet require this will do so as well. Test the fracking hell out of it in the simulation, test the limits of the system on a track, and then have drivers carefully supervise the vehicles during extensive on-road testing with simulated normal traffic conditions, and THEN test in real traffic (still with careful driver supervision and recording, logging, debugging all turned on). This crap is too important to sell X-thousand cars and have them out on the roads with families in them, only to discover a major flaw resulting from some "unlikely" corner case you didn't think of but that actually ends up happening very often given a large enough population of vehicles and varying road, and environmental conditions, etc.
Think about it, you want to give a teenager a license after he proves he can drive... in a simulator? Hell no! He needs real-world supervised experience for 6-12 months in most states, and that's a good thing. A new AI is much like a teenager in that regard. Now what level of changes should require re-certification, and how many miles, hours, etc, and under what conditions is something for others to determine, but in general that regulation makes perfect sense and arguing to remove it is stupid.
"Steering around a problem is, on average, safer than applying the brakes."
We could even just leave it at that, it is a factual statement and is supported by the actual data versus random speculation.
Secondly "you" (in this case your self-driving car) cannot control whether the car behind you is following too closely (especially if you're already in the "slow" lane), and not all cars will be automated cars for many decades, if ever. In addition, the car has sensors beyond just visual and can detect obstacles such as the cliche "expectant mother" (seriously, give me a break) and avoid colliding with her. The car behind you has a human driver who can't see well in the rain, might be distracted, and is likely to not be a particularly good driver in the first place. The most logical way to avoid an accident in that case is to get out of their way if possible.
You don't need to own your own race track to race on one, nor do you need to own your own test facility to prove the road-worthiness of a new automobile model. You simply pay someone to use their facility for a while, if not contract with them them to do that actual compliance testing itself.
In addition, if you're a startup with an algorithm, you're going to have to partner with a vehicle manufacturer, and/or sell or license your software to them anyway. They have the resources for the testing.
Perhaps you forgot to take into account HOW THE SOFTWARE BEHAVES when there is a fault in design or manufacturing. You're not testing the car for mechanical faults, you're testing the system's ability to deal with such faults, which is most definitely unique to self-driving-cars. With standard cars, it might be somewhat beneficial to plan for, and perhaps test, what an average person might do in the event of a failure and make it easier for them to avoid an accident, but that's a different issue alltogether. With a self-driving car you're not just selling the nuts and bolts you're selling the intelligence too.
Bit-flip error in specific hardware triggered by the 2022.3.5 version driving in Death valley for over 6.5 hours.
But on the mandatory test track you normally do thing irrelevant for automatic cars. accelerate, turn, brake, stop, etc. Every automatic car will be able to do that (well hopefully) so I don't think this type of testing being useful for verification of the autopilot
In addition to passing a suit of actual real-world tests the software should be required to pass a shitload of standarized simulated tests, which all cars must pass before being allowed on the market. This test suit should grow constantly, e.g by entering recorded accidents, and every year the cars software should be able to pass that years test. It should also be illegal to make cars which can not have its firmware updated to the newest software which passes the newest test.
Someone COULD have pulled off onto the shoulder in front of you.
Someone DEFINITELY is in the lane in front of you. "Could" is less likely than "definitely" . The shoulder is the therefore the safer bet.
"Or the car behind you is following too closely " - it normally is, most of the time. Especially considering that the driver of tge car behind you may well not be focused 100% on driving. If they are turning down the radio because they're calling in to try to win Aerosmith tickets, 1/4 mile is too close for conditions.
The article is pointless. Okay, Google is trying to replace the current "controlled" road test with a simulator. The article goes on to say how wonderful simulators are. So what? It says *nothing* about the current regulations. What are they intended to test? Are they done once per model? For every firmware revision? Every individual vehicle? Are they meant to be fully exhaustive or are they more on the order of the driving test a person must take to get a license? Without knowing what the current tests are there's no way to judge whether a simulator is an adequate substitution.
Pure personal speculation: I suspect that the main goal of the testing requirement is to give a warm fuzzy feeling to a non-technical person. It lets them see a tangible object responding to tangible threats. It probably puts the car through a series of common scenarios and some uncommon but easily imagined dangers, so the non-technical human can see the car dodge obstacles and walk away confident that the robot responds like a human would. If that's the case, a simulator will never be sufficient since it's not really a test of the car's performance, but a test of the human's confidence in the car.
Chelloveck
I give up on debugging. From now on, SIGSEGV is a feature.
We LIVE in a simulation!!!
There's an intersection near my house where cars crash every so often.
A driver could go through there once a day. Cars go through there many times each second. How many millions of cars go through without a crash happening? How much more attention should each driver pay to prevent it?
Could a robot car be programmed to drive through there perfectly every time?
This one time I was driving along angry in the rain, the car in front of me stopped suddenly, I jammed the brake pedal to the floor, felt the ABS go thud-thud-thud and my car had superficial damage.
The robot car doesn't have to be perfect to be better than a human.
There is only human error. Gimme.
As people have pointed out, simulation doesn't require permission, so what if they were lobbying it as a requirement rather than just optional? I imagine this is actually an attempt at creating regulatory capture. If the tests are made very very difficult, it would give the entrenched players a far greater advantage because they got in first.
Hard tests are good. However, if there is any way for manufacturers are able to put in their own tests (even by corrupt means), companies can set themselves up to already have a solution, while others scramble to get theirs working.
We want good algorithms, but with the broken patent system, good solutions are expensive, and arguably unreasonably so.
I am blind and therefore need to use the computer with the aid of specialized software, but hey look at the abilities NOT the disabilities.
I live in the UK.
I'd like to know why a Blind Person cannot use one of these vehicles I purposefully did not say "Drive" as Driverless means just that no driver just a user!
If as we are told these cars are shortly to be available on UK roads then can someone explain to me WHY ordinary sighted folk will be able to use them and not VIPs (Visually Impaired Person)?