Slashdot Mirror
The Problem With Self Driving Cars: Who Controls the Code? (theguardian.com)
schwit1 writes with Cory Doctorow's story at the Guardian diving into the questions of applied ethics that autonomous cars raise, especially in a world where avoiding accidents or mitigating their dangers may mean breaking traffic laws. From the article: The issue is with the 'Trolley Problem' as applied to autonomous vehicles, which asks, if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do? The problem with this formulation of the problem is that it misses the big question that underpins it: if your car was programmed to kill you under normal circumstances, how would the manufacturer stop you from changing its programming so that your car never killed you?
...I find it very hard to understand how people drive in modern cars which have so much closed source programming.
How modern are you talking? I have cars which speak directly to this; all I'm lacking is a fully computerized car. I have a W126 300SD, which is all-mechanical down to a vacuum shutoff on the engine. I have a D2 A8 Quattro, which is fully electronically regulated and can't run without the PCM, but which still has a transmission (ZF5HP42) with an actual shift linkage and a limp mode, and hydraulic power steering (which is lovely, but ironically not quite as communicative as the W126, even though that has a recirculating ball box.) A truly modern car with e.g. a ZF9 doesn't have a shift linkage, and if the TCM goes up in smoke, so do your hopes of driving home. It also doesn't have hydraulic power steering, so if the power goes out while you're doing something tricky, it's going to get trickier. At which point do you pucker?
In the D2 you can still conceivably replace literally all of the computers with devices of your own design. In some european markets there is even a heater-only control unit that operates the flaps with bowden cables, as in cars of yore, which puts it ahead of the W126 body. The AT can be replaced with a six-spool which doesn't require any power, and people have megasquirted the ABZ before. You need either the ABS or an adjustable proportioning valve, though. The car doesn't have one because it has EBD, which you will be throwing away. It is possible to source a LSD for the rear, though; you can use your ring and pinion with the guts from the Audi V8's rear diff, which is a LSD (Torsen, IIRC, like the center.)
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
or how does the manufactorer stop me from simply driving too fast? in an age where most cars have country-specific software & hardware modifications, it makes zero sense for a car to be able to go (much) faster than the maximum allowed speed limit.
Would you buy a car that came equipped with an explosive that would, under certain circumstances, explode and kill the driver?
This whole "trolley problem" is bullshit.
From TFA, Doctrow uses the "trolley problem" to get to a different point:
Nice switch there but the basis is still bullshit. No one will buy a machine that has code in it specifically designed to kill them.
And the main reason for that is that the programmers would have to be 100% certain of EVERY SINGLE SITUATION and implement that code with 100% reliability. And that is impossible.
The moment such a machine mistakes a toy doll for a child and INTENTIONALLY injures/kills the occupant(s) is the moment that company will be sued out of existence.
Not only that, another big problem with the "trolley problem" is that it doesn't pass the "could a human driver do better" test.
It assumes that you have lost control of the vehicle to the extent where you can only select between two choices. While every driver will claim that they are superior it is all just bullshit and they will be unable tho make a choice at all in those situations.
The main point of automatic drivers are to not get into or cause situations where you don't have control of the vehicle and that includes keeping the speed limit and being more aware of the surroundings.
When the automatic driver comes into a situation where a "hard choice" would have to be made then it is past the point where the human driver would already have flattened that teleporting, wheelchair bound, ball playing child that suddenly appeared in front of the car.
Who controls the code? Maybe you, maybe them. If you tamper with it, you're responsible. Otherwise, they will probably be responsible for its behavior. But the computer is not going to be "programmed to kill you", that is bollocks. The computer is going to be programmed to follow the law. That means that it's going to be less likely to be at fault in an accident to begin with, that it's going to be more likely to successfully mitigate the accidents it does get into, and it means that rather than being programmed to kill you, it's going to be programmed to stay in the lane and hit the truck rather than swerve and hit the pedestrian because to do otherwise would be illegal — not just because of the pedestrian, but because of the lane marking. That is not remotely the same thing.
The car will be programmed to do its best not to kill you, and that's going to take the form of yielding gracefully to fuckheads rather than getting in an accident to begin with.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Exactly: there is never a such clear-cut decision between "my owner or that bus full of nuns". I can imagine that cars will be programmed to make decisions based on certain basic principles, but those principles will be nothing like the 3 laws of robotics. Taking drastic action to avoid an accident may lead to a worse one, and it'll be a long while before our machines will be anywhere near able to make such complex decisions.
To begin with: if everyone sticks to the rules of the road and drives normally, there is very little chance of an accident occurring. If an exceptional situation occurs, the fault of an ensuing accident primarily lies with whomever caused that exceptional situation (even if it's unintentional). If someone's tyre blows and they swerve into your lane as a result, if a child chases a ball into the road, or if a cyclist runs a red light in front of your car, you'd (probably) do everything to avoid a crash and so should a self-driving car, but you are not under any moral obligation to drive yourself into the side of a building in order to avoid the other car, child or cyclist. Self driving cars should operate under the same premise: it should never be considered necessary to sacrifice the driver.
If something unexpected happens, cars might follow a protocol similar to this one:
1) Stay in your lane and come to a controlled stop.
2) If a controlled stop will not prevent a collision (and this is something that self-driving cars should be able to assess fairly accurately), change to a different lane if there is an unobstructed one.
3) If there are no unobstructed lanes but the road ahead is clear and the local speed limit is below x, change into oncoming traffic.
4) If all else fails, reduce speed as much as possible and allow the collision to happen.
These are not meant to be complete and valid for all situations, it's just to give an idea of how such "laws" could be formulated, in the form of a decision tree that self-driving cars would be able to follow, without having to make complex judgment calls or difficult moral decisions. And I can well imagine that a basic set of such rules will be set into law so that all self-driving cars will follow the same basic protocol. As a driver you'd have little incentive to change the programming in your favour, and if you did, it would become immediately apparent as soon as you're involved in an accident.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
" it makes zero sense for a car to be able to go (much) faster than the maximum allowed speed limit."
Sorry I don't accept that limitation. If my Corvette could only go 80 or 90 why would I of bought it? At that point there would be nothing to distinguish it from a Prius.
I grew up in a racing family. Horspower and Torque are fun. Cars that are speed limited or drive themselves equal zero fun. Lifes too short to deprive yourself of an flying down the road in an open cockpit car on warm summers day. Damn, I hate winter.
Donald Trump, on a crusade to make Nixon look respectable
Who controls the code? They do. Just like they do now. How many people here are actively changing the code in their cars? Legally they are responsible for it.
Oh you chipped your car? Well now you're responsible.
Seriously the anti-car crap is getting ridiculous, as is the question of ethics. Car making a decision to kill the driver? Car breaking the road rules? When every car is driven perfectly according to the rules the death rate will be decimated and bystander accidents will be treated in the same way as any other idiot stepping into a cage with a running robot arm.
I don't understand how people have made this so complicated.
Now I know how users must feel when I try to explain virtualization. I know most of these words but it is very hard to makse sense out of them.
if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do?
How about you tell us what should a HUMAN driver choose in a similar situation first, before you ask what should a computer do?
These kind of stupid questions are well, stupid. And they come up often simply because there is no real valid worry about autonomous cars. Humans make lots of mistakes and having a computer drive would remove a whole range of avoidable accidents. Worrying about a few boundary cases is as stupid as all the "what if my car is burning and I need to get out quickly?!" objections to wearing seat belts. It is unfounded fear that is not based on facts.
Oliver.
How much sleep have you lost over the engineering decision to make trains so large and heavy that the simply CAN'T stop for pedestrians and other vehicles. Yeah, I thought so. People will kvetch about how self driving cars are programmed right up until they become every-day objects an after that they'll be just as accepted (benefits AND dangers) as trains are today.
The Trolley Problem is a red herring that distracts from the real danger: government remote-controlled detainment of political opponents, as depicted in Minority Report. Plus, any number of variations: script-kiddies hacking, drug cartel kidnapping, kidnapping/trafficking of women/children, murder-for-hire (drive off cliff), nation-state espionage and assassination. When major crimes, and not just credit card scams, become available to the push of a button, the risk threshold to the criminal is lowered for heinous crimes.
I'd also have it sound the horn and flash the lights.
And this is, IMO, exactly what will happen.
Remember, these vehicles have complete records of everything that is happening around them at all times. Everything that can be recorded, that is. So the insurance companies will have exact records of how the robot was 100% within the law AND had taken every REASONABLE response to mitigate the collision.
The robots do not have to be 100% at determining whether your life is worth more/less than someone else's. They just have to be 100% at showing that they were following the law and attempting to avoid the collision.
The legal system and the insurance companies will sort out the rest. And the insurance companies will pay to have the legal system write the new laws to reflect that.
That's precisely the point: sideswiping the car next to you is such a risky manoeuver that machine nor man can probably make that judgment call very well, with any vehicle. So an automatic car wouldn't. Take evasive action if it's safe (more or less within traffic rules, i.e. without hitting anything else; the car should be able to make that call), stop in your own lane if possible, or slow down as much as you can and crash into the obstacle in your lane. Avoid the impending accident if you can do so safely, or let it happen if you can't. Of course there may be exceptions explicitly defined if they fit into a simple decision tree, like driving into the guardrail. But mostly the car shouldn't go looking for the lesser accident or the harder target, because that's where the complexity (and liability) comes in.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
Would you buy a car that came equipped with an explosive that would, under certain circumstances, explode and kill the driver?
You mean like an airbag? https://www.google.co.uk/searc...
I am sure that there are many other solipsists out there.
Who said selectable? In any event it's safe to assume the fully automated car is obeying all related traffic laws etc. Obeying all the laws does not mean it will never be in a situation where it has to choose. Easy example is 50mph speed limit rural roads guard rails on each side, kid pops out through a hedge and runs into road after a ball or whatever, you have an oncoming truck doing 50 do you hit the kid or do a head on into the truck you have no time to effectively break or any forewarning of the event. It's perfectly reasonable to hit the kid in some cases you will even be sited for hitting the truck.
What you want is automated taxies that's fine sell it as a service. If you own it you're entitled to modify it. We can not nor should we restrict travel it's a basic right so should be access to any shared property like roads. Restricting people because something might happen is immoral the whole evil of thought crimes. If somebody mods their vehicle and that's the primary reason it injures another sure thats civil and possibly criminal. But to say you do not have freedom of movement unless your using state sanction software for safety reasons is just far far to easy to be abused.
No sir I dont like it.
but you are not under any moral obligation to drive yourself into the side of a building in order to avoid the other car, child or cyclist. Self driving cars should operate under the same premise: it should never be considered necessary to sacrifice the driver.
While you are correct, don't think for a minute that there won't be a lot of pressure to give a heirarchy weighting to decisions made by the cars.
We make fun all the time by saying "Think of the Children", but I'll bet if you polled a sizable group of people in some hypothetical You could save one person, an adult or a child scenario, probably most would save the child and leave the adult to die.
All of which is to say, if there is the possibility to apply weighting to a vehicle's collision avoidance system, the is no question there won't be a lot of agitation to do so.
Which then beings up the Escalade set. In a NatGeo article a few years back, they had a feature on petrofuels. They had a short bit in it interviewing a woman who owned a Hummer. When asked why she made the choice of vehicle, she stated "Because if I get into an accident, I want to know I'll win!"
If there's any justice in this world, that pathological bitch will be run over by a autonomous M1A1 Abrams tank. But I hear the same sentiments, if not so mememememe! oriented, from the owners of those vehicles. Their safety is paramount, and when I ask if they are okay with killing the person in the normal sized car they hit, they say they are okay with it, because the person in the normal sized car should have bouth what they are driving (they ignore the vehicle size arms race problem)
I can see it now - the 2025 Cadillac Escalade with MyFamily! autonomous driving protection.
I can see the slogan now - When all is chaos, your family makes it though with MyFamily!
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
I expect that hacking a car's vehicular AI will change its insurance rates. Insurance companies charge insurance based upon known risk. A vehicular AI that has been rated by an independent rating agency (UL, etc) is a known quantity and an insurance company will be willing to insure it.
A home spun vehicular AI will be considered an unknown risk and will be treated as such by the insurance companies. If you want to roll your own vehicular AI, feel free but you'll be responsible for having the AI rated and certified. Otherwise the vehicular AI will be treated as a "High Risk" AI and your insurance will reflect that.
Architectural plans are like computer source code with a couple of differences: You only compile once.
For instance, the proper response to an impending accident might be to sideswipe the car in the adjacent lane, but only if it's actually a vehicle that could take the hit, as opposed to a motorcycle. ...
Driving is a far more complex activity than a lot of people realize,
Which is why we drive over a group of nuns right now. Driving may be complex but human decision are ultimately entirely based on luck, self preservation (which is why the seat BEHIND the driver is the safest in the car), and whatever you're able to do with your hands in a sheer moment of thoughtless panic.
The perfect computer will be no different. It won't come down to side swiping or some major calculation onto the future prospects or occupancy of the lane beside you, it won't be a case of can you safely side swipe in a way that no one is killed but a few people are injured. Not in our lifetimes. Not even remotely. Heck in the next 10 years computers are going to have enough trouble determining with certainty if the lane is clear or not.
The computers won't ever make complex judgement calls because quite frankly they lack the information required to make a complex call. The calls will be simple: is the lane left to me clear yes / no. That's as much information that the computer will have with many thousands of dollars worth of sensors. Likewise thing on the road in front of you, can I break / avoid yes / no? Computers aren't going to determine if it's a child, a small chimpanzee, a nun, president Obama, etc.
We are holding computers to such an incredibly high standard above our own driving in emergency situations in a world where emergency situations are likely to be decimated in probability. All this talk of judgement calls and ethics is an exercise in pointless navel-gazing.
The problem with any of those rules is that they will never be used and reflect a human driver decision based on human reaction time.
A self driving vehicle can anticipate an accident much faster than any human driver and you can often (as a human) come to a complete stop or decelerate enough in cities before the worst has happened.
Sure the kid and cyclist might get hit (or more likely and in most situations they hit your car) but they most likely won't die from the impact (and if they do its not your fault) whereas a human driver might continue full speed ahead between the realization of a hit and a maneuver to stop and that is where most of the damage occurs.
The other occurrence is malfunction (tyre blows etc) and then your insurance or the car company is on the line, even then, a computer can often correct better than a human. My VW already corrects the direction of my car when I go full speed over a bump or slip on ice. It also corrects against sudden overreacting on the steering wheel or slamming the brakes. Adaptive ABS in these cars will allow for even shorter brake distances.
Custom electronics and digital signage for your business: www.evcircuits.com
There's no reason to make the software non-free. It's not like the code will have some SacrificeBusfullOfChildrenToSaveDriver variable that some idiot can change, and even if there were hardly anyone capable of reprogramming the car would be stupid enough to risk their lives on their own untested edits to the program.It would probably be a good idea to have signed software for security purposes, but that's different and compatible with free software.
There is no need to worry about the moral dilemma of choosing who to sacrifice. The answer is simple and obvious to everyone -- pick the driver who never drives drunk, never drives sleepy or otherwise impaired, never gets distracted, has lightning reflexes, always drives carefully, and is less likely to kill everyone. Morally speaking, we want to start using self-driving cars, even if they are worse than the average driver, starting first with replacing arthritic old grandma with failing eyesight*.
* Roughly speaking, the morally correct thing to do is replace any driver who's insurance premiums (aka professional estimate of actual driving ability) are higher than those of a self-driving car.
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
If a child can run into the street from a blind spot faster than you can break, you're driving too fast. The autonomous car will not drive too fast to break in such a situation.
If the child is deliberately hiding in a place it shouldn't be, near a higher speed road and manages to quickly cross what should usually be a wide clear area around such roads (or in an unpopulated area), then there won't be time for a human to react at all. An autonomous car could probably cut down the speed a bit, but avoiding people who deliberately try to throw themselves in front of traffic simply isn't doable or even something to care that much about. You're not going to be able to avoid jumpers, or the human cannonball either, nor is an autonomous car. As the other reply pointed out, whether it's wildlife or people, usually the best option is to simply do a predictable controlled break unless it's basically a slow-motion situation playing out over many seconds (such as road conditions making braking very ineffective, while speed isn't particularly high).
Why is there a car in the adjacent lane in a high-speed situation with objects that can conceivably exhibit behaviour that could cause in impact faster than you can do a controlled break? Sounds like you're driving too fast and tailgating someone while you're passing someone else. How about, you know, not doing that? Accident avoided.
The trick to avoid serious accidents is not to be able to make complex judgement calls in an emergency, humans suck at that, and life isn't Groundhog Day where you get to practice a dozen times 'til you get that call right. The trick to avoid serious accidents is to do your best to ensure you're in a controllable situation as much as possible. You should be keeping enough distance to be able to break when someone more than slams their breaks. You should keep enough clearance to be able to accelerate or decelerate if someone starts moving into your lane. You should keep excessive distance to cyclists or pass them at a controllable speed.
An autonomous car can keep safe margins far better than a human can, and it's much more capable to actually keep the situation within the actual limits of what it can deal with. Because, above all, it wont delude itself into thinking it can actually make perfect complex judgement calls within fractions of a second.
What you are taught when you learn how to drive is "always brake, never swerve". Which also happens to be exactly what a self-driving car would and should do in an emergency situation.
Reformulated as a trolley problem, this is: do not pull the lever.
Why? Because swerving will almost always put the car in more danger for itself and others, since it may cause the car to spin, or approach cars in incoming lanes or pedestrians on the footpath, or any number of bad things.
Since, by law, other cars must maintain a safe distance behind you, slamming on your brakes is always a safe response to an obstruction. Any other response, such as changing lanes, should be considered if it is safe or not. If it is unsafe, then you shouldn't do it, things like "busses of schoolchildren" is completely outside the parameters of the problem.
This whole self driving car issue took a wrong turn the day that someone had decided that they had the moral imperative to break the law in order to prevent crashes. You do not have that imperative yourself, you have the imperative to follow the law in order to prevent crashes. The reason that two drivers can pass each other at enormous speeds without prior communication is that the law forces them into their lanes. The law is the protocol that all drivers follow to interact, since they are unable to talk to each other and as soon as you break it, you expose yourself and others to untold danger. If the law says "don't leave your lane" and best driving practices says "don't swerve", then you don't swerve.
When Argumentum ad Hominem falls short, try Argumentum ad Matrem
You're going down the road, you'll hit the guardrail at an angle and be just fine. Also, you should have noticed the kid climbing up and getting on the guardrail. If you can't see that, you're driving too fast for the conditions.
You might not be familiar with my comments but I've been rather staunch, to the point of offering bets, that claim AV is not going to get here for a very long time - if at all. (Partially autonomous is already here, fully autonomous isn't going to be here on the road, in any great numbers, for a very long time.)
But, to go back... Almost every accident is caused by someone driving too fast for the conditions. (Read that carefully if you're confused.) The situation you mention should not happen with a human or computer driver. You've got time to see the kid climbing up onto the guardrail. You've got time to see them approach. You had ample time to slow down and be able to make a panic stop. A computerized vehicle is going to go slow when there are dangers around. It will, ideally, notice something on the side of the road, long before it reaches the road, and adapt to that situation.
A human should have done the same thing. If you add, "well we're going around a blind corner" then you don't negate my point - you prove my point. Accidents aren't accidents in almost every single case. The result may not be the intended result but it sure as hell wasn't an accident. The operator drove too fast for the conditions. If you have a child appear in the road and are unable to stop then you damned well are at least partially culpable (in my opinion - we're talking morals and not laws) because even if you were going less than the speed limit, you were still driving too fast for the conditions.
If you can't stop your vehicle in time then you need to have already slowed down before you reached that point. A good example is your contrived highway. With modern car illumination, ABS, and average human response times - you're already nearing the point were you're driving too fast to stop, completely, should an obstruction appear even at the limit of your lighted vision. (I believe modern cars, with ABS, average about 60 MPH before this threshold is reached.)
If you're driving too fast for the conditions then it's not "the other guy" who is putting you at risk and taking the lives of your children into their hands - in the vast majority of cases, there are some (a tiny percent) which actually qualify as accidents but, more often than not, those are also easily attributed to people not doing things like maintaining their vehicle properly.
Here's one more indication for you. I repeat, you're driving too fast for the conditions. If you're going 5 MPH and slide off the road because of snow and ice, you're driving too fast for the conditions. If you are driving so fast that you're unable to come to a safe and complete stop for anything that might enter your travel lane then you're driving too fast for the conditions.
An AV will not do that. The software will be written by people smarter than that. However, you still mash the hell out of the guardrail to avoid the children because the fault is pretty much entirely yours for failing to maintain control of your vehicle. No matter how fast you go, if you have an accident then you were driving too fast for the conditions. I don't care if you're drunk and going 1 MPH and bump into another car as you attempt to park. You were driving too fast for the conditions. And yes, sometimes the appropriate speed is a grand total of zero miles per hour.
"So long and thanks for all the fish."