Slashdot Mirror
The Problem With Self Driving Cars: Who Controls the Code? (theguardian.com)
schwit1 writes with Cory Doctorow's story at the Guardian diving into the questions of applied ethics that autonomous cars raise, especially in a world where avoiding accidents or mitigating their dangers may mean breaking traffic laws. From the article: The issue is with the 'Trolley Problem' as applied to autonomous vehicles, which asks, if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do? The problem with this formulation of the problem is that it misses the big question that underpins it: if your car was programmed to kill you under normal circumstances, how would the manufacturer stop you from changing its programming so that your car never killed you?
...I find it very hard to understand how people drive in modern cars which have so much closed source programming. As far as I'm concerned, if I can't take something apart and have a reasonable idea of how it works - which I can just about do, with an old enough vehicle - I'm not going to trust my life with it so directly. I similarly don't accept elective surgery from a proprietary robot which carries on the procedure without step-by-step oversight from other expert humans in the room.
nowadays you got the freedom to drive like an asshole - just give the car a user selectable setting, if it should preserve your life at all costs, preserve the life of others or make a decision that minimizes overall damage (but may harm you) - if you select the "me first" option, you are responsible for your car mowing down a row of krishnas. "simple" as that.
Either annual or ongoing, if easily enough done the police would probably do it. You change the code so the car doesn't, say, drive off a cliff instead of straight into the middle of a class of school girls (just to make it clear, I'd drive into the kids. It's my car after all and facing the choice between killing a dozen kids and me, the rugrats croak), then this is an illegal modification of your car and it is no longer considered safe for traffic and shut down.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
- how does the manufacturer stop me today from modifying my car so it endangers others (e.g. mount a flamethrower on it) ?
Another thing. How are these various autonomous car software platforms going to interact with one another.
It's one thing to build in recognition protocols for your own vehicles, so that multiple vehicles of the same type act in a concerted manner.
But what happens where you have four or five different codebases? How are the notoriously closed car manufacturers going to deal with car behavior from another system?
I can foresee some rather nasty interactions. Head on collisions where one car tries toavoid by a juke left to avoid while the other tries to juke right.
Chas - The one, the only.
THANK GOD!!!
This is just sensationalism. The real issue is that, if people are willing to give up their responsibilities to control a vehicle, they necessarily give up their freedom to decide how that vehicle behaves in certain situations. If you want to decide how a vehicle behaves there are probably two options: get a manually operated vehicle, or build your own "automatic" vehicle with your own rules. But good luck on that latter; just as there are regulations on acceptable behavior with manually-operated vehicles*, I bet there are going to be regulations on the "rules" automatic vehicles are going to have to include.
*Yes, you can choose to save yourself and drive into a trolley or crowd. But you will also have to suffer the consequences of that decision. You might be alive, but if you intentionally chose to harm others, you might not like where you spend your life.
"There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
People, in America at least, will absolutely bust a gasket once the first actual deaths roll in and some egghead behind a desk in some remote part of the world "decides" who lives and who dies. Until it happens people won't care. Americans are very independent and a me first kind of crowd so it may be the righ ^h^h^h^h profitable choice for manufacturers is to simply protect the occupants no matter the collateral damage as long as they can't be held liable.
Who controls the code? Maybe you, maybe them. If you tamper with it, you're responsible. Otherwise, they will probably be responsible for its behavior. But the computer is not going to be "programmed to kill you", that is bollocks. The computer is going to be programmed to follow the law. That means that it's going to be less likely to be at fault in an accident to begin with, that it's going to be more likely to successfully mitigate the accidents it does get into, and it means that rather than being programmed to kill you, it's going to be programmed to stay in the lane and hit the truck rather than swerve and hit the pedestrian because to do otherwise would be illegal — not just because of the pedestrian, but because of the lane marking. That is not remotely the same thing.
The car will be programmed to do its best not to kill you, and that's going to take the form of yielding gracefully to fuckheads rather than getting in an accident to begin with.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
... there are some freeways where the top speed is 55 and others where it's 70+.
My Garmin frequently gets the speed limit wrong for a road and if fully automated cars rely on that data, then I'll have to override the car in order to comply with the law. Because, the manufacturer will NOT be responsible for any tickets.
Who controls the code? They do. Just like they do now. How many people here are actively changing the code in their cars? Legally they are responsible for it.
Oh you chipped your car? Well now you're responsible.
Seriously the anti-car crap is getting ridiculous, as is the question of ethics. Car making a decision to kill the driver? Car breaking the road rules? When every car is driven perfectly according to the rules the death rate will be decimated and bystander accidents will be treated in the same way as any other idiot stepping into a cage with a running robot arm.
I don't understand how people have made this so complicated.
if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do?
How about you tell us what should a HUMAN driver choose in a similar situation first, before you ask what should a computer do?
These kind of stupid questions are well, stupid. And they come up often simply because there is no real valid worry about autonomous cars. Humans make lots of mistakes and having a computer drive would remove a whole range of avoidable accidents. Worrying about a few boundary cases is as stupid as all the "what if my car is burning and I need to get out quickly?!" objections to wearing seat belts. It is unfounded fear that is not based on facts.
Oliver.
In Switzerland, you are required by law to help if you see a person in danger. However, it is understood that you are to make sure that you can operate safely first. It makes no sense to go in with the best intentions only to produce a second victim for the firefighters to rescue.
Thinking that further, it is clear that your car cannot take responisibilty for other participants in traffic since it cannot control them. It will save your life at all cost. Now if the decision lies between possible injury of yourself and death of another, you WILL take a broken leg and cracked ribs.
Human nature dictates that self-preservation comes first. Nothing will ever change that on any kind of broad, fundamental level.
How much sleep have you lost over the engineering decision to make trains so large and heavy that the simply CAN'T stop for pedestrians and other vehicles. Yeah, I thought so. People will kvetch about how self driving cars are programmed right up until they become every-day objects an after that they'll be just as accepted (benefits AND dangers) as trains are today.
The Trolley Problem is a red herring that distracts from the real danger: government remote-controlled detainment of political opponents, as depicted in Minority Report. Plus, any number of variations: script-kiddies hacking, drug cartel kidnapping, kidnapping/trafficking of women/children, murder-for-hire (drive off cliff), nation-state espionage and assassination. When major crimes, and not just credit card scams, become available to the push of a button, the risk threshold to the criminal is lowered for heinous crimes.
Why would an autonomous car allowed to reach such deadly speeds around groups of pedestrians without knowledge of fully working controls and brakes? The safety systems should trigger long before such problem presents itself, and if the worst does happen, the priority should always be to protect the driver and the vehicle first to ensure it remains under control and stops as safely as possible in order to lessen additional damage occurring outside of the narrow decision space. Otherwise there will be mannequins, robots and possibly robotic mannequins on the roads. And blood, lots of blood.
I'll give you a hit we have fully computer controlled manual gearboxes for a long time now. Running a clutch and changing gears it not that hard of a problem. You get all the efficiency of a stick shift with none on of the annoyance factor.
No sir I dont like it.
What ethics underpin the training sets used to develop the cars' rule base? Given a 'Trolley Problem', what scores were assigned to killing one group vs the other? What scores were assigned to killing you (the car's occupant) vs a pedestrian or cyclist?
Have gnu, will travel.
If I am forbidden from hacking my car's software will I be unable to stop it when:
Our society will probably never accept the principle that pedestrians are responsible for their own safety around cars. Whenever I drive near shopping areas, people just walk out like they own the road. Even at a cross walk, they often just start walking without any regard for courtesy or the laws of physics. Heck, even the weather often doesn't stop them such as thinking that maybe it's not wise to jump out in front of any driver when the road is wet or icy. A lot of people are just too stupid to be trusted around human-driven vehicles. Self-driving vehicles will kill them left and right, and so the law will have to be firmly on the side of the vehicle.
it misses the big question
A better question for an IT forum would be to ask how the hell do you test whether the implementation (of which party to kill) works as designed?
It should be immediately obvious to anyone in a capitalist society that who dies is a cost-option. Let's say that opting to save the car's occupants comes at a $1million price premium on the cost of the vehicle.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Luddites had it right.
I expect that hacking a car's vehicular AI will change its insurance rates. Insurance companies charge insurance based upon known risk. A vehicular AI that has been rated by an independent rating agency (UL, etc) is a known quantity and an insurance company will be willing to insure it.
A home spun vehicular AI will be considered an unknown risk and will be treated as such by the insurance companies. If you want to roll your own vehicular AI, feel free but you'll be responsible for having the AI rated and certified. Otherwise the vehicular AI will be treated as a "High Risk" AI and your insurance will reflect that.
Architectural plans are like computer source code with a couple of differences: You only compile once.
A human driver will always choose self preservation even if it means killing others, so why should an autonomous car behave any differently?
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
VW (and others?) were caught tampering with the engine code to get more performance while cheating on the emissions. It's problems like this that make the question of who owns the code, or who reviews it, very relevant...
That's precisely the point: sideswiping the car next to you is such a risky manoeuver that machine nor man can probably make that judgment call very well, with any vehicle.
That depends - if there is an oncoming lorry in your lane and the only way to avoid it is to side swipe the vehicle next to you that's what I would do to avoid what looks like impending death. That's what I would expect a self-driving car to do to: take whatever it perceives to be the lowest risk to the health of the car's occupants because that is what a human driver would do instinctively. To do otherwise and you are only one step above Futurama's suicide booths. How long will it be before you get some idiot who walks out in front of car which then causes it to swerve into an oncoming lorry, killing the occupants, where a human driver may just have hit the idiot? The advantage that a smart car would have though is that it could perhaps tell the lorry what it is going to do, get confirmation back from the lorry that it too will swerve and then both vehicles synchronize their motion to minimize casualties...but preservation of occupants must still come first as it would for a human.
Don't allow individuals to own self driving cars. Only allow them to be rented as taxis or leased by the company that designed and built them. Not only is this more efficient from a utility standpoint (the taxi part, anyway), it solves the legal liability problem. There is precedent for this in the GM electric car as well as the Ferrari 599XX.
We make fun all the time by saying "Think of the Children", but I'll bet if you polled a sizable group of people in some hypothetical You could save one person, an adult or a child scenario, probably most would save the child and leave the adult to die.
True, but suppose we change the "adult" to your wife/husband who is in the car with you and the "child" to a 12 year-old running away from a policeman and who dashes into the street. Do you swerve into the oncoming lorry and kill the adult you love or hit the child who was probably a delinquent? In that situation I think you'd get far more people saying they would protect the adult they love over the child: the closeness of the relationship with the people affected is a huge factor.
This is the problem with a car making life and death decisions which might prioritize the lives of those outside the car higher than those in it. The people you are driving with are those you are far more likely to have close relationships with and so want to prioritize. Indeed even if we stick to the purely abstract case suppose the car had a child in it? How would it know even that unambiguously and, without that, even the simple abstract case becomes impossible since it might be a choice of a child vs. another child.
No, it can not be users selectable unless cars that have chosen the user first mode are not allowed on public roadways. When you drive on public roadways you accept that you have to obey certain "rules of the road", such as speed limits.
You do realize that this would automatically disqualify all human drivers right? Humans will always prioritize themselves and will not always obey the rules of the road. A computer which prioritizes is occupants but which always obeys traffic rules would still be a huge improvement.
Is to allow the driver to answer a set of preferences including the question of should the car give higher priority to preserving other lives or the drivers.
Then it's the driver's choice just as it would be if they were driving the car.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
Why in the world would we require autonomous cars to answer hypothetical questions on morality?
"Grab them by the pussy" -- President of the United States of America
I expect it will go far beyond that in two ways.
(1) The vehicle will no longer be in compliance with regulations that permit the privilege of taking your vehicle onto a public road. Merely driving it on a public road may make one vulnerable to civil charges, loss of driving privilege, confiscation of vehicle, etc. Much like drunk driving regardless of whether an accident occurs or not.
(2) It will void your insurance coverage and make you fully liable for anything that occurs. There will probably be no "hacking" coverage available to a regular person. Only certain research entities, including the auto company's, will probably be able to get coverage regarding software related incidents.
Luddites had it right.
Stick with horses? That's also a self driving vehicle that you merely issue driving commands to.
There's no reason to make the software non-free. It's not like the code will have some SacrificeBusfullOfChildrenToSaveDriver variable that some idiot can change, and even if there were hardly anyone capable of reprogramming the car would be stupid enough to risk their lives on their own untested edits to the program.It would probably be a good idea to have signed software for security purposes, but that's different and compatible with free software.
There is no need to worry about the moral dilemma of choosing who to sacrifice. The answer is simple and obvious to everyone -- pick the driver who never drives drunk, never drives sleepy or otherwise impaired, never gets distracted, has lightning reflexes, always drives carefully, and is less likely to kill everyone. Morally speaking, we want to start using self-driving cars, even if they are worse than the average driver, starting first with replacing arthritic old grandma with failing eyesight*.
* Roughly speaking, the morally correct thing to do is replace any driver who's insurance premiums (aka professional estimate of actual driving ability) are higher than those of a self-driving car.
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
The three laws of robotics state:
To me, the interesting ramifications of these laws in many stories, and one movie involving Will Smith, are more than enough to answer all questions regarding autonomous technology and responsible use. I have no idea how, as a programmer, these would actually manifest themselves in code...
The article brings into light a problem without actually naming it: law. I think that this problem is that different societies will choose different approaches to regulating, controlling, and outlawing technologies, and the same society can make different decisions depending on it's point in time. The society now in the United States makes the most sense as a place to try and sell new technologies to, since it's the richest and has historically been permissive towards the rights of the creators'. Without being able to answer financial liability questions in a meaningful way, this may not continue to be the best place to develop new, and risky, technologies, in the future.
I don't believe the problem of who owns the code is actually a problem. We have many, many different models of code ownership in effect, and the entity which creates the code can choose how they want to "own" it. The actual problem to me seems to be who do I sue when I don't like an outcome?
I pray the 0th law never comes into effect.
I believe that automotive impact testing done using this Poster, and any supportive souls as, and pardon the pun, crash dummies. By concentrating only on cost one easily losses focus on perspective benefits. Cost Only Solutions are starting to go on its long path to obsolescence, but it is clear that it is already headed in that direction.
Will Linux be used in the analysis?
if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do?
How about you tell us what should a HUMAN driver choose in a similar situation first, before you ask what should a computer do?
^ THIS.
Cory's article completely misses the point. Or rather, he brings up the Trolley Problem and then moves on to his own point. The reason it's an ethical dilemma is because it brings up ethical issues. That dilemma doesn't change just because a computer is involved, it just shifts the burden to the system. An obvious solution that would probably occur for the first 10-15 years is "Transfer control back to the human in the event of an emergency", which of course just puts right back where we started.
My biggest concern is that we'll get a bunch of hot shot programmers who don't have the training to UNDERSTAND that they're encoding ethical decisions into their code, or don't understand the weight of them, before they do so. "Solutionism" on the part of Bay Area techies who've never taken a philosophy or ethics class in their life but are convinced science can solve all the problems in the world is how we'll get into messes like anomalous cars, systemd, and sex-changing velociraptors.
Hire a Linux system administrator, systems engineer,
Car DRM = dealer only service and based on how evil they want to get all the way down to tires, windshield wipers, oil changes, lights.
The problem with the trolley problem is that the answers to the trolley problem are much, much simpler than the answers to "How does a car recognize a trolley problem scenario with almost zero false positives?" How often do you read of humans committing actual suicide by driving off a road to save a pedestrian? That's the ceiling for allowed false positives.
That is true. Also some car manufactures (only Honda I know for sure) has put a great deal of R&D into making their cars safer on the outside incase of running into pedestrians.
Website Just Down For Me? Find out
But what about Muslims programming cars to deliberately kill people
If terrorist activity is detected, should the AI drive the car to the incident so the driver can assist in fighting the terrorists, or should it flee the area?
I'm voting for everyone piling on, and the AI could allow access to a locked weapon compartment.
That is true. Also some car manufactures (only Honda I know for sure) has put a great deal of R&D into making their cars safer on the outside incase of running into pedestrians.
If you want to sell a car in Europe or the USA, you have to consider not only what happens in a crash, but also if you bounce a pedestrian off the front of it, or even off of one of the side mirrors.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The problem is more: what about American companies programming cars to kill "terrorists" ? Would Iran buy cars programmed in the US ? And Russia ? And China?
repost (Herblock is on vacation)
I know autonomous cars will be "oh so safe". At the moment I'm just as worried about what these things will make people do to people.
[OPENING OVERTURE ]
Your driver liability insurance policy has come up for review. We have been recently been acquired by AAAA, the quadruple-A company -- the "Autonomous AAA of the future" and what that means for you as a member is -- it has never been easier to upgrade to an a-car! Financing is available! [link] Due to increasing pressure in the political, legal and underwriters' arenas, we regret to inform you that the cost of your driver policy will be rising this quarter in order to begin collection of fees for the Federal National Driver Insurance Pool, and rising at a steady rate thereafter. It will continue to rise over time despite your [good to excellent] driving record. Now that the Autonomous Vehicle Safety Act is law, and blanket liability accident investigation procedures have been approved by Congress, the legal liability of autonomous vehicles is capped nationwide. While this grants the manufactures freedom from risk of direct criminal penalty and potentially unlimited civil liability, it places human drivers in a difficult position. Most a-car accidents will, of historical necessity rather than actual circumstances, be "no-fault". Since human drivers and any victims claiming injury from them are still obliged to use traditional law enforcement and legal means of redress -- and the cost of these continues to rise -- underwriters are pressuring insurance companies to drop human drivers altogether. We do not intend to do this, but we can no longer provide policies for extended periods. Your new maximum policy period is now [one month]. Thank you for insuring with AAAA.
[INTERMISSION \]
Meanwhile...
Dear editor: DRIVERS cause accidents. A-CARS prevent them. That's what the billboard says -- and if Howard County Referendum passes this September manually operated cars will soon be a thing of the past here. What started as a discussion at a hearing after last year's tragic accident grew into a full heated debate, and to think it all started with the parents who provide their children with a-cars pinning the blame squarely on other peoples' children. But then, after co-opting the national campaign with its slick literature and canned answers for everything -- NOW the fault is with human drivers themselves. And then in an astounding feat of lunacy they claim that it's only fair to place the blame on everybody. Not just the drunk, the aged or infirm, the inexperienced, the distracted or the just plain stupid. But no one's stupid in their book, we're just behind the times is all. They are like the drum majorettes of some utopian humanist parade. I say, SAVE US from these rich hippies, their weird toys and their broken ideals. Now I know a lot of these people, even like some of 'em, but aside from this national 'sideline the humans' campaign they're pushed at us (and WHO is paying for those TV spots I wonder) let's not forget that this debate started around kids. Kids who need to learn to drive as surely as they need to learn to push a pen and spell their name. It's like swimming, who would discourage their own children from practice in swimming, to become expert swimmers, because water is dangerous?? Every kid will need to drive some day, or suffer harm or hardship by not knowing how. These a-car parents even forbid their kids from riding in cars being driven by folks they've grown up with, trusted for years. At the parent conferences we even sit on opposite sides of the table, we can barely be civilized even, because this crap has gotten so deep. Well I say they are making a big mistake and don't seem
<blink>down the rabbit hole</blink>
but it might have just that switch.
if the driver is found to have been driving a car that had been modified as such, then he should go to jail of course.
I mean, the rules should be in the law.
but this exact thing has been on slashdot for more times than I care to count for already so this article can go to hell and cory doctorow too.
world was created 5 seconds before this post as it is.
and even if there were hardly anyone capable of reprogramming the car would be stupid enough to risk their lives on their own untested edits to the program.
You've never met anyone who has modded cars have you?
Machine learning? You can't really predict the results of that, so that would be a whole lot of liability for whoever's responsible for the software when it errs and runs a nun over. If you can show what the software requirements are, and that the software meets the requirements, and the requirements are audited, you've got a much higher chance of winning the lawsuit.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
How often do human recognize the trolley problem and consciously make a decision. My guess is that it is pretty close to never. We react in those sort of time frame, we don't recognize.
If information wants to be free, why does my internet connection cost so much?
Self-driving cars are essentially taught by example
This is what we in the biz call a training set. So, if a hobo jaywalks in front of my self driving car, too close to stop, what is the score for squashing the hobo? What is the score for swerving off the cliff to save the wino and kill me?
Have gnu, will travel.