Slashdot Mirror
The Problem With Self Driving Cars: Who Controls the Code? (theguardian.com)
schwit1 writes with Cory Doctorow's story at the Guardian diving into the questions of applied ethics that autonomous cars raise, especially in a world where avoiding accidents or mitigating their dangers may mean breaking traffic laws. From the article: The issue is with the 'Trolley Problem' as applied to autonomous vehicles, which asks, if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do? The problem with this formulation of the problem is that it misses the big question that underpins it: if your car was programmed to kill you under normal circumstances, how would the manufacturer stop you from changing its programming so that your car never killed you?
Make the program non-free of course. We may even have to make a law that car control software should always be non-free.
...I find it very hard to understand how people drive in modern cars which have so much closed source programming. As far as I'm concerned, if I can't take something apart and have a reasonable idea of how it works - which I can just about do, with an old enough vehicle - I'm not going to trust my life with it so directly. I similarly don't accept elective surgery from a proprietary robot which carries on the procedure without step-by-step oversight from other expert humans in the room.
nowadays you got the freedom to drive like an asshole - just give the car a user selectable setting, if it should preserve your life at all costs, preserve the life of others or make a decision that minimizes overall damage (but may harm you) - if you select the "me first" option, you are responsible for your car mowing down a row of krishnas. "simple" as that.
Either annual or ongoing, if easily enough done the police would probably do it. You change the code so the car doesn't, say, drive off a cliff instead of straight into the middle of a class of school girls (just to make it clear, I'd drive into the kids. It's my car after all and facing the choice between killing a dozen kids and me, the rugrats croak), then this is an illegal modification of your car and it is no longer considered safe for traffic and shut down.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
- how does the manufacturer stop me today from modifying my car so it endangers others (e.g. mount a flamethrower on it) ?
that the U.S government will eventually require manufacturers to leave the car "open", so they can stop any self-driving vehicle with a remote signal, or even alter its software while on the road. For reasons of national security, of course.
Another thing. How are these various autonomous car software platforms going to interact with one another.
It's one thing to build in recognition protocols for your own vehicles, so that multiple vehicles of the same type act in a concerted manner.
But what happens where you have four or five different codebases? How are the notoriously closed car manufacturers going to deal with car behavior from another system?
I can foresee some rather nasty interactions. Head on collisions where one car tries toavoid by a juke left to avoid while the other tries to juke right.
Chas - The one, the only.
THANK GOD!!!
This is just sensationalism. The real issue is that, if people are willing to give up their responsibilities to control a vehicle, they necessarily give up their freedom to decide how that vehicle behaves in certain situations. If you want to decide how a vehicle behaves there are probably two options: get a manually operated vehicle, or build your own "automatic" vehicle with your own rules. But good luck on that latter; just as there are regulations on acceptable behavior with manually-operated vehicles*, I bet there are going to be regulations on the "rules" automatic vehicles are going to have to include.
*Yes, you can choose to save yourself and drive into a trolley or crowd. But you will also have to suffer the consequences of that decision. You might be alive, but if you intentionally chose to harm others, you might not like where you spend your life.
"There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
People, in America at least, will absolutely bust a gasket once the first actual deaths roll in and some egghead behind a desk in some remote part of the world "decides" who lives and who dies. Until it happens people won't care. Americans are very independent and a me first kind of crowd so it may be the righ ^h^h^h^h profitable choice for manufacturers is to simply protect the occupants no matter the collateral damage as long as they can't be held liable.
Who controls the code? Maybe you, maybe them. If you tamper with it, you're responsible. Otherwise, they will probably be responsible for its behavior. But the computer is not going to be "programmed to kill you", that is bollocks. The computer is going to be programmed to follow the law. That means that it's going to be less likely to be at fault in an accident to begin with, that it's going to be more likely to successfully mitigate the accidents it does get into, and it means that rather than being programmed to kill you, it's going to be programmed to stay in the lane and hit the truck rather than swerve and hit the pedestrian because to do otherwise would be illegal — not just because of the pedestrian, but because of the lane marking. That is not remotely the same thing.
The car will be programmed to do its best not to kill you, and that's going to take the form of yielding gracefully to fuckheads rather than getting in an accident to begin with.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
... there are some freeways where the top speed is 55 and others where it's 70+.
My Garmin frequently gets the speed limit wrong for a road and if fully automated cars rely on that data, then I'll have to override the car in order to comply with the law. Because, the manufacturer will NOT be responsible for any tickets.
Who controls the code? They do. Just like they do now. How many people here are actively changing the code in their cars? Legally they are responsible for it.
Oh you chipped your car? Well now you're responsible.
Seriously the anti-car crap is getting ridiculous, as is the question of ethics. Car making a decision to kill the driver? Car breaking the road rules? When every car is driven perfectly according to the rules the death rate will be decimated and bystander accidents will be treated in the same way as any other idiot stepping into a cage with a running robot arm.
I don't understand how people have made this so complicated.
Would you buy a car that came equipped with an explosive that would, under certain circumstances, explode and kill the driver?
Are you talking about 1234yf?
if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do?
How about you tell us what should a HUMAN driver choose in a similar situation first, before you ask what should a computer do?
These kind of stupid questions are well, stupid. And they come up often simply because there is no real valid worry about autonomous cars. Humans make lots of mistakes and having a computer drive would remove a whole range of avoidable accidents. Worrying about a few boundary cases is as stupid as all the "what if my car is burning and I need to get out quickly?!" objections to wearing seat belts. It is unfounded fear that is not based on facts.
Oliver.
In Switzerland, you are required by law to help if you see a person in danger. However, it is understood that you are to make sure that you can operate safely first. It makes no sense to go in with the best intentions only to produce a second victim for the firefighters to rescue.
Thinking that further, it is clear that your car cannot take responisibilty for other participants in traffic since it cannot control them. It will save your life at all cost. Now if the decision lies between possible injury of yourself and death of another, you WILL take a broken leg and cracked ribs.
Human nature dictates that self-preservation comes first. Nothing will ever change that on any kind of broad, fundamental level.
Clearly a question posed by someone that thinks autonomous cars a just a list of deterministic instructions.
Machine learning is information theory, it will act in unexpected ways and it will make mistakes - it has to if you expect it to solve intractable problems.... if you design the NN well, then it will do far less unexpected things and make far less mistakes than a human.
How much sleep have you lost over the engineering decision to make trains so large and heavy that the simply CAN'T stop for pedestrians and other vehicles. Yeah, I thought so. People will kvetch about how self driving cars are programmed right up until they become every-day objects an after that they'll be just as accepted (benefits AND dangers) as trains are today.
Self-driving cars are largely built using machine learning; they aren't "programmed" to make such choices explicitly. Corey Doctorow should perhaps retire from science related writing if he doesn't understand such basics.
It's also doubtful that most human drivers are capable of going through a careful ethical analysis in the middle of an accident.
The Trolley Problem is a red herring that distracts from the real danger: government remote-controlled detainment of political opponents, as depicted in Minority Report. Plus, any number of variations: script-kiddies hacking, drug cartel kidnapping, kidnapping/trafficking of women/children, murder-for-hire (drive off cliff), nation-state espionage and assassination. When major crimes, and not just credit card scams, become available to the push of a button, the risk threshold to the criminal is lowered for heinous crimes.
Why would an autonomous car allowed to reach such deadly speeds around groups of pedestrians without knowledge of fully working controls and brakes? The safety systems should trigger long before such problem presents itself, and if the worst does happen, the priority should always be to protect the driver and the vehicle first to ensure it remains under control and stops as safely as possible in order to lessen additional damage occurring outside of the narrow decision space. Otherwise there will be mannequins, robots and possibly robotic mannequins on the roads. And blood, lots of blood.
I'll give you a hit we have fully computer controlled manual gearboxes for a long time now. Running a clutch and changing gears it not that hard of a problem. You get all the efficiency of a stick shift with none on of the annoyance factor.
No sir I dont like it.
When people release new custom ROMs for these cars it will be nice. ..... *CRASH*
"Sorry, the OS encountered a problem: Accellerate.app is not responding"
Do you want
I'll give you a hit we have fully computer controlled manual gearboxes for a long time now. Running a clutch and changing gears it not that hard of a problem. You get all the efficiency of a stick shift with none on of the annoyance factor.
One person's annoyance is another person's fun. A self driving car would be really annoying to me.
If I am forbidden from hacking my car's software will I be unable to stop it when:
Simple: ROT13 (or something like it) it and invoke DMCA. If the car than does not kill you the lawsuit will whipe you out.
And yes, I wish that would be a joke. :-(
Our society will probably never accept the principle that pedestrians are responsible for their own safety around cars. Whenever I drive near shopping areas, people just walk out like they own the road. Even at a cross walk, they often just start walking without any regard for courtesy or the laws of physics. Heck, even the weather often doesn't stop them such as thinking that maybe it's not wise to jump out in front of any driver when the road is wet or icy. A lot of people are just too stupid to be trusted around human-driven vehicles. Self-driving vehicles will kill them left and right, and so the law will have to be firmly on the side of the vehicle.
Does it tell me when to change gears or will the gearstick flop around by itself and do I have to move my foot out of the way so it doesn't get hit by the clutch pedal?
Nope, while manual they are not mechanically linked to the gearbox. Pressing the clutch and flailing the gearstick will switch the bits, telling the electronics to switch gears. The actual switching is delayed a revolution or two to not damage the gearbox.
Of course the gear can be changed without the pedal and stick being involved.
it misses the big question
A better question for an IT forum would be to ask how the hell do you test whether the implementation (of which party to kill) works as designed?
It should be immediately obvious to anyone in a capitalist society that who dies is a cost-option. Let's say that opting to save the car's occupants comes at a $1million price premium on the cost of the vehicle.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Luddites had it right.
I did some quick math, I think $17.5 billion plus covering medical costs (per incident) sounds reasonable.
I expect that hacking a car's vehicular AI will change its insurance rates. Insurance companies charge insurance based upon known risk. A vehicular AI that has been rated by an independent rating agency (UL, etc) is a known quantity and an insurance company will be willing to insure it.
A home spun vehicular AI will be considered an unknown risk and will be treated as such by the insurance companies. If you want to roll your own vehicular AI, feel free but you'll be responsible for having the AI rated and certified. Otherwise the vehicular AI will be treated as a "High Risk" AI and your insurance will reflect that.
Architectural plans are like computer source code with a couple of differences: You only compile once.
Unfortunately for you, everybody else has ruined driving for you by being a dick on the road. Welcome to the future of self driving DRM locked cars!
A human driver will always choose self preservation even if it means killing others, so why should an autonomous car behave any differently?
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
VW (and others?) were caught tampering with the engine code to get more performance while cheating on the emissions. It's problems like this that make the question of who owns the code, or who reviews it, very relevant...
That's precisely the point: sideswiping the car next to you is such a risky manoeuver that machine nor man can probably make that judgment call very well, with any vehicle.
That depends - if there is an oncoming lorry in your lane and the only way to avoid it is to side swipe the vehicle next to you that's what I would do to avoid what looks like impending death. That's what I would expect a self-driving car to do to: take whatever it perceives to be the lowest risk to the health of the car's occupants because that is what a human driver would do instinctively. To do otherwise and you are only one step above Futurama's suicide booths. How long will it be before you get some idiot who walks out in front of car which then causes it to swerve into an oncoming lorry, killing the occupants, where a human driver may just have hit the idiot? The advantage that a smart car would have though is that it could perhaps tell the lorry what it is going to do, get confirmation back from the lorry that it too will swerve and then both vehicles synchronize their motion to minimize casualties...but preservation of occupants must still come first as it would for a human.
Don't allow individuals to own self driving cars. Only allow them to be rented as taxis or leased by the company that designed and built them. Not only is this more efficient from a utility standpoint (the taxi part, anyway), it solves the legal liability problem. There is precedent for this in the GM electric car as well as the Ferrari 599XX.
We make fun all the time by saying "Think of the Children", but I'll bet if you polled a sizable group of people in some hypothetical You could save one person, an adult or a child scenario, probably most would save the child and leave the adult to die.
True, but suppose we change the "adult" to your wife/husband who is in the car with you and the "child" to a 12 year-old running away from a policeman and who dashes into the street. Do you swerve into the oncoming lorry and kill the adult you love or hit the child who was probably a delinquent? In that situation I think you'd get far more people saying they would protect the adult they love over the child: the closeness of the relationship with the people affected is a huge factor.
This is the problem with a car making life and death decisions which might prioritize the lives of those outside the car higher than those in it. The people you are driving with are those you are far more likely to have close relationships with and so want to prioritize. Indeed even if we stick to the purely abstract case suppose the car had a child in it? How would it know even that unambiguously and, without that, even the simple abstract case becomes impossible since it might be a choice of a child vs. another child.
No, it can not be users selectable unless cars that have chosen the user first mode are not allowed on public roadways. When you drive on public roadways you accept that you have to obey certain "rules of the road", such as speed limits.
You do realize that this would automatically disqualify all human drivers right? Humans will always prioritize themselves and will not always obey the rules of the road. A computer which prioritizes is occupants but which always obeys traffic rules would still be a huge improvement.
Is to allow the driver to answer a set of preferences including the question of should the car give higher priority to preserving other lives or the drivers.
Then it's the driver's choice just as it would be if they were driving the car.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
I think it has something to do with it being a symbol of masculinity or something.
Wrong. The car is a tool of freedom. Not a symbol, an actual tool. Not a male thing, but freedom for both sexes. There is no shortage of females who had strong emotions for their first car, that first experience of a very tangible freedom.
Muscle cars often have a mostly male attraction but it is not necessarily a symbol of masculinity, something that changes the interpretation others have for the owner/driver. For example muscle cars don't "change" one's status in high school cliques. One would simply be the weakling, geek, etc with a nice car. Where masculinity comes into play is more of an internal thing. A muscle car can deliver fun, excitement, adrenaline, etc.
If they keep building straw men this size, I'm gonna invest in straw.
No car is going to decide whether to kill you or kill someone else. It doesn't know who you are - or if it does, it's legally it's own person, and will take it's own responsibility.
In addition, we are talking about accidents, in which, by definition, it's difficult to know what's going to happen. Differing programming instructions probably won't make a hill of beans worth of difference, because at some point, the car is out of control. The operator inputs, be they artificial or corporeal, don't really matter anymore.
But by all means, keep acting as if you, the car, or anyone else can control the outcome. I'm sure it will make you feel better.
Why in the world would we require autonomous cars to answer hypothetical questions on morality?
"Grab them by the pussy" -- President of the United States of America
there is a line of reasoning used by some people. they buy the biggest vehicle- SUV, truck, car. the thinking is if theyre in a crash with another vehicle theyll survive because they have the bigger vehicle. too bad so sad for the other vehicle & its occupants. so in a sense some cars onthe road already are pre-programmed to only choose safety of driver >> safety of anyone else on road.
I expect it will go far beyond that in two ways.
(1) The vehicle will no longer be in compliance with regulations that permit the privilege of taking your vehicle onto a public road. Merely driving it on a public road may make one vulnerable to civil charges, loss of driving privilege, confiscation of vehicle, etc. Much like drunk driving regardless of whether an accident occurs or not.
(2) It will void your insurance coverage and make you fully liable for anything that occurs. There will probably be no "hacking" coverage available to a regular person. Only certain research entities, including the auto company's, will probably be able to get coverage regarding software related incidents.
Luddites had it right.
Stick with horses? That's also a self driving vehicle that you merely issue driving commands to.
It's just like cell phones. You'll root your car, install a backup system, then you'll install cyanogenmod, and that's that. It could be called something else, but the software will work the same basic way.
The three laws of robotics state:
To me, the interesting ramifications of these laws in many stories, and one movie involving Will Smith, are more than enough to answer all questions regarding autonomous technology and responsible use. I have no idea how, as a programmer, these would actually manifest themselves in code...
The article brings into light a problem without actually naming it: law. I think that this problem is that different societies will choose different approaches to regulating, controlling, and outlawing technologies, and the same society can make different decisions depending on it's point in time. The society now in the United States makes the most sense as a place to try and sell new technologies to, since it's the richest and has historically been permissive towards the rights of the creators'. Without being able to answer financial liability questions in a meaningful way, this may not continue to be the best place to develop new, and risky, technologies, in the future.
I don't believe the problem of who owns the code is actually a problem. We have many, many different models of code ownership in effect, and the entity which creates the code can choose how they want to "own" it. The actual problem to me seems to be who do I sue when I don't like an outcome?
I pray the 0th law never comes into effect.
I believe that automotive impact testing done using this Poster, and any supportive souls as, and pardon the pun, crash dummies. By concentrating only on cost one easily losses focus on perspective benefits. Cost Only Solutions are starting to go on its long path to obsolescence, but it is clear that it is already headed in that direction.
Will Linux be used in the analysis?
if your car has to choose between a maneuver that kills you and one that kills other people, which one should it be programmed to do?
How about you tell us what should a HUMAN driver choose in a similar situation first, before you ask what should a computer do?
^ THIS.
Cory's article completely misses the point. Or rather, he brings up the Trolley Problem and then moves on to his own point. The reason it's an ethical dilemma is because it brings up ethical issues. That dilemma doesn't change just because a computer is involved, it just shifts the burden to the system. An obvious solution that would probably occur for the first 10-15 years is "Transfer control back to the human in the event of an emergency", which of course just puts right back where we started.
My biggest concern is that we'll get a bunch of hot shot programmers who don't have the training to UNDERSTAND that they're encoding ethical decisions into their code, or don't understand the weight of them, before they do so. "Solutionism" on the part of Bay Area techies who've never taken a philosophy or ethics class in their life but are convinced science can solve all the problems in the world is how we'll get into messes like anomalous cars, systemd, and sex-changing velociraptors.
Hire a Linux system administrator, systems engineer,
Car DRM = dealer only service and based on how evil they want to get all the way down to tires, windshield wipers, oil changes, lights.
What should a car be programmed to do? Agreement on this question for along term goal would be nice.
Some discussion of what options would be possible in the next ten to twenty years might be interesting.
Is analyzing input from sensors ( that can be place onto a car ) and detecting a situation where someone will soon be fatally injured or instantly killed a solved problem? Is it expected to be soon?
The world is such a complex place, that AI seems to be having the most success using learning algorithms instead of approaches such as coding all possible cases or basing actions off of details simulations. I'm under the impression self-driving cars currently are based off learning algorithms. If I'm right on this, then a self driving car can't simply be given the instruction "kill the driver if it will result in minimum deaths". The algorithms would need to be given real or simulated input of situations where a death is unavoidable and shown how to handle the situation.
Sadly, it was only at this point in writing my response that I skimmed the full article. My points are still valid but off the main topic of the article.
If terrorist activity is detected, should the AI drive the car to the incident so the driver can assist in fighting the terrorists, or should it flee the area?
I'm voting for everyone piling on, and the AI could allow access to a locked weapon compartment.
Hit the bus. That was easy.
Mu. First, you would never design the car to murder its driver. And if you accidentally did, then you would also make sure the driver can maintain its code, in order to correct this error and any other such errors.
Article then goes on to talk about a lot of crazy things ("make cars suck as much as iPhones!") based on the silly premise.
All this and more can be solved by putting automated cars on RAILS. With sensors and braking systems designed in both rail and car.. If the car and rail exchange data...
repost (Herblock is on vacation)
I know autonomous cars will be "oh so safe". At the moment I'm just as worried about what these things will make people do to people.
[OPENING OVERTURE ]
Your driver liability insurance policy has come up for review. We have been recently been acquired by AAAA, the quadruple-A company -- the "Autonomous AAA of the future" and what that means for you as a member is -- it has never been easier to upgrade to an a-car! Financing is available! [link] Due to increasing pressure in the political, legal and underwriters' arenas, we regret to inform you that the cost of your driver policy will be rising this quarter in order to begin collection of fees for the Federal National Driver Insurance Pool, and rising at a steady rate thereafter. It will continue to rise over time despite your [good to excellent] driving record. Now that the Autonomous Vehicle Safety Act is law, and blanket liability accident investigation procedures have been approved by Congress, the legal liability of autonomous vehicles is capped nationwide. While this grants the manufactures freedom from risk of direct criminal penalty and potentially unlimited civil liability, it places human drivers in a difficult position. Most a-car accidents will, of historical necessity rather than actual circumstances, be "no-fault". Since human drivers and any victims claiming injury from them are still obliged to use traditional law enforcement and legal means of redress -- and the cost of these continues to rise -- underwriters are pressuring insurance companies to drop human drivers altogether. We do not intend to do this, but we can no longer provide policies for extended periods. Your new maximum policy period is now [one month]. Thank you for insuring with AAAA.
[INTERMISSION \]
Meanwhile...
Dear editor: DRIVERS cause accidents. A-CARS prevent them. That's what the billboard says -- and if Howard County Referendum passes this September manually operated cars will soon be a thing of the past here. What started as a discussion at a hearing after last year's tragic accident grew into a full heated debate, and to think it all started with the parents who provide their children with a-cars pinning the blame squarely on other peoples' children. But then, after co-opting the national campaign with its slick literature and canned answers for everything -- NOW the fault is with human drivers themselves. And then in an astounding feat of lunacy they claim that it's only fair to place the blame on everybody. Not just the drunk, the aged or infirm, the inexperienced, the distracted or the just plain stupid. But no one's stupid in their book, we're just behind the times is all. They are like the drum majorettes of some utopian humanist parade. I say, SAVE US from these rich hippies, their weird toys and their broken ideals. Now I know a lot of these people, even like some of 'em, but aside from this national 'sideline the humans' campaign they're pushed at us (and WHO is paying for those TV spots I wonder) let's not forget that this debate started around kids. Kids who need to learn to drive as surely as they need to learn to push a pen and spell their name. It's like swimming, who would discourage their own children from practice in swimming, to become expert swimmers, because water is dangerous?? Every kid will need to drive some day, or suffer harm or hardship by not knowing how. These a-car parents even forbid their kids from riding in cars being driven by folks they've grown up with, trusted for years. At the parent conferences we even sit on opposite sides of the table, we can barely be civilized even, because this crap has gotten so deep. Well I say they are making a big mistake and don't seem
<blink>down the rabbit hole</blink>
they cannot stop someone with the skills to rewrite the code.
binary key and locks are finite in their nature that they must occupy a memory space and if this memory is accessible by the system well its game over , sure you can offshore the code and only have the loaded copy in runtime memory but anykind of security starts with physical security , and whatever physical security you put on a car wont last long in from of a propper cutting torch in my garage.