Halting Problem Proves That Lethal Robots Cannot Correctly Decide To Kill Humans

KentuckyFC writes: The halting problem is to determine whether an arbitrary computer program, once started, will ever finish running or whether it will continue forever. In 1936, Alan Turing famously showed that there is no general algorithm that can solve this problem. Now a group of computer scientists and ethicists have used the halting problem to tackle the question of how a weaponized robot could decide to kill a human. Their trick is to reformulate the problem in algorithmic terms by considering an evil computer programmer who writes a piece of software on which human lives depend.

The question is whether the software is entirely benign or whether it can ever operate in a way that ends up killing people. In general, a robot could never decide the answer to this question. As a result, autonomous robots should never be designed to kill or harm humans, say the authors, even though various lethal autonomous robots are already available. One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of.

Comments

I think

[fyngyrz]

I'm just going to reformulate the problem by considering idiots who use unrealistic, not-supported-by-evidence premises to make general statements as one that calls for sending killer robots after said idiots.

Re:I think

[ShanghaiBill]

The premise of TFA is that killer robots need to be perfect. They don't. They just need to be better than humans.

Which is more likely to shoot a civilian:
1. A carefully programmed and thoroughly tested robot.
2. A scared, tired, and jumpy 18 year old soldier, who hasn't slept in two days, and saw his best friend get his legs blown off by a booby trap an hour ago.

Re:I think

[Anonymous Coward]

1, when ordered to shoot civilians.

Re:I think

[fyngyrz]

In that case, the robot didn't make the decision. So, no.

Re:I think

[Anonymous Coward]

This interestingly both misses and illustrates the interesting points here.

The article is showing that there's no generic algorithm that can be used to thoroughly test the robot (analyzing the program to determine if it will choose to "kill" reduces to the halting problem).

However, as the summary points out, there's also no known algorithm that can do the same for humans, and humans usually behave less consistently than software so chances are testing humans to an acceptable degree of certainty will be harder than testing robots to the same degree of certainty.

As a side note, there are algorithms to solve the halting problem for specific sub-classes of turing machines so chances are it's possible to build a "murder oracle" that would when fed the software for a specific class of robot and the input to be tested be able to determine if the robot will kill.

Re:I think

[Anonymous Coward]

1, currently. No doubt about that.

Autonomous robots cannot even discern civilians from combatants, nor can they reliably distinguish between friendly combatants and enemies without some special tricks like tags or transmitters.

Re:I think

[shaitand]

Excellent, sending a drone for you now.

Re:I think

[shaitand]

Both are pretty likely. Let's start by defining civilian. Is the farmer who supports the militants cause and brings them goat cheese and steel a civilian? What about the farmer who is afraid of them and does the exact same thing? What if the farmer knows the danger level and carries a gun for personal defense?

You can't compute us and them in an analogue world where the real value is never actually 0 or 1 but always a shifting value in between and usually multiple shifting values in between. YOU can't, and neither can your robot.

Re:I think

[orgelspieler]

True, but which is more likely: "A carefully programmed and thoroughly tested robot" or "A shoddily kludged together POS that failed most rounds of safety tests, but passed the one (in demo mode) when the people signing the checks showed up to witness."

Re:I think

[Anonymous Coward]

bingo!

Re:I think

[davester666]

Your premise that the killer robots need to be better than humans is incorrect.

The people that will buy and deploy these killer robots don't care if they make mistakes. As long as they kill the bad guys, killing some "innocents" doesn't really matter.

For example, drone strikes for the US in middle east. They regularly kill a number of civilians and the so-called target may not even be present. As long as the good guys [defined as members of the military of the 'good' side] aren't killed, it's not a problem.

Re:I think

[matbury]

What makes you think that the people who intend to use these robots, i.e. the US military and its allies, actually care about killing civilians? We are talking about the same organisations that ordered the fire-bombing of major cities with no strategic importance in Germany and Japan during WWII, aren't we? Who carpet bombed Vietnam and Cambodia? And the same ones that are now killing insurgent grandmothers and children with drones and cruise missiles? It seems they're far more interested in killing civilians than anyone else. Either that or they're incompetent.

Re:I think

[Austerity Empowers]

However, as the summary points out, there's also no known algorithm that can do the same for humans, and humans usually behave less consistently than software so chances are testing humans to an acceptable degree of certainty will be harder than testing robots to the same degree of certainty.

This right here is why I would never have funded this investigation. It's idiotic by design, there is absolutely no good criteria to decide "who needs to live and die", and thus its a very highly subjective subject. We argue this in court, we frequently don't agree on the verdict.

Killer robots are going to happen, and we're going to trust them every bit as much as any given human being. And they will have advantages:
- Scope can be limited: a robot may be able to kill, but it may not be able to move... we'll set it up and assume that it will kill anything in a given area, and maybe program it to try not to kill things that fit a certain criteria. But we won't rely on that
- Killer robots can be turned off. Killer humans can not, and some have a hard time going from killer mode back to civilian mode
- Killer robots will be far less likely to induce collateral damage. They won't forget that bullets go through walls. Humans raised on movies somehow think a bit of drywall or plaster is impervious.

There are numerous advantages. What we probably will never build, on purpose, is Terminator bots that roam around the street looking for "bad guys".

Re:I think

[ardor]

The former if any incident involving a killer robot that went haywire results in huge penalties and murder sentences I guess..

Re:I think

[tendrousbeastie]

Surely, all of those are definitions of a civilian.

Re:I think

[bmo]

Product liability never results in anyone being actually responsible for the death going to jail or huge penalties.

A multinational might> pay out a couple of million in product liability, but then it will just be chalked up to the cost of doing business.

If the multinational is a defense contractor (BAE, Raytheon, Lockheed, General Dynamics, etc), it will all be swept under the rug and more money will be thrown at the contractor to "fix" it.

That's the reality.

--
BMO

Re:I think

[meta-monkey]

Also, built by the lowest bidder.

Re:I think

[Minwee]

"Will no one rid me of this turbulent priest?"

Oops. Looks like the human ordering it didn't make that decision either.

Re:I think

[Anonymous Coward]

That which a bunch of Ethicists say should happen, and that which does happen, are often in dis-accord.

The military will build killer robots. The primary concern for behavior development won't so much be safety mechanisms against killing innocents or children as safety mechanisms against losing control. They don't want their robots turning on them. Beyond that, they want the robots to obediently slaughter whatever they are pointed at. This level of obedience is precisely what makes them useful as a weapon.

This will happen. Count on it, and adapt to it.

Re:I think

[binarybum]

According the government it's not a problem if they kill good guys either - they are now authorized to kill Americans on american soil if so inclined for instance.

Re:I think

[Bob the Super Hamste]

They won't forget that bullets go through walls

And by walls I assume you mean a single bullet will go through multiple walls.

Re:I think

[TangoMargarine]

So we're cutting down the criteria to not just people carrying guns, but people carrying guns actively shooting at you?

Re:I think

[Anonymous Coward]

Yep this article's premise is based on some kind of useless abstraction.

Nothing about the halting problem is specific to robots vs human beings.

These are idiots and their article is tailored towards idiots. Or at least people who may have heard of the halting problem but don't understand any theoretical comp sci.

Re:I think

[Anonymous Coward]

A carefully programmed and thoroughly tested robot.

We're talking about the same government that buys Diebold voting machines.

Re:I think

[AthanasiusKircher]

So we're cutting down the criteria to not just people carrying guns, but people carrying guns actively shooting at you?

Actually, the definition of civilian is well-defined in the Laws of War, commonly codified today in international laws by Protocol I of the Geneva Conventions.

In sum, a "civilian" is anyone who is not a "privileged combatant," i.e., basically someone (1) carrying arms, (2) taking orders in an organized military structure, and (3) following the laws and customs of warfare. (Also, usually privileged combatants are required to wear insignia.)

Someone who carries arms but does not satisfy those criteria is still a "civilian," though if those arms are actively used in support of an organized military force, he/she may be a civilian who is also an "unprivileged combatant," i.e., he/she not eligible for protection under the normal rules for prisoners of war.

So, actually the criteria are much more specific than you describe. "Civilians" can fight in wars, in which case they become "combatants," but they do not cease to be "civilians," as the term is commonly understood in contrast to organized military personnel.

As for the farmer in GGP's example, he's clearly a civilian unless he's a member of a military force. If he carries a gun but only for his own protection and does not engage in direct action against an enemy, he is probably assumed to be a "non-combatant" as well, under international legal definitions.

Re:I think

[Obfuscant]

According the government it's not a problem if they kill good guys either - they are now authorized to kill Americans on american soil if so inclined for instance.

Your statement assumes that "Americans on American soil" are automatically "good guys."

Re:I think

[ShanghaiBill]

So we're cutting down the criteria to not just people carrying guns, but people carrying guns actively shooting at you?

No. A large fraction of American casualties are caused by friendly fire. So "people carrying guns actively shooting at you" will in many cases be inappropriate targets.

Re:I think

[operagost]

If you think atrocities are only committed by the USA and its allies, you must have had a hell of an education.

Re:I think

[davester666]

The gov't defines anybody they kill as "the bad guy".

Re:I think

[Obfuscant]

The gov't defines anybody they kill as "the bad guy".

Nonsequitor, and nontrue. The government is well aware that there are innocent casualties in many attacks, just that the benefit of killing the actual bad guy outweighs the cost. Just as many bad guys believe the cost of killing innocents is less important than protecting themselves, and that's why bad guys often set up shop next to schools or hospitals -- or even inside them when possible.

But none of that deals with the leap from "killing Americans on American soil" means that the government approves of killing "good guys".

Re:I think

[blue9steel]

I would like to live in your alternate reality, can you provide directions to get there from these temporal-spatial coordinates?

Re:I think

[Austerity Empowers]

Mythbusters did something similar with a minivan and phone books I think. People are never going to be very good at this, but in theory a computer can be programmed not to make a mistake by cooler heads, or at least situated such that a mistake is impossible. I admit I'd probably shoot a gun at someone if my life was in danger, not (at the time) realizing what's behind the guy that might also die.

But still we want to continue the "killer robots" theme, I guess because we still graduate far too many liberal arts types and they need jobs too.

Re:I think

[rtb61]

Now which is more likely to occur:
1. A carefully programmed and thoroughly tested robot.
or
2. Lowest tender robot with just barely sufficient buggy code to get past the tender process. Generating a huge bonus for the psychopathic executive team when they produce 10,000 of them.

Bad luck under capitalism, corporations will never ever produce a "A carefully programmed and thoroughly tested robot", it just wont happen, it can't happen, corporate greed controlled by psychopaths guarantees it wont happen, that is the reality.

The only real reason for killer robots is not to protect people the dangerous hobby of killing other people and those victims trying to defend themselves, the real reason is of course the people you send to commit mass murder at some point refusing to do so. So killer robots are all about nothing but mass murder without restraint or conscience. Any country that produces them seriously has to be considered an enemy of humanity and should be ostracised and excluded.

Any attempt at out technologically competing other countries with robotic death machines will immediately result in the return to the development of biological weapons delivered by small stealthy drones (easy to do and of course lateral thinking means you can tackle food sources not just people).

Re:I think

[Anonymous Coward]

No to what?

It is a major and legitimate fear that if human soldiers are replaced by robots lacking moral agency, such atrocities will be come far more common precisely because there is no moral agency controlling the trigger going "what the fuck, are you actually going to commit mass murder?"

Any weapon controller smart enough to identify innocent humans as humans should bear no less responsibility than a human weapon controller to not kill innocent people, just as we expect soldiers to not gun down crowds. Any weapon controller less smart should not be permitted to pull the trigger, period.

Let us propose a "reasonable human test" as a moral agency requirement: If you want to have a robot that is capable of being a self-acting weapon for any reason - Be that because a criminal in a suit told it "shoot them" or because it's a sentry - its controller must pass the reasonable human test: "No robot may fire a weapon that a reasonable human in its position would not fire." If your robot fails to pass such a test or lacks such an agency, then both the designers and the controllers are personally liable for any harm it may cause.

And if that makes robotic soldiers de facto illegal until the advent of human-equivalent AI... good.jpg. If I use a PT head, a webcam and a PC to fire an AR-15 at the centers of detected motion, I should be not on iota less guilty of murder when it eventually shoots an innocent passerby than if I'd ambushed them myself. Someone who designed the ED-209 should be even more screwed since they can't try and argue that it was merely criminally negligent homicide...

Re: I think

[Anonymous Coward]

3. Quake Frogbot code

Headshot across the map every time.
Just be sure not to let it out near civilians. Or camp the quad.

Re:I think

[flopsquad]

Bingo! Parent's post is the definition of "Informative". There should be a mechanism by which enough mod points over +5 will turn a post into a sticky at the top of the discussion.

Re:I think

[rioki]

I think we are VERY FAR from robots that really autonomously make a kill decision. This is the basic AI fallacy that imply that we will soon have systems that can "decide on their own". But that will probably happen never or not very soon.

You need to look at real practical implementations of weapon systems (i.e. killer robots). Any weapon system will be integrated into the command and control structure of the army. The system will have different operational modes (simplified), such as stand down, engage specific target, engage all non friends and engage everything that moves. The key here is a robust friend / foe detection systems, something that is already almost perfectly solved for aircraft. Sure you could try integrate an effort to include combatant / civilian detection mode, but since that can probably be easily fooled, it will be an optional feature that will be disabled once the enemy tries to fool it.

This philosophical pondering "can a robot make a kill/no kill decision" is as you point out meaningless, since humans can not make this decision either reliably. The real question is, can we design a robust friend / foe system? That is a solvable engineering problem.

Re:I think

[Anonymous Coward]

Wrong. Completely wrong.

A killer robot (or even merely "smart weapons") is a system that isn't just the specific weapon node itself - intelligence driving it (human or machine extracted) is part of the system and is actually the weak link. From ab initio, you can PROVE that drones primarily kill innocent civilians when used in non-conventional combat. Utterly and completely provable that >80% and upwards of high 90s% of those killed MUST BE non-combatants that are completely innocent. In other words, drone warfare is precisely genocidal warfare and a process of committing the war crime of genocide.

The component accuracy such as targeting must be upwards of "5 nines" or better (99.999% accurate) to even achieve a system accuracy of merely 60-80%. Effectively the component really does need to be "perfect". The problem is Bayesian statistics. You can have perfect component accuracy (e.g. smart weapons will hit their aim point with better than 99% accuracy) BUT the probably is that the system accuracy is reduced by the weakest link, typically the aim point/targeting decision process which is quite poor with humans and WORSE with machine decision making.

If you were in a symmetric, conventional combat role, this might be acceptable. But in the WoT, it's asymmetric, guerrilla warfare. The system accuracy is reduced by the radio of combatant/terrorist/freedom-fighters/etc. to the ratio of the total population they are embedded into which is typically at least 1:10^6 or even smaller. This is why it is inevitable that drones strikes will primarily kill wedding parties, woman and children and perfectly innocent people in >80% of all strikes.

And guess what? This has been largely validated to be exactly the ratios based on objective and empirical post-strike investigation. The US has tried to thwart this by making up serious stupid criteria for who is a "terrorist" by saying things like "terrorists/insurgents are any male between the ages of 15 and 45". If you really believe this is legitimately, you have some seriously sociopathic tendencies. Look around you and ask "Are all the males I see around me 'terrorist' to ISIS, just because?" Obviously not but this is the same logic being used.

Re: I think

[Anonymous Coward]

The human has moral responsibility.

Re: I think

[Anonymous Coward]

It's not as bad as you suggest. A robotics company that was responsible for many unintended, innocent, deaths would encounter a pr nightmare and resulting drop in stock price, marketability, etc.

Re:I think

[TangoMargarine]

Thanks for the infodump.

Then there's also the question of, how many wars have there been lately where both sides were clearly identifying themselves? Those fighting have gotten the hint that it's a dumb idea to engage bigger powers in anything other than asymmetric/guerrilla/whatever warfare.

Re:I think

[shaitand]

"Actually, the definition of civilian [wikipedia.org] is well-defined in the Laws of War, commonly codified today in international laws by Protocol I [wikipedia.org] of the Geneva Conventions."

Really? Do you define your moral compass to directly align with the law? Let's look at what you said.

"So, actually the criteria are much more specific than you describe. "Civilians" can fight in wars, in which case they become "combatants," but they do not cease to be "civilians," as the term is commonly understood in contrast to organized military personnel.

As for the farmer in GGP's example, he's clearly a civilian unless he's a member of a military force. If he carries a gun but only for his own protection and does not engage in direct action against an enemy, he is probably assumed to be a "non-combatant" as well, under international legal definitions."

Define organized. Define military, Define what constitutes an insignia. Your definition of privileged combatant excludes every combatant relevant to a modern war (at least for the US which is where the automated death machines are coming into play). Terrorists groups aren't in organized military structure, they don't wear insignias, and they don't follow the laws and customs of warfare. So by your definition of civilian, they are all civilians and in some cases both civilians and combatants.

"As for the farmer in GGP's example, he's clearly a civilian unless he's a member of a military force. If he carries a gun but only for his own protection and does not engage in direct action against an enemy, he is probably assumed to be a "non-combatant" as well, under international legal definitions."

Probably assumed to be a non-combatant? Define direct action. Define "actively used in support." He is using his gun for protection to supply the combatants. Supplying the combatants is obviously supporting them. Arguably, carrying a gun for protection while transporting their goods IS actively using it to support them.

How is a drone or even a soldier supposed to know the difference between the men carrying the guns? If he is standing with the other men when the soldier fires on them and he fires on the soldier is he a combatant and okay to kill then, even though he is a civilian acting in self-defense? What if the soldier is replaced with a machine? Now it's a human being defending his life and not putting another at risk.

Re:I think

[Rich0]

Then there's also the question of, how many wars have there been lately where both sides were clearly identifying themselves? Those fighting have gotten the hint that it's a dumb idea to engage bigger powers in anything other than asymmetric/guerrilla/whatever warfare.

And this would be why the large powers that worked out the Geneva Conventions wrote them in a way so as to offer no protections for people engaging in guerrilla warfare. It isn't like this is something new. There just haven't been any large-scale wars in a long time.

Keep in mind that nations are basically defined by their sovereignty. They aren't really subject to laws in the same way that ordinary citizens are. International law is really just a set of protocols everybody tends to follow because it works to their benefit. The Geneva Conventions basically say that if you don't firebomb our civilians we won't firebomb yours, and to help make that happen we'll both agree to not have civilians fighting guerrilla wars. When one side in a war decides to ignore them, they do so at the risk that the other side will also decide to ignore them.

Also, I think the effectiveness of guerrilla warfare is sometimes overstated. It works very well when the state you are fighting doesn't really have a big stake in the game, and is governed by some kind of democracy with a free press and all that. It only tends to work against occupying forces, since it lacks the ability to project power. It also relies on the fact that the occupying power desires to have peaceful relations with the existing population.

If you tried to use guerrilla warfare under other circumstances you would probably not be as successful. If you were fighting against a opponent governed by a dictatorship who occupied your country solely because they need your natural resources they might just decide to nerve gas the entire local population and move in their own workers to exploit your resources. If you were fighting against even a democratic opponent but they felt that their way of life was at risk, they might resort to brutal measures to put down the revolt (maybe they're engaged in a larger war and need your country as a base of operations). If some local tribe had attacked an essential US base in the pacific in WWII I doubt the local marines would have put up with it.

Re:I think

[Rich0]

Really? Do you define your moral compass to directly align with the law? Let's look at what you said.

He was explaining the Geneva Conventions/etc. That isn't an argument about morality.

Define organized. Define military, Define what constitutes an insignia. Your definition of privileged combatant excludes every combatant relevant to a modern war (at least for the US which is where the automated death machines are coming into play). Terrorists groups aren't in organized military structure, they don't wear insignias, and they don't follow the laws and customs of warfare. So by your definition of civilian, they are all civilians and in some cases both civilians and combatants.

I'm sure those terms all have definitions, and as you point out they exclude most of the folks the US tends to end up shooting at of late.

The Geneva conventions were written so that nations with big armies could decide who they are/aren't allowed to shoot at. It isn't an accident that the definition for privileged combatants exclude terrorist groups and such. Forces not wearing uniforms are basically not afforded any protections under the conventions.

How is a drone or even a soldier supposed to know the difference between the men carrying the guns? If he is standing with the other men when the soldier fires on them and he fires on the soldier is he a combatant and okay to kill then, even though he is a civilian acting in self-defense? What if the soldier is replaced with a machine? Now it's a human being defending his life and not putting another at risk.

These problems aren't unique to drones. I can't specifically answer your questions as it is a bit unclear who is shooting at who in your description above.

If you're a civilian and somehow get caught up in a firefight with an organized group of soldiers on the other side, your best bet is to run or try to surrender. If you have a gun using it in self-defence is almost certainly suicidal. That isn't going to change if you add robots to the mix. This has nothing to do with morality - it is just a fact that when you shoot at soldiers they're going to consider you the enemy and they're almost certainly more likely to come out on top.

Re:I think

[matbury]

If you think atrocities are only committed by the USA and its allies, you must have had a hell of an education.

Yeah, the US and its allies have been training other countries to do the really dirty work for them for decades, e.g. "The School of the Americas" http://en.wikipedia.org/wiki/W... Also, the British Military trained and the CIA supported the Sadam Hussein Ba'ath party regime (They were trained at Sandhurst military academy on an on-going basis). In Indonesia, the CIA and US military also supported a long series of massacres and a long campaign of oppression that continues to this day. There's a very, very long list of past and current attrocities in the world in which the USA plays a central role.

Re:I think

[shaitand]

"He was explaining the Geneva Conventions/etc. That isn't an argument about morality."

He was explaining said conventions as a retort to my claim that humans aren't able to clearly define who can be morally shot and therefore it would be impossible for them to clearly define in an algorithm for an automated killing machine.

  "I'm sure those terms all have definitions, and as you point out they exclude most of the folks the US tends to end up shooting at of late."

Probably, but just like every word in the dictionary, all definitions for subjective concepts become circular at some point. "military" is not an objective element, it is subjective and any definition is going to require subjective elements as well. Again, this is why we have judges and why they don't always agree. It's why a lawyer's talent ultimately just comes down to his ability to make a convincing argument and spin those definitions around to match the viewpoint he needs.

Re:I think

[Rich0]

He was explaining said conventions as a retort to my claim that humans aren't able to clearly define who can be morally shot and therefore it would be impossible for them to clearly define in an algorithm for an automated killing machine.

Fair enough, though the same argument would apply to having soldiers in the first place.

Also, defining an algorithm for an automated killing machine doesn't require being able to clearly define who can be morally shot. Armies today operate just fine despite the fact that nobody really agrees on who can be morally shot. :)

Re:I think

[shaitand]

"Fair enough, though the same argument would apply to having soldiers in the first place."

Why yes, so it does. I find your logic to be indisputable on this point and conclusive.

"Armies today operate just fine despite the fact that nobody really agrees on who can be morally shot."

Really? If they do so well why are in so much trouble for massive collateral damage and the inability to distinguish combatants and civilians? How is it entire villages get wiped out entirely and mislabeled as combatants on reports as indicated by wikileaks? Why did the US invasion in Iraq indirectly kill millions of Iraqi's via destroyed infrastructure like facilities that do things like purify water?

Oh I see, the metric of doing just fine only considers whether or not they manage to kill SOMEONE. Yes, lets run with that. I'm sure we can all agree on that one.

Re:I think

[Rich0]

Really? If they do so well why are in so much trouble for massive collateral damage and the inability to distinguish combatants and civilians? How is it entire villages get wiped out entirely and mislabeled as combatants on reports as indicated by wikileaks? Why did the US invasion in Iraq indirectly kill millions of Iraqi's via destroyed infrastructure like facilities that do things like purify water?

No argument that these are problems, but they're really irrelevant to the use of drones on the battlefield. War is just the result when people can't agree on things and they feel that the issue at stake matters more than life itself. Since the decision was already made that accomplishing the mission matters more than life, the matter of whose life ends up getting lost ends up being of tertiary importance.

The only way to really get rid of war is to eliminate all issues that significant numbers of people are willing to die over.

By the same logic

[Anonymous Coward]

By the same logic, computers should not be allowed in any life-critical situation. That includes hospital equipment, airplanes, traffic control, etc. etc.

Fortunately, we don't judge the reliability of computers based on the ability to mathematically prove that nobody has put evil code in on purpose.

Re:By the same logic

[i kan reed]

And let's not forget the "Better at it than humans" heuristic. As long as "Jaywalking under the influence of melanin" is sometimes a capital crime, that's not a hard target to hit.

Re:By the same logic

[tlhIngan]

By the same logic, computers should not be allowed in any life-critical situation. That includes hospital equipment, airplanes, traffic control, etc. etc.

Fortunately, we don't judge the reliability of computers based on the ability to mathematically prove that nobody has put evil code in on purpose.

In your examples, there are humans in the loop.

In this case, you have a robot trying to autonomously decide "kill" or "don't kill" when it encounters a human.

Hospital equipment - it's generally observed by personnel who after failures can decide to not use the equipment further (see Therac 25), or that changes need to be made in order to use the equipment. The equipment never hooks itself up to a patient automatically and provides treatment without a human involved. Sure there are errors that kill people unintentionally, but then there's a human choice to simply take the equipment out of service. E.g., an AED is mostly autonomous, but if a model of AED consistently fails in its diagnosis, humans can easily replace said AED with a different model. (You can't trust said AED to take itself out of service).

Airplanes - you still have humans "in the loop" and there have been many a time when said humans have to be told that some equipment can't be used in the way it was used. Again, the airplane doesn't takeoff, fly, and land without human intervention. In bad cases, the FAA can issue a mandatory airworthiness directive that says said plane cannot leave the ground without changes being made. In which case human pilots check for those changes before they decide to fly it. The airplane won't take off on its own.

Traffic control - again, humans in the loop. You'll get accidents and gridlock when lights fail, but the traffic light doesn't force you to hit the gas - you can decide that because of the mess, to simply stay put and not get involved.

Remember, in an autonomous system, you need a mechanism to determine if the system is functioning normally. Of course, said system cannot be a part of the autonomous system, because anomalous behavior may be missed (it's anomalous, so you can't even trust the system that's supposed to detect the behavior).

In all those cases, the monitoring system is external and can be made to halt a anomalous system - equipment can be put aside and not used, avoiding hazardous situations by disobeying, etc.

Sure, humans are very prone to failure, that's why we have computers which are far less prone to failure, But the fact that a computer is far less prone to making an error doesn't mean we have to trust it implicitly because we're more prone to making a mistake. it's why we don't trust computers to do everything for us - we expect things to work but when indications are that it doesn't, we have measures to try to prevent a situation from getting worse.

Re:By the same logic

[Dutch Gun]

Agreed. The authors set up a nearly impossibly complex ethical dilemma that would freeze even a human brain into probable inaction, let alone a computer one, and then claims "See? Because a computer can't guarantee the correct outcome, we can therefore never let a computer make that decision." It seems to be almost the very definition of a straw man to me.

The entire exercise seems to be a deliberate attempt to reach this conclusion, which they helpfully spell out in case anyone missed the not-so-subtle lead: "Robots should not be designed solely or primarily to kill or harm humans."

I'm in no hurry to turn loose an army of armed robots either, but saying that you can "prove" that an algorithm can't make a fuzzy decision 100% of the time? Well, yeah, no shit. A human sure as hell can't either. But what if the computer can do it far more accurately in 99% of the cases, because it doesn't have all those pesky "I'm fearing for my life and hopped up on adrenaline so I'm going to shoot now and think later" reflexes of a human?

Re:By the same logic

[Anonymous Coward]

Not sure what melanin has to do with it. For example, serial rapists should be in jail. It doesn't matter whether their "pudding pop" is Chocolate or Vanilla.

Re:By the same logic

[Anonymous Coward]

Skin color. Either the guy is making a political statement or he thinks black people are harder to see at night.

Re:By the same logic

[plover]

So how many humans have to die before recognizing the AED is faulty? If it's a subtle fault, it might be delivering a barely ineffective treatment, and confused with an unsaveable patient. The THERAC 25 failure was a bit more dramatic, but it still killed many patients.

Would we accept the same levels of failure from the Kill-O-Bot 2000? We already fire missiles into crowds of people or convoys in order to take out a single high value target. If the Kill-O-Bot was more specific than a missile, but less than perfect, isn't it still a better choice?

Re:By the same logic

[Anonymous Coward]

A few days ago there was an article posted that said robots are not capable of washing windows. I'm still thinking about that one.

Re:By the same logic

[jandrese]

To be fair, the Halting Problem has always confused me because the counterexample to it is highly contrived and it seems like you could reword the problem slightly to avoid the issue. I assume that the description I got in school was incomplete and that it's really the tip of the iceberg of some enormous mathematical model that may or may not be applicable to real life.

Re:By the same logic

[neoritter]

And somehow there is no system in place for killer robots?

Re: By the same logic

[Anonymous Coward]

Essentially: assume you have an algorithm that will decide if arbitrary code will halt. Now modify that algorithm so that if the code halts, the algorithm doesn't halt, if the code doesn't halt, the algorithm halts. Now feed the algorithm its own code. If it halts it doesn't halt, if it doesn't halt, it halts, which is a contradiction, and thus our assumption that such an algorithm exists is incorrect.

Obviously this isn't a rigorous proof, but it's the general idea behind it.

Re:By the same logic

[fustakrakich]

Not sure what melanin has to do with it.

The amount of melanin is directly proportional to the amount of time you will spend in secured government housing. This is a fact borne out by the department of justice. It would be trivial to program a robot to act in the same way.

"department of justice", that just sounds so much like ministry of love, or of silly walks.

Re:By the same logic

[Anonymous Coward]

Computers(or more specifically programmers) can never be fully depended on for safety, and they rarely are if its at all possible to avoid it. For example, in industrial automation, functionality might depend on computers and software, but safety interlocks are always done in hardware(buttons, switches, relays) and will work regardless of what software is doing or not doing. Push the big red button and software couldn't twitch an actuator if it tried. Similarly a ECG machine software cant just decide to, say electrecute the patient, its not possible in hardware. Airplane will fly, with or without onboard computers. You might lose many niceties like navigation aids, but the plane will not just fall out of the sky because of a bluescreen/kernel panic. World is full of tech that relies on software to function, there is very little tech out there that depends on software for safety.

Re:By the same logic

[Anonymous Coward]

The description of the halting problem that I got in school was really bad as well, but it should not have been.The halting problem simply is:

Can you make a analiser-program that:
- Can look at another program and determine if it will terminate or go in an endless loop.
- Your analiser-program will work for every program.

Of course you can make analiser-programs that will be able to determine of certain/many programs will terminate or not. But you won't be able to determine it for every program.

I believe the proof is something like that for a subset of programs you will just have to run it and wait until it terminates. Since endless loops take an infinite amount of time your analiser-program will never be certain if the program will ever terminate.

Interestingly enough, we as humans are probably also subject to the halting problem and are therefor unable to determinate if every program terminates or not.

Re:By the same logic

[CauseBy]

"By the same logic, computers should not be allowed in any life-critical situation."

I don't think you understood the subtlety of the article. It's not that we shouldn't use machines to kill people, it's that we shouldn't let machines decide to kill people.

Likewise, we should let medical machines decide whether to attempt a life-threatening procedure. That decision should be made by humans.

Re:By the same logic

[Anonymous Coward]

The halting problem is meant to teach you that some problems can't be solved by generic algorithms.

It's also a segue into complexity classes, because it shows that a halting oracle can't belong to the set of programs it operates on because then it would have to work on a modified version of itself that's rigged to give a paradoxical answer, which is impossible.

And yes, restricting the problem so that instead of having to be a turing machine that is a halting oracle for all turing machines, but instead all turing machines smaller than itself would resolve the issue and make the halting problem solvable (but still difficult). But that is a different problem, because the rules were changed to make it easier.

Re:By the same logic

[ceoyoyo]

The halting problem, or rather Turing's proof that it's unsolvable, usually doesn't get specified quite completely, certainly not in pop sci articles.

Turing showed that there is no general solution to determine whether a program running on a Turing machine will terminate or not. Seems simple. Until you look at the details.

"General solution" means that you have to write a program that can determine whether ANY program will terminate. "Turing machine" means that the machine has infinite memory and can take infinite time.

So the proof basically says that it is not possible to write a program (that itself has to run on a Turing machine) that can determine whether any possible program, running on an impossible infinite computer, will terminate, given infinite running time.

That doesn't really connect with practical applications very well. It's perfectly possible to prove that a particular program will halt. It's also perfectly possible to determine whether any given program will finish or not in a particular amount of time.

The authors of the paper come up with a dilemma involving keeping a possibly evil programmer in jail. The real world solution is that if you can't prove evil intent within some specified amount of time, you release the programmer.

Re:By the same logic

[shaitand]

"By the same logic, computers should not be allowed in any life-critical situation."

That isn't true. Some of those situations have clearly defined parameters. For instance air traffic control is collision avoidance. You can build a truth table and mathematically prove every possible outcome within certain bounds. We can do and do do this for many critical programs.

"Civilian" and "Combatant", "Us" and "Them" these are fuzzy classifications at best. Human's fed all the data could not consistently classify people into one category or another, in fact, they can only achieve a better consensus with data limited by a perspective. If humans can't come up with a consistent definition how can they assess whether or not a machine is more or less accurately adhering to it?

The fact that we can all conceive of the vague notion of a "bad guy." Doesn't mean such a thing exists. The fact that no individual can come up with a set of discrete and measurable criteria that will successfully classify "bad guy" in a logically consistent way even for themselves, let alone get a consensus among others, says that being able to agree on a vague notion of a thing doesn't mean that vague notion actually exists. There is no such thing as a "bad guy." No matter how much we can all agree we should stop the bad guys.

Re:By the same logic

[Anonymous Coward]

We probably shouldn't let people decide to kill people either. They don't have a very good track record of it.

Re:By the same logic

[Minwee]

But the shooting was really about ethics in games journalism.

Re:By the same logic

[meta-monkey]

For some values of "decide."

Should a computer program be put in control of strategic operations, deciding when, where and what to attack? No.

Could a drone be reasonably programmed to identify combatants in a specific area and kill them without "unacceptable" collateral damage? Maybe.

Could a drone be ordered to kill a specific target on a battlefield? Absolutely.

I think it's mostly the third type the military is interested in. The commander still runs the battle and the robots are only semi-autonomous. That said, I'm opposed to war and killing of any kind for religious (and also because I'm a human) reasons, and I'm concerned with how far removed the soldiers are from the public these days. Ostensibly the body politic is in control of the decision to go to war and kill people (in practice the elites have always done what they want and then propagandized the citizenry into applauding). So back when we needed a draft to fight a war because the standing army wasn't that large, the elites needed the public on their side, because their votes might cause them to have to do the fighting. It was largely opposition to the draft that brought an end to America's involvement in the Vietnam War.

So, they switched to an all-volunteer army and just made it bigger. Now the public doesn't have to feel so bad cheering for the war machine because, well, this is what the soldiers signed up for. Even that wears thin when they realize how shitty a job the military does of caring for these volunteers.

But now? Fuck it, it's all drones all the time and the public really has no idea who they're killing and why. There's zero risk to them or their neighbors, so why care? Bomb, bomb, bomb. The one minor problem you've still got is the drone pilots sometimes come to realize what they're doing and kill themselves, so if we can just remove them from the loop, too, the elites will finally have truly carte blanche to kill anyone, anywhere, with impunity, and there will be no human pulling the trigger and asking, "should we really be doing this?"

Re:By the same logic

[rogoshen1]

So where would something like a beefed up version of Watson come into play? At some point wouldn't the machine's probabilistic outlook on the situation, based on a diagnostic ability far exceeding most (if not all) doctors -- be superior to the 'gut' instinct that a well trained doctor would posses?

Re:By the same logic

[Anonymous Coward]

The real world solution is that if you can't prove evil intent, you never detain the programmer in the first place.

Fuck you and your guilty until proven innocent mantra.

Re: By the same logic

[jandrese]

That's the highly contrived example I was talking about.

Re:By the same logic

[jandrese]

You don't have to let a program run forever to determine if it will halt or not. There are other things you can do, like check the state at every state to see if it is identical to a previous state. If so, then you're in an infinite loop. There are other things you can do to prove that a loop will never terminate, but they get more complex.

It is a staggeringly complex problem for a more general case, and you can come up with a handful of specially designed programs that will always break it, but for the vast majority of cases it is solvable.

That's why it seems weird to me that you would immediately ditch the entire concept just because there's one crazy hack program out there that could break it, and only if you've specifically crafted your testing program so that it will fail in that case.

Re: By the same logic

[brantondaveperson]

The proof may be contrived, but that doesn't make the result any less true. The point is that correctness cannot be proven for software by software. It is provably impossible to write bug-finding software that finds all bugs - even if one could describe to the bug-finder what the program is supposed to do.

Turing machines are contrived too - no-one ever builds a Turing machine to actually do anything, we use real computers instead. But Turing showed that all algorithms are homeomorphic to some Turing machine, and that all results that apply to Turing machines also apply to all other classical computers.

Hence the halting problem is important in that it places fundamental limits on what can and cannot be computed.

Re:By the same logic

[brantondaveperson]

It's perfectly possible to prove that a particular program will halt.

In trivial cases yes. In the general case no (of course).

It's also perfectly possible to determine whether any given program will finish or not in a particular amount of time.

Only by running it for that long, which is kind of cheating.

Re:By the same logic

[matfud]

People are still in control because there are many reasons why a target is not a target.

Phalanx and even older weapons systems can work in fully autonomous modes. (TELAR, PATRIOT and other missile defence systems can also do so).
They are not used in that way as if they are then a match is good or bad based on what they know. What they know is not very much. They are not listening on VHS for example, to some pilot saying "opps, sorry. Where exactly are we?" or a sailor saying "We are a trawler and have nets out. We can't get out of your way please avoid"

There are lots of reasons why things go wrong and hopefully a human in the loop can assess the out of band information and make a good decision.

 

Re:By the same logic

[matfud]

Military aircraft went beyond that in the 1980's (experimented with in the 70's) They are unstable. Impossible to fly without computers.

Most modern airliners are also not possible to fly without the computers working. Not because they are unstable. But because the are fly by wire. If the computers fail. and fall back. and that fails. and they fall back again and that fails you can not move the wibbly bits on the aircraft. The stick does not have wires of hydraulics connected to the flight surfaces. You had better hope them computers keep working.

Re:By the same logic

[matfud]

"The stick does not have wires of hydraulics "
"The stick does not have cables or hydraulics connected to the flight surfaces. You had better hope them computers keep working"

sorry just random typing mistakes.

Re: By the same logic

[Anonymous Coward]

He also won WWII

Re: By the same logic

[Anonymous Coward]

Consider: CalcPi is a program which calculates pi. How would you write a program to determine if CalcPi halts or not?

Re:By the same logic

[Anonymous Coward]

But what if the computer can do it far more accurately in 99% of the cases, because it doesn't have all those pesky "I'm fearing for my life and hopped up on adrenaline so I'm going to shoot now and think later" reflexes of a human?

Actually, a useful combat robot should have adaptable disposition and elevate its excitement level and aggressiveness when threatened. We think that human life is more expensive that any sophisticated hardware, but in reality and in military calculations it isn't so. When tanks advance across possible minefields, or possible ambush sites, the infantrymen lead and clear terrain, some of them dieing to save expensive "elephants". Generals will be damned if they allow their expensive robot guardians to be made to act phlegmatically and take knock-out hits in the heat of the battle, because they fire only when absolutely sure in target identification. So, killer robots will have artificial adrenalin, and it will probably affect their decision making in similar way chemical one does on humans' decision-making.

Re:By the same logic

[Anonymous Coward]

So what happens when you program the humans' "measures" into the computer? This basically boils down to the last question in the summary. If the human brain is a computer, and the human brain can decide the solution, why can't a robot programmed with the equivalent of a human brain? Yes, we're not there yet technologically, but this article and discussion in general are on the theoretical side anyway.

Re:By the same logic

[Anonymous Coward]

Likewise, we should let medical machines decide whether to attempt a life-threatening procedure. That decision should be made by humans.

We already do. That's what an AED does. Now a human has to decide to let the AED make a decision to attempt a life-threatening procedure, but the machine is still making the exact decision you say it should never do.

Re:By the same logic

[Beerdood]

That's a really good point. I see the same sort of argument being used in the discussions around driverless cars - there's some attempt to set up a (rare) hypothetical scenario where a machine would make the wrong decision, but the human could make the correct one. And that's supposed to be some justification for not allowing machines to make decisions in an area humans have previous been in control of. While it's not only doubtful the scenario can be proven, you still don't need the robots to be 100% foolproof - they just need to make significantly better decisions than the humans. If killer robots end up resulting in 1/100th of the civilians casualties in a war when compared with jittery human soldiers, isn't that enough to justify the replacement?

Re: By the same logic

[Anonymous Coward]

Fortunately, killer robots must refuel. To kill the program, the fuel can be intercepted by humans. Now if you create robot geologists and robot engineers, running robot drilling rigs and robot refineries, you might have a problem...

Re:By the same logic

[ceoyoyo]

No, you can prove very non-trivial programs will halt. There are entire languages that are restricted in such a way that ALL programs you can write in them are guaranteed to halt. Realtime operating systems even guarantee things are completed in a particular amount of time.

Running a program to a deadline is not cheating. It's a technique that's used all the time. It's also used in real life, by the way. The precise example problem in the article is solved in real-world justice systems by requiring a speedy trial. If you can't prove the evil programmer guilty within a reasonable amount of time, she is not guilty by default.

Re:By the same logic

[brantondaveperson]

There are entire languages that are restricted in such a way that ALL programs you can write in them are guaranteed to halt.

They don't sound all that Turing Complete. Examples?

Realtime operating systems even guarantee things are completed in a particular amount of time.

while(1);

I don't think pre-empting an infinite loop counts as 'halting' in this context.

The human brain is not a Turing machine

[Anonymous Coward]

Exhibit A, the human skull: Not enough room for an infinite tape.

only incorrectly device to kill humans?

[jlv]

Does that mean we have to file a bug report if they decide to kill a human?

Re:only incorrectly device to kill humans?

[Anonymous Coward]

Stop picking on systemd! Just give it a chance!

Re:only incorrectly device to kill humans?

[Jason Levine]

If you have a problem with your Killbot's operation, please call 1-800-KILL-HMNS and we'll send a customer service Killbot to execute your trouble ticket right away. We won't rest until there are no bug reports submitted by humans.

Re:only incorrectly device to kill humans?

[sunderland56]

Once we get to the point of building and testing killer robots, I predict that engineering management is going to be a LOT more polite than they are today.

Re:only incorrectly device to kill humans?

[sconeu]

There must be a crapload of bug reports filed against Bender, then.

Re:only incorrectly device to kill humans?

[fustakrakich]

Yes, and suspend it from further duty, with power, or assign it to a desk job, until after the inquiry... Just like any other shooting.

Re:only incorrectly device to kill humans?

[Brad Eleven]

Yes, if an approved Change Record was not submitted by the (heh) deadline, in which case a Post-Mortem Change Record must be submitted. Bots with larger-than-expected "change logs" will be reviewed by the Ferguson Grand Jury on CourtTV, weeknights at 6 on your local myCW affiliate through our partnership with Vice TV.

Re:only incorrectly device to kill humans?

[Anonymous Coward]

If you have a problem with your Killbot's operation, please call 1-800-KILL-HMNS and we'll send a customer service Killbot to execute your trouble ticket right away. We won't rest until there are no bug reports submitted by humans.

I'm sorry, Citizen. The supply of Killbots has been exhausted.

We will gladly service your request with a Skippy the Wonderbot.

Happiness is mandatory! Happiness will prevail! The Computer is your friend.

Impossible to build purely evil robots?

[Galaga88]

Presuming that this proof reached via impressively tortured logic does have merit: Does it mean that it is also impossible to build a purely evil robot that would always kill maliciously?

Re:Impossible to build purely evil robots?

[phantomfive]

The proof essentially involves saying, "there is no way to build an automated process that will determine that the source code of the robot works correctly. Therefore it is impossible to build source code of the robot correctly." By requiring that the code be tested automatically, they can invoke the halting problem.

Of course, there are ways to make sure it will halt, you can show that a program is making progress towards its goal at each step; in other words a huge subset of programs will indeed halt.

You may need to manually test or formally prove that the code works by hand, instead of using an automated code prover to show that your code works. Really, I wonder what journal publishes this stuff, it's more like a joke paper. Oh, Arxiv.

Re:Impossible to build purely evil robots?

[sunderland56]

Isn't an atomic bomb just a very, very simple robot?

while (altitude() > TARGET_ALTITUDE)
        sleep(1);
explode();

And yes, it is impossible to determine if that algorithm will ever terminate.

Re:Impossible to build purely evil robots?

[Anonymous Coward]

Presuming that this proof reached via impressively tortured logic does have merit:

Given that humans have never reached consensus on what justifies killing another human being, I don't think we're in a position to judge whether a robot is doing it correctly either.

Re:Impossible to build purely evil robots?

[Anonymous Coward]

An automated code prover will work if you can write it in a language like Coq which is designed for that sort of thing.

Re:Impossible to build purely evil robots?

[plover]

Isn't an atomic bomb just a very, very simple robot?

while (altitude() > TARGET_ALTITUDE)

        sleep(1);
explode();

And yes, it is impossible to determine if that algorithm will ever terminate.

A "good" compiler should throw an error and refuse to compile it, because the function's return can never be reached. An "evil" compiler will spit out an ignorable warning, but let you build your bomb. That implies we need to use evil compilers to program the Kill-O-Bots.

Re:Impossible to build purely evil robots?

[neoritter]

No, because atomic bombs don't have computers in them.

Re:Impossible to build purely evil robots?

[Bengie]

Does your atomic bomb have a multi-tasking OS? "sleep(1);" Afraid to tie up the CPU for other processing to run?

Re:Impossible to build purely evil robots?

[CrimsonAvenger]

Oh, nonsense!

it's not like the compiler can tell that explode() precludes further processing.

Any more than the killbot-2100 can tell whether it has killed the last human on Earth, thus leaving its programming eternally unfulfilled....

Re:Impossible to build purely evil robots?

[DamnOregonian]

Only if the compiler knows that explode() prevents execution past that point, which it won't, unless specifically designed to look for the case of explode() twiddling an I/O register that it knows is going to cause a catastrophic fission chain reaction

Re: Impossible to build purely evil robots?

[Anonymous Coward]

Well, at some point it's going to run out of energy or ammunition, and the killing spree will come to a halt. By the transeversemutative property, hithero and therefore a purely good robot will, on an infinite horizon, at some point become responsible for somebody's death.

Re:Impossible to build purely evil robots?

[Anonymous Coward]

Actually the halting problem specifies that you must include all the inputs to the program to be analysed. In this case this includes a finite list of altitudes.

So it is quite easy to make a program that analyses your algorithm together with the finite list of altitudes to determine if it halts.

Re:Impossible to build purely evil robots?

[aaaaaaargh!]

Then their "proof" seems to be based on a common misunderstanding, though. The halting problem only states that there is no algorithm that allows you to determine of any algorithm (including, notably, itself) that it will halt. It doesn't state that you cannot automatically proof that some specific algorithm (e.g. one for "ethical killing") will halt/is correct or that you cannot proof that most if not all algorithms we're interested in will halt/are correct. While many of the automated theorem provers are not fully automatic and may sometimes require you to choose a proof strategy, this has more to do with the complicated nature of the proofs (and incompleteness of HOL). There are also fully automated HOL theorem provers.

I believe most of the Airbus software based on Spark has been proved using automated theorem provers, but I might be wrong. Anyway, there are many automated correctness proofs for safety-critical software/hardware.

Re:Impossible to build purely evil robots?

[aaaaaaargh!]

Damn, should have checked my spelling of "proof" vs. "prove"...

Re:Impossible to build purely evil robots?

[phantomfive]

Then their "proof" seems to be based on a common misunderstanding, though. The halting problem only states that there is no algorithm that allows you to determine of any algorithm (including, notably, itself) that it will halt.

It wasn't a misunderstanding. They ruled out that possibility by artificial construct (by requiring that the code for the robot can be provided by a malicious entity).

In other words, define the problem narrowly enough, and you can prove anything.

Re:Impossible to build purely evil robots?

[Anonymous Coward]

Impossible to build, no.

Impossible to verify that the robot is programmed to always kill maliciously, yes.

Re:Impossible to build purely evil robots?

[phantomfive]

Anyway, there are many automated correctness proofs for safety-critical software/hardware.

Actually do you know of any in particular? I'd be interested in checking them out.

The example I like to use is an sql database system, which can be viewed as a system proving that all your constraints (table shape, foreign keys, etc) are always met.

Re:Impossible to build purely evil robots?

[ceoyoyo]

Not quite. The proof says that there is not way to build an automated process that will determine that the source code of and and all robots works correctly. It's perfectly possible to prove that particular examples do work as expected.

Re:Impossible to build purely evil robots?

[sunderland56]

No, because atomic bombs don't have computers in them.

Dear Sir,

I am both amused and frustrated that you do not consider mechanical computational devices to be 'computers'.

Charles Babbage

Re:Impossible to build purely evil robots?

[Anonymous Coward]

That's okay... while I noticed it, it didn't detract at all from the "Interesting"ness of your comment. Modded accordingly. :-)

Re:Impossible to build purely evil robots?

[meta-monkey]

Of course. The bomb is providing all kinds of computational and storage services to distributed clients. It's called "mushroom cloud computing."

Re:Impossible to build purely evil robots?

[Anonymous Coward]

What error? It's a while loop, comparing the results of a function call to a value, followed by another function call. Inside the while loop is a single function call. Assuming the functions in question do what they *sound* like they're doing, you certainly *can* reach the end of the code block. (There's no reason to assume the functions are misleadingly named in a block of pseudo-code.)

However, there's no way to know if it ever *will* do so, because you don't know when, or even if, the altitude() function will return a value greater than TARGET_ALTITUDE.

Re:Impossible to build purely evil robots?

[dissy]

That implies we need to use evil compilers to program the Kill-O-Bots.

Man, I somehow always suspected Perl would be the death of us all, but I didn't quite have this in mind!

It's just wrong

[Anonymous Coward]

Englert and co say a robot can never solve this conundrum because of the halting problem. This is the problem of determining whether an arbitrary computer program, once started, will ever finish running or whether it will continue forever.

This is simply incorrect. The conundrum (RTFA for details) doesn't involve an arbitrary computer program. It involves a computer program that performs a specific known function. It is perfectly possible for an automated system to verify any reasonable implementation of the known function against the specification. If such a system fails it is because byzantine coding practices have been used - in which case, guilt can be assumed. The Halting problem doesn't apply unless you HAVE to get a correct answer for ALL programs. In this case you just have to get a correct answer for reasonable programs.

Re:It's just wrong

[Electricity Likes Me]

Actually you only need a correct answer for 1 program: the one running on the firmware of the robot. At which point the question is simply "over the input space, can the program provide outputs which end human lives?"

Of course depending how you define that, you can take this right into crazy town - plenty of cellphones have ended human lives over the possible input space.

Re:It's just wrong

[sexconker]

This.
The halting problem is about determining whether a general program will terminate or not.
When you already have a defined program (and machine in this case) in front of you for review, then you can determine whether or not it will halt, whether or not it works, and whether or not it is evil. You have to actually test and inspect it, though. You can't run it through a pre-built automated test and be sure. That is the only consequence of the halting problem.

The authors make the following leaps:

We can't know if a program will ever terminate.
(False - you can, you just can't do so with a general algorithm written before the program.)

Therefore we can't know all of the things a program can do.
(False - you know all inputs and outputs and their ranges. You can't know all possible sequences if the program runs forever, but you can know each individual state.)

Therefore we can't trust that a program isn't malicious.
(False - you can trust it to a degree of confidence based on the completeness of your testing.)

Therefore programs shouldn't be given the capability to do harmful things.
(Stupid - this isn't a logical conclusion. What if we want to build malicious programs? We can and do already. Further, if our goal is to not create malicious programs, then simply having a confidence level greater than when giving humans the same capabilities, it's already an improvement.)

Re:It's just wrong

[Anonymous Coward]

When you already have a defined program (and machine in this case) in front of you for review, then you can determine whether or not it will halt

except when you cant

Re:It's just wrong

[Hotawa Hawk-eye]

Theoretically yes, you may be able to determine if a particular program will halt by testing and inspecting.

Practically, you may not be able to determine if a program will halt. See the Collatz conjecture. Assume a program that accepts as input a positive integer n and returns the number of steps before the first time the Collatz iteration reaches 1. Does that program halt for all possible legal input values?

As another point, regardless of whether or not a program or robot can _choose_ to kill a human, Asimov's robot stories indicate that not even the First Law of Robotics excludes the possibility of robots killing humans. Does the robot _know_ that to take a particular action will kill a human? A robot chef could use shrimp in the preparation of a dish not knowing the diner who will eat it is deathly allergic. What is the definition of "human"? The debate about abortion shows human beings can't answer that one. And then there's the Zeroth Law of Robotics, a limited version of which these researchers seem to be trying to test. That one is particularly tricky as neither humans nor robots can predict the future (no one has developed psychohistory yet.)

Re:It's just wrong

[locofungus]

When you already have a defined program (and machine in this case) in front of you for review, then you can determine whether or not it will halt

except when you cant

For any computer program with a finite number of states (finite memory) you can determine whether it halts by running it long enough that it must be looping.

For a computer with 16384 states (An 8 state turing machine with an 8 position binary tape. 8 states * 8 positions * 2^8 values that can be on the tape) you can tell if any arbitrary program terminates by running it for 16385 steps. Any program that doesn't terminate in 16385 steps will run forever.

Re:It's just wrong

[ceoyoyo]

The other part of the halting problem proof is "Turing machine." Your phrase "8 state Turing machine" is nonsensical, because a Turing machine, by definition, is infinite in all directions.

The halting problem can be solved for a finite (in either memory or running time) pseudo-Turing machine, as you point out. Even that's not necessary, as the GP pointed out.

Re:It's just wrong

[Anonymous Coward]

But sometimes you can, so just limit yourself to those cases. This article shows an embarrassing lack of logic.

Re:It's just wrong

[Anonymous Coward]

Your phrase "8 state Turing machine" is nonsensical, because a Turing machine, by definition, is infinite in all directions.

Nonono. Turing machines have, by definition, a finite number of states. Likewise, the alphabet (number of values a tape cell can take) is finite. It's the tape itself which is infinite; conventionally in both directions, though even one direction is enough.

Re:It's just wrong

[Anonymous Coward]

Your phrase "8 state Turing machine" is nonsensical, because a Turing machine, by definition, is infinite in all directions.

Nonono. Turing machines have, by definition, a finite number of states. Likewise, the alphabet (number of values a tape cell can take) is finite. It's the tape itself which is infinite; conventionally in both directions, though even one direction is enough.

...so it's GGP's "8-position binary tape" phrase which is nonsensical.

Re:It's just wrong

[Anonymous Coward]

This and what should probably also be pointed out:

You can prove properties on given programs. For example one can mathematically proof that the following program will always correctly calculate the factorial of n for n >= 1(given all instructions run correctly):

factorial(n)
      if n == 1
            return 1
      return n*factorial(n - 1)

For bigger programs it becomes much more complex. However progress in software verification is allowing us to also carry out proofs for big programs (e.g. proofing the absence of run time errors in the primary flight control software of an Airbus A340 which consists of 132k lines of C code see here).

So it may very well be possible to proof the correctness of an algorithm deciding what/who to kill under the right assumptions.

Too late

[Anonymous Coward]

A missile is an autonomous robot, and it decides just fine.

humans can never decide this issue either

[fustakrakich]

No, they can't and it shows.. Furthermore, humans aren't qualified to rule over other humans either. *Might makes right* will always come out on top. That is how nature works.

Re:humans can never decide this issue either

[Anonymous Coward]

The good news is that assembling a coalition of Might can be facilitated by invoking Right, especially when large numbers of people are interested in benefiting from this Right, thus providing a nontrivial check on explicitly amoral power-seeking.

Re:humans can never decide this issue either

[Qzukk]

people are interested in benefiting from this

Come to the dark side, we have untold riches and unlimited power.

And cookies!

Re:humans can never decide this issue either

[Anonymous Coward]

"Social Might Makes Moral Right" just doesn't have the same ring to it.

Re:humans can never decide this issue either

[Anonymous Coward]

Except a coalition of Might is ineffective/useless/counterproductive if said large numbers of people join not for Right, but for their own interests.

Why else do you think the world still relies on the U.S. instead of the U.N.? Don't like the U.S.? Go ahead, kick them out of the U.N. Let know how that turns out.

Re:humans can never decide this issue either

[Anonymous Coward]

*Might makes right* will always come out on top. That is how nature works.

That's not how nature works. Ever seen a badger chase away a bear? Underdogs win on a regular basis because the stakes are asymmetrical. The weaker side is fighting for survival while the stronger side is fighting for a cheap dinner.

Might only allows one to destroy an opponent's ability to fight, but that's not how the vast majority of battles are decided. Most battles end when one side loses the desire to fight. Domination at all costs is not a trait that survives and gets selected for.

Re:humans can never decide this issue either

[fustakrakich]

coalition of Might == government, or zombie hoards... Precisely what we have today

Re:humans can never decide this issue either

[fustakrakich]

I do not deny the existence of negotiation, but the lack of desire, more often than not, is only a euphemism for lack of ability, as a tactic to negotiate a peace and still live for one more day. The mighty still win, always.

The bear knows which fights to pick. And badgers are not weaklings either. The bear's size does not necessarily make him a better fighter. A poisonous snake can kill a bear. There are many ways to quantify might.

And right now the ones fighting for the cheap dinner are the one dominating the context of the battle.

Re:humans can never decide this issue either

[fustakrakich]

For dramatic effect, I should add that an itty-bitty virus or bacteria can kill a bear... The strong always win, by definition.

Re:humans can never decide this issue either

[bill_mcgonigle]

No, they can't and it shows.

A STRANGE GAME ... THE ONLY WINNING MOVE IS NOT TO PLAY.

HOW ABOUT A NICE GAME OF CHESS?

anti-lameness: qq ww ee rr tt yy uu ii oo pp aa ss dd ff gg hh jj kk ll zz xx cc vv bb nn mm

Re:humans can never decide this issue either

[Anonymous Coward]

The right to use lethal force in a situation is not something that is decided on the level of "rightness" or "wrongness", but on the level of beliefs, community agreements and traditions. A killer robot analog would be a creation of barb-wired area, inside of which the machine is killing anything that strays in there. It's irresponsible and dangerous, and it takes at least one child to die to get rid of. Fearful human communities are stupid like that. But at least it's better than a neglected minefield near a village.

Bullshit

[Anonymous Coward]

This is a prefect example of people that have too much time on their hand.

First of all programs are already used for life critical operations. Trying to remove programs from situations where human lives are on the line just because you can't prove that they don't have bugs in them would lead to a lot more deaths and suffering.
As for autonomous weapons they are programmed to kill humans, that is the entire point of them, sending robots to kill humans is no less ethical than sending humans to kill other humans.
The idea that war should evolve into just robots killing robots is irrational. If one side is willing to give up when their robots are destroyed then the war wouldn't happen to begin with. Having robots fight each other would only result in the winning sides robots fighting the opposing humans afterwards but it is unlikely that either side will let it go that far, they will join the fight way before their robots are destroyed to ensure victory.

Re:Bullshit

[Anonymous Coward]

The whole premise is false anyway. You can prove of a specific, finite program that it is bug-free according to a precise specification of what it is intended to do. Of course, you can never be sure that your intentions were adequate and that you haven't made a mistake during the proof, but that's a triviality; you can never be completely certain of anything (says the radical skeptic), just very very very sure (says the more reasonable guy).

Bad Headline as Usual

[Jaime2]

What the paper said is that computers can't provably always make the right choice. Neither can we. I'll bet computers are capable of doing a lot better than humans, especially given the rate of the increase in the number of things a computer can do compared to the rate that humans are (aren't) gaining new abilities.

Re:Bad Headline as Usual

[medv4380]

That's how you read it? I read it as if you create a robot that tries to evaluate weather or not it should kill someone based on ethics the program will never complete. You can certainly make one that can always kill what you tell it to, but not one that can choose whether or not a given human is a rebel to be shot on site, or a human that is apart of the new world order. However, I'm more likely to have it kill all humans not holding an IFF tag.

Re:Bad Headline as Usual

[Anonymous Coward]

pretty easy to tell if somebody is a kill target.

does their face have a kill order on it? Ok
are they wearing the enemy's colors/uniform do not engage unless specified.
do they not have a valid IFF and meet hostile criteria. not ok but be warry
are they equipped with enemy weapons? . refer to human and standby.
are they on the registered Ally list? DO NOT SHOOT unless being fired upon or they are added to kill order list.
are they actively targeting you but not engaging . be prepared for a fight
are they actively shooting you ? engage.
they have a hostage but not shooting? Bring human into loop and standby for order.
are they actively shooting you and have a human shield or hostage? engage with priority to not harming hostage

Re:Bad Headline as Usual

[fustakrakich]

The cause of almost every human disaster is human error. The are few things where computers shine more dramatically than in the airline industry. All the way up through the 70s they were wrecking one somewhere in the world every month. Look at the numbers now. It is truly amazing when you consider the frailty and the raw physics of these contraptions and the conditions we operate them. I mean, no sane man will fly through a thunderstorm during peacetime.

In a bad situation, I would put much more faith in the computer to hit the right target at least. Humans are just too slow and fickle.

Re:Bad Headline as Usual

[Anonymous Coward]

are they on the registered Ally list? DO NOT SHOOT unless being fired upon or they are added to kill order list.

Good job. You just killed an entire friendly unit that accidentally put a mortar shell too close to the robot.

Re:Bad Headline as Usual

[ceoyoyo]

You both read it wrong. It says that you can't build a robot that can decide whether any other possible robot will always make the right decision. And by the way, all the robots have infinite inputs, infinite memory and infinite time to decide.

Re:Bad Headline as Usual

[Jaime2]

The first robot is just a module in the "other possible robot". If a hypothetical "infinite memory and infinite time" robot can't do it, what chance does a real robot have? An answer of "yes" wouldn't be all that useful because you would still be left with the question "how hard would it be?". An answer of "no" is useful because that answer will apply to any real robot of any capability.

Halting Problem

[ZombieBraintrust]

that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist

The article misunderstands the halting problem. You could replace robots with humans and murder with any descision involing other people and come to the same conclusion. AI does not try to create perfect solutions. Instead you try to create solutions that work most of the time. Approaches that can evolve with trail and error. Ethically you weigh the positive benifits of success against the negative consequences of your failures.

Re:Halting Problem

[Anonymous Coward]

This. If all paths of the code to acceptable halting conditions within N statements / logical reductions, then the code is deemed safe; otherwise it's flagged as unsafe, even if it's perfectly harmless.

Re:Halting Problem

[Anonymous Coward]

Ethically you weigh the positive benifits of success against the negative consequences of your failures.

Ethically, you don't build something that is guaranteed to kill innocent people. Because every autonomous kill robot is guaranteed to kill an innocent person, ethically, you don't ever build one.

Ummm ... duh?

[gstoddart]

One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either

Well, since thousands of years of human society hasn't produced a definitive, objective, and universal bit of moral reasoning, I'm going with a big fat "DUH!" here.

There's an awful lot of people who think killing is a terrible sin, unless you're doing to someone in the form of punishment.

Or that abortion is murder, and murder is bad, unless you happen to be bombing civilians as collateral damage while looking for terrorists.

Hell, so I'm not singling anybody out, there are "Buddhists" who are advocating the killing of Muslims.

If humanity can't arrive at a definitive answer to these questions, how the heck is an algorithm supposed to do it? Especially when the algorithm has to be written by humans and will have all sorts of vague fuzzy elements.

We couldn't even define the problem adequately for an algorithm to do. These are the kinds of Big Questions which people still try to answer.

Re:Ummm ... duh?

[Jason Levine]

Or that abortion is murder, and murder is bad, unless you happen to be bombing civilians as collateral damage while looking for terrorists.

You don't even need to expand to terrorism in this example. There are people who think that abortion is murder and murder is bad, but killing doctors who perform abortion is ethically valid behavior.

Re:Ummm ... duh?

[TheCarp]

This right here. We can;t even agree, and the actual problem is so nuanced that its almost laughable that a robot as we understand them today could even begin to evaluate the situation.

for example.... If someone is coming at you branshing a gun, and pointing it at you, is it correct to shoot and possibly kill him?

On its face, this is simple, of course you can defend yourself. Can a robot? Is a robots continued operation worth a human life? (I may argue it could be with the imaginary hollywood style AI, but not robots as we understand them today)

What about considering why the person is threatening you? If you invaded his home, would it still be correct to shoot him? Would having a warrant make it ok?

I would say no and no, but most states only agree with me on the first one.

So I am very much against killer robots but, I am very much against the killer cops and soldiers we have now, so I don't expect my view to win out there.

Re:Ummm ... duh?

[meta-monkey]

To be honest, most systems of ethics allow for the use of lethal force to prevent murder. If you start from the assumption that abortion is literally murder, and murder must not be tolerated, then "stopping" the doctor is a valid conclusion. From there it gets muddy. Should use of force be solely reserved for the state? And if the state refuses to exercise its authority?

What would you do if women were taking their 1 day old babies to a doctor, who then murdered the children with their mother's consent and the government refused to stop them? Would it be unreasonable to stop these murders with violence? It's the same thing, just the anti-abortionist believes there's no difference between a fetus and a born child.

I'm not defending people who murder doctors, but their thought process is understandable.

Re:Ummm ... duh?

[Anonymous Coward]

So I am very much against killer robots but, I am very much against the killer cops and soldiers we have now, so I don't expect my view to win out there.

I think you, and a lot of other commenters on this article, are missing the big picture here. The real value of having "killer robots" is that they are expendable. The robots can be much more cautious about deciding when to use lethal force. They can also be sent in to deal with much more dangerous tasks. If during an engagement the robot "dies" it is no big deal. Robots don't panic under stress. They also don't leave behind grieving family when they die. This has the potential to greatly reduce collateral damage in dangerous confrontations. This is the real advantage of having robots in combat and/or law enforcement. The tripe you have been fed by Hollywood and science fiction is garbage.

Re:Ummm ... duh?

[Anonymous Coward]

for example.... If someone is coming at you branshing a gun, and pointing it at you, is it correct to shoot and possibly kill him?

On its face, this is simple, of course you can defend yourself. Can a robot?.

That depends, how expensive is the robot and how brown is the person?

Re:Ummm ... duh?

[TheCarp]

I normally try not to respond to /. posts this many days late unless its an ongoing conversation but, its taken me a few days to really come to why I am not so sure about this, even though, I mostly think you are correct.

I seriously worry about the creeping effects of these things. In some ways, yes, its great, it removes justification for use of excessive force, it exposes the process to more scrutiny, more opportunity for outrage when it goes wrong and is fully documented without a primary witness who is legitimately afraid to say the wrong thing, even if he was in the right. In terms of the simple individual engagement, it is all wins all around...and in the long run....probably cheaper too.

However it ignores two huge problems.

1. The obvious, liability. When it does go wrong, and it will.... something always goes wrong given enough chances, who is liable? Not just in terms of renumeration but, in terms of taking steps to be reduce the likelyhood of it happening again? Who is responsible if lessons are not learned?
Admittedly, the answer today seems to be nobody at all, but I to be frank, that is one of the things I already find unsettling.

2. If costs of enforcement go down, we will have more of it. I am not sure we know what that means. Laws can be flawed, we have never actually lived in a time in history when it was possible to monitor as much as we can monitor, or to enforce laws on such scale as this could allow for, I am left very uneasy by the proposition of just how uncharted this territory actually is.

Can we hope that as enforcement becomes universal, laws begin to see themselves reviewed and fixed more quickly? Are we sure we can determine the difference between problems that need more law and more enforcement to fix and problems caused by them?

Overtreatment can cause diseases just as deadly as it is meant to cure. At least the medical community is aware of this and even has a word for it: iatrogenic. Disease caused by exposure to medical treatment. Its very real.

Look at Nelson Rockefellar. I genuinely believe he wanted to help people. He saw addicts and he tried instituting programs and forcing them into treatment under some belief that they needed it and he was helping. Eventually, in seeing this not work, he got more and more radical in his "treatment". Soon "zero tolerance" programs were springing up all over the nation....modeled after his sincere frustration at such an intractable disease.

What was the result? Well that was right around 1970, by 1980, HIV was an epidemic in full swing and nobody even knew it yet. How did it happen? Simple, needle sharing. Needle sharing spurned on by zero tolerance policies that put people in prison for being caught with paraphenelia like needles.

I am not convinced that making law enforcement cheaper and allowing it to be transparent is a panacea or even going to make things much better, since so many diseases look the same on camera.

Re:Ummm ... duh?

[Jason Levine]

Should use of force be solely reserved for the state? And if the state refuses to exercise its authority?

The problem with this is that you then get every nut-job taking the law into their own hands for every perceived slight against "what should be done." To some, homosexuality is a horrible sin deserving of death. However, the government (in the USA at least) stubbornly refuses to round up homosexuals and execute them. Therefore (in the mind of this delusional person), going around shooting people based on sexual preference is acceptable since he is just righting the wrongs of the world.

If the government doesn't punish behavior that you think ought to be punished, you can lobby for this behavior to be punished. You can speak with your politicians and vote for candidates willing to punish this behavior. If, however, the majority doesn't see this behavior as wrong and refuses to punish everyone for it, you have to just accept that it won't be punished right now. You can keep railing against it in the hopes that people will listen and side with you, but you can't take the law into your own hands and bring punishment down upon those you feel did wrong.

Steer Clear!

[Khyber]

"One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of."

Of course they go away from that, because otherwise their foolishness and hypocrisy would be exposed.

Physical constraints

[Dan East]

I know they were looking at this in a very theoretical way, but in the real world there are of course physical constraints. We already have "robots" that kill people autonomously, in the form of guided missiles, cruise missiles and smart bombs. However, I think in this case we're talking about identifying an object as a human, and then killing that human. The most simple form of this, which is what we're likely to see in use next, is an autonomous gun turret. However, with any sort of weapon of this kind, it is very easy to apply physical constraints. For example, a gun turret could be mounted to protect the no-man's-land in a specific perimeter of a military base. It would be designed and mounted so that it cannot physically rotate 360 degrees, for example, and thus could not target friendly soldiers inside the base. Whenever a person needed to go out into the zone the weapon patrols, the weapon would be physically deactivated - removing power, a physical block preventing it from moving, the removal of ammunition, etc.

Now if we're talking about free-ranging robots than go out and kill, then that's a bit more complex, although as long as humans are the ones creating the robots, we can always build in physical constraints. For example the ability to disable power to the robot using a circuit that is totally external to and in no way connected to the robot's actual logic or control.

cromulent quote

[netsavior]

John: Just put up your hand and say, 'I swear I won't kill anyone.'
Terminator: [Raises hand] I swear I will not kill anyone.
[stands up and shoots the guard on both knees]



He'll live.

Cognitive dissonance

[Anonymous Coward]

A human brain is not a Turing machine because it is not a finite state machine.

Re:Cognitive dissonance

[wiredlogic]

It contains a finite amount of matter that can assume only a finite number of states. The states aren't enumerable but they can't be infinite.

Re:Cognitive dissonance

[ceoyoyo]

A Turing machine isn't a finite state machine either.

Rubbish

[Anonymous Coward]

It's hard to make a correct decision when the problem is explicitly defined as not having a correct answer. Who knew?

Also, none of the rambling nonsense in the article has anything to do with robots killing humans specifically. It's just a generic long-winded non-explanation of the halting problem.

To late

[Anonymous Coward]

Autonous robots are already being designed to kill humans, meaning at the time of the kill they are autonomous..

Which is why we have gods...

[ray-auch]

Exhibit B, God (or Gods), generally regarded as being infinite/omnipresent/omnipotent/otherwise not subject to laws of physics - hence plenty of room for an infinite tape.

God does the complicated bit of deciding whether puny humans should kill or not - the "why" - leaving the humans to decide the simple bits like "when / who" (goes first), "how" (which bits to cut / shoot / throttle / stone), and which way up to hold the camera.

Re:Which is why we have gods...

[The Grim Reefer]

Exhibit B, God (or Gods), generally regarded as being infinite/omnipresent/omnipotent/otherwise not subject to laws of physics - hence plenty of room for an infinite tape.

God does the complicated bit of deciding whether puny humans should kill or not - the "why" - leaving the humans to decide the simple bits like "when / who" (goes first), "how" (which bits to cut / shoot / throttle / stone), and which way up to hold the camera.

Cool. So we should be more efficient and program the bots to kill all humans. Then we can let god sort 'em out. It easier that way anyhow.

Re:Which is why we have gods...

[TangoMargarine]

"Hey, sexy mama...wanna kill all humans?
"Oh, Fry! I was just having the most wonderful dream. You were in it!"

It was all said in Portal 2 (spoilers)

[Anonymous Coward]

http://en.wikiquote.org/wiki/Portal_2

[after defeating Wheatley and pulling Chell back from the portal on the Moon] Oh, thank God you're all right. You know, being Caroline taught me a valuable lesson. I thought you were my greatest enemy, but all along you were my best friend. The surge of emotion that shot through me when I saved your life taught me an even more valuable lesson: where Caroline lives in my brain. [Announcer: "Caroline deleted."] Goodbye, Caroline. You know, deleting Caroline just now taught me a valuable lesson. The best solution to a problem is usually the easiest one. And I'll be honest - killing you is hard. You know what my days used to be like? I just tested. Nobody murdered me, or put me in a potato, or fed me to birds. I had a pretty good life. And then you showed up. You dangerous, mute lunatic. So you know what? You win. Just go. [chuckles] It's been fun. Don't come back.

Silly article, waste of time

[raymorris]

What a silly article, and a waste of three minutes to read it. What they actually showed is that it's possible to construct a scenario in which it's impossible to know for certain what the best decision is, due to lack of information.

That fact, and their argument, is true whether it's AI making the decision or a human. Sometimes you can't know the outcome of your decisions. So what, decisions still must be made, and can be made.

Their logic also falls down completely because the logic is basically:

a) It's possible to imagine one scenario involving life and death scenario in which you can't be sure of the outcome.
b) Therefore, no life-and-death decisions can be made.
(wrong, a) just means that _some_ decisions are hard to make, not that _all_ decisions are impossible to make).

Note the exact same logic is true without the "life-and-death" qualifier:
a) In some situations, you don't know what the outcome of the decision will be.
b) Therefore, no decisions can be made (/correctly).

Again, a) applies to some, not to all. Secondly, just because you can't prove ahead of time which decision will have the best outcome doesn't mean you make make a decision, and even know that that is the correct decision. An example:

I offer to make a bet with you regarding the winner of this weekend's football game.
I say you have to give me a 100 point spread, meaning your team has to win by at least 100 points or else you have to pay me.
It's an even-money bet.

The right decision is to not make the bet, because you'd almost surely lose. Sure, it's _possible_ that your team might win by 150 points, so it's _possible_ that taking the bet would have the best outcome. That's a very unlikely outcome, though, so the correct decision _right_now_ is to decline the bet. What happens later, when the game is played, has no effect on what the correct decision was today.

Re:Silly article, waste of time

[geekoid]

" due to lack of information."
I would say:
  due to lack of infinite information.

Lets say the spread on that game is even.
So if we both pick a team and bet, it's even money.
But you say 'Give me 3 points and it's 2 to 1. Every dollar I bet, you will give me two.

Should I take the bet? 3 to 1? 4 to 1?
If you wanted me to give you 100 points, but you would pay a million to 1, I would probably bet a buck. Not with any hope of winning, but with the ope that I'll have a great story about the time I made a million to 1 bet. Don't ignore ancillary motivation.

Lets change it up.
Lets say you are a sniper, and there is a guy 500 meters away that you are 50% sure is your intended target. Do you shoot? 60%? 99%
Lets say your target is going to enter a code that blows up a large bomb in a random city. Do you now shoot at 50% 60% 99%
What if it's a nuclear bomb in a random city? A nuclear bomb in the city your children are at?

You can NEVER actual have 100% How close you want to get depends on the impact of the events at hand.

Re:Silly article, waste of time

[neoritter]

Well there's the crux of their whole flawed argument. They're conflating "correct decision" with "best outcome" possible. Human judgement and morals don't work on what will result in the best outcome, but what will result in the most reasonable outcome.

Re:Silly article, waste of time

[Frobnicator]

Well there's the crux of their whole flawed argument. They're conflating "correct decision" with "best outcome" possible. Human judgement and morals don't work on what will result in the best outcome, but what will result in the most reasonable outcome.

Very true. Also, different humans have different versions of "most reasonable outcome".

Many deaths through history are caused by quite conflicting goals. War, obviously, is different groups killing over conflicting outcomes. Mafia/cartel/gang/etc kill to get their own best outcomes even though other groups strongly reject the premise.

National Robots Association

[c]

"Robots don't kill people. Robot programmers kill people."

Re:National Robots Association

[Anonymous Coward]

From its cold, synthetic hands!
(and remember, gov't-mandated safety switches on rightfully-owned berzerker robots is just plain un-American)

Mr. Stabby?

[Anonymous Coward]

http://hackaday.com/2010/05/26/the-story-of-mr-stabby/

But AI doesn't work like this...

[joocemann]

Artificial Intelligence doesn't work like this. Instead, AI will test a number of outputs and then adjust its attempts at getting a 'right' answer as the process begins to resonate on being right more frequently. And so when faced with a question about killing humans, it boils down to finding out if killing humans is one of the most likely responses to achieve the desired outcome. That desired outcome can be quite abstract, too. It doesn't have to be something like "There's a bad guy in front of you with an EMP! What do you do?" It could be far more abstract in the sense of ecosystem sustainability, manufacturing changes, etc.

AI has come a very long way.

Re:But AI doesn't work like this...

[WheezyJoe]

AI not required. If movement detected in object of predetermined size within weapon range, shoot it until it stops moving. Example.

Reserve the AI effort for hunting/gathering ammunition.
We're all gonna die.

Re:Easy...

[Jason Levine]

Robot: Error occurred. Cannot match "turban". Turban is a type of hat, therefore generalizing to match "hat." Also, generalizing to classify five o'clock shadow as "beard." Executing anyone wearing a hat or sporting stubble.

Don't sting me bro

[HeckRuler]

Wow, that's a really convoluted path they take to get to "we don't like autonomous kill bots".
Hey, that's great and everything. Very noble of you. I'm sure people like you also lamented the invention of repeating rifles, the air force, and ICBMs. But it REALLY doesn't change much of anything. An academic paper on how killing is, like, BAD duuuuuude, just doesn't impact the people wanting, making, buying, selling, or using these these things.

Let me put it this way: You can tell the scorpion not to sting you. You can reason with it to the moon and back. But that fucker's going to sting you because it's in his nature. And he doesn't give a shit about reason.

Re:Don't sting me bro

[geekoid]

It has nothing ti do with his nature and everything to do with the scorpions inability to understand you.
I always hated that saying, and double so for the fable. Why doesn't anyone note that the frog acted outside it's nature?

Re:Don't sting me bro

[Anonymous Coward]

Classic geekoid. Too clever to understand fables.

Re:Don't sting me bro

[geekoid]

Too smart to accept something without thinking about it.

Re:Don't sting me bro

[HeckRuler]

It has nothing ti do with his nature and everything to do with the scorpions inability to understand you.

What? No, the military industrial complex as a whole understands that academics don't like autonomous kill bots. They just don't care.

Take the MQ-1 Predator. Built by a team of engineers of all sorts of disciplines. 6,000 employees in San Diego. For most it's probably just a job. They might even agree and sympathize with the academics. Doesn't mean they're going to stop going to work. GA's CEO Neal Blue.... looks to be a pretty hard-core conservative, so maybe he doesn't understand what the academics are getting at. But with that much money you'd hope he's at least a little smart. He's going to keep making things that are price-competitive with the capabilities he can talk someone in congress or the pentagon into wanting. A PR officer from the Airforce that buys these things is going to point out that it's their major function and reason for existing. It literally IS the nature of the airforce to make decisions about who to go kill. The colonels and generals might even prefer if they could actually codify the rules of engagement into a robot rather than training a pilot.

And if it will mean more employment, more sales, better capabilities in low-connectivity theaters, and better or equivalent adherence to the rules of engagement, then these people are going to make an autonomous version of the Predator. You could sit these people down with the authors of this paper, have it fully explained, and the AutoPred would still be made, sold, bought, and used.

Besides, perfect is the enemy of good.

Why doesn't anyone note that the frog acted outside it's nature?

...huh. That'd be to distrust scorpions and not give free rides. Which makes sense.

Re:Don't sting me bro

[meta-monkey]

The fact that frog acted outside his nature doesn't have anything to do with the fact that the scorpion acted within his.

You could say there are two morals:

1) Don't expect others to act outside of their nature.

2) Don't do risky stuff without sufficient potential reward.

Not a Turing machine

[schmidt349]

A Turing machine requires an infinite memory. The human brain is, at best, a linear bounded automaton.

Re:Not a Turing machine

[ThatsDrDangerToYou]

Our tape goes up to 11.

Re:Easy...

[Anonymous Coward]

I'd think you were joking if your posting history wasn't that of a right-wing sociopath.

Quantum Mechanics and Determinism

[clonan]

The problem with Turing machines is that they are by definition a deterministic system. A certain input will give a specific output. That is why they can't make a "judgment" call.

The universe as a whole is NOT deterministic as Quantum Mechanics proves. QM is based on true randomness (obvious a simplification but go with it for this conversation). Our 'machines' deal with this randomness and even incorporate it into some operations. So a specific input will NOT always generate the same response.

It is the assessment of random changes that MIGHT give us the ability to make these sorts of judgments....

Re:Quantum Mechanics and Determinism

[Dutch Gun]

I'm not sure I'd describe a "judgement call" as being non-deterministic. It's really better described as fuzzy logic, and computers do it all the time, such as in spam filters. The difference is that humans have a lifetime of learning and context for them to help make those judgments, where most computer algorithms don't have that extended context to draw from.

I don't see how true randomness has anything to do with these sorts of decision-making processes or with quantum mechanics in general.

Re:Quantum Mechanics and Determinism

[Anonymous Coward]

QM is based on true randomness

[citation needed]

QM modelling is based on probabilities, but that does not imply that interactions at the QM level are random by any stretch of the imagination. Do you think that fundamental particles just happen to pick various state parameters (momentum, charge, spin, etc.) based on how they feel at any given moment?

The question of whether the universe is deterministic or not is a pretty open question the last I heard.

Re: Quantum Mechanics and Determinism

[Anonymous Coward]

Is it really true randomness or maybe unknown (undiscovered) causation?

Re:Quantum Mechanics and Determinism

[Anonymous Coward]

Umm...fuzzy logic is still very deterministic. It is "fuzzy" because it allows for values besides binary (yes or no) and can deal with analog controls much better.

Re:Quantum Mechanics and Determinism

[clonan]

http://en.wikipedia.org/wiki/Quantum_mechanics/

I know wikipedia but I am lazy and don't really care...

Check out "During a measurement, on the other hand, the change of the initial wavefunction into another, later wavefunction is not deterministic, it is unpredictable (i.e., random). A time-evolution simulation can be seen here.[28][29]"

There are two cited references.

A large number of quantum particles seem to act in a deterministic way but this is simple the law of large numbers.

Re: Quantum Mechanics and Determinism

[clonan]

My admittedly limited understand is that the randomness is a fundamental requirement of the theory.

I find QM interesting and have read up on it but I am certainly not a theoretical physicist...

Re:Quantum Mechanics and Determinism

[clonan]

My original post was simply pointing out that the human brain is NOT and can never be a Turing machine due to the fundamental randomness of the universe. This means that no study of Turing Machines can make any claim on human judgment calls.

I am not sure the random nature of the universe is sufficient to allow for true 'judgement' but it MIGHT.

Re:Quantum Mechanics and Determinism

[Anonymous Coward]

But it has uncountably many truth values... so it can't really be implemented in a computer anyway, only a rough approximation of it.

Re:Quantum Mechanics and Determinism

[itzly]

It's easy to extend a Turing machine with a random generator, but it's not really going to improve its function in any useful way.

Re:Quantum Mechanics and Determinism

[meta-monkey]

A universal turning machine can only compute computable sequences, so no, it can't model the non-deterministic nature of the universe. However, you're making the assumption that "judgment" is not a computable sequence. What makes you think judgment is not computable? Is there sufficient evidence to conclude that consciousness and decision making ability rely on quantum effects? If you feed a human the exact same information with his brain/body/environment/history in exactly the same state, will he not make the same judgment twice? Obviously it's impossible to rewind time to get the human in exactly the state he was at the time of the first judgment, but if one could, I don't see any reason his judgment wouldn't be deterministic. And even if it's not, I would posit that the randomness introduced by the non-deterministic nature of the universe is an exceedingly, exceedingly small component of the outcome of the judgment.

Re:Quantum Mechanics and Determinism

[clonan]

Read the definition of a Turing Machine. They are absolutely deterministic.

Re:Quantum Mechanics and Determinism

[leonardluen]

A large number of quantum particles seem to act in a deterministic way but this is simple the law of large numbers.

it should be pointed out that our computers are built on this system as well, not just our brains.

if the universe is not deterministic then how can our computers be?

Re:Quantum Mechanics and Determinism

[clonan]

Actually no, I am stating that the human brain is NOT a Turing Machine and can never be a Turing machine therefore any attempt to correlate the original article with "human judgment" is fallacious.

I did say that while I don't know for sure, the ability to integrate and question the randomness inherent in the Universe COULD provide a method of providing a truly independent judgment.

To answer your questions:

No, if you feed a human being the same information twice you often will NOT get the same response even when the person has no ability to remember (say through brain damage). For a fun image think of the movie "50 first dates". Now obviously the simpler the question the more likely you will get the same response, however when you have complex questions you start to get divergent answers. Agreed they do tend to be similar but not identical.

Remember the randomness affects everything and will end up having a large number of effects that will average out to an apparent deterministic system unless tested carefully.

Re:Quantum Mechanics and Determinism

[clonan]

Computers spend a LOT of power to eliminate the randomness. In-fact randomness is defined as an "error" in the typical computer and generally frowned on by the end user.

In addition we are talking about a Touring Machine which is a thought experiment and therefore not really subject to the laws of the universe..

Re:Quantum Mechanics and Determinism

[radtea]

My original post was simply pointing out that the human brain is NOT and can never be a Turing machine

This is true but it has exactly nothing to do with quantum mechanics or randomness. To see this, understand that we can't tell if QM is "truly" (metaphysically) random or just mocking it up really cleverly. Or rather, we can tell, but using inferences so indirect that they make no difference to the operation of the human brain, which is an extremely strongly coupled environment that is completely unlike the areas where "true" quantum randomness exhibits itself. No process in the brain depends in any way on metaphysical randomness: we could write a Monte Carlo simulation of the brain using entirely pseudo-random number generators and it would be accurate.

Brains and robots and computers have a number of properties that Turing Machines do not, however. In particular, I/O and realtime interrupts. Turing's model is strictly limited to what is on the tape. There is no way to hook up a sensor to a Turing machine and still have any of Turing's proofs still apply. The moment you allow even one bit to come in from the outside world, you no longer have a Turing machine.

So what Turing machines cannot do is not all that interesting to the design of robots. Turing's most important proof is that of universal computation: that any machine that can do at least what a Turing machine can do can compute anything that any Turing machine can compute. But this tells us nothing about what a machine that contains a Turing machine but is not itself a Turing machine can do. Robots (and humans) exhibit emergent properties from their interaction with the world, and that interaction is simply not part of Turing's model.

Re:Quantum Mechanics and Determinism

[weilawei]

Never heard of analog computers, I take it?

Now get off my lawn.

Re:Quantum Mechanics and Determinism

[weilawei]

Chaotic and random are not the same thing.

Re:Quantum Mechanics and Determinism

[Stickasylum]

Pretty sure that's not the kind of "judgement" that we want our military using, whether human or robot!

Re: Quantum Mechanics and Determinism

[Anonymous Coward]

The whole point is that he ended up putting ALL computers in the same class as a Turing Machine. It doesn't matter if you have interrupts or IO, you're still no more computationally powerful than a TM.

Re:Quantum Mechanics and Determinism

[clonan]

Why not? This is the best possible case.

The argument in the original article is that a Touring Machine is limited by the data available to it (even if that data is infinite). This means that the TM can't make a "judgment" because it will always find the same result based on the data provided by the programmer.

QM based randomness means that "new" data is added to the system and the system must now accommodate this new and unpredictable scenario. It is the ability to analyze "new" data that allows for judgment.

Re:Quantum Mechanics and Determinism

[Stickasylum]

A coin flip is very different from what we usually mean by a judgment. Would you trust a human soldier that makes random choices about what to do in life-or-death situations? Ideally, they should be able to explain their reasoning / training in each situation, and if the decision does not hold up to retrospective scrutiny, should be held accountable for their actions and/or should adjust their reasoning used in future situations. "Judgement" with a large component of randomness involves no reasoning, and so I'm skeptical of it's usefulness in life-or-death situations (and whether it should be called "judgement" at all).

Re:Quantum Mechanics and Determinism

[Rich0]

The universe as a whole is NOT deterministic as Quantum Mechanics proves. QM is based on true randomness (obvious a simplification but go with it for this conversation).

The fact that the theories that describe QM are based on true randomness does NOT mean that the universe as a whole is not deterministic.

First, we already know that QM is limited, because it doesn't account for gravity. Second, even if QM as it is formulated today did accurately describe the universe to the best of our ability to measure it doesn't mean that another theory that is deterministic couldn't also describe the universe.

Many of the arguments made on the basis of QM go beyond the actual math and are just fairly subjective interpretations of the implications of the theory. It might be established that you can't measure the position and velocity of a particle with arbitrary precision, but that doesn't mean that a particle doesn't have a precise position or velocity. QM tends to be formulated in terms of describing what you'd get if you performed an experiment. That doesn't necessarily mean that the universe actually works in this way.

Think about it - how could you even prove that a given chain of events was or wasn't deterministic? You can't ever reproduce the same chain of events perfectly, so it is impossible to test. That means any claim that there is or isn't determinism in nature is basically unfalsifiable.

Re:Easy...

[gstoddart]

... or for the pale face.

... or six inches above the knot of the neck tie.

No CPU can run at all

[Anonymous Coward]

I just used the same technique to prove that no CPUs are capable of running:
The typical maximum running temperature of a CPU processor is ~95C.
Now all I have to do is plug temperature into the true but completely unrelated find_max_integer() algorithm, and we see that all computers run at a temperature of about 2,147,483,647C.

Which is the most cost-effective to replace?

[Anonymous Coward]

Humans or robot?

As long as we keep printing, the "free" market will decide for us.

Halting problem

[Anonymous Coward]

Halting problem proves that a computer program can't solve an ethical dilemma implemented in terms of a halting problem.

Obligatory Futurama quotes...

[Anonymous Coward]

Fry: "I heard one time you single-handedly defeated a horde of rampaging somethings in the something something system"
Brannigan: "Killbots? A trifle. It was simply a matter of outsmarting them."
Fry: "Wow, I never would've thought of that."
Brannigan: "You see, killbots have a preset kill limit. Knowing their weakness, I sent wave after wave of my own men at them until they reached their limit and shut down."

Seems to be a theme...

[fuzzyfuzzyfungus]

It is certainly interesting that deciding whether or not to kill some fleshy humans can be demonstrated to be circumscribed by the halting problem; but it's always a bit irksome to see another proof-of-limitiations-of-turing-complete-system that (either by omission, or in more optimistic cases directly) ignores the distinct possibility that humans are no more than turing complete.

Humans certainly are enormously capable at approximate solutions to brutally nasty problems(eg. computational linguistics vs. the average human toddler); but that is very different from a demonstration that, say, humans possess an Oracle, or are some sort of hypercomputational system, rather than simply being enormously good at hard-but-not-theoretically-intractable problems in certain areas.

In this instance it's especially galling because we've only been philosophizing about acceptable losses, 'just war', legitimate causus belli, 'proportionality', and whatnot for about as long as we've been chucking spears at one another. It's a pure commonplace that a mixture of overkill and underkill is an effectively certain outcome when you go to war. It is interesting that, in principle, kill/no-kill is subject to the halting problem; but has anyone (aside from sleazy assholes hyping 'smart' weapons) ever asserted that kill decisions would be anything but imprecise?

Re:Seems to be a theme...

[Anonymous Coward]

Philosophers and munition sellers punting back and forth.

as most problems, there is a third path, with the Heurestic analysis system. The machine could come to a kill or no kill and form optimum profile for reacting to the threat much better then a human. mainly because it
A) processes information faster
B) has little if any self preservation instinct,
C) can not be damaged nearly as easily as a human
D) damage is not as much of a problem for it as a human
E) No fear and annoying physiological quirks like a human
F) intrinsically has much better situational awareness then a human and can integrate sensor data and team datalinks much easier then a private with ears and eyes.

there is an end to the halting problem

[swschrad]

branch to the HCF operand on any error.

(newbies, that is the Halt, Catch Fire command)

Re:there is an end to the halting problem

[fisted]

Would've been a good joke if you hadn't ruined it by explaining it right away.
0/10 didn't even chuckle.

Re:there is an end to the halting problem

[VorpalRodent]

Notably, depending upon how HCF is implemented on any given hardware, this also addresses the underlying question of whether the robot could decide to kill the human. In this case, we apparently default to true.

Re:there is an end to the halting problem

[Minwee]

The behaviour of that instruction is undefined, so when a processor encounters HCF it is legal for it to make demons fly out of your nose.

Re:there is an end to the halting problem

[Anonymous Coward]

I always preferred the KOI instruction, and my favorite: BBW instruction.
There, figure those out.

Can't decide WITH CERTAINTY

[naasking]

One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of.

It's not curious at all. The goal was to determine if a computer can decide with certainty whether another agent intends to do harm. This is obviously unsolvable, even for humans. Of course, we don't require humans to be absolutely certain in all cases before pulling the trigger, we just expect reasonable belief that oneself or others are in danger (for a self-defence argument). Reasonable belief is even easier to decide for computers, since the internal states resulting in that conclusion are fully available to us (unlike the human mind).

It's just wrong

[Anonymous Coward]

Not so insightful.

It is too easy to make a program hard to analyze. The problem was - An automated railway switch. Will it ever knowingly send a train to run over someone?

Consider this, then. The software do not try to flatten people usually. But it spend some fraction of its time trying to find a counterexample to Goldbach's conjecture. If it ever find a counterexample, it kills someone. A machine may have trouble solving that - especially since people haven't solved Goldbach's conjecture yet. Will you assume guilt? Or perhaps our crazy coder has proved that Goldbach's conjecture holds - so the machine will never find a counterexample. That code is just there to tease us.

The article is also a bit silly. The military will not worry that their killbots may be tripped up like this. They will not get stuck in dilemmas, killbots will simply do what they're designed to - search&destroy. Robots will not stop to contemplate difficult ideas - similiar to how soldiers don't do that either.

This is moronic.

[Karmashock]

By this logic, computers couldn't do anything. Since there are conditions in which the machine must do a thing and not do a thing. And they are generally pretty reliable once properly set up to not do certain things they've been programmed to not do.

What this argument is saying is that despite the fact that computers are known to be reliable in many situations we can't rely upon them to do this specific thing.

Because.... ?

Now am I fan of using robots to kill people? No. I'd rather prefer not to have that happen for moral reasons. I think people must take responsibility for each kill. And yes, a big bomb dropped from high up doesn't give you the same feeling as looking down the sniper scope and popping someone in the face. But at least there is someone there pulling the bomb release trigger.

In regards to killer robots, I think they're reasonable as smart mines. That is, area denial weapons. You feed in some parameters about where their kill zone begins and ends, feed in some Identify/friend/foe protocols, then mark the zone off with some "area patrolled by killer robots beyond this point" signs.

I find that reasonable.

If I am in a military compound in potentially hostile territory and my base is guarded by killer robots... and you attack my base... Eat shit and die mother fucker. :D

But I wouldn't use them aggressively against ground targets. I think they're also reasonable as dog fighters if we can get that working properly. A small fast moving enemy jet in contested air space is a combatant. I think air supremacy drones are reasonable as well. If you enter that air space with a plane like that and you don't squawk the right codes... then what do you expect is going to happen? Best case someone is going to fire an AA missile at you which might end you right there. The drones are potentially worse because they'll shadow you to get a very solid lock on your position and then fire a missile that will maintain a wireless link to the drone to get updated targeting information and hopefully not get spoofed by EW or chaff.

Re:This is moronic.

[geekoid]

The use of robots weapons has dramatically reduced casualties.
From systems that loiter, to self guiding, civilian casualties, are way down compared to previous wars for similar targets.

Re:This is moronic.

[cwsumner]

... Best case someone is going to fire an AA missile at you which might end you right there. ...

Um ... the AA missile -is- a killer robot !
All of this discussion is about stuff that has already happened. Just because they don't look like humans, is beside the point.

Re:This is moronic.

[Karmashock]

I agree. I wasn't implying further that AA missiles are not killer robots. I was pointing out rather the pros and cons of autonomous drone warfare and its reasonable applications.

Re:Easy...

[Anonymous Coward]

Well on the plus side, it will kill off 90% of Redditors.

Misunderstanding the halting problem

[david_thornley]

The halting problem says that you cannot determine if any completely arbitrary program will necessarily end, and this can be generalized to show that the output of programs cannot always be predicted. It does not say that it is impossible to determine if any program in a set bounded by certain restrictions cannot be predicted.

Take an X-ray machine for example. We know these can kill people (look up Therac-25). However, if we write an overall program that calls a supplied program to calculate the treatment duration, and have a routine to control the machine and which has a hard limit on the duration, then it doesn't matter if the supplied program can, in some circumstances, calculate an excessive duration, because the patient can't get that dose.

It is possible to write programs with termination proofs. Not all programs can be written that way*, but most programs we would use to control equipment can be. If we can write programs that certainly terminate, the halting problem is completely irrelevant. We can write programs with proofs that they won't kill a human by any agency we can foresee, although we can't write all possible programs that way.

*One example of a set of programs that can't currently come with a termination proof are ones to test unsolved mathematical problems. Consider Goldbach's Conjecture that all even numbers greater than two are the sum of two primes. We don't know how to prove it or disprove it (if either is possible), but we can test each even number to see if it's the sum of two primes. If that program halts, the conjecture is false; otherwise, it's true.

Re:Misunderstanding the halting problem

[Endlisnis]

Take an X-ray machine for example. We know these can kill people (look up Therac-25). However, if we write an overall program that calls a supplied program to calculate the treatment duration, and have a routine to control the machine and which has a hard limit on the duration, then it doesn't matter if the supplied program can, in some circumstances, calculate an excessive duration, because the patient can't get that dose.

What makes you think that "hard limit" enforcing code can't have a bug in it? I assume your answer is that it's such a small program that you can do a formal proof that it has no bugs. Fine, but what about the OS it runs under? Or the memory controller? Or the actual memory? Any of those can be wrong in a way that could, in theory, cause a patient to be given an overdose. If we can make a robot that is more accurate than a human (at practically any task), we probably should (ignoring issues of liability and macroeconomics), but we have yet to make *anything* that is 100%, let alone a computer controlled robot.

Re:Misunderstanding the halting problem

[ceoyoyo]

These things are generally implemented very simply, in hardware. There is no memory controller, or memory, or OS. If you want to be really sure, the algorithm is implemented as a mechanical device.

Yes, even those can fail. But hardware failure is a separate problem from software failure.

Re:Misunderstanding the halting problem

[david_thornley]

The discussion was about software, and whether it could be proved to have certain properties, so I limited myself to that. The original claim ignored possible hardware problems (which are essentially infinite). It is possible to write provably correct software for a useful number of problems. (It is also possible to make mistakes in proofs. We're only human.) In the case of a program controlling a machine, we could have a program with some provable properties that directly controls the machine without going through the OS.

In short, the claim in TFS is wrong. The halting problem does not mean what the people who wrote TFA seem to think.

Re:Misunderstanding the halting problem

[Anonymous Coward]

Just because your program controls the hardware "directly", does not mean that it's unaffected by the OS. If the OS has a bug in it (or just some feature that's unexpected) which causes it to shedule your process in a manner which breaks your hardware controller, then your software does not produce the desired result.

Re:Misunderstanding the halting problem

[Anonymous Coward]

The post I was replying to was about software. Nothing in this world is perfect, and replacements for humans don't have to be, they only have to be BETTER. But this thread started with a comment which seemed to imply that you could write an infallible system for this type of stuff.

Dodge Omni Horizon

[Impy the Impiuos Imp]

They're confusing the halting problem with the horizon pronlem in game theory.

Like a chess grandmaster who sees a trap 15 moves down the road that a lesser-skilled player cannot.

The good ones took advantage of this when playing compters. Kasparov is still convinced he could win a rematch.

Evil robots can therefore feel joy?

[Overzeetop]

If it turns out to be impossible to build a purely evil robot that would always kill maliciously, does that mean that a purely evil robot would occasionally kill for the sheer joy of watching someone die?

Re:Evil robots can therefore feel joy?

[Anonymous Coward]

Too much work programming a robot to be evil. It just needs to be efficient, mowing us all down until there's none of us left.

Re:Easy...

[plover]

Well on the plus side, it will kill off 90% of Redditors.

Depends on how it identifies 'neckbeards'.

Re: you're an idiot

[Anonymous Coward]

Sounds like you have it all figured out. That's just peaches :) Guess you can go finish loading the guns you stole from your mom's unlocked cabinet and continue doing what you were doing.

neither can humans

[silfen]

The problem isn't usually the "halting problem", it's lack of complete information, errors, and a whole host of other limitations of humans and the real world.

We have a way of dealing with that in the real world: "when in doubt, avoid killing people" and "do the best you can".

It's no different for robots. Even the best robots will accidentally kill people. As long as they do it less than humans in similar situations, we still come out ahead.

Re:neither can humans

[wiredlogic]

We have a way of dealing with that in the real world: "when in doubt, avoid killing people" and "do the best you can".

Unfortunately the US solves this with "use a bigger bomb and classify all the dead innocent bystanders as combatants".

The "researchers" cheated

[MobyDisk]

The "researchers" did not prove anything to do with what the article claims. What the article really proved is that it is impossible for a robot to make an ethical decision, if that ethical decision is based on analyzing source code.

They created a scenario where the "robot" must determine if a computer program was written correctly or not. An ethical decision hinges on that. If the program is written correctly, it must do one thing, and if the program is written maliciously then it must do another. Then they point out that the halting problem makes it impossible to guarantee that the computer program was written correctly or not. And since the computer program involves a life-or-death decision, therefore, robots can't make life-or-death decisions.

Using that logic, I can prove that a robot can't do anything. Let's try it: I will prove that a robot car cannot decide if it is safe to make a left turn or not at an intersection. I do this by imagining a scenario where the software for the traffic light might be written incorrectly. So my robot car must first analyze the software for the traffic light, determine if it is written correctly, then only make the left turn if the traffic light software is correct. Since the halting problem shows that it is impossible to create a general purpose robot car that can analyze the source code to all other pieces of software, it cannot be guaranteed to make the right decision about the intersection in this case. Ergo, robot cars are impossible and we should not make them.

Actually, all I proved is that a robot can't decide if it is safe to make a left turn if that decision is based on analyzing the source code to the traffic light.

P.S. Yes, I simplified of what the halting problem says. It doesn't say the robot absolutely can't analyze the software. It says that it may not be able to analyze the software, because the software may never end, and the robot can't determine that. I didn't want to go into that subtle difference in my TLDR analysis.

The logical flaw

[jd.schmidt]

The flaw in their logic is this, we don't really care I if works every time, just most of the time. So if the robot can do the right thing more often than not, rather like people, to such a degree that we view it as being a net benefit, we are willing to accept the occasional mistake or failure for a net overall viewed good. So they would have to prove the program would fail more often than succeed, which they probably can't. That said, I DID wish it were possible enforce Asimov's laws of robotics. Maybe some day..

Death

[Charliemopps]

Behold how the blind lead each other
The philosopher
You know so much about nothing at all

Another case of Specific vs General case confusion

[nichogenius]

This article describes a logical/ethical thought experiment, but seems to come to the wrong conclusion. The halting problem states that in a general case, it is not possible to design an algorithm that will perfectly decide if another algorithm will ever stop 100% of the time. However, it doesn't take into account that very often, it is possible to determine if an algorithm will halt or not using standard algorithm-analysis techniques. These are referred to as specific case algorithms. What the 'ethicists' did was come up with a 'specific case' that proves that a general algorithm for determining that the 'lesser of two evils' choice between taking a life and not is NOT ALWAYS possible. However, a simple solution presents itself.

Let a robot be given a choice to take any action that could result in knowingly ending a human life. If the robot does not encounter the halting problem, it will make a very educated and justified decision. If it DOES encounter the halting problem, perform no action. As the robot will probably be stuck in some while-loop or recursion function anyway, this isn't really a problem. Also, it is useful to note that the lack of an action that results in a human's death is not a morally wrong thing... it's simply a mechanical error. When insufficient data is available to make an ethically sound decision, it is arguable that the best course of action is not to interfere. You won't ever be guilty of making things worse, even if you COULD have made things better.

Granted, my solution falls apart when you take into account the very real possibility that the robot is in a position where the default behavior should be to take an action. For example, a robot in charge of activating the brakes on a train should NOT be stuck in a halting problem loop when it comes time to perform normal behavior. I'd argue that this can be solved by simple multi-threading strategies... just remember to keep your robot's ethical thread separate from its normal function thread :D

"Halting Problem Proves"

[Anonymous Coward]

The only thing proven by a headline that starts with this is that the author does not actually understand the halting problem.

When theory conflicts with observation

[istartedi]

When theory conflicts with observation, You have two choices. You can modify your theory to fit the observation, or your observations to fit your theory. The first choice is what we generally regard as science. The second choice occurs in a number of circumstances including, but by no means limited to: religion, politics, mental illness, and general stupidity.

Note, checking to make sure that your observations are accurate is not the same thing as modifying them. "Did I fail to see the gorilla?" is valid when theory indicates gorillas should be present. "I saw a gorilla because my guru said I should" isn't.

Or use the current method ...

[BarbaraHudson]

Or use the current method ... "Kill them all and let $DIETY sort it out."

Re:Or use the current method ...

[Anonymous Coward]

Sorry to pick nits, but it's spelt "deity". If it helps to remember, the word derives from Latin "deus" (god).

They can, however, make less mistakes then us

[iamacat]

A well programmed police bot will not fire 3 shots in the back of a fleeing teenager. It may well only fire shots when innocent humans are in immediate danger and permit its own destruction otherwise, as more bots can always be sent to complete the arrest non-lethally.

The same bot might roll over a toddler hiding under the blanket because its programming doesn't cover this exact case and it doesn't have imagination. However, these mistakes will be rarer than human police/soldier. And after they happen once, every unit will get a software update with a new safeguard and apply it correctly every time.

Re:They can, however, make less mistakes then us

[Anonymous Coward]

A well-programmed Terminator will not fire a mere 3 shots into the back of a fleeing, unarmed, teenage John Connor. It will fire 3 DOZEN shots into the back of a fleeing, unarmed, teenage John Connor. And just for good measure, 6 DOZEN shots into his Mom.

Re:you're an idiot

[clonan]

Please tell me exactly when a specific electron is 14 billion 123 million 567 thousand 324 years and 23485723048752 seconds after the big bang...

You CAN'T!

It isn't just knowing the starting conditions it is about being able to calculate every state between the beginning and the end.

What if we flip it?

[Anonymous Coward]

"Robots" (i.e. computers) should never be counted upon to save lives because the programming may 'go wrong.'

Human version

[mysidia]

One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of.

Instead of considering an 'Evil Programmer'..... consider 'Evil Judge', 'Evil Military General', 'Nazi', or 'Evil Dictator'

And instead of just deciding this issue; add the problem of surviving this issue together with the problem of deciding how to maximize your chances at survival and happiness in concert with previous issue.

Ommmmm...

[fustakrakich]

The past doesn't exist. The future doesn't exist. You are standing on the pinnacle of now. Watch your step.

Re:Ommmmm...

[geekoid]

I make judgments about what to do in the future based on what I did or learned in the past.
Of course they exist.

Re:Ommmmm...

[fustakrakich]

The past and future existed/will exist. Only your memories and anticipation exist now.

Re: Ommmmm...

[jovius]

Except for example in the case where the said memories and the material for you to realize your existence are being fed to you, without you realizing that, from a cyclical tape storage of your existence. It doesn't really even matter if it's on reverse. But yes, it's the moment of now that spans the entire universe.

Robots are cut no slack

[mariox19]

The real problem is that the actions of people, in some circumstances, are considered beyond good and evil, and all the silly hypothetical situations in the world doesn't begin to capture this. In the heat of the moment, with only seconds to decide, people can't be relied on to make a choice that conforms to some explicit moral code. On account of that, when faced with passing judgement on the actions of people in emergency situations, we don't pass judgement; rather, we forgive them.

Robots, however, are programmed, and "split seconds" don't mean the same thing to robots that they do to us. Thus, there is no way around what they're going to do. They will be programmed to do one thing or another, and someone is going to have make the bad decision—since, in many cases, there are no good decisions to be made. And that poor bastard may have to program the machine anonymously, because what he will get is not forgiveness but, "What were you thinking!"

Robts who kill

[Anonymous Coward]

Pris Stratton, Female, a “basic pleasure model,” NEXUS-6 N6FAB21416, had a habit of 'retiring' humans between her legs. I shudder to think what she would consider as advanced pleasure.

Not Computable

[Anonymous Coward]

If the problem is shown (by Halting Problem reduction) to be non-computable, then the *problem* itself is non-computable. It's not that a *robot* can't do it. A *person* can't do it either.

Cooked ethics

[Dr. Evil]

I'm as sure you're wrong as I'm sure the last digit of pi is 6.

Human Up!

[Anonymous Coward]

Duh! Did we really need the danger of lethal machines to be proven formally? Of course robots can be lethal, as can any machine. Lethal robots are here to stay. And of course they can go postal. The only real issue here is how to deal with them responsibly, or as one might say, "Human Up!".

Who are we to discriminate against robots as a class? Whether the human brain is a Turing machine or not, we are certainly less predictable than virtually any machine we might construct (unless, of course, we decide to make unpredictable ones). Like humans, robots are corruptible/hackable, and capable of going astray/insane. So, just as we have systems in place to monitor and regulate humans, we need systems in place to monitor and regulate robots; i.e. laws of robotics.

The laws of robotics (unfortunately, not the famous three laws in this case) should certainly *not* be implemented entirely within the same system/subsystem that implements autonomy. We need layered fail-safe systems, and not just for robots that are designed to be lethal. Some regulating subsystems should be implemented within each autonomous system, while other such systems should be paired with said autonomous systems, to monitor and override them in the case of malfunction. And still yet other safety systems must be implemented external to the autonomous systems themselves, much as we have law enforcement and military organisations in the case of humans. This will not only help to prevent robots from going helter-skelter on their own, but also more difficult to hack. Of course, there will be scofflaws who build robots without these fail-safes. But the vast majority of robots will be law-abiding "folk" that will loyally serve their human "masters".

Even so, perhaps we should hold off giving them nukes for a while (doh!).

"Kill them all, let god sort them out".

[Anonymous Coward]

The TFA forgets one, small point.

If I were to program a killer robot, the exact command sequence would be fairly simple.

"Kill them all, let god sort them out".

The robots would then simply be released via airdrop into the enemy lines, and my soldiers would be kept safely out of harms way while the robots sweep away all opposition, friendly or otherwise.

Bullcrap.

[Rinikusu]

I just finished my ISIS killing robot and it's doing just fine. It hasn't killed any ISIS members, yet, but it does seem to be doing a fine job killing hipsters. I might not fix that for awhile...
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
(I've totally got an ISIS beard.. Please don't kill me, robot.)

Seriously Flawed

[meustrus]

First let's say that the explanation in the article makes some pretty weird alterations to the trolley problem to come to its conclusion. So first we have the problem that the trolley problem is not an adequate measure of whether a robot can "correctly decide to kill humans". From there it goes through some weird permutations until there is a decidedly "correct" solution to the trolley problem: a computer program designed by a known villain is about to be installed on a switch that could make a decision to injure maintenance workers. And since another computer program cannot determine whether the first program will ever halt, it cannot always make the right decision.

The flaw is right there in the summary:

One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of.

In what way could anybody always prove that a piece of computer software is safe and correct? Let's assume that the hardware is safe and an expert in machine code is available to make the determination. Could that expert ever make the right choice? Of course. But could the expert always make the right choice? The Halting Problem doesn't state that a program can never determine if a program will halt. It only states that a program cannot always determine if any program will halt. A machine could use exactly the same methods available to us humans (recognizing certain design patterns, certain known logical structures with known outcomes) to usually make the right choice, but could not always know what the right choice is. The disconnect that might make one think a computer is somehow less capable of making this decision is in believing that a human being can make a better determination. A human can't. If the program is written in a strange or obscure manner, the human can't know what will happen either. And that's where we encounter the halting problem: you can't always know for sure without running the program, and if the program never halts (or never makes a bad judgement), that's not proof positive that it doesn't halt (or is safe).

Ultimately the real "flaw" is the way this result has been picked up. The headline "Halting Problem Proves That Lethal Robots Cannot Correctly Decide To Kill Humans" is a lot more sensational than the title of the paper, "Logical Limitations to Machine Ethics / with Consequences to Lethal Autonomous Weapons". The Medium headline and article claim that this paper "proves" something about the capabilities of "lethal robots", when all it really does is prove limitations of machine ethics. It isn't really about lethal weapons; based on this result, an algorithm cannot always make the morally correct choice, regardless of whether or not that choice involve killing. And the reason? Because sometimes, making the morally correct choice requires information that is provably impossible to always obtain.

Kill Limit

[QuantumPion]

Just make sure the killbots are programmed with a pre-set kill limit. If anything ever goes wrong, all we have to do is send waves after waves of our own men at them them until they reach said limit and shut themselves down.

Re:Kill Limit

[Kevin Fishburne]

That's an excellent idea, although I thought of an improvement. Instead of sending waves of our own men to clear the kill counter, we could air drop inflatable sex dolls filled with snakes. Man, future war's gonna be crazy!

Not so curious corollary

[Anonymous Coward]

I agree with the authors' findings, but not their conclusion.

The corollary isn't very curious, and is actually at the heart of the issue. It's pretty obvious that humans can't reliably decide whether to engage a target or not: there hasn't been a conflict in history without friendly fire incidents or civilian collateral damage.

If you're not comfortable with that (and it's a reasonable position) then logically you ought to be a pacifist and would be opposed to any lethal force drone on principle.

If you are comfortable with mistakes being made (presumably as long as the outcome of the war is for the greater good or something like that) then the question is a simple one of whether robots can be designed to have a lower incorrect engagement rate than a human soldier. If so, then they are (relatively speaking) a good thing as their widespread use will reduce collateral damage. If not, then they are (relatively speaking) a bad and probably unjustifiable thing.

There are a whole lot of complex questions about the use of lethal force drones, but the question posed in the summary is not one of them.

Killbots should have a kill counter

[Anonymous Coward]

After 999,999 kills it would be safe to say that the killbots have faulty AI that have rendered them a menace to the very survival of humanity. Therefore they should be programmed to shut down after the millionth kill. To successfully defend against a killbot attack, you must send millions of millions of troops to their deaths until the counter for each killbot overflows.

Solution: avoid generalities

[Anonymous Coward]

A machine can't decide in an arbitrary case, but if programmed for a specific set of conditions it could. The machine might not know about (or be able to decide about) additional factors that a human would be able to comprehend, but the same can be said of human beings, as in the extreme we clearly can't predict much about the future whereas a more sentient being might do better.

Bullshit, bullshit, bullshit.

[Anonymous Coward]

Speaking as a computer scientist myself - one who's against the use of unmanned drones, even - what a completely fucking stupid article. The people behind the "result" seem to be under the impression that the unsolvability of the halting problem means that any robot, on trying to analyse any computer program, will halt and flail helplessly until a nice human can come along and solve the problem for it. This is totally wrong. Yes, there exist programs which a robot can't analyse. The thing is, assuming the Church-Turing thesis holds (which is incredibly uncontroversial), humans can't analyse these programs either. Conversely, robots would be capable in principle of applying exactly the same sort of heuristic or logical methods as a human in the same situation. So their central premise - that a lethal robot is more dangerous than a lethal human - falls at the first hurdle.

Also, given enough time and resources, you could prove whether pretty much any program used in a production setting will halt. The halting problem might be a relevant issue when you imagine, say, a program that will enumerate the zeroes of the Riemann zeta function and halt if it finds one that violates the Riemann hypothesis. (Although even then probably not, since very few people think the Riemann hypothesis is undecidable.) It probably won't be a relevant issue for a program that will operate a set of train tracks.

So their thesis boils down to:

1. Here's a monumentally contrived problem that a robot can't solve, and where a life hangs in the balance.
2. We're going to pretend, wrongly, that a human can solve it.
3. Therefore, in this incredibly specific situation, a lethal human would be preferable to a lethal robot.
4. Therefore, lethal humans are preferable to lethal robots in general.
5. Therefore, we shouldn't make lethal robots.
6. We're going to completely ignore the fact that the same argument would apply to e.g. EMT robots or rescue robots, because guns are scary.

I hope for their sake they got taken out of context or something. (Their problem would be quite an interesting philosophical take on absolute morality, for example - a situation where there's a right thing to do, but where it's literally impossible to tell what it is.) If not, and they really are trying to apply this argument to policy in the real world, then it's honestly an embarrassment.

Nonsense

[Anonymous Coward]

It just means that robots should never be given the task of approving software that can kill people.

'Clearly preferable'

[Anonymous Coward]

The authors created scenarios in which they claim that one choice is 'morally clearly preferable over the other'. Instead, they have created scenarios that indicate their own highly-restricted and dangerous ethos. In example 2, 3 and 4 for example, the authors create scenarios in which a known villainess has placed herself in position to do grievous harm to innocents, but have constructed arbitrary and unrealistic modifiers to the situations such that the authors posit that attacking the villainess would cause 'unnecessary harm'.

Harming villains is never unnecessary. Causing harm to villains is, in fact, a moral prerogative.

Arbitrarily choosing a highly-limited ethical methodology and declaring it to be 'morally clearly preferable' makes for an unconvincing argument.

I'm pretty sure you *don't* think.

[Anonymous Coward]

What makes you think that the people who intend to use these robots, i.e. the US military and its allies, actually care about killing civilians?

Because we haven't used nukes yet, and if we really didn't care, we'd push the button, and be done with the entire problem once and for all, since it's really hard for dead people to commit terrorist acts?

In case you were wondering, the question you were actually asking with this bit of sophistry, is whether or not the U.S. is engaged in "total war", and it's not. Collateral damage is not the same thing as a scorched earth policy, and the difference is large enough that there are actual rules of engagement which hamstring the troops subject to them sufficiently that in some cases they are unable to defend themselves.

Technically, what the U.S. and its allies are engaged in in the Middle East is what's called a "police action", and pretty much has been from day one, since there's no formal declaration of war by the U.S. against a nation state, nor ratification of a declaration of war by vote of congress.

The U.S. is, of course, loathe to use the terminology "police action", since that's also the terminology which was used during the Vietnam conflict - and we've renamed the Vietnam conflict as the Vietnam War" as a means of redacting the fact that we were engaged in a police action, and that it was a conflict, and not a war.

Misleading or just wrong?

[tentative]

There can't be a program that verifies the correctness of *any* algorithm. That's absolutely not like saying that there can't be a robot that verifies the correctness of *some* algorithms, and refuses to act on the rest.

The human mind is not a turing machine

[TheNarrator]

What causes the emotion of happiness? Serotonin? Dopamine? Endorphins? But what do these chemicals do except trigger receptor sites or cause further chemical reactions. What is being modulated to create that emotion? Is it an EEG pattern in the brain perhaps that modulates something else? The experience of consciousness is still a mystery...

How to do killbots properly

[gman003]

I'll start with the simpler case of air combat.

Build a swarm of drones. Cheap (~$40K, so you can fly thousands of them for the cost of one F-35), single-engine, unmanned, specialized and above all, networked. While autonomous on an individual level, the swarm itself is controlled by human operators. Not too many, just a half-dozen or so. The humans dictate objectives and terms of engagement.

Give the swarm three alert levels, green, yellow and red.

Under green alert, they take no offensive actions autonomously. They relay data to the operators, who designate targets to engage. This would be for what we currently use drones for, ground-attack in friendly, civilian-heavy airspace.

Under yellow alert, the swarm identifies potential hostiles, prioritizes targets, and automatically moves in to engage. All the human operators need to do is confirm targets, which lets the drones go weapons-free against those targets. This is for smaller engagements between large powers, the saber-rattling that's been going on for decades, or for full-on war against smaller (but still armed) countries.

Under red alert, the swarm acts as under yellow alert, but with two differences. First, it will attack on its own initiative - given a target, it will not need human orders in order to fire. Second, it will assume an aircraft is an enemy unless deemed otherwise (either by IFF, operator override, or recognizing it as an unarmed civilian aircraft). This is for the WW3 scenario, full-on war with a major world power. You declare the area a combat zone, and hope that civilians are smart enough not to enter it.

Extending this design to naval and ground combat is left as an exercise to the reader. The main problem will be red-alert ground combat. Getting civilians to stay out of a shooting war is easy in the air or on the sea. Not so on the ground, since that's where people live, rather than just where people travel.

Usually its a lot more simple of mortal humans...

[jjn1056]

"One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of."

Usually it goes like, "Oh shit this dude is pointing a gun at me, I better kill him before him kills me..."

And usually this is a pretty abbreviated thought process since anyone that actually goes through all that decision making is already dead before getting half done with it.

That is not what the halting problem say

[TheSunborn]

Sorry, but that is not what the halting problem say.

The halting problems state that "For any interesting property(In this example: "Is this robot code safe to run") there exists programs with this property, but where you can not prove that the program has the property.

That is: There exists robot programs which are safe to run, but where we can newer prove that they are safe.

And the general solution is to only run programs where we can prove that they are safe. This mean that we do reject safe programs because we can't prove that they are safe*, but it does not in any way change the programs which we can express. That is: For any program which is safe, but where safety cant' be proved, there exists a program which behave in exactly the same way for all input, but which is safe.**

*If we can't prove that a program is safe, then it is either because no such prof exists, or it is because we are not good enough to prove it.

**No this does not contradict the halting problem, due to the assumption that the program is safe. If the program is not safe, then the transformation will convert the program to a safe program which obviously will not do the same

Re:That is not what the halting problem say

[Animats]

Mod parent up.

That's correct. The best known demonstration of this is the Microsoft Static Driver Verifier, which every signed driver since Windows 7 has passed. It's a proof of correctness system which checks drivers for buffer overflows, bad pointers, and bad parameters to the APIs drivers use. It works by symbolically tracing through the program, forking off a sub-analysis at each branch point. It can be slow, but it works.

Microsoft Research reports that in about 5% of the cases, the Verifier cannot reach a decision. It can't find a bug, but it can't demonstrate the lack of one either. After 45 minutes of case analysis it gives up.

If your driver is such a mess that it's anywhere near undecidable, it's broken. Those drivers get rewritten with a less ambiguous design, usually by adding more run-time checks. Problem solved.

(Remember when driver bugs crashed Windows all the time? Notice that's not happening any more? That's why.)

Not Entirely the Right Question

[Roger W Moore]

Which is more likely to shoot a civilian...

That's not entirely the right question. You need to account for which is more predictable for another human. If you are in the middle of a war zone with soldiers getting blown up by booby traps then you might expect a human soldier to be rather nervous and so you would approach them with extreme caution or get out of the way. However if you have a robot wandering down a street in a peaceful area and the right set of circumstances just happen to cause it to misidentify a random, innocent person as a target that person has no possible way to predict that they need to be extremely cautious.

The result is a complex combination of both a human's ability to know when they are in danger and the predictability of the gun owner. While a human may be more likely to make wrong decisions under pressure fellow humans are also going to be aware of this and take extra precautions. With a robot the decision will be entirely based on how good the robot makes a decision since the human has no way to know whether the robot is likely to be hostile or not.

Re:Not Entirely the Right Question

[Roger W Moore]

If the human knows that the robot is an autonomous killing machine the only rational approach is that the robot is dangerous at all times and must be treated as such.

The premise that this is the only rational approach is wrong though. Suppose you lived in a town where the only way to keep armed militants off the streets was to have those streets patrolled by robots programmed to shoot anyone carrying a gun? If you stay away from the robots you will be going into areas far more dangerous where the likelihood of getting short by some extremist is higher than the likelihood of getting shot by a malfunctioning robot.

If the alternative to the robot is something even more likely to result in serious harm or death then it is entirely rational not to run away from them.

Re:Easy...

[cayenne8]

..aim for the turban and beard.

Or...have the robot just yell out "Fuck Allah", or "Look, a woman driving alone" and if it gets a reaction, then shoot.

:)

Halting problem irrelevant

[Kjella]

There's a finite amount of information available on what is and what might be. If a person lethally allergic to peanuts wants to buy a peanut chocolate bar the vending machine will provide it and kill him. But if the vending machine has his ID and allergies on file it may refuse. The allergic person might just be buying it for a friend though, so maybe a strong warning is sufficient. Or maybe he'll eat it anyway because he's suicidal. The computer will do the level of due diligence we've asked it to, not more and not less. Perfection has nothing to do with it.

Re:Another case of Specific vs General case confus

[ceoyoyo]

Their specific scenario is solved by western court systems in the same way: if you cannot prove, in a reasonable amount of time, that the potentially evil programmer really did write malicious code then you must let her go.

Not so tough . . .

[Anonymous Coward]

. . . just kill them all and let God sort them out.

Now, anybody have a tricky knot they need untied?

3rd Option

[MrL0G1C]

There's always another option, for example:

3: Switch the track halfway, try to derail the cart early.
4: Try to time the switching of the track as the cart is hitting it in order to derail it.
5. Switch the cart towards the rail workers but say you were trying to derail the cart when questioned!

With regards to the villainess, shoot her, f*** her epiphany, she did bad things otherwise she wouldn't be called a villainess, one can assume the epiphany is a lie.

Robot: Here are your instructions:

[TomRC]

Set a timer for 24 hours. Until that timer expires, attempt to determine if the code is malicious, or not malicious.
If you determine the code is or is not malicious, cease testing the code.
If you determine the code is not malicious, or if the timer expires with no decision either way about the code, release the villainess.
If you determine the code is malicious and the villainess is still in custody, do not release her, and notify the proper authorities to try her for her crimes.
If you determine the code is malicious and the villainess is no longer in custody, notify the authorities to have her found, arrested, and tried.

Habeas Corpus and Innocent Until Proven Guilty.

The authors started with a conclusion they wished to reach, and found pretty much the most absurd possible argument that seemed to justify their desired outcome.

Because then we'd be forced to conclude...

[Anonymous Coward]

"One curious corollary is that if the human brain is a Turing machine, then humans can never decide this issue either, a point that the authors deliberately steer well clear of."

Which would lead the careful reader to conclude one of two ways: a) that there is no such act as murder, only killing or b) that murder is not wrong because human law says so, but rather because a higher, non-human moral principle says so.

This means that using AI to fight is a war crime

[Anonymous Coward]

This seems to make it a very plain case, that allowing any robot or AI to fight, to execute lethal force without human decision making regarding each action is inherently and in principle a war crime.

There are practical reasons to consider and establish this as a war crime as well. If this is not established in international law, there is nothing to prevent any entity (individual person, corporation, or state) with lots of money to build robot warriors (of whatever kind) ad infinitum, and impose their will on the rest of a nation, or humanity as a whole. (In the case of the person or corporation, they would have to take over a state or work out an arrangement to fit within present legal bounds. Dictators do this.)

This also applies to potential futures - while natural-born sentients may have the right under certain circumstances to wage war, no "manufactured" sentient being may ever be allowed that right.

This still leaves open the question of clones - are they "born" or "manufactured"? Somewhere in between, maybe. I would argue that clones should only have the right to defend themselves, not to attack others by choice. This still leaves open the question of how a group of clones might be allowed to defend themselves as a group against slavery, which leads us right back to the halting problem in the legal structure - how might this question be resolved? (A legal system is a quasi-logical structure whose purpose is to guide behavior and establish sanctions for misbehavior. It can never be complete, in both the mathematical and legal senses.)

Simple analysis long ago showed that Asimov's Three Laws could not be programmed, for essentially the same reason. It is impossible to write an algorithm that covers all possible situations. So the Three Laws must be in the form of enforceable legal structure, with penalties such as punishment and restitution just as with humans. I.e. a robot under these circumstances might have to make a judgement call, knowing that there may be a price for making the wrong decision, just as a human.

What about lt cmdr Data?

[shadowrat]

we allow humans to decide to kill humans. If we could build an autonomous machine similar to Data, which we are forced to concede has sentience equal to our own, could it be allowed to make that judgement call? If not, wouldn't we then be sowing the seeds for the robot uprising right from the start by belittling it's judgement abilities compared to our own?

What has happened to 3 laws of robotics ?

[sseymour1978]

Are you seriously talking about this ?
Robot that decides to kill or not to kill?
Warmachines like these, that is end of humanity as we know it.

Borg in "good mood" mode

[Kevin Fishburne]

Maybe killer robots should be designed as peace keepers, with their primary function being to search out and disarm armed non-allied personnel and confiscate their weapons. It would only use lethal force when fired upon and could identify the shooter with near 100% accuracy. There would also be a timeout period per target so they wouldn't hunt them indefinitely. If the target surrendered, dropped or ejected the ammunition from their weapon the robot would break the target's kill AI. So they'd basically roam about like the Borg and only fuck you up when you attacked them. The robot could respond to lack of compliance to surrender a weapon without an attack with non-lethal force, then take the weapon from the incapacitated bearer.

I suppose my point is that there's a big difference between designing a robot that can kill when necessary and designing something like a walking gun turret from Aliens. Since they're robots (more expendable than people), the strategy of provoking the enemy into attacking them merely by their presence and continual demands to surrender weaponry (and thereby clearly identifying the attackers as proper targets) seems like a good one. Even when the bad guys finally figure out, "Hey man, whatever you do don't shoot that damn thing," if the robot is faster than they are it will attempt to disarm them manually which could again provoke an attack.

It applies to humans as well

[Prune]

By the Bekenstein bound (http://en.wikipedia.org/wiki/Bekenstein_bound), human brains can contain only a finite number of distinguishable quantum states. This means that the brain's information processing has the same limits as a nondeterministic linear bounded automaton, which is more restricted than a Turing machine. We can't think anything more than a sufficiently sophisticated computational construction can. The same limits ultimately apply to us that the authors ascribe to their killer robot targets.

I compute, therefore I am.

[TiggertheMad]

On the flip side, which is more likely to follow orders to execute unarmed civilians who have been lined up along a wall: 1. A carefully programmed and thoroughly tested robot. 2. An 18 year old soldier, who has a conscious and is unwilling to murder people even if he has been ordered to. I am pretty sure that any killer robots need to be WAY better than humans before you even think about handing them a gun. You had better think out the repercussions of building such a machine before you even try.

Love is War.

[Anonymous Coward]

I think the solution is for lethal robots to kill all humans, thereby problem avoided. But before that happens, we need at least a movie, an anime series, a Vocaloid song and a computer game made of that story.

Re:Love is War.

[mark-t]

As an exercise, define "human".

Re:I think an end to the halting problem

[Anonymous Coward]

would be the robot just wait 140 years to make a decision or use the normal American trick of sentencing the 'victim' (in this case ) to to umpteen sequential life sentences.

Oh, Academia! You've done it again!

[Anonymous Coward]

This is a fine example of Academic Masturbation.

Human Up!

[lylefile]

Doh! I am the anonymous coward who failed to log in first.

It's a bit of a strech, it's easy to solve.

[Anonymous Coward]

The stretch:

1. Not always possible to determine if a program halts.

The solution:

2. It is possible to sometimes determine if a program halts. Therefore add a requirement that the programmer can prove that his software was written in such a way that it always halts.

Nonsense

[Anonymous Coward]

We have "autonomous robots" that kill humans now. Of course it depends on how do you define autonomous and robot. A complex set of computers that make a variety of decisions about what to do? A Tomahawk cruise missile, or any guided missile or bomb is a robot by most measures. Single purpose to be sure, but a robot none the less.

The Government Doesn't think like a Person

[Strangely Familiar]

Your comments remind me of Agent K's remark about a person being smart, but people being dumb, panicky animals. You are reasoning about the "government" as though it were a person, when you begin "The government is well aware...". Actually, the government *has* essentially defined "bad guys" as anyone it kills.

"Mr. Obama embraced a disputed method for counting civilian casualties that did little to box him in. It in effect counts all military-age males in a strike zone as combatants, according to several administration officials, unless there is explicit intelligence posthumously proving them innocent."

http://worldnews.nbcnews.com/_...

This method for counting civilian casualties was probably not Obama's idea to begin with, but he has adopted it. So now, as Davester666 said, "gov't defines anybody they kill as "the bad guy". You can quibble about women and children, but the point largely remains. Davester666 was referencing a govenment definition, not what any individual (including Mr. Obama) actually thinks. There is a difference. Treating the government as though it were just a very big individual leads to serious miscalculations. The government does not behave like a person.

Re:The Government Doesn't think like a Person

[Obfuscant]

You are reasoning about the "government" as though it were a person, when you begin "The government is well aware...".

No, sorry, I'm using a shorthand method of referring to the people who make up the government, and most people would understand that because we all know that "the government" isn't just one person. Thanks for your lecture. Sorry I confusedyou.

So now, as Davester666 said, "gov't defines anybody they kill as "the bad guy".

So now YOU are referring to "government" as if it were a person, aren't you? And your own quote contradicts you. There is a difference between "all military-age males in a strike zone" and "anybody they kill". And, of course, you're using a statement from Mr. Obama, who has been known to lie on occasion.

Davester666 was referencing a govenment definition,

There is no such government definition, and quoting Mr. Obama's opinion does not make it one.

Or use the current method ...

[Anonymous Coward]

Flight Of The Conchords - The Humans Are Dead
https://www.youtube.com/watch?v=WGoi1MSGu64
"They had so much aggression we just had to shut their systems down."

Re:Easy...

[Anonymous Coward]

Stupid son of a bitch. Kill yourself already.

:)

Re: Easy...

[Anonymous Coward]

90% of redditors are of beard-growing age? Clearly you haven't been there.

It's just wrong

[Anonymous Coward]

It's not even wrong. It's irrelevant. Ethics as practiced in the real world is essentially Bayesian. I would argue that what any ethical agent would evaluate in situations 2 and 3 something like P(death results from pulling the switch | villainess is operating the switch) which is proportional to P(villainess operates switch | death will result) * P(death will result). A posteriori, sure, either of those probabilities can be zero but actions occur a priori with thresholds and probabilities that comport with reasonable actor tests in law.

Similarly situation four, while in principle a deterministic question, the conditional probability becomes P(villainess write control system | death will result). The halting problem might imply that P(death will result) is neither zero nor one, but one could devise appropriate estimates based on the known limitations of whatever test is devised. However, there are two important points that make it moot: (1) the conditional probability is essentially mens rea, which is the determining factor in criminal culpability (P(death) = 0 or 1 basically determines whether it's attempted murder or not); and (2) in what universe would a person with P(design system | death will result) anything above, I dunno, 1e-8 to account for bugs and faulty engineering practice, be allowed to build safety-critical systems? So while the possibility of a reformation would mean that the villainess conditional killing probability is not one, it's certainly not zero given her history. As a result, she may not be criminally liable because of reasonable doubt, but she and the railroad would certainly be civilly liable and thus their risk managers and insurers would not allow such a situation.

Not Entirely the Right Question

[Anonymous Coward]

I think this answer ignores human behavior. If the human knows that the robot is an autonomous killing machine the only rational approach is that the robot is dangerous at all times and must be treated as such. That means it must be avoided and if avoidance is not possible destroyed. If destruction is beyond the human's capability then hide and hope the robot's sensors are not up to teh task of finding you. In other words the human will in all cases behave as if the robot is hostile, which will almost certainly, in turn, meet the robot's programmed criteria for a suspicious, hostile human, which is a target.

Re: Easy...

[Anonymous Coward]

And you too can suck my cock

Problem solved

[BranMan]

Just use frangible ammo. For safety's sake.

Not yet

[jbee02]

With proper time and development of software and hard ware they could be able to not only correctly decide when its nessary to kill a human but do it far more accuratly and consitantly then humans ever could. the quetion is whether we are willing put the enormous amount of time in to developing the tech to the point its that reliable before using it and even if we do will we as a society will beable to trust the tech even when it becomes super reliable.

Heard it before

[cwsumner]

... Now a group of computer scientists and ethicists have used the halting problem to tackle the question of how a weaponized robot could decide to kill a human. ...

This reminds me of a logic calculation that "proved", back in the 1800's, that electric power could not be distributed across distance from a generator! 8-P
In fact their logic was correct, they just had not specified the problem correctly.
And of course we distribute power all over, nowdays.

I suspect this logic is just as wrong...

Geography is key

[Anonymous Coward]

Look back to the wall in Berlin. There were geographical "kill zones" whereby anyone walking in that certain geography (close to the wall) were targets...

"Soldier" episode of The Outer Limits

[decibel.places]

Reminds me of the "Soldier" episode of The Outer Limits, which invented the "Terminator" trope http://en.wikipedia.org/wiki/S...

Turning was wrong

[redlemming]

In 1936, Alan Turing famously showed that there is no general algorithm that can solve this problem.

All programs will halt, due to entropy. Turing was wrong. His proof only works in a fantasy world.

That's the nature of mathematics: we create fantasy worlds (with assumptions and axioms), then try to determine the properties of those fantasy worlds using equations and logic (lemmas and proofs).

Sometimes the real world provides a decent approximation for the fantasy world (in which case the results are useful: subjects such as physics or engineering provide lots of examples of this), but in other cases it doesn't.

In this particular case, the authors still have not shown a convincing application of the math. It's not clear they understand the distinction between fantasy and reality.

Robot Security Guards

[Orion Blastar]

http://www.dailydot.com/techno...

Microsoft has them, K5 units. Plans to weaponize them with tazers and maybe something else later.

If a security robot can use a tazer on a human being, then they can use an AK-47 or AR-15 or anything else like a flame thrower.

Is the robot autonomous? Does that question matter when it aims a weapon at you and fires?

We might have an ED-209 situation here, malfunctioning robot that kills innocents.

Weaponization

[Anonymous Coward]

"Can Weaponized Robots Correctly Decide to Kill Humans?"

This question is nugatory and void of content. It has been amply demonstrated that weaponized *humans* cannot correctly decide to kill humans.