New Report Cites Dangers of Autonomous Weapons

HughPickens.com writes: A new report written by a former Pentagon official who helped establish United States policy on autonomous weapons argues that autonomous weapons could be uncontrollable in real-world environments, where they are subject to design failure as well as hacking, spoofing and manipulation by adversaries. The report contrasts these completely automated systems, which have the ability to target and kill without human intervention, to weapons that keep humans "in the loop" in the process of selecting and engaging targets. "Anyone who has ever been frustrated with an automated telephone call support helpline, an alarm clock mistakenly set to 'p.m.' instead of 'a.m.,' or any of the countless frustrations that come with interacting with computers, has experienced the problem of 'brittleness' that plagues automated systems," Mr. Scharre writes.

The United States military does not have advanced autonomous weapons in its arsenal. However, this year the Defense Department requested almost $1 billion to manufacture Lockheed Martin's Long Range Anti-Ship Missile, which is described as a "semiautonomous" weapon. The missile is controversial because, although a human operator will initially select a target, it is designed to fly for several hundred miles while out of contact with the controller and then automatically identify and attack an enemy ship. As an alternative to completely autonomous weapons, the report advocates what it describes as "Centaur Warfighting." The term "centaur" has recently come to describe systems that tightly integrate humans and computers. Human-machine combat teaming takes a page from the field of "centaur chess," in which humans and machines play cooperatively on the same team. "Having a person in the loop is not enough," says Scharre. "They can't be just a cog in the loop. The human has to be actively engaged."

Comments

The *US* missile is "controversial"?!?!?!

[Anonymous Coward]

What about the KH-22 (or AS4 "Kitchen") that the Soviets/Russians have actually fielded - since 1962.

The Kh-22 uses an Isayev liquid-fuel rocket engine, fueled with TG-02 (Tonka-250) and IRFNA (inhibited red fuming nitric acid), giving it a maximum speed of Mach 4.6 and a range of up to 600 km (320 nmi). It can be launched in either high-altitude or low-altitude mode. In high-altitude mode, it climbs to an altitude of 27,000 m (89,000 ft) and makes a high-speed dive into the target, with a terminal speed of about Mach 4.6. In low-altitude mode, it climbs to 12,000 m (39,000 ft) and makes a shallow dive at about Mach 3.5, making the final approach at an altitude under 500 m (1,600 ft). The missile is guided by a gyro-stabilized autopilot in conjunction with a radio altimeter.

Fly 600 KM - then hit whatever it happens to find. Potentially with a nuclear warhead.

Oh, that's right. That doesn't fit into typical thoughtless anti-US bullshit. Sorry to mess up your narrative.

Re:The *US* missile is "controversial"?!?!?!

[rtb61]

I could give you a current real world autonomous weapon system the US has not only fully funded but put quite chaotically into the field. That would be proxy terrorist fighters, quite the mess they made with that autonomous weapon system and a real warning of what can happen when you attempt the same digitally. Of course we have yet to see the full repercussions of that, say a tow missile on a power boat taking down an oil tanker, either manned by those the weapon was given or those it was on sold to (you know it is just a matter of time, what were they thinking, obviously not thinking at all).

Autonomous system can go wildly out of control or just be completely shut down, not by those deploying it of course. However war is organised murder so it is already fucked up. So hunker down, let loose your massed autonomous systems and hope for the best, defensively not a real problem because you head them off in one direction, away from you and should they turn around and get closer, you are in a better position to shut them down. The further they get from you, the less of a problem they become for you and well, let's not lie, war is organised mass murder orchestrated by psychopaths to feed their ego. So autonomous weapons suck offensively (lose control) but are pretty good defensively ie the remain locked up and shut down unless psychopathic mass murders decide to attack and well, which is worse the autonomous weapons or psychopathic mass murders, which is least controllable, which is more dangerous, which is more destructive and which can be more readily shut down.

The US military are more likely to be saying they are bad, really bad because of course they feel threatened by them should other countries deploy them defensively ie I answer you army of 1 million with 10 million automated aerial mines (cheap when mass produced) and a approximate? designated kill zone and bunch of service technicians and a much expanded police force with a secondary roll as a defence force manning automated combat systems (highly trained professionals). So can a drone swarm comprising tens of thousands of drones wipe out a modern carrier attack fleet, not the first few thousand but certainly those that follow up behind an approaching cloud of exploded debris can (the idea of launching a few hundred thousand drones at once far enough apart to prevent a chain reaction, at an approaching military force, is the whole idea, if not the first wave, then the tenth, meh just a cheap automated device so who cares as long as it is cheaper than the approaching hardware and can continue to function with defence forces hidden in shielded bunkers). Automated weapons 100% favour the defenders and that does not sit will with an imperialists state nor a grossly obese military industrial complex (it is way cheaper and defensively far more effective ie only used in times of war and then all bets are off anyhow).

u.s. has had them for decades

[iggymanz]

land mines are autonomous weapons, no human is in the decision loop to fire when the preset conditions for detonation are met.

http://www.un.org/en/globaliss...

The differences

[DrYak]

The missile is guided by a gyro-stabilized autopilot in conjunction with a radio altimeter.

Fly 600 KM - then hit whatever it happens to find.

That is the main difference between classical intercontinental ballistic/guided missiles and the autonomous weapons mentionned here.

Classical missile mainly flight to a specific point (which was decided in advance by a human being) a go ka-boom on whatever happens to be at that point.
If the intelligence on which the human was acting is precise (i.e.: exact coordinate of the position of the targets are known) the missile exactly hits the target that the human intended. If the intelligence is wrong, the missile still goes exactly where it was asked to, it's the human who asked the wrong thing.
Think throwing a rock on a target, shooting a target with an arrow. Only with more complex gadgets.

Autonomous weapon on the other hand a deployed or reach a region (which is what was decided by the human being) and then on *their own* start looking around to find potential target that they engage on their own autonomous decision. The human being is not the own who is taking the final decision in the grand scheme of things, it's the AI running inside the autonomous weapon. The weapon is at risk of misinterpreting what it perceives and wrongly take decisions to engage.
Think Aliens movie-style automatic gun turrets.

So the historic precedent of such unwanted destruction isn't as much classical missile that you mention (where the commander giving the order to fire more or less knows what is going to happen).
The closest historic precedent are *mines*. Object that are left on order by human, but then would activate and explode without much control by the ordering humans. With very strong risks that they'll end up harming the wrong target (left over mine that explode and maim local civilian population long after the conflict is finished). That's why mines got banned by several countries.

That's why it's really risky to leave an AI (That could be hacked or spoofed) to make the decisions.

Still, the best analogy

[DrYak]

Autonomy implies some sort of ability for decision making and control, which is far more desirable than how mines actually operate (although some do have the ability to self-deactivate after a set time).

Though we must concede that you're right in that mine are really primitive mecanisme that don't exactly have an AI and thus are far from autonomous...

A human makes the conscious decision to attack anything that enters the area when the mines are placed. Just because it may be years before that happens does not mean mines are autonomous, just delayed.

...mines are still the best historical analogy that we have for problems brought by autonomous weapon.

In both situation, human have only a vague input about the region that should be attacked.
- mines are deployed over an area
- autonomous weapons are sent to seek for potential target in a designated area

In both situation the human ARE NOT the one making the decision about the detonation.
- mine detonate on their own when they sense some form of proximity
- autonomous weapons are autonomous, they are suposed to pick up and engage their target on their own without further human input

In both situation things can go horribly wrong
- mines have been left over for long period of time and have often maimed innocent civilians long after the conflict is finished.
- AI can go wrong in lots of ways (wrong instruction, or plain hostile hacking/spoofing) and end up engaging the wrong target.

Currently mines are banned by lots of countries.
Same should be done with pure autonomous unsupervised weapons.