Slashdot Mirror
New Report Cites Dangers of Autonomous Weapons
HughPickens.com writes: A new report written by a former Pentagon official who helped establish United States policy on autonomous weapons argues that autonomous weapons could be uncontrollable in real-world environments, where they are subject to design failure as well as hacking, spoofing and manipulation by adversaries. The report contrasts these completely automated systems, which have the ability to target and kill without human intervention, to weapons that keep humans "in the loop" in the process of selecting and engaging targets. "Anyone who has ever been frustrated with an automated telephone call support helpline, an alarm clock mistakenly set to 'p.m.' instead of 'a.m.,' or any of the countless frustrations that come with interacting with computers, has experienced the problem of 'brittleness' that plagues automated systems," Mr. Scharre writes.
The United States military does not have advanced autonomous weapons in its arsenal. However, this year the Defense Department requested almost $1 billion to manufacture Lockheed Martin's Long Range Anti-Ship Missile, which is described as a "semiautonomous" weapon. The missile is controversial because, although a human operator will initially select a target, it is designed to fly for several hundred miles while out of contact with the controller and then automatically identify and attack an enemy ship. As an alternative to completely autonomous weapons, the report advocates what it describes as "Centaur Warfighting." The term "centaur" has recently come to describe systems that tightly integrate humans and computers. Human-machine combat teaming takes a page from the field of "centaur chess," in which humans and machines play cooperatively on the same team. "Having a person in the loop is not enough," says Scharre. "They can't be just a cog in the loop. The human has to be actively engaged."
The United States military does not have advanced autonomous weapons in its arsenal. However, this year the Defense Department requested almost $1 billion to manufacture Lockheed Martin's Long Range Anti-Ship Missile, which is described as a "semiautonomous" weapon. The missile is controversial because, although a human operator will initially select a target, it is designed to fly for several hundred miles while out of contact with the controller and then automatically identify and attack an enemy ship. As an alternative to completely autonomous weapons, the report advocates what it describes as "Centaur Warfighting." The term "centaur" has recently come to describe systems that tightly integrate humans and computers. Human-machine combat teaming takes a page from the field of "centaur chess," in which humans and machines play cooperatively on the same team. "Having a person in the loop is not enough," says Scharre. "They can't be just a cog in the loop. The human has to be actively engaged."
Having any type control link is susceptible to multiple types of attacks. This will drive the push for more autonomy and AI.
It decided not to hit the computer center it was sent to destroy and flew on to a maternity ward.
They needed a high level official report to figure this out?
I don't read your sig. Why are you reading mine?
Remember that time the military was going to build a nuclear-powered cruise missile that would leave a cloud of fallout behind it, drop nuclear warheads and crash into something when it was done?
Autonomous weapons would not make mistakes. They would do their jobs -- too well.
After reading the last line I can only remain hopeful. Are we one step closer to getting more info on the military's use of neural feedback interfaces, transcranial electrical stimulation, magnetic alteration of live neural pathways, etc?
CAPTCHA: entice
What about the KH-22 (or AS4 "Kitchen") that the Soviets/Russians have actually fielded - since 1962.
The Kh-22 uses an Isayev liquid-fuel rocket engine, fueled with TG-02 (Tonka-250) and IRFNA (inhibited red fuming nitric acid), giving it a maximum speed of Mach 4.6 and a range of up to 600 km (320 nmi). It can be launched in either high-altitude or low-altitude mode. In high-altitude mode, it climbs to an altitude of 27,000 m (89,000 ft) and makes a high-speed dive into the target, with a terminal speed of about Mach 4.6. In low-altitude mode, it climbs to 12,000 m (39,000 ft) and makes a shallow dive at about Mach 3.5, making the final approach at an altitude under 500 m (1,600 ft). The missile is guided by a gyro-stabilized autopilot in conjunction with a radio altimeter.
Fly 600 KM - then hit whatever it happens to find. Potentially with a nuclear warhead.
Oh, that's right. That doesn't fit into typical thoughtless anti-US bullshit. Sorry to mess up your narrative.
land mines are autonomous weapons, no human is in the decision loop to fire when the preset conditions for detonation are met.
http://www.un.org/en/globaliss...
Autonomous Weapons are high value targets for hacking. More so than banking. I don't envy poor souls that were tasked with meeting such design challenges.
Imagine you had to design portable ATM that has to operate flawlessly even when moved to a crack den without having reliable connectivity to C&C.
https://www.youtube.com/watch?v=l0WG0B2JYLQ
Hire a Linux system administrator, systems engineer,
Just thought I'd throw that out there.
There was a autonomous gun system demo'ed for the DMZ between the Korea's. Don't know if they ever deployed it, but it "locked" on to anyone who moved in the target zone and fired.
..who mistook a maternity ward for a computer center...
"His name was James Damore."
PEOPLE kill people.
Duh.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
no bomb us more mom us,, hugs not thugs.. cease fire.. spiritual (r)evolution underway... ladies first...
They kill, injure or destroy stuff.
Back in the days they always said Phoenix missile on the F14 can take out an enemy aircraft 120 miles away (or some long distance like that). There are other missiles of this capability so seeing a blip on radar but what is it really? Enemy aircraft or something else like a civilian airliner or a UH60 carrying UN officials? There are many other cases of friendly fire, what thought has been put into this (like everyone else, I didn't RTFA).
mfwright@batnet.com
In terms of operating the weapon, there's little difference between the new LRASM and a classic Tomahawk or any other cruise misslle made in the last 30 years. It's just HughPickens being a sensationalist.
The actual report talks about AI based systems with kill authority (aka SkyNet).
As I recall, the US's excuse for shelling an MSF hospital for 30 solid minutes was essentially "no one bothered checking that autonomous systems were correct."
Basically someone called in an air strike, a plane was diverted, they pointed it in the direction of the actual target, but the hospital was in the way so the system locked onto that instead and no one noticed. Then the system just kept on automatically firing on the target it was given because, after all, the computers had verified the target and computers don't make mistakes.
https://www.youtube.com/watch?...
It's pretty nice as an intermediary step before "cyborg" (which seems like it should need direct connection to the nervous system to apply as a term, despite the way usage has expanded recently to include heavy cell phone usage).
Nice try, the system didn't 'lock' on to anything. It hit the GPS coordinates it was told to. You need to check with whomever relayed those coordinates. Hint: They weren't US FACs.
I thought autonomous weapons were 100% safe...?
I mean, they're only *weapons*, after all! ...
nice headline
That's why the best autonomous weapon is big, dumb bomb. You drop it and it autonomously drops and levels the area.
I've got to say neigh to that one.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
Anyone who has ever been frustrated with an automated telephone call support helpline, an alarm clock mistakenly set to 'p.m.' instead of 'a.m.,' or any of the countless frustrations that come with interacting with computers, has experienced the problem of 'brittleness' that plagues automated systems,
While true, I can also recount numerous frustrations originating from human interventions that lead to disaster such as initiating an emergency procedure ultimately leading to a nuclear reactor explosion, failed controlled burns or environmental disasters. Even in everyday life, trying to reason with a customer rep from bank A or government department B, can be as frustrating an unhelpful as trying to figure out which number I should push. As such, the existence of issues in automated system is hardly a justification disregard issues that keeping humans in the loop introduces, with the inconvenience that humans cannot be patched easily: they will keep making the same mistakes. I'd be interested in having statistics about the number of errors over a certain number of years between a fully automated system and human-included system to fully appreciate the benefits of one or the other.
While I'm all for overview and proper design, automation will become inevitable because of the advantages it can provide in certain type of conflicts - namely with technologically advanced adversaries. While some militaries may afford to have large amount of man-power and resources to maintain all these systems, countries with lower GDPs, large territories to defend, growing ambitions and lower ethical concern about consequences of potential errors will likely have automated defense systems to offset the support costs of human operators. In turn these systems will have a faster decision-making loop, providing an advantage over non-fully automated systems.
Of course the introduction of automated systems introduces the risk of hacking and thus the cost-saving of implementing automated systems will somehow go into stronger network defenses. However keep in mind that while totally possible to hack these system to actually leverage them against the users, this is not a trivial task either and requires skilled hackers, not your typical certification-hunting pen tester. However, network defenses are being automated as well, for the better or worst. A large chunk of network defense can be done by civilians (and probably will have to be given the competitive salary of the industry).
In any case, yes, we do need to careful with these systems and yes they have a lethal power, but so does many other systems, including systems with humans "in the loop". This should not prevent the development of automated systems, much like I don't believe it will stop the development of automated cars, planes and trains, much like it didn't stop the automation of the stock market despite glitches, which can also have tragic consequences. It needs constant testing, updating and training to new, unexpected issues.
@cyberrecce
The missile is guided by a gyro-stabilized autopilot in conjunction with a radio altimeter.
Fly 600 KM - then hit whatever it happens to find.
That is the main difference between classical intercontinental ballistic/guided missiles and the autonomous weapons mentionned here.
Classical missile mainly flight to a specific point (which was decided in advance by a human being) a go ka-boom on whatever happens to be at that point.
If the intelligence on which the human was acting is precise (i.e.: exact coordinate of the position of the targets are known) the missile exactly hits the target that the human intended. If the intelligence is wrong, the missile still goes exactly where it was asked to, it's the human who asked the wrong thing.
Think throwing a rock on a target, shooting a target with an arrow. Only with more complex gadgets.
Autonomous weapon on the other hand a deployed or reach a region (which is what was decided by the human being) and then on *their own* start looking around to find potential target that they engage on their own autonomous decision. The human being is not the own who is taking the final decision in the grand scheme of things, it's the AI running inside the autonomous weapon. The weapon is at risk of misinterpreting what it perceives and wrongly take decisions to engage.
Think Aliens movie-style automatic gun turrets.
So the historic precedent of such unwanted destruction isn't as much classical missile that you mention (where the commander giving the order to fire more or less knows what is going to happen).
The closest historic precedent are *mines*. Object that are left on order by human, but then would activate and explode without much control by the ordering humans. With very strong risks that they'll end up harming the wrong target (left over mine that explode and maim local civilian population long after the conflict is finished). That's why mines got banned by several countries.
That's why it's really risky to leave an AI (That could be hacked or spoofed) to make the decisions.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Autonomy implies some sort of ability for decision making and control, which is far more desirable than how mines actually operate (although some do have the ability to self-deactivate after a set time).
Though we must concede that you're right in that mine are really primitive mecanisme that don't exactly have an AI and thus are far from autonomous...
A human makes the conscious decision to attack anything that enters the area when the mines are placed. Just because it may be years before that happens does not mean mines are autonomous, just delayed.
...mines are still the best historical analogy that we have for problems brought by autonomous weapon.
In both situation, human have only a vague input about the region that should be attacked.
- mines are deployed over an area
- autonomous weapons are sent to seek for potential target in a designated area
In both situation the human ARE NOT the one making the decision about the detonation.
- mine detonate on their own when they sense some form of proximity
- autonomous weapons are autonomous, they are suposed to pick up and engage their target on their own without further human input
In both situation things can go horribly wrong
- mines have been left over for long period of time and have often maimed innocent civilians long after the conflict is finished.
- AI can go wrong in lots of ways (wrong instruction, or plain hostile hacking/spoofing) and end up engaging the wrong target.
Currently mines are banned by lots of countries.
Same should be done with pure autonomous unsupervised weapons.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
For the ruling class. As a member of the ruling class the only real threat to your never-ending rule is the military. It's just too tempting to cut them out of the loop....
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
program that Lockheed Martin and all the other defence contractors are always working on: W.E.L.F.A.R.E., the Welfare Every Last Fucking Asshole Republican Earmarks.
As soon as it develops an Austrian accent shut it down quickly.
> does not have advanced autonomous weapons in its arsenal.
Not really true. It has missiles and torpedoes which are autonomous. They are kind of dumb though and have done things like mistakenly shot a passenger aircraft out of the sky because their human told it too and the missile wasn't bright enough to say 'that's passenger jet lol dumb human I'm not doing this.' And drones are autonomous because they stay airborne if they lose communications and can in theory be taken over by black hats.
The idea of trying to hold a nation or area by using a free fire zone grid is not new.
https://en.wikipedia.org/wiki/...
The French and US in "Vietnam", the French in Algeria https://en.wikipedia.org/wiki/..., the US and NATO and their Middle East and North Africa drone zones.
Using an AI or humans to kill everything in a "free fire zone" is not a new US tactic. The results of past wars and regime change/US backed coups tactics should by now by understood. by the smarter mil and contractors the US gov seeks advice from.
Does the US expect different results from an AI ready systems to remove the human control aspect of drone use? Having a human controlled or AI drone fly over an area and target anything moving is not a new idea.
Will the results be any different? The same winning by now body count math as in the Vietnam years.
US AI systems will not be any more secure than todays easy to redirect drones. The US contractors use systems per drone to ensure the mission works and healthy profits ie flying prototypes with weapons systems added. The computing power for ever more on board AI security would detract from the mission and if lost would give insights into US crypto.
Better just to have standard equipment fail and fall into other nations hands within a drone than offer up advanced AI crypto links and the needed long range global networks. Losing a simple upgraded AI drone is easy, losing the keys back up to a wider control network is hard.
The plans, tactics and results will be the same. Just with an AI to take the command from special forces, signals intelligence finding any cell phone "on", or a free fire grid pattern to be active in, or local "moderates" with the freedom to call in an AI drone.
Domestic spying is now "Benign Information Gathering"
Remember: guns don't kill people. People kill people.
But if it's an autonomous gun, well, they really kill people.
Or, O'Toole's commentary on Murphy's Law: Murphy was an optimist.
Hmm. Actually some of that Philip K. Dick stuff wasn't that bad, even pretty good.
someone ordered an airstrike there, military didn't check the coords from another source if it's a no shoot coordinate.
basically the problem is that us military works as order-an-explosion service for whoever social engineers how to get them to shoot. whoever provides them with the 'intel' gets to enjoy the benefits. like all the yemeni 'rebels' getting hellfired. who fingers them? their local rivals, duh. neither the fingerer or the one who gets exploded is particularly pro-US or anti-US in any way and because US makes 0 effort in apprehending the 'suspects' or even notifying them that they're 'suspects' then anyone who says they're an ally of US can just order strikes on their local rivals and nothing gets fact checked. just a waste of money and lives and results just in bad will against the US, so entirely pointless.
world was created 5 seconds before this post as it is.
I hate to tell them this but the majority of weapons in use already follow an autonomous path once targeted and launched. anti-aircraft missiles, hellfires, anti-ship missiles, air dropped and ship launched torpedoes, submarine torpedoes once the wire is cut. CIWS is autonomous once activated in defense mode, ram rolling airframe missiles are autonomous once put in defensive mode as well. There are many other examples as well. That's the way smart weapons work. You target them, or let the computer target for you, and give it launch permission and it'll do it's job.
Naval warfare between fleets is going to be skies full of missiles trying to overwhelm the defensive bubbles of each ship or fleet. There's no way a human can be in the loop for every weapon in an electronic warfare environment. The weapons have to be autonomous once launched and you have to launch a lot of them at once to have any chance of getting past defenses. Sea to shore is a bit different as the latest cruise missiles can already be retargeted if needed and there are less of them.
Will be doing *exactly* as programmed and ordered.
That sentience and self-awareness stuff? Completely and utterly detached from any human-slaughtering.
The only thing that ever went wrong was that those who made it act as it did forgot that they were viable human targets too, no matter how far above the laws they may have been and so didn't slip in any exceptions.
Send such weapons to target with no outside communications what-so-ever. Any open port for communications makes hacking much more likely. But if it is a set it and forget it device,it will do what it is supposed to do. Drones are now saving the lives of our soldiers and they are also saving the lives of innocents, If we did not use drones we would be bombing cities and suburbs and killing huge numbers of civilians to get the bad guys.
People are wasting entirely too much time panting about the scary idea of autonomous weapons turning evil and hunting people, when a worse thing already exists: evil humans unleashing death on each other.
People fretting about some future robot-driven apocalypse, while ignoring all the current ACTUAL mass-death in Africa and the middle east and all the gang violence in big American cities, and all the migrant-driven violence in Europe are displaying an amazing degree of moral blindness.
SciFi novels and films aside, the simple fact is that no machine will ever actually be intelligent - all AI is just the simulation of intelligence. It's all a game of getting machines to appear to be intelligent to human observers and to perform complex tasks as if they were intelligent so they can replace human workers in certain situations. A computer will eventually have a big enough dictionary, complex enough tables, and complex feedback loops in its code to fool people into thinking it is intelligent, but no computer will ever actually KNOW anything, in the GROK sort of way. Make no mistake, every AI technique is just a version of one or more feedback loops, sometimes aided by lookup tables and sometimes a random number generator or two. A computer that simulates intelligence sufficiently will seem to know what a ball is and be able to recite descriptions, recognize one, cite possible uses, bet but will never actually KNOW what a ball IS. No computer will every "get it". If you want to worry about people being killed by machines, worry about the most-likely version: programming errors in automated weapons that someday will be entrusted to use AI while selecting and destroying targets. Such machines are most-likely to kill the wrong targets by accident due to programming errors. I've been coding since the 1970s and have seen all the fads come and go, and I also have written code that currently is in one of the unmanned aerial systems the military currently uses. Modern shoddy coding, not smart machines turned evil, is the real worry in this field.
I know my transistors and I know my bits, and there is no way ANY number of them will become intelligent or magically spring to life. Worry about this if you want to. Knock yerself out worrying about a zombie apocalypse or an alien invasion too if that floats your boat, but I'll not join you in worrying that a few million light switches will become intelligent and go on a killing spree.
how about deciding at last that killing each other (by whatever means) is not the optimum way to resolve arguments?