What Will Happen When Killer Robots Get Hijacked? (marketwatch.com)

"Imagine an artificial-intelligence-driven military drone capable of autonomously patrolling the perimeter of a country or region and deciding who lives and who dies, without a human operator. Now do the same with tanks, helicopters and biped/quadruped robots." A United Nations conference recently decided not to ban these weapons systems outright, but to revisit the topic in November.

So a MarketWatch columnist looked at how these weapons systems could go bad -- and argues the risks are greater than simply fooling the AI into malfunctioning. What about hijacking...? In warfare, AI units can function autonomously, but in the end they need a way to communicate with one another and to transfer data to a command center. This makes them vulnerable to hacking and hijacking. What would happen if one of these drones or robots was hijacked by an opposite faction and started firing on civilians? A hacker would laugh. Why? Because he wouldn't hijack just one. He would design a self-propagating virus that would spread throughout the AI network and infect all units in the vicinity, as well as those communicating with them. In a split second, an entire squad of lethal autonomous weapons systems would be under enemy control... Every machine can be overridden, tricked, hijacked and manipulated with an efficiency that's unheard of in the realm of human-operated traditional weaponry.

However, the U.S. government remains oblivious. DARPA (Defense Advanced Research Projects Agency) has already announced a $2 billion development campaign for the next wave of technologically advanced AI (dubbed "AI Next"). One of the goals is to have the machines "acquire human-like communication and reasoning capabilities, with the ability to recognize new situations and environments and adapt to them." I may be overreaching here, but the UN meeting on one end and this announcement on the other, make me think that the U.S. government isn't just pro-robotic -- it may already have a lethal autonomous weapons ace up its sleeve.
The article ends with a question: What do you think about killer robots replacing human combatants?

And what would happen if killer robots got hijacked?

Comments

Well, there's the one thing

[Krishnoid]

A hacker who identifies with Thanos' agenda could start 'solving' the overpopulation problem by themselves.

Security Time

[Anonymous Coward]

So any reasonable person would say that we should be able to design and implement inexpensive and ubiquitous secure systems before trying our hand in weapons? That we shouldn't run Wintel ecosystem as it is on those killer bots and always prefer feature creep of insufficiently engineered, too complex features over security?

Re: Security Time

[phantomfive]

Security is so far from a priority for most companies that it's not even an afterthought. Talk to administrators and you'll hear people say, "we're just not a big enough target for hackers." They will keep that attitude even when the company is a big enough target. No one cares if their system gets hacked, even credit agencies. They just put a vague fix in and move on. No questions about whether their problem is more systemic, and how to avoid those problems in the future.

Or cars

[Anonymous Coward]

You don't need to steal weapons from the military to get killer robots. You just need to botnet cars. They already kill people without help.

Let me driver-assist you to safety.

Re:Or cars

[Hognoxious]

Yeah, apart from the tiny detail of having to be physically there those are exactly the same thing.

let's play global thermonuclear war

[Joe_Dragon]

let's play global thermonuclear war!

Re: let's play global thermonuclear war

[Wycliffe]

We should designate an official war zone over an uninhabited part of the Pacific Ocean where million dollar robots can fight each other to the death. At least that way instead of having an expensive chess match where real people die, we just have the expensive chess match.

Re: let's play global thermonuclear war

[Anonymous Coward]

We do that with the economic model already, it's giant economic robots tackling each other.

Re: let's play global thermonuclear war

[NicBenjamin]

Like most ideas about revolutionizing warfare, this is something that's very old. Many many cultures would have champions fight to the death so the army did not have to.

For that to work couple things have to be true:
1) Both sides have to have a roughly equal chance of success. That is the champions have to have comparable combat power. Back then, nobody's going to agree to wager their war on a naked guy with a pointy stick vs. a Knight in full plate. With robots, many countries won't have a robot capable of going toe-to-toe against us.
2) Both sides have to want whatever they're fighting enough to raise an army to fight for it, but not so much that they'll actually die for it. If they're gonna die for it regardless of the outcome of the fight, the fight is pointless. If the result of losing a robo-war is that the President gets executed, his top guys imprisoned, and everyone else gets fired, your worst enemies get to run the country, etc. then when the champion loses you fight anyway.
3) Both asides have to trust that the other side will abide by the deal. You're not spending $1 Billion on robo-champion if you think those bastards will fight even if their robot loses.

Just go through recent US-involved wars in your head. how many are all three things true of both sides?

The Nazis and Japanese could have made some pretty cool robots, but we'd have fought the actual war regardless of who won the robo-battle. The Koreans, Vietnamese, Iraqis, Taliban wouldn't have robots anywhere near our league, so they'll lose the war and then fight anyway.

Re:let's play global thermonuclear war

[BlueStrat]

I'll just leave this here.

https://youtu.be/wv6pfQTl-d4

"My boy!"

Strat

Re: let's play global thermonuclear war

[dcollins]

I have this as item #3 on my list of "Top Geek Myths".

Counterargument: People don't submit to perceived tyranny because their material stuff got destroyed; rather, the opposite.

Also: "What robot soldiers could do is just as scary, though: Make outright colonialism a practical option again." War Nerd, 2014.

Wrong question

[rsilvergun]

the question we should be asking ourselves is what will happen when the ruling class doesn't need the working class to keep the military class in check? Right now we've got a bit of a balance going on. The Army protects the ruling class but the working class keeps an eye on the army and going into a decent civilian life after some time in the army gives them something to do besides run a Junta. Because of that there's a floor on how bad the ruling class can treat the army and the working class.

All that goes out the window when they've got a robot army. The robot army will never betray them. Sure, there's some engineers keeping it running, but they're nerds and typically lack the drive and charisma to overthrow the ruling class. Those kinds of coup are pulled off by charismatic generalissimos. So you're gonna have the ruling class, a small, well paid merchant class to keep the killer robots going and everybody else. That's you and my, btw. And the ruling class won't need us to buy their crap to be rich either. They'll own everything and have factories to build it. I suppose there'll be a few positions for their doctors and sex slaves. The rest of us get abandoned, sorta like how we ignore starving people in Africa.

Re:Wrong question

[SqueakyMouse]

Do they have more of an incentive to abandon people, and look like jackasses, or provide for them, with their abundance of resources, and be seen as heroes? If the premise is that wage slaves are now obsolete, then the ruling class no longer needs to keep them poor. That was only necessary when the only way to get someone to do the less desirable jobs was to keep them poor enough that they'd otherwise starve. In this scenario, there are robots to fill this gap, so the ruling class may as well keep people happy and have someone to admire the statues they build of themselves, keeping their egos inflated. They'd still that buzz of having power over people, since the recipients of their services have a lot to lose by not complying.

Re:Wrong question

[Anonymous Coward]

This is a great point - it's unclear to me what would stop them. By now we know a few things about the bulk of the owners of capital:
1. They don't believe in Democracy
2. They are motivated, and don't allow pesky things like laws to stand in their way
3. They have the patience to work across generations

Think of Trump and his children. These people want to bring about a new feudalism and bring us all back to Game of Thrones, and the Dragons are the killer robots. Human nature doesn't change.

We need to outlaw this now, before it is too late.

Same as today

[wolfheart111]

CARNAGE... Lots and Lots of Carnage. Its so easy to kill people. :)

the military has much experience with this risk

[RhettLivingston]

There is no new risk here. Virtually every modern weapon system is robotic and many are mostly autonomous with heavy usage of data channels for control. If hackers could take them over, we'd have long ago suffered these imagined catastrophes.

Re:the military has much experience with this risk

[mentil]

But... but... all it takes is 2 cans of Mountain Dew and 30 seconds of rapid-fire typing and then you say "I'm in!" and now your bar tab is in the negative and Terminatrices warm your bed at night.

Re:the military has much experience with this risk

[RhettLivingston]

What about all of our fighters? Most can be slaved to other fighters, like F22s, right now and the new ones have no fully mechanical controls. They have extensive communications so that they can fight as automated teams, utilize each other's sensor suites, etc. If a hacker could get in, they could theoretically have a squadron at their control with pilots along as helpless hostages. I guess the pilots could destroy the flyability by punching out. It would be a very expensive loss if a whole squadron punched.

Or what about our guided missile frigates. Again, tons of communication channels and computer control. If the network could be penetrated, a hacker could wipe out a fleet with the ordinance a single frigate carries.

In many cases with current systems, the people are only in control because the software has been written to put them in the loop, and there are often modes of operation available but not used where they are only in the loop for the initial command. We even have systems where the people who are in the loop are at some location thousands of miles away. People in Washington can push the button to drop a bomb from a drone halfway around the world.

The point is, we already have highly automated, massively destructive systems that could be turned against us if the networks and systems could be hacked. This is nothing new. I worked on manned systems in the 90s that had enough automation and remote coordination that network security was a serious concern. We have been doing this for a long time.

Moreover, secure communications is something that has been critical to military operations forever. The first line of security is not the drivers and OS on a device, it is the network itself. Just disrupting the communications of advanced military devices is hard enough. Actually injecting your own traffic into these channels is way past the skill necessary to break into virtually anything else out there.

And once you're in, you're not usually going to find Linux or anything else you know. I worked on a mission computer operating system. It was 100% custom. There wasn't a single line of commercial code in that system. A hacker would find nothing familiar.

Re:the military has much experience with this risk

[Whorhay]

For the time being I think the risk of such hacking is only a major concern when it comes to other nation states. Sure an individual or small group of hackers might find their way into a .mil system every once in awhile but they aren't likely to find a way into anything mission critical that way. Other nation states though have the resources to both develop intelligence gathering networks and possibly acquire example systems. I suppose if such systems were involved in conflicts in enough numbers with enough losses then you might see civilian hackers begin to work their way in. As it is the surface area that gets exposed to private individuals with the right tools and the inclination to poke at such systems is going to be pretty small.

Thats easy

[MichaelSmith]

<tars>Nothing Good</tars>

ha ha

[AndyKron]

There's nothing that can be done and I'll be dead by then so have with this.

Robo-Coup

[mentil]

And what would happen if killer robots got hijacked?

When killer humans (i.e. a military) gets 'hijacked', it's called a coup.
It's unlikely the autonomous weapons would be hacked to fire on civilians. It'd be MUCH more effective to turn them into robotic sleeper agents that target military officials/visiting politicians (thanks to facial recognition). OTOH, firing on civilians would ensure a) civilians demand their removal, ensuring no further hacking can take place; b) their hacking is discovered immediately, ensuring the vulnerability is quickly patched, and c) an opportunity to take out more-important targets is squandered.

Re:Robo-Coup

[umghhh]

Is it all that simple?
As for people (being targets etc) can have control - that is a fairy tale. This does not work properly in majority so called democracies in the West where democracy has been replaced (for good reasons) by representative type of it. The result in US is for instance that majority probably knows that war on drugs is senseless and rather counterproductive. It goes on however even if in some places you can sell and buy ganja now and do not fear immediate legal consequences. There are other examples and there are other countries where the same happens. One may argue of course that this is not the same as immediate threat to all humans which brings us all on the boat etc. Yet who said that the 'revolt of the machines' will be of massive military type that we all associated with warfare. What about efficient, targeted and limited actions against some individuals? As an example - if a protest movement grows big enough it may indeed change course of history, if you take wind from the sails by assassinating one of its leaders you may prevent said change of the course of history. I would not think there would be mass protests in such case and even if there were properly controlled by propaganda actions they just die out. For this you just need enough power. We have seen how that worked in Iran with half of the country taking to the streets in early 2000s. But even the military coup that took place in Chile in the 60ties has succeeded. Who says it is not possible then? Even skynet like 'revolt of the machines' is in some sense possible - the swarms may be difficult to stop once put into action and some of these actions may be difficult to recover. The potential is there. Even if the likelihood of utter demise of humanity in any particular year is small - there are plenty of years ahead.

To UN I have one thing to say - it may work as a forum for information exchange between countries but it is already overtaken by the countries which do not share values majority people in the West does consider vital. My point is: what qualifies UN to any decision in the name of humanity?

Re:Robo-Coup

[Kjella]

It'd be MUCH more effective to turn them into robotic sleeper agents that target military officials/visiting politicians

How do you figure that? The first time you use it there'll probably be eye witnesses, security cameras, ballistics etc. pinning the robot as the guilty one. After that they will take the model out of service and scour through everything until they find out how you did it.

Compare that to a thousand units going berserk all at once at many different locations. Most of them will probably be stationed at military bases, everyone there is a target as either military service personnel or NGOs. Lots of expensive equipment too. And if some are on guard duty in a public location, well shit here's your "protector" going on a rampage. It could do some real military damage, it'd be a PR nightmare at home, a PR nightmare abroad and afterwards they'd be nothing but junk because you had to stop them by force. And a lot simpler, a simple countdown and then start shooting at every heat signature you see.

If I wanted to use them to find some high-level target for assassination well then I wouldn't use my eyes and ears to execute it. Then I'd just use that inside information to plant an IED or something that wouldn't compromise my intelligence asset. To do that it'd have to be very, very important and no other way to get it done.

Re:Robo-Coup

[mentil]

All good points. OTOH, if you're able to put them going berserk on a timer, why not time it to coincide with when your intelligence says a VIP is going to be present?
Bonus points if the killbot has IoT flaws that allow for backdoor access to the military network.

Bad official policy

[Todd Knarr]

Replacing human combatants with robots is a bad policy decision in general. The reason is simple: one of the main reasons restraining governments from waging war is the cost in terms of their own citizens killed in the fighting, removing humans from the fighting removes that restraint. We'll have enough problems with governments that already don't care about their own citizens, we don't need to add every other government to that list until we figure out another way to discourage them from starting wars any time they don't get their way.

On top of that, the possibility of robots being subverted by attackers isn't in any way overstated nor are the reactions to the possibility over-reacting. Look at our computer networks today and try to convince me that we can somehow make botnets and malware vanish overnight, and then picture a world where "distributed denial of service attack" translates to "security guards shooting any human who enters the shopping mall".

We already have plenty of killer robots

[iceco2]

People keep talking about this like this is some futuristic threat. Yet we are living with this threat(and managing it) for a while.
We have weaponize drones and other remotely controlled platforms for many years. I am aware of one incident where the Iranians managed to hijack a US drone, and force it land. But this was presumably done with GPS spoofing and did not give them sufficient control to attack anything.

Air to air missile systems have fully autonomous modes of operation, and we trust them not to shoot airliners out of the sky.

The future is here, has been here, and the the risks are manageable. Of course we need to make sure we have adequate safeguard against hackers or simple malfunction, but we are far away from a Terminator scenario.

Re:We already have plenty of killer robots

[Krishnoid]

Not only is the future here, so is the video.

What does it mean to 'ban' them?

[Wrath0fb0b]

"Imagine an artificial-intelligence-driven military drone capable of autonomously patrolling the perimeter of a country or region and deciding who lives and who dies, without a human operator. Now do the same with tanks, helicopters and biped/quadruped robots." A United Nations conference recently decided not to ban these weapons systems outright

Forgetting for a moment the logistics of actually enforcing such a ban, what would such a ban actually entail?

Presumably the ban would not apply to the mere act of developing a drone/robot or the AI to make it move around and perceive the environment. That is, it's not a ban on any of the fundamental building blocks of robotics or AI. Nor would it apply to actually building a search-and-resce bot or teaching it to navigate unknown or difficult terrain. So countries interested in such things would continue to build them just short of actually being weapons (with laughable charades)

After that, what's left? Attaching a gun to it? IFF? It seems fairly clear that this is not the technical challenge here. So even if there's a 'ban', countries will be free to develop technologies and, if they master those problems, will credibly be a screwdriver away" from having full on AI weapons.

wrong

[Tom]

Every machine can be overridden, tricked, hijacked and manipulated with an efficiency that's unheard of in the realm of human-operated traditional weaponry.

It isn't really all that difficult to buy, bribe, threaten or convince a human.

The difference is scale. Humans are polymorphic, so they are not exact copies of each other and the identical exploit will work on one, but not others. So you need to customize your exploit for each of them, which makes mass hacks difficult. That is the reason social engineering works, but is rarely used large-scale.

Re:wrong

[DCFusor]

And you're not thinking big enough even with that! Why do we let others create war machines - using our resources - and suck us into dying for them? Are we so lazy as to need to be told what to think, because we can't be bothered to learn HOW to think? /rhetorical
Back up and think why there are migrants in the first place as a small example for what is a far larger issue than them and Europe. When has the West missed a chance to ruin what little they have, make enemies with "collateral damage" that from afar, seems anything but accidental? Ruin their ability to do well where they are...kill their relatives, innocent or not, give them a reason to hate that's very palpable (you killed my family) and then open your doors...because you're humanitarian - unlike that government you let run your own show that causes this in the first place.
.

The cognitive dissonance implied boggles the mind. First you let your governments create huge problems, then you try to fix it, not by putting a stop to that, but by taking action to accept the consequences for something you never asked to have happen. How many people in the West really want to destroy the middle east? Only the ones who gain money and power by it, I'd wager, or mostly them. I think I see where the problem is...but knowing what it is isn't going to feed the kids of a weapons builder who doesn't seem themselves as driving or enabling great evil, and who will insist on their important job...

Re:wrong

[Tom]

That was not social engineering, it was propaganda. Similar, but not the same thing.

Re:wrong

[Tom]

It's a bit more complicated than that.

During the Cold War, the West meddled in various countries affairs to prevent that they become too friendly to Russia. This was a huge part of why Afghanistan, Pakistan, Iran and a bit later other countries as well slid into Islamism.

Once Islam had taken over, some smart people understood just to what degree they had fucked up and that Islamic countries can never be allowed to become rich, properous or influential. That is why Saudi Arabia... oh... wait... I'm afraid the smart people were busy doing other things and nobody realized just how serious the fuck-up was, until the wake-up call of 9/11. Then, finally, the sources and nests of islamic terrorism were wiped ou... wait, what? They attacked Iraq? The Saudis are still good friends? Ah, my mistake. It took until the Syrian war before... not? The Jemen war? Seriously? Still not?

Re:wrong

[DCFusor]

Yep, but I'm trying to learn how to avoid TL;DR...often failing. Depends on the audience too.
Everything in the ME is complicated...seems everyone involved likes it that way. Which is yet another mistake.

Re:wrong

[drew_kime]

Every machine can be overridden, tricked, hijacked and manipulated with an efficiency that's unheard of in the realm of human-operated traditional weaponry.

It isn't really all that difficult to buy, bribe, threaten or convince a human.

Yes, a human. OP said every machine. Learn to subvert one, and you've got them all. Efficiency. That's the problem.

Re:wrong

[Tom]

Uh, you did read the part of my posting that you didn't quote, where I'm literally saying the exact same thing?

Re:wrong

[Tom]

I disagree. The Middle East isn't complicated at all.

It's an area where several different tribes, cultures and religions are all mixed up. As long as everyone is tolerant of everyone else, they all profit from the exchange of ideas. Whenever some intolerant version of someones religion becomes dominant, things go to hell.

The primary problem being that the dominant religion for the last thousand years explicitly hates another major religion, and is explicitly intolerant of all the others, so it takes a bit more of the don't-take-it-too-serious approach to live a workable version of it. That's why things go to hell more often than elsewhere in the world, and more badly.

Re:wrong

[drew_kime]

So how was the part you quoted wrong?

Re:wrong

[DCFusor]

I didn't say (or at least mean to) that it was complex, only that people "make it out" to be so. Though it is interesting that various factions can't seem to let go of the idea that one particular spot there was the "holy home of their religion". As if you had to be on it for your religion to be valid or something.
The fact that pretty much any of the interested parties has once controlled that same spot of land, and considers it all-important to get control of it back is a complication...perhaps unwillingness to accept the flow of history is a "simple" problem to define, but not so simple to solve when to the players, everything is utterly black and white. And everyone's grandpa has killed everyone else's grandpa, with memory all too long. I liked Henry Rollins' take on it - no matter what the causes are/were floating around - do you really want your kids exposed to living in hatred for the rest of forever? I was glad to see him put that message out to all the sides of that issue on a standup comedy tour. It sure is a waste of energy as it is.
I agree that taking it all too seriously - self-important hubris - is probably the largest part of the problem. Overcompensated inferiority complexes?

Re:wrong

[Tom]

Though it is interesting that various factions can't seem to let go of the idea that one particular spot there was the "holy home of their religion"

Because when those religions came into existence, all religions were tribal religions. The christian and islamic idea that everyone can become a member is an innovation in the sphere of religion.

I agree that taking it all too seriously - self-important hubris - is probably the largest part of the problem. Overcompensated inferiority complexes?

I'm quite sure there is a strong correlation between strength of religious feelings and dick size, yes.

Re: wrong

[edris90]

What's the model for different people interacting in the world in general anytime a singular culture becomes dominant it becomes detrimental to the majority over time. Once a culture run out of other cultures to demonize, it turns on itself, splintering off segments of itself and creates new villains from its own numbers, to maintain artificial distinction and conflict and collective ego. Is only mutually assured destruction that keeps the peace with humans. It is only prevention of any dominant pattern the law allows people to choose peace instead of War.

Re:Drones

[NicBenjamin]

IIRC the Iranians forced one down. The Russians have managed to jam everything, including GPS, over Syria several times. But in terms of actually hacking the drone, so you can take control of it and use it against it's official owner?

Nope.

The gov remains oblivious to what?

[Lije Baley]

Who really thinks that any one country has any say over whether such weapons will be developed? BTW: How is this different from worrying about the same kind of scenarios involving, say, a bunch of F-35s? Not really, I would say, as those killer robots wouldn't get far either without ample human arming and maintenance.

More realistic scenario

[Max_W]

What if a SUV which has got 500 horse powers and weighs three tons gets hijacked? One needs to study for years to pilot an UAV, but everybody can drive.

In my opinion there should be a legal limit to personal car power and weight; say 100 hp and 1500 kg.

And it is not "maybe", it is happening. About one and a half million(!) people are being killed on roads each year by cars globally. Times more badly injured. These are figures of a WW3. These accidents are related to drinking, suicides, mental illness, terrorism, drugs, etc.

And practically nothing is done about it. Even more powerful and massive cars hit the market. So, please, stop blaming drones. Let us first learn how to handle the real problem at hands.

Horizon Zero Dawn

[F34nor]

Basically exactly fucking that.

We don't have to wonder; we've seen it.

[Hallux-F-Sinister]

At best, a robot is basically something with at least a rudimentary decision-making system, taking input data, performing computations upon it and/or comparing it to previously stored data, that then arrives at a conclusion, and as a result makes and then implements a decision. In the case of a killer robot, (which we must recall is NOT a robot that does nothing whatsoever but kill from the instant it is created, to the instant it is destroyed, but rather a robot that is capable of killing, and which may, under the right circumstances, do so,) we're just talking about something that presumably will only kill under a certain specific set of circumstances, which upon being 'hijacked,' (by, one presumes, someone with malicious intent,) begins killing people whom it was not originally intended to kill, or to kill under circumstances other than those the people whose killer robot it was, intended.

What is a robot? It's something with at least a rudimentary decision-making system, within or controlling a chassis that can, in theory, in some cases, when designed and built or modified to be able to do so, move autonomously about. What is a killer robot? A robot that beyond the aforementioned requirements to be a robot, is physically capable of killing (people).

I remember this one time, I saw all over the news a couple of years ago, reports of this one day when a chassis, whose controlling decision-making system had gone haywire, (in this case, the killer robot was made of meat, but that's not really relevant,) went into a hotel room in Las Vegas, Nevada, high above and in sight of a nearby music venue. The rudimentary decision-making apparatus driving it, which had, up until that point only killed people when it was appropriate. (In this one instance of this model of killer robot, I believe that to the best of anyone's knowledge, the circumstances had never previously occurred in proximity to it since the day it went operational, decades earlier, when it would have been appropriate for it to kill, and so it had never killed anyone before that day. I could be wrong on that point, but what had happened before is not really relevant to the case, so... moving on... but if it had, it would probably not have been free to move about as it was, so for the sake of argument, let's say it had not.)

But on this fateful day, this killer robot's programming somehow got screwed up, and it decided that it needed to kill a large number of people, and picked the people at that music venue as its target. (I'm not going to speculate HOW or WHY its programming got screwed up, but suffice it to say that despite all that it had likely been through over decades of operation, it had never previously gone nuts and killed anyone,) and so it came to pass that it concluded that it was right and proper for it to use one or more weapons of war, either designed, modified, or rigged-up to maximize lethality, to kill and maim, or at least grievously wound, a large number of people in the music venue across from the hotel.

Now... you may dispute my equivocating a (human...ish, anyway,) mass-shooter to a robot, but at the end of the day, there's no meaningful difference. You have who or what was NOT supposed to kill random (or specific) strangers, and had for a long time managed somehow not to, and then one day, it does, and it kills a BUNCH, owing to being hijacked either by a hacker who figures out how to remotely control it, or a collection of bad ideas, improperly dealt-with anger or stress, or a desire to harm as many other people as possible. Again, the result is the same. Large numbers of dead and injured being made out of previously healthy, presumably happy individuals, by someone or some thing that is difficult to stop. (Recall the efforts that had to be made to stop the guy. Suppose he got away? Suppose he'd managed to take out the cops in the hallway, jump out the window and parachute to a waiting motorcycle, detonating the room and evidence behind him, and speeding into the night o

Re:How could you possibly hack a killer robot

[Anne Thwacks]

Once the sloc reaches a million, what chance is there that it WILL do exactly as its told?

Re:How could you possibly hack a killer robot

[Cmdln Daco]

Checksums are trivial to defeat. Hashes, a little harder.

Why just killer robots?

[aglider]

What about cars, trucks, trains, airplanes, medical devices, home automation, mobile phones, smart TVs ... you name it?

They will kill something

[Hognoxious]

They will kill something. It'll just be a different thing than if they hadn't got hijacked.

Think a little

[Wizardess]

There is one sinple general rule I've learned about life in general over 70+ years. It reads, "If it can be done, it will be done." It's one of Joanne's laws, I guess. I have many I've derived over the years. This one is the most generally applicable. If there is a market of people willing to pay for something and any hole of any size through which to slip product, the market will be fulfilled. Drugs are the most obvious such. Guns are another obvious market filled because there was a way to fill it, a way to do it. Scientific curiosity is another way things that can be done get done. "What happens if I ....?" "Oh, neat!" or "Oh well, I can buy a new lab."

AI weapons can be done. They will be done. That is a simple application of Joanne's Do-it law.

So my first question is, "If you make it illegal, what magic will you apply to prevent it from happening anyway?"

That leads directly to my second question. "Are your principles so strong you are willing to NOT be on the side that does it first and most effectively?"

(And as an aside, a Computer Organized Operation that takes over a government is a Coo-Coupe.)

{^_^}

Start learning German !!

[pablo_max]

Here in Germany, our population is only about 80 million.
Regrettably, as we have shown in the past, this number is sufficient to achieve world, or even regional domination for the long term.
It seems to me these robots you speak of is the answer to our past failings.

Sign up for Duolingo my friends and start learning German now.

Who is regulating killer cars?

[misnohmer]

Vehicles can be used as lethal weapons. Imagine all those self driving vehicles which suddenly flip one "if" statement, and rather than avoiding pedestrians they aim to hit them. Better yet, imagine a more advanced hack which will perform face recognition and only target specific pedestrian, or even a group of pedestrians. Organized car attacks could be used to attack infrastructure too. 100 million cars suddenly used as weapons might present more danger to the people than a few thousands border patrol robots. While people are thinking about regulating military robots, I don't see any lawmakers worry about cloud connected, self driving capable cars as lethal weapons.

What Will Happen When Killer Robots Get Hijacked?

[CptLoRes]

I imagine they will just continue with the killing? Unless it happens to be a pacifist hacker that is..

This is easy

[StikyPad]

Locking down a platform to respond only to authorized access is pretty easy. Microsoft is, ironically, a perfect example. Using cryptographic signing, its mass-update feature has never been compromised. The same robust authentication mechanisms are almost certainly in place for military hardware, making it statistically improbable that anyone can "hijack" the controls without physically taking them over. And if the enemy can take over your command center, then all bets are off anyway.

The ruling class always needs poor

[rsilvergun]

because if nobody is poor than nobody is _rich_. Being wealthy isn't just about material wealth. It's about the political power that comes from deciding how resources are distributed and, more often than no, who lives and who dies.

Re:The ruling class always needs poor

[SqueakyMouse]

In the scenario I described, there was a 'ruling class' with political power who controlled how resources were distributed, and could potentially control who lives and who dies. My question is why would they use that power in the way you're describing? What do you think their goals are exactly, and how does 'abandoning' people help toward these goals? Do you not think you have more power over people if they're dependent on you than if you cut yourself off from them?

Strobe Lughts

[bill_mcgonigle]

The most startling fact I've heard about the robot army scare is that Boston Dynamics, or whatever they're called now, has as a goal, making that mule robot so fast that a human will need to use a strobe light to see its movements clearly - it'll be a blur to the human eye otherwise, when it's done. Keep that in mind when you consider the idea of a human army fighting a robot army. Now make those robots autonomous and more intelligent than humans.

Coup

[pikine]

At least when humans are couped, there will be some resistance. Robots don't know their rights and wrongs, so creating an army of killer robots is just a roundabout way to voluntarily give away all your military power to someone else. The money US spends on AI military now is the money Russians will save when they turn the robots back against the US.

They might as well save everyone the hassle and kiss up to the Russian overlords now.

Re:Coup

[mentil]

So you're saying that in Russia, killer robots hack YOU?!

Already did. They voted for Trump.

[bussdriver]

Sorry, couldn't help but join in. Even though it seems there are more payed trolls posting on Trump side... robots?

Re:Unhackable

[PPH]

We begin our counter-offensive on Patch Tuesday.

To live like Gods

[rsilvergun]

and I don't mean materially. I mean being treated like a God. To have the entirety of human civilization bent not just to your will, but to exist for the sole purpose of improving your personal quality of life.

It's a level of narcissism and power that's hard to really imagine. It's like trying to get a grasp on a google (the number, not the company). The human brain isn't well equipped for it.

And they're not going to cut themselves off from everyone, just most everyone. Also, they're not intentionally cutting themselves off from those people. At that level of wealth 95% of the population becomes like ants. You just don't think about them, again, any more than you and me spend our days thinking about the kids currently getting bombs dropped on them in Yemen. It's too far removed from your life. And the remaining 4% of the population (which is still several million) spend every waking moment trying to get into or stay in your good graces. Again, it's the closest thing to Godhood humans can imagine.

Not much difference

[nehumanuscrede]

Between a hijacked " killer " robot and today's method of calling the police for the sole purpose of sending a Swat team to the targets home.

One is directly under your control, the other indirectly. The outcome is pretty much the same.

Re:Drones

[Iamthecheese]

The US military has greatly shortened its time to equipment improvement. Strategic purchases are now requested, implemented, and delivered in months instead of years when they're urgently needed.

easy

[ziggy182]

They will fight Chuck Norris and loose!

First things first

[McFortner]

I, for one, welcome our new robot overlords.

Re:Drones

[Hognoxious]

If you jammed them you can always bring them in and replace the electronics in them.

Jamming prevents it contacting its base. It doesn't allow you to control it.

Re:Drones

[peas_n_carrots]

"Never" been hacked before, so must be unhackable.

Famous last words...

Its a Liability bypass,

[edris90]

Once they can get assault bots mainstreamed, then they can hack their wayaround ,oversight ,accountability. now that they can always just say that they got hacked they can do anything they want and make up any excuse and scapegoat anybody for it and then continue to do it. The hackability to the military is a feature they can use to distort the public View

Re: the military has much experience with this ris

[edris90]

The hackers and threats from other countries are not the biggest danger.the biggest danger is our own military overstepping its own bounds. concern is the our soldiers may fail to disobey their commanding officers when appropriate and through orders betray the interest of the Common Man.

Re:Drones

[NicBenjamin]

I didn't say they were unhackable. However, the nature of those hacks makes seizing control fo a combat drone from a human operator and shooting at it's troops much harder then most hacks.

The drone is loitering above the combat zone for a limited period of time before it runs out of fuel/weapons/etc. and has to return to base. It has multiple systems you'd need to crack. You'd probably need access to it's cameras to aim it's weapons properly, which means you have to hack the drone's transmission hardware. To send the commands you also have to hack either the receiving hardware or spoof the signal. The weapons, engine, flaps, etc. may or may not be on the same system so it's entirely possible you could steer the drone but not fire the weapons or control the speed. And if the operator notices he can just get on his radio and tell the troops to shoot their drone down. So you have to hack multiple systems, in roughly a 24-hour-period, and not be noticed. Possible, but more then a little tricky if the drone-wielders don't suck.

That does not mean they're perfect. The Army and/or Air Force should be spending a fairly significant amount of money figuring out a way to make jamming harder, because a jammed drone is useless. The Iranians clearly had something interesting going on or they couldn't have forced that landing. But hacking a US drone and using it to bomb US marines is a lot harder then you are implying.

Re:Drones

[peas_n_carrots]

Someone wanting to take over an autonomous army wouldn't be hacking individual drones (unless maybe they're all in the same spot). The lowest barrier to takeover is to hack central command. This could be done from the inside or out. Protecting against all avenues of attack is incredibly difficult.

Your rationale is exactly how companies get into trouble with security. Consider only the hardest avenues of attack and claim it's secure. Hackers don't go after the the hardest vectors, they go after the most vulnerable.

Re:Drones

[NicBenjamin]

Central computer?

Pretty sure the technical name for that is "the pilot's head."

Hacking the central computers of Creech airbase you'd be able to play hell with the contracting system, but that's about it.

The pleasures that come with power

[rsilvergun]

Sex slaves. Monuments built in your name and image. Worship. Scientific advancements to improve your life and longevity. Good 'ole sadism, or maybe just the opposite: feeling like a white knight when you swoop in to save the day with food donations.

Power isn't a means to an end, it's an end in itself. If you're not smart enough to immerse yourself in the wonder of the universe like Einstein did and you don't need to work for a living there's not much else left.