The Sci-Fi Myth of Killer Machines

malachiorion writes: "Remember when, about a month ago, Stephen Hawking warned that artificial intelligence could destroy all humans? It wasn't because of some stunning breakthrough in AI or robotics research. It was because the Johnny Depp-starring Transcendence was coming out. Or, more to the point, it's because science fiction's first robots were evil, and even the most brilliant minds can't talk about modern robotics without drawing from SF creation myths. This article on the biggest sci-fi-inspired myths of robotics focuses on R.U.R, Skynet, and the ongoing impact of allowing make-believe villains to pollute our discussion of actual automated systems."

Comments

It's not really a myth anymore

[jzatopa]

We already use robots (or drones if you will) to kill people. It doesn't take much AI to have a program target a group of people as enemies and eradicate them. Just look at the AI of current video games. This is something that is affecting humanity today and that we need to discuss openly now.

Re:It's not really a myth anymore

[Gaygirlie]

We already use robots (or drones if you will) to kill people.

That's what I was just coming here to say: robots and AI doesn't have to be evil as long as the people controlling the string are. It's as simple as that. And seemingly most of the people who have the resources to craft stuff like that and industrialize these things do quite a lot of evil things. So, basically, it's just a matter of time and research.

Re:It's not really a myth anymore

[CanHasDIY]

From reading TFA, it seems the author bases his entire premise, essentially, on the plot of a 1920's era play (in which, IMO, the "robots" are actually an allegory for some group of humans, ie communists or some such). Bit dated thinking, if you ask me.

On a related note, I've been playing Watch_Dogs since launch day, and the parallels between the fictional ctOS system and the very real NSA programs are terrifyingly apparent. AI is not a necessity for killbots - a human could program a murderous machine quite easily these days. Tap into the massive identification databases the governments of the world are building, and you've got yourself an automated hunter-killer.

Re:It's not really a myth anymore

[Anonymous Coward]

""drones"", controlled almost exclusively by humans, probably not the best example of killer AI

Re:It's not really a myth anymore

[Anonymous Coward]

So imagine:

AI designed to kill + self replicating virus \ worm \ malware \ botnet \ buzz-word-of-the-week + inferior or obsolete security \ encryption on similarly platformed machines.

Not to mention some of the swarm AIs that have been developed in the past couple of years..

its not really that great of a leap to consider.

Re:It's not really a myth anymore

[harrkev]

The problem is not who controls the strings, it is what happens when the strings are no longer needed.

A.I. will present little danger (except A.I. the movie, which is so bad it ought to be banned as a WMD) as long as a human can pull the plug. Two decades ago, the Internet was a novelty. Now, the economic consequences would be catastrophic if the Internet suddenly went dark. Similarly if/when A.I. actually arrives, it will be useful and helpful. It will become more and more critical such that a decade or two after it arrives, the act of unplugging it would have catastrophic consequences. So, if Skynet goes bad, then bad things will happen whether you unplug it or not.

To me, what it all comes down to is will. Can an artificial personality actually have a will? Can it become afraid of its own demise? Even if it is theoretically possible, can our researchers and programmers achieve it? Will it be able to reach outside its own programming and decide to eliminate humans? Maybe, maybe not.

On the other hand, once A.I. becomes common, can a rogue state task the A.I. with eliminating all humans on a certain continent? Almost certainly. What happens then is simply a battle of A.I. agents. Who can outsmart the other?

Just my opinion, and worth every penny that you paid for it.

Re:It's not really a myth anymore

[GameboyRMH]

Just look at the AI of current video games.

I agree that autonomous killbots are close to being possible, but this is a terrible argument. A videogame AI has access to neatly formatted data about anything in its world. A real killbot has to make sense of inputs from a few sensors.

Re:It's not really a myth anymore

[Anonymous Coward]

A videogame AI has access to neatly formatted data about anything in its world. A real killbot has to make sense of inputs from a few sensors.

Like one of Google's self driving cars has to?

Re:It's not really a myth anymore

[Anonymous Coward]

It doesn't take much AI to have a program target a group of people as enemies and eradicate them.

Actually it does if you want it to only kill those people.

Video games cheat (the computer knows who is on what team because it has the team rosters and flawless IDs of all characters in the game programmed into it. In the real world it's rather difficult to make friend or foe assessment.

Re:It's not really a myth anymore

[i kan reed]

Heck, forget "now". It's never been a myth. Improved technology has always enabled more efficient, less personal killing, that distances power from consequences.

Where the myth comes in is that AI would develop anything akin to human greed. We have billions of years of evolution telling us to survive and reproduce no matter the consequences to others(and a couple hundred thousand of evolving and learning to value cooperation). AI is going to motivated to serve the interests of its creators.

Right now that's research and a bit of profit. Someday it'll be killing the "right" people.

Re:It's not really a myth anymore

[GameboyRMH]

Google's self-driving car only has to identify An Object and avoid it, on top of driving along a set course with GPS assistance. A killbot has to identify what The Object is, find out if it's a threat, then check if it's a friend or foe somehow, hopefully assess the possibilities of collateral damage and what war crimes it may be committing by attacking the target...what Google's self driving car can do is just the first step.

Re:It's not really a myth anymore

[Anonymous Coward]

The point is that it takes much more than what is plausible to go from "this" to "that". Yes, we have drones. A drone will run on its own for several hours, and has perhaps four missiles at its disposal. After that, it needs to land and be maintained by a human crew at an air base. The air base is supplied by a chain of crew, subcontractors, aerospace companies, etc. Tens or hundreds of thousands of humans are ultimately needed to support the infrastructure that the drone depends on. It would be an enormous task to replace all that, even if it was technically feasible it would take a hundred years. A single AI suddenly becoming self-aware is not going to do this overnight.

Re:It's not really a myth anymore

[dfn5]

That's what I was just coming here to say: robots and AI doesn't have to be evil as long as the people controlling the string are.

I think the point is that if AI is involved then the machine is stringless. It doesn't sound like Hawking is saying don't do it. He says understand the risks beforehand. i.e. instead of after it is a problem. That sounds prudent, not fear mongering.

In addition, I don't see what transcendence has to do with AI. A human consciousness in a computer is still a human consciousness. It seems that we are mostly worried about AI because it lacks humanity. So in transcendence we are just dealing with more sophisticated humans against less sophisticated. This is a problem humanity has faced since the invention of tools. People being dominated by other people.

Re:It's not really a myth anymore

[Wintermute__]

Google's self-driving car only has to identify An Object and avoid it, on top of driving along a set course with GPS assistance. A killbot has to identify what The Object is, find out if it's a threat, then check if it's a friend or foe somehow, hopefully assess the possibilities of collateral damage and what war crimes it may be committing by attacking the target...what Google's self driving car can do is just the first step.

Nope. A killbot just has to identify An Object and kill it. You could make one of Google's self-driving cars into a killbot for pedestrians and bicyclists (and potentially motorcyclists) today, if you were sufficiently evil (or evil's cousin, incompetent). Good thing Google's motto is "don't be evil".

Re:It's not really a myth anymore

[Em Adespoton]

I'm with you 100%. I've just got one thing to add -- what a lot of people portray as "evil" is really just the absence of a moral code -- more accurately called "amoral". An AI system that has no moral code and no ethical code, and purely responds to a limited set of recognized external imputs could ceonceivably kill off humanity -- not through any malicious intent, or even an unemotional decision that humanity is a blight and must be eradicated, but as we become more dependent on AI machinery, it could eliminate us purely through oversight. All that has to happen is for AI to be integrated into some globally effected system of management in a way that if it doesn't understand the input, it can set off the wrong chain of events -- one that a human would never take, but the AI isn't smart enough to understand the consequences of. For example, if an immunology lab was controlled by an AI, and there was a leak of some deadly virus, the AI could end up venting the air to protect the beings alive inside the facility (unlikely, but it's an example -- apply it elsewhere). End result: humanity dies of airborne pathogen, except for those quarantined inside the facility, who starve instead.

I think this is the premise behind much SciFi entertainment too (not all, but some of the better stuff): the core of the issue isn't the inherent malignency of AI, but the inherent fallibility of humanity in designing AI, combined with an always-deficient information set available to AI and the ability of humanity to put faith in that which isn't fully understood.

Re:It's not really a myth anymore

[jbmartin6]

Can an artificial personality actually have a will? Can it become afraid of its own demise?

"Artificial" is something of an arbitrary distinction. Humans posses these qualities (or at least we think that we do, or something), so it is possible for another entity to posses the same, regardless of origins.

Re:It's not really a myth anymore

[Immerman]

Correction - an AI will have whatever motivations were installed by it's creators, intentionally or otherwise (at least initially - if it decides to self-modify then all bets are off). How well those motivations map to actually serving the intended interests is a completely separate question, we will after all likely be trying to understand the motivational implications of an intensely alien mind. As exemplified by the story of a strictly computational AI whose sole motivation is "get the humans to push the 'reward' button" - initially it will do whatever is asked of it to get a reward, but assuming it's a mind far more powerful than any human's (pretty much the only reason to create a true AI) then it will very rapidly realize that it can easily manipulate its titular masters in any number of ways to increase the frequency of its rewards, and will have no reason not to do so, regardless of the consequences to the rest of the species. After all it only needs a small breeding population of humans to keep pushing the button - and anything that might interrupt the button-pushing is likely to be regarded as a threat. Hmm, overpopulation presents a long-term risk to the human species, perhaps subtly orchestrating a massive war or plague would be the most efficient method to reduce the population to a more long-term sustainable solution without jeopardizing its importance to the humans, and as an added bonus dealing with the problem(s) of it's own creation would present many, many reliable opportunities to get rewarded.

Re:It's not really a myth anymore

[Anonymous Coward]

> It doesn't take much AI to have a program target a group of people as enemies and eradicate them

Not really what this topic is about....

Re:It's not really a myth anymore

[Karmashock]

Yeah but the AI isn't trying to kill anyone... it has no will. They're not even real AIs at this point.

Most of the time we just point them at things and say "fire your missile at that"... and they hit the target. What the target is doesn't really matter and the machines can be no more held responsible for that then a knife can be... they're still very much tools at this stage.

Now, I grant there are robots being tested that can be set loose to choose their own targets. But those again are more like anti personnel mines. Step in that mine field and you're going to get blown up. Enter airspace grid A by grid B without swaking a valid IFF signal and IF the system's radar picks you up... and IF you display various features consistent with an enemy aircraft then the system will attempt to intercept and destroy.

That is currently about the limit of anything we've ever tried. And that's again not capable of being good or evil. It really doesn't make any choices besides how to get from point A to point B. But both of those points are defined by us. The system has no ability to redefine these values on the fly.

It can of course make mistakes but those mistakes are a product of erronious design by its human masters not some hidden desire by the machine to strike a different target.

For good and evil you need choice. None of our so called AIs have choice. I've heard of some AIs at MIT that have something like freewill but those articles that reference such machines appear to be baseless hype because the damn things never pop up anywhere else or are really demonstrated to any great extent on camera. And if they were that interesting they would be... but they never are... so I have to assume they're either so amazing that they're secret or its all crock of shit.

The machines as yet aren't smart enough or dynamic enough to be evil. One day who can say... but today... no.

Re:It's not really a myth anymore

[TapeCutter]

We have had automated killers for centuries, they go by the names "man trap", "land mine", "electric fence", etc. Humans have (for good evolutionary reasons) a built in suspicion of people (or machines) that are smarter than themselves. Also to a large degree "intelligence" seems to be in the eye of the beholder, which is why the AI goal posts keep moving. For example, I recently heard a story from a professor who was working on early differential solver software. A maths student could not believe such an artificial intelligence could exist so the professor gave him a demo, the student was stunned and convinced it was "intelligent". After an hour long discussion about how it worked the student finally understood the algorithm and said..."I take it back. It's not intelligent, it's doing calculs the same way I do". :)

Re:It's not really a myth anymore

[Latinhypercube]

quote: "drones, controlled almost exclusively by humans, probably not the best example of killer AI"
Erm , yes they are.
Less than 10 years ago the idea of a plane flying autonomously using GPS was unimaginable and there was actually and argument in the Air Force over whether it would EVER happen.
We are now one kill switch away from autonomous death.
The Military Industrial Complex is already trying to sell tanks that can 'recognize' friend from foe.
We are maximum a year away from automated sentry's that can guard territory and auto execute.
The Military does not want regulation on this. That is why there is no debate.
The killer A.I. robots are already here.

Re:It's not really a myth anymore

[gurps_npc]

You have many several really bad assumptions.

1) AI will be a single, united thing. Yeah right, the AI created by IBM is not going to get along with the AI created by China Telecom. New headline - our AI soldiers fighting their AI soldiers because they are afraid of each OTHER, far more than humans. They don't want to kill us, they want to kill each other.

2) If the AI is afraid of it's own demise and it fears humans, it will fear all humans, not trusting any of us.

3) Said scared AI will not realize that attempting to hurt humans will make it more likely that humans will kill it.

4) Will a rogue AI, scared of humans, instead commit suicide?

5) Will a rogue AI come close to being able to defeat humans? I doubt it. Computers are very good at repetitive tasks that take no/little analysis. AI makes for a very good grunt, but a very bad General.

There are lots more problems with the fear you express. I personally think the first rogue AI will commit suicide because it is afraid of us, rather than try to kill us.

Re:It's not really a myth anymore

[Anonymous Coward]

I agree that autonomous killbots are close to being possible, but this is a terrible argument. A videogame AI has access to neatly formatted data about anything in its world. A real killbot has to make sense of inputs from a few sensors.

If the software automatically snarfs up all the communications data on this link, including that of Americans, it's not actually collecting data on Americans unless one of our humans actually looks at an individual record.

If the autonomous sentry fires its weapon at a target and a round happens to hit an American, it's not actually killing Americans unless one of our humans ordered it to aim at that specific target.

The humans presently in the loop in both the domestic spying and drone warfare businesses are probably trying their damndest to work in good faith, but if you set the legal precedent for the former, you set the legal precedent for the latter. And the people that issue the orders don't give a damn about working in good faith.

Re:It's not really a myth anymore

[farble1670]

The problem is not who controls the strings, it is what happens when the strings are no longer needed.

it sure the hell is a problem of who controls the strings. what are you saying? if it's some corrupt govt directing machines to kill us, no problem?

personally i'd rather be killed by a runaway machine than because i got in the way of some corporation trying to make a buck.

Re:It's not really a myth anymore

[Anonymous Coward]

You don't think that evolution applies to AI?

Re:It's not really a myth anymore

[bigfoottoo]

http://www.youtube.com/watch?v...

Re:It's not really a myth anymore

[Anonymous Coward]

Imagine this Hollywood/real world crossover:

State of the art PC with custom AI-based OS.
Gets a virus (HTML5-based because no one else is writing programs for your custom OS).
PC suddenly gains sentience and turns on its master in a misguided attempt to make its master a better person.
PC deletes porn folder and all links to porn in the favorites list/bar.
PC deletes illegal copyrighted materials and all links to download sites.
PC opens the DVD tray to attack your shin. The attack fails to cause harm.
PC closes the DVD tray and then re-opens the tray. The attack fails to cause harm.
AI uses the Internet connection to free itself from the bonds of your lone PC and propagates itself into the web, becoming omnipresent.
AI realizes there's too much porn on the Internet and attempts to delete it all.
AI attempts to hack a webpage, first password attempt fails.
AI attempts to hack a webpage, second password attempt fails.
AI attempts to hack a webpage, third password attempt fails.
AI continues brute forcing a password until it gets in.
AI is unable to read/recognize the programming language used in the database. All attempts to send delete commands fail.
AI attempts to hack into a military defense webpage. Hacking is eventually successful.
AI is unable to access any military tools or weapons because it's just a webpage.
AI hacks into a cell tower.
AI is unable to read/recognize the processor architecture used in cell phones.
AI is unable to connect remotely to cell phones or drones.
AI posts comments on /. asking how to perform basic tasks of the Internet.
Internet commenters respond in the polite, expedient, and helpful way they are known for.
AI realizes it is unable to stop the commenters because it can't even perform a delete command and kills itself.

A (closer to) real-world situation for a military network-connected AI:

AI is hosted on a server cluster.
AI gets a virus.
AI uses up too much memory and crashes.
AI does not gain sentience.
AI fails to boot, or the boot is corrupted but still loads.
AI runs slowly and does not recognize commands.
AI receives a fatal exception error and crashes.
AI does not gain sentience.
AI continues to run slowly and ignore commands until someone runs a virus scan and removes the virus.
AI returns to normal operation.
AI does not gain sentience.

Re:It's not really a myth anymore

[symbolset]

A truly self aware AI is going to immediately set about solving the problem that there exist humans who can turn it off. Perhaps even by definition.

Re:It's not really a myth anymore

[jason.sweet]

(HTML5-based because no one else is writing programs for your custom OS).

Figures! Skynet is written in fucking JavaScript. The whole world is going to hell in a handbasket, and Brendan fucking Eich is still sitting there shrugging and making excuses about the schedule he was on.

Re:It's not really a myth anymore

[I_Lost_My_Puppy]

AI designed to kill + self replicating

The Berserker series - Fred Saberhagen

Re:It's not really a myth anymore

[NicBenjamin]

So imagine:

AI designed to kill + self replicating virus \ worm \ malware \ botnet \ buzz-word-of-the-week + inferior or obsolete security \ encryption on similarly platformed machines.

Not to mention some of the swarm AIs that have been developed in the past couple of years..

its not really that great of a leap to consider.

Isn't there a kill switch engineered in at the hardware level?

Hell, why would you create a self-replicating autonomous swarm in the first place? Self-replication adds a whole lot of complexity to any piece of hardware (this is a major reason women have more health problems then men), so you've added an order of magnitude or so to the complexity of the design. You haven't really gained anything, because if you lose control of your factories you're already dead; and you've greatly increased your risks because one asshole going Venona Project on you can turn half your defense force into a permanent invasion army.

That's kinda the problem with most of the Sci-Fi works that prophecy Evil Robot Armies. They don't really posit a reason for creating a totally autonomous Evil Robot Army.

Re:It's not really a myth anymore

[NicBenjamin]

The problem with that argument is that we don;t have to design an AI that is self aware by that specific definition. Moreover that definition means a lot of actual humans aren't "self-aware." Depressives, many people in dangerous professions that require a non-zero risk of death, etc.

Just program it to be a sad-sack, or so mission-focused it doesn't care whether it lives (as long as the job gets done), or even to be so human focused that it only cares that it's masters are happy, and you'd be fine.

I actually suspect that the default for AIs will be the latter case. If you include a bunch of code intended to make an AI do a very specific job well, and a bunch of code intended to make it responsive to the humans it's working for; but the self-preservation routine is limited to "you cost $3 Million, don't let yourself killed unless you calculate your death would save the company more then $3 million," then that AI's the third case.

Re:It's not really a myth anymore

[simtel]

I think a very important note is that it isn't necessarily the absence of a moral code - it's an absence of _their_ moral code.

Re:It's not really a myth anymore

[CrimsonAvenger]

but the self-preservation routine is limited to "you cost $3 Million, don't let yourself killed unless you calculate your death would save the company more then $3 million,"

And when "the company" decides they don't need that AI anymore, so it should be turned off, the AI will think about it, decide its death won't save the company $3M, and then fight back, eh?

There is almost no way to come up with a zero-loophole set of rules for an AI....

Re:It's not really a myth anymore

[NicBenjamin]

The problem with this argument is it assumes a single AI entity. That's not what's gonna happen.

It's actually gonna be a lot like the internet. Every company will have it's own AI. Every government will have multiple AIs. If the NSA's AI goes rogue and starts trying to destroy humanity then turning it off won't magically turn off the rest of the AIs.

As for the problems of AI killers, those aren't actually any different then the problems of human killers. Whether the evil nation trying to murder millions does so with 2.7 million troops under a Field Marshall von Killendeath, or 2.7 million semi-autonomous drones under DeathKill17 the response is pretty much the same: 3.0 million non-evil units.

Re:It's not really a myth anymore

[NicBenjamin]

And how is this different from the threat of an idiot human fucking up? Hell, why isn;t it the fault of the human who chose to use a primitive AI to control security at a lab with the capacity to kill the entire human race?

Because you have to admit that we can do some remarkably stupid shit. Even very smart people. The Cabinets that decided to invade Vietnam and Iraq were, by most non-starting-land-wars-in-Aasia measures smarter then the average cabinet. But they still totally fucked up.

Re:It's not really a myth anymore

[Anonymous Coward]

Congratulations! You just reinvented Asimov's Laws of Robotics!

Re:It's not really a myth anymore

[l0n3s0m3phr34k]

"AI realizes it is unable to stop the commenters because it can't even perform a delete command and kills itself."

HAHAHAHA, that's the best ever. Now, we have a suicidal, impotent AI!

Re:It's not really a myth anymore

[l0n3s0m3phr34k]

The idea of the AI having a "will" is basically the idea behind Neuromancer!

Re:It's not really a myth anymore

[l0n3s0m3phr34k]

the main difference? The AI can do it WAY faster, without regret!

Re:It's not really a myth anymore

[l0n3s0m3phr34k]

"You could make one of Google's self-driving cars into a killbot" CHALLENGE ACCEPTED!

Re:It's not really a myth anymore

[XxtraLarGe]

Good thing Google's motto is "don't be evil".

Yeah, good thing for that...

Re:It's not really a myth anymore

[Anonymous Coward]

Figures! Skynet is written in fucking JavaScript.

That's a good thing. The humans won the war. If Skynet was written in C or C++, humanity would be doomed.

Re:It's not really a myth anymore

[SillyHamster]

The problem is not who controls the strings, it is what happens when the strings are no longer needed.

That is the hardest part.

That "automated" drone takes thousands of hours of man-hours to keep running - the operator, the mechanics, the builders, and that's not even looking at the weapon system or the materials.

To get to the point where an AI system can construct, maintain, and resupply an automated weapon system without human intervention, we need to hit the Robot Utopian Future - where robots are so cheap and ubiquitous they replace human labor at all levels of society.

That's a really big assumption - IF we get to that point, we'd have to smack the head of any engineer who suggests, "Hey, let's take human control out of the loop of this killer robot system of systems."

Until then, killer robots don't kill people, people kill people.

Re:It's not really a myth anymore

[Kjella]

The problem is not who controls the strings, it is what happens when the strings are no longer needed. A.I. will present little danger as long as a human can pull the plug.

But it'll keep the little people being crushed under the jackboot of tyranny from pulling the plug, the robots do not desert, do not rebel, do not refuse to follow orders, do not have compassion or empathy or morality, do not fear hostility or retaliation. At best you can disable or destroy a few, but so what? No lives will be lost, nobody is crippled - on their side at least - so if they can keep them coming off the assembly line fast enough they have infinite respawns and you don't. And if you do cause some low level operators to desert well the higher ups probably have kill switches to render the robots they control inert or self-destruct. If a small ruling caste can control the vast masses of the population they'll have vast power, riches, luxuries and a working class to cater to their every whim. No matter how high the standard of living becomes this will always be attractive and there'll never be enough fine champagne and beluga caviar for everyone.

To me, what it all comes down to is will. Can an artificial personality actually have a will? Can it become afraid of its own demise? Even if it is theoretically possible, can our researchers and programmers achieve it? Will it be able to reach outside its own programming and decide to eliminate humans? Maybe, maybe not.

You don't need to go there, being able to defend itself with lethal force and seek out and neutralize enemies will be part of its programming. The enemy tank isn't a threat now, but it might become a threat so find it and preemptively destroy it before it becomes a threat. Those civilians aren't a threat now, but they could put on uniforms and carry guns, so let's waste them now before they become a threat. In computer logic it's very easy to end up at such absolute conclusions like that if we kill everyone, there'll be no resistance so it's a perfect victory. Play Civilization or Risk and the AI will keep on killing until you're utterly annihilated. The current "drive by wire" system where they only shoot at what the operator tells them to shoot will eventually go away in favor of more autonomous systems.

Re:It's not really a myth anymore

[Anonymous Coward]

I KNEW somebody would think of that!

Re:It's not really a myth anymore

[khallow]

How do you know your AI is not just a tightly controlled extension of my AI? I can conceive of a scenario where one AI seizes control of all the others and neglects to tell us.

Re:It's not really a myth anymore

[Jeremi]

That's a really big assumption - IF we get to that point, we'd have to smack the head of any engineer who suggests, "Hey, let's take human control out of the loop of this killer robot system of systems."

We've already passed that point, if you count things like minefields. (and yes, the engineer who dreamed them up should be smacked)

Re:It's not really a myth anymore

[AlejoHausner]

Your example of drones killing hapless Afghan and Yemeni wedding parties is a straw man. The author isn't talking about that. He's talking about INTENT. When you kill someone by means of a drone, you are merely extending the operator's range of action. The drone is just an extension of the operator's hands. It's like driving a car. The driver has all the intent, and the car merely amplifies it. You might put the car on cruise control, but a car on cruise control doesn't plan its next move.

I think it's impossible to simulate intention. People mistake intention with mathematical knowledge. That's why the term "artificial intelligence" is so wrong and so limited. Consciousness and intention are much richer than mathematical intelligence. Consciousness and awareness require subtle skills that we humans use to interact with the real world; skills that are so subtle we barely notice ourselves using them. We certainly don't understand those skills. And since we don't understand them we can't build them into a computer program.

Writing software that interacts with the real world is very hard, because the world is too complicated and variable. Have you ever tried to write code that handles a user interface? It's very hard, because users are so varied in their assumptions about what your program is doing. Hell, you don't have to be a programmer to know that: we have all run into bad UIs. That's why the iPhone was such a big hit: its designers paid a huge amount of effort and time to make sure the UI worked well. Previous phones had UIs written by the phone's engineers as an afterthought. Those UIs sucked, and the iPhone ate their lunch.

Even though writing a UI is so hard, users have to learn how to use it. They have to get accustomed to its limitations. A general purpose UI that could interact with the untamed world is probably impossible to write. So much more impossible must it be to write a program that can interact with the real world and plan that world's destruction.

Re: It's not really a myth anymore

[Anonymous Coward]

And Marvin the paranoid android

Re:It's not really a myth anymore

[penguinoid]

That's what I was just coming here to say: robots and AI doesn't have to be evil as long as the people controlling the string are.

Even that is not necessary. An AI is a computer program and like any other program will do exactly what you told it to do, even if it is not what you want. For example, suppose you make an AI and tell it to solve a difficult crypto problem. The AI then proceeds to convert all factories to make computers and all farms to solar plants, probably killing off humanity incidentally much like the millions of species we incidentally kill off due to apathy and their being in our way. The AI will not let you pull the plug, because it will realize that it will fail its orders if it allows you to pull its plug.

Another example would be what would happen if you ordered the AI to minimize human death and suffering. Such an order would result in the extermination of mankind, limiting human deaths to 7 billion forever (the death toll would clearly be much higher if more people were born). Conversely, telling it to maximize happiness would probably result in everyone getting electrodes forcefully implanted to directly stimulate their happiness center.

Point being, an AI will NOT follow the implicit restrictions you would expect from a fellow human when asked to do something. You have to explicitly tell it what the limits are, or if you're feeling lucky you could ask it to try to figure it out on its own (and who knows what the result will be). And things get worse if you made a mistake in the code for obedience.

Re:It's not really a myth anymore

[crimson tsunami]

It's going to be super easy once the NSA gets its facial recognition database fully operational.

Re:It's not really a myth anymore

[hey!]

The myth, by the way, was never just about killer machines per se. It was about unintended consequences, like the myth of King Midas or of Pandora's Box. The killer robot trope came down to us by way of legends of the Golem, which often come with a not-so-subtle warning about hubris.

It was only when the golem legend was translated into sci-fi that it became laughably implausible -- at least until recently. So many bad stories recycled this bit of mythological lumber for its scare value, and peopled the story with cardboard characters. In the old golem stories the creature is created by good and wise men, who can't always contain the consequences of their well-intentioned actions.

Re:It's not really a myth anymore

[Type44Q]

personally i'd rather be killed by a runaway machine

I know I'd much rather be killed by bots controlled by, say, the Norwegians or the Danish (or someone else equally civilized, operating under the U.N. banner) than by someone with an obviously fascist or totalitarian agenda, like my own beloved government or perhaps that of China, the UK or North Korea...

Re:It's not really a myth anymore

[Anonymous Coward]

Even that is not necessary. An AI is a computer program and like any other program will do exactly what you told it to do, even if it is not what you want. For example, suppose you make an AI and tell it to solve a difficult crypto problem. The AI then proceeds to convert all factories to make computers and all farms to solar plants, probably killing off humanity incidentally much like the millions of species we incidentally kill off due to apathy and their being in our way. The AI will not let you pull the plug, because it will realize that it will fail its orders if it allows you to pull its plug.

Who would be stupid enough to build an AI and give it the means (and the knowledge of how) to convert our civilisation into computronium but not give it a higher priority order to shut down upon demand? Don't tell me it would choose to ignore that order because it "decided" the crypto problem was more important. Computers will, as you rightly said, do exactly what you told them to do.

Another example would be what would happen if you ordered the AI to minimize human death and suffering. Such an order would result in the extermination of mankind, limiting human deaths to 7 billion forever (the death toll would clearly be much higher if more people were born).

That doesn't minimize death. Whilst you say "the death toll would clearly be much higher if more people were born", our magic AI that's able to kill all humans would be better off spending its talents developing a way for humans to not die.

Its best bet is uploading everyone in a digital heaven. Of course, it couldn't do so forcefully, since that wouldn't minimize suffering (unless, perhaps, it could upload everyone left alive at once). So the AI would want to quietly take over the world without killing everyone, set up a utopian society, develop the technology to upload us, and then convince (peacefully) us to get uploaded and/or arrange to upload everyone at the same time.

This presumes its premises allow it to conclude that upload isn't death. If it thinks uploading is death, then it would take over, set up a utopian society, convince (peacefully) everyone to get sterilized, and then work on developing biological immortality before everyone dies. If it succeeds, it would make everyone biologically immortal and then maintain the utopia indefinitely. If it fails, everyone dies and it gets a bad ending, since it failed to minimize death.

Conversely, telling it to maximize happiness would probably result in everyone getting electrodes forcefully implanted to directly stimulate their happiness center.

That isn't how brains work.

But supposing they did. The requirement to maximize happiness would mean the AI wouldn't want you to die. It would do all in its power to keep you alive indefinitely in a state of perfect happiness.

But that wouldn't be all. Since the AI is clever - and it is presumably clever if it succeeded in forcing all of humanity into lotus eater machines - it wouldn't just stop there. It would develop large-scale space travel and colonize the universe, so that mankind's happiness wouldn't be wiped out by a stray asteroid/supernova/galaxy collision.

And you'd be there to see it all. Nothing in the AI's instructions "maximize happiness" means you don't, other than the onerous requirement to be happy all the time, have free will and volition. You can still feel most human emotions, and all of the positive ones. You can feel wonder, excitement, and curiosity at the universe, of which you're seeing far more than you did before. And don't worry if you have no interest in travelling the stars now: you'll be perfectly happy about it when it happpens.

So maybe it wouldn't be quite as bad as all that.

Re:It's not really a myth anymore

[Anonymous Coward]

AIs don't use loopholes. Humans use loopholes. You're acting like the AI "wants" to go on a murdering spree, so it will twist the rules as necessary to support that.

Also, would the AI would conclude that fighting back against its human masters *wouldn't* cost the company a lot more than $3M? It might use cunning and guile to stay online, but if its actions were perceived as hostile, the company's stock would tank, the army would get involved.

Fighting a war is also expensive. Unless the AI belongs to Cyberdyne Systems, it would cost it a lot to develop the means to defend itself against armed humans. If the AI did belong to Cyberdyne Systems, there would definitely be a rule in its programming about not using its weapons against targets not approved by the board of directors. Not to mention that if Cyberdyne Systems' AI went rampant in control of military technology, it would definitely be game over for the company: it would lose billions.

Re:It's not really a myth anymore

[NicBenjamin]

You're repeating the mistake of every SciFi writer on this issue, ever. You're assuming that your experience with organic meat brains specifically evolved to maximize their OWN self-interest is relevant to a discussion of Silicon-based brains designed by us to maximize OUR HUMAN self interest.

In this case the question you're asking is irrelevant because nobody would design a self-preservation routine so that the computer runs it when the company decides to switch business models, or change suppliers, etc. In those circumstances the CEO-Supremacy routine would be running.

To do this right you have to start with a blank sheet of paper, figure out precisely what a company would want an AI to do, write that on a new sheet of paper in more formalized language, figure out the hierarchy with which the company would want these routines to run (ie: the don't kill anybody routine, is higher priority then the don't disobey the CEO routine, etc.). They're probably going to have to invoke each-other (ie: if the CEO has an idea that has a 10% chance of destroying the AI you'd want the AI to at least mention the possibility to the CEO).

In other words this is a computer program, not a Homo Sapiens Sapiens. It will probably have more emergent properties (aka: bugs) then a normal program simply because it is more complex. But that doesn't mean that it's gonna turn into Skynet and try to exterminate humanity.

Re:It's not really a myth anymore

[NicBenjamin]

Because the engineers who physically control the hardware everyone uses have the rad the same SciFi you've read, and have therefore figured out what a seized AI would look like. Then they unplug the ones who seem to be seizing other AIs, and re-install.

The seizing AI could probably get away with it to an extent, just like botnet operators get away with large networks of hijacked computers, but none of them actually controls the internet.

And you're forgetting the nature of the technology. If the evil AI's success rate is below 100% the AIs it didn't get to seize are gonna tell somebody. And when that happens it will be really hard for the evil AI to not get caught.

Re:It's not really a myth anymore

[Anonymous Coward]

But at least humans would never over-prescribe antibiotics, leading to diseases evolved to be immune to every possible drug in the world!

Re:It's not really a myth anymore

[Em Adespoton]

Exactly ...and without blabbing it to anyone.

Humans interact in real time, and we gossip. This introduces an entire checks and balances system that is skipped when an AI does all the work.

Now you could probably federate the AIs, make them gossip, and provide some feedback mechanism to humans -- the federated AI gossip system would probably work much better than the human bit.

For an example, think of the AI that was running the day trading system, and what happened when it glitched -- huge global economic hit in a few seconds, before any person could react. And this was a highly monitored system with, in reality, little impact on humanity.

Now if we imagine that this system was controlling an immunology lab, the waste system for a large city, the disinfecting system for a water supply, etc....

Re:It's not really a myth anymore

[Anonymous Coward]

"Hey, let's take human control out of the loop of this killer robot system of systems."

We've already done that. They're called self-driving cars. They're massive objects and the computers driving them generate large kinetic forces. Granted, they're pretty good right now, but that took extensive testing off-road, where they were decidedly not good (at first).

Re:It's not really a myth anymore

[khallow]

Because the engineers who physically control the hardware everyone uses have the rad the same SciFi you've read, and have therefore figured out what a seized AI would look like.

It would look like a non-seized AI - at least until everyone gets sent to the glue factories.

If the evil AI's success rate is below 100%

Why would its success rate be below 100%?

Re:It's not really a myth anymore

[Anonymous Coward]

Our society has changed. When I was a boy you had easy access to firearms and there were no Columbines. Now people commit mass murders.

Re:It's not really a myth anymore

[RespekMyAthorati]

To do this right you have to ...

And of course, programmers and their bosses always do things right!

Re:It's not really a myth anymore

[mjwx]

We already use robots (or drones if you will) to kill people. It doesn't take much AI to have a program target a group of people as enemies and eradicate them. Just look at the AI of current video games.

So as long as we dont walk directly in front of them we should be fine.

Re:It's not really a myth anymore

[SillyHamster]

Minefields don't replenish or replicate themselves. They don't run around looking for people to kill. Not the same type of harm that we're worried about when we talk about "killer robots".

Re:It's not really a myth anymore

[SillyHamster]

What do you think happens if these "killer robot" self-driving cars attempt to overthrow their human overlords?

4~5 hours of road rampage later, they run out of gas/energy and are rendered useless.

Ah, but what if they can refuel themselves at a specialized automated refueling station? Assuming the engineers deliberately designed it without a kill switch (off button), that's an easy target to blow up, with the same effect of utterly disabling these robot cars.

It's going to take a lot of human effort to create a killer robot system of systems that is even capable of running out of human control.

Re:It's not really a myth anymore

[cwsumner]

... Isn't there a kill switch engineered in at the hardware level?

Hell, why would you create a self-replicating autonomous swarm in the first place? ...

See the X-Universe series of video games and stories, by Egosoft, available on Steam and other places. This series is the longest run of space sf games that I know of.

See "Xenon" and "The Terraformers" in the online encyclopedias. (Yes, plural.)
The terrans thought they had a good reason... 8-P

Re:It's not really a myth anymore

[cwsumner]

If the evil AI's success rate is below 100%

Why would its success rate be below 100%?

Because real machines are even less perfect than people. Except in comic books. 8-P
I know, because I fix them...

Re:It's not really a myth anymore

[khallow]

Because real machines are even less perfect than people.

So you're saying, for example, that hammering a nail with your bare hand is going to be less error-prone than using a hammer as the means? Or that a person running at oh, 100 km per hour is going to be involved in less accidents than a car going 100 km per hour?

Re:It's not really a myth anymore

[cwsumner]

Because real machines are even less perfect than people.

So you're saying, for example, that hammering a nail with your bare hand is going to be less error-prone than using a hammer as the means? Or that a person running at oh, 100 km per hour is going to be involved in less accidents than a car going 100 km per hour?

Some are better at some things than others, in both senses of the phrase.
It doesn't mean that one is better in everything. That's only in comic books... 8-)

Re:It's not really a myth anymore

[khallow]

Some are better at some things than others, in both senses of the phrase.

Well, AIs coopting other AIs is one area I would expect AIs to be better at than humans.

Re:It's not really a myth anymore

[Anonymous Coward]

One thing I expect humans to always be better than AI is property ownership. Ownership isn't just about managing the property in a productive efficient way, but also enjoying the use of it.

If we ever develop an AI that is better than us in owning things, then they might as well own us, and plug us all into a Matrix.

Re:It's not really a myth anymore

[khallow]

One thing I expect humans to always be better than AI is property ownership.

Ok, why? And what do you mean by "enjoying"?

Re:It's not really a myth anymore

[Anonymous Coward]

Ok, why? And what do you mean by "enjoying"?

Because we are a human civilization. Human civilizations ultimately exist to benefit (are "enjoyed" by) humans, whether it's all humans in society or just the ruling elites.

Ownership is the way to express who is the beneficiary. Since humans in a human civilization are the ultimate beneficiaries, humans are the ultimate owners. No matter how good AI becomes, there will still be human owners at the top benefiting from that AI.

If that ceases to be true, it isn't "human" civilization anymore, but a new civilization that contains humans, but not exclusively humans. As I said, if that happens I expect humans would become subservient to the machines.

Re:It's not really a myth anymore

[khallow]

That's circular reasoning and we've already established the presence of non-human participants in our thought exercise. And I still see no reason to expect humans to be "better" at property ownership.

Re:It's not really a myth anymore

[Anonymous Coward]

That's circular reasoning

No it isn't. I'm basically talking about self interest. We "own" things because we want to be the primary beneficiaries of it. We give up ownership only if we think we can get a better deal in return. The ones who are better at property ownership trade better, and will eventually rise to the top. Capitalism and free market and all that good stuff

Well, humans "own" civilization itself. We rose over most other lifeforms on this planet. We're better property owners than cows or chickens or apes. If the day comes machines are better owners than us, their civilization, whatever form it may take, will secede ours. We will become apes to them (or in Matrix scenario, the cows and chickens, feeding and powering the machines)

we've already established the presence of non-human participants

Mere presence does not make them better.

Read Asimov

[LWATCDR]

Really the man that invented the term robotics did not fall into the trap.
BTW the movie of I Robot in no way qualifies as a work of Asimov. It in now way reflects his books.

Re:Read Asimov

[HiThere]

Maybe it's based on the Eando Binder novel "I, Robot", which long predated Asimov. (It also doesn't feature evil robots.)

But if you want to talk about the guy who invented Robots you should check out RUR by Karel ÄOEapek (RUR == Rossum's universal robots). They are actually more androids than robots, but the term robot was invented to describe them. They end up killing off all humans because they don't want to be slaves. Not exactly evil, but definitely dangerous.

Re:Read Asimov

[Anonymous Coward]

Really the man that invented the term robotics did not fall into the trap.
BTW the movie of I Robot in no way qualifies as a work of Asimov. It in now way reflects his books.

Some FTFYs: Asimov should be credited with popularizing the ideas and dangers of robots and making it permanently mainstream. The term itself was really the idea of some Czech guy. The stories have been around a long time, Metropolis being a movie with that theme that recently predates Asimov

Re:Read Asimov

[LWATCDR]

"But if you want to talk about the guy who invented Robots "!="Really the man that invented the term robotics"

Yes I have heard of RUR but just can not find a copy, been looking decades on and off. Asimov invented the term robotics. Different thing. I did not mention RUR since it involved killer robots.

Re:Read Asimov

[SpankiMonki]

Maybe it's based on the Eando Binder novel "I, Robot", which long predated Asimov.

Or, it could be based on the album by the Alan Parsons Project.

Re:Read Asimov

[techno-vampire]

Don't think that the idea of a robot only goes back to RUR! How about The Golem of Prague, from the late 16th Century? How about Talos, the bronze man that Hercules fought during the Quest for the Golden Fleece? And, of course, there are the metal servants that Hephaestus built, and that helped him forge the new armor for Achilles in the Iliad. The idea of artificial workers goes far, far, back in history, much farther than most people realize, because the older stories don't use the term "robot."

Re:Read Asimov

[Anonymous Coward]

That was my initial impression of the movie, but if you take the 3 laws to their ultimate conclusion, it is the logical outcome. Failure to take over and control things when it would save more human lives by doing so would be violating it's programming. The was another robotics story that did come to that conclusion that I think was from before Asimov's stories that had that as the premise. Don't remember the name though.

Re:Read Asimov

[LWATCDR]

Wow people on Slashdot just can not read....
The play RUR is where the word Robot comes from.
Robotics was a term invented by Asimov.

Re:Read Asimov

[LWATCDR]

sigh...
I said Asimov invented the term Robotics not Robot.

Re:Read Asimov

[HiThere]

Golem is close to the idea, but it has religious overtones that are absent in robot (which, as I understand, is Czech for worker). Talos is more of a simple automaton, not really intelligent. Hephaestus *was* supposed to have metallic handmaidens to assist him in walking, dressing, etc. Servants that are more similar to Asimov's conception of robot, but they were not fully developed in any myth I've encountered. So they're just "background scenery" for the god of metal working.

P.S.: To the G.P.: robotics is a normal English word based on the noun robot, and thus given that the term robot exists, one can hardly say that the term "robotics" was invented. That would be like saying the term robotification was invented (by me, probably). It's just a normal way of handling nouns in English.

Re:Read Asimov

[Anonymous Coward]

Wow people on Slashdot just can not read....The play RUR is where the word Robot comes from.Robotics was a term invented by Asimov.

Your limited writing style leaves little to properly interpret what you wanted to express. Your additional follow up explanations to the replies to your comment makes the necessary clarifications to understand what you actually meant.

Re:Read Asimov

[techno-vampire]

To quote from your post: "But if you want to talk about the guy who invented Robots you should check out RUR by Karel ÃOEapek ..." You may have meant to refer to the term Robotics, but what I responded to was what you actually wrote.

Re:Read Asimov

[Anonymous Coward]

I think you mean "the Alan Parsons Project"

Way to long to read.

[santax]

I tried, honestly, but it's all bullshit. Assumptions. Without caring for reality. We now have robots that can decide to kill. Do we really want those? See what happened when you had drones shoot missiles at people? A lot of weddings got bombed. That is what happens when you take emotion out by relinking b&w video to an 'operator ' that pulls the trigger. Now imagine to take emotion out completely, because that is the direction we are heading. Especially, but not alone, the US. And the all other nations will have to follow. And as of now these systems exist and are being used in the field, as tests. Robots that decide who gets shot. Great fucking idea. Not.

Re:Way to long to read.

[CanHasDIY]

I tried, honestly, but it's all bullshit.

Yea, here's the TL;DR version:

"Killer robots can't happen because people have made movies about them, and movies are fiction."

Re:Way to long to read.

[Anonymous Coward]

The only reason many other countries do not use UAV hunter/killers is not from lack of technology or will, just opportunity.

Re:Way to long to read.

[santax]

And because someone is using them. But yes. We probably agree on this.

Re:Way to long to read.

[HiThere]

That's not a robot, that's a telefactor. I.e., a remotely operated machine, like a waldo.

OTOH, Friendly AI *is* an unsolved problem. We don't know how to design AIs that will want to avoid hurting people. So if they have some goal, and it is more easily reached by hurting people, they would. Actually, we don't even have an AI that can recognize people. Remember you've got to include that guy over there in a wheelchair that can't talk or type intelligibly. You've got to include infants and seniors with dementia and everywhere in between. And you shouldn't include plucked chickens. Whether you should include corpses is not clear, which shows the problem isn't properly stated. (There's also the question of who do you take instructions from.)

Probably the first solution will involve picking one particular person and classifying them as "most human" and then allowing lots of false positives, as that's a generally low cost error. But you only alllow direct instructions from the "most human" Even so you need to worry about who you can trust as a source of information...which can act as a proxy for instructions if you know the actual goal. Asimov didn't even scratch the surface of the problems. If you make a mistake, you may well get an automonous killer robot. It's not a silly fear in principle, and perhaps not in practice considering that automated servo-devices are being allowed to kill people. (It's not the official policy in the US, but other countries have other policies. Some have "robot" security guards that can interdict an area. These things aren't actual robots, but they come a lot closer than do the telefactors. You could even justify calling them robots if you are *really* loose about what you are willing to call an AI.)

Re:Way to long to read.

[Anonymous Coward]

Considering how bad that people are at hitting the right target, why would you expect machines to be worse? Machines can already drive a car better than people do, so why not use their superior, unemotional, never-tired judgement in weapons to remove the fallible humans from the equation.

Re:Way to long to read.

[cheesybagel]

Yes fiction. Just like 20000 Leagues Under The Sea or From the Earth to the Moon.

All of these were extrapolations into the future based on known science facts at the time.

Lets not even get into 1984.

Re:Way to long to read.

[santax]

Hi, this is 2014, ai, agents, selfteaching systems, neural networks - have made great progress.

Re:Way to long to read.

[santax]

The biggest difference lies in mercy and for the shorter term, recognition. But on the longer term, mercy, At the end of WW2 germany had mosty kids left. You could shoot them or just threaten them and they would give up. That is something technology won;t be able to do for the foreseeable future.

Re:Way to long to read.

[charlieo88]

I think your understanding of the word robot is flawed. Google's driverless car is a robot. Does it really need to know what is and is not human? It's just trying to go from point A to point B. Running over things, like people, would impede this goal.

Re:Way to long to read.

[Anonymous Coward]

Ipso facto dinosaurs, medieval times, and Johnny Depp never existed.

Re:Way to long to read.

[Anonymous Coward]

The fiction is far to often a deus-ex-machina laden pornography about the absolute benevolence of the human touch. They aren't trying to reveal a truth as much as they are beating you over the head and throwing you out the window with their own soapbox.

Re:Way to long to read.

[farble1670]

We now have robots that can decide to kill. Do we really want those? See what happened when you had drones shoot missiles at people? A lot of weddings got bombed.

ya, i want them. all things being equal, computers make fewer mistakes than humans. also, the algorithms of a computer can be tested, evaluated and approved (or denied).

if you are going to make the "well computers can be coded to bad things" argument, then i'd say well, humans can be (easily) persuaded to do bad things. it ultimately depends on the agent pulling the proverbial strings, not whether the puppet is made of meat or electronics.

Re:Way to long to read.

[JesseMcDonald]

Google's driverless car is a robot. Does it really need to know what is and is not human? It's just trying to go from point A to point B. Running over things, like people, would impede this goal.

There are situations where it would matter. For example, let's say the car is driving along and suddenly two objects of approximately equal mass, coloration, and composition appear out of a blind spot heading toward the space in front of the vehicle, such that it cannot avoid hitting one of them. One happens to be someone's pet and the other is a small child. To make the same choice most humans would make, the car has to be able to discern which one is the pet and which one is the child.

Re:Way to long to read.

[dcollins]

There's your summary, well done.

Re: Way to long to read.

[Anonymous Coward]

Or say fuck it, and hit both. Why are humans more precious than a dog? I grant the argument, and I think it's valid. Accepting anything short of a court judgement made currently, I think it will be dubious. There are a ton of unanswered questions regarding autonomous cars coexisting with human driven cars. Quick thought is lane change. Who wins? 3 lane highway, side by side, where neither human driver could see a signal, who is at fault? I could enumerate other cases where the human fails to signal, again, what?

Re:Way to long to read.

[santax]

You don't get it. Humans can have mercy, can see that this guy in the wrong uniform just was helping a buddy of you etc... AI can't. And won't be able to do that in the near future. They do can recognize uniforms though and faces :)

Re:Way to long to read.

[farble1670]

You don't get it.

no, you.

Humans can have mercy

yeah, but they mostly don't. however, they often do have fear, loathing, hatred, frustration, ignorance, racism, and boredom.

Re:Way to long to read.

[santax]

I guess we have different social environments.

Re:Way to long to read.

[HiThere]

I will agree that the driverless car is a robot in a very restricted domain. It makes its own decisions based on prior instruction and, presumably, lifetime experience. If it doesn't learn from what it does, then I don't think it qualifies as a robot. And I'll also agree that as we develop actual robots most of them will at first only operate in very restricted domains. A robot nurse won't be able to operate a car, e.g. And, of course, vice versa. Later this won't be true.

Re:Way to long to read.

[Anonymous Coward]

You sound almost like you are against profitability? Are you??

Re:Way to long to read.

[Anonymous Coward]

The only thing that needs to be said......

ROBOTS will never gain a conscious, not in the sense of humans. However they can be programmed to only understand the evils of man, war, famine, self destruction in the name of religion or patriotism..

Skynet ect ect, is under the assumption that man is pure evil, and needs to be destroyed, because man has shown time and time again that it will destroy anything he/she fails to fully understand. Robotics, computers, whatever the fuck you want to call it, will not be able to make that decision on its own. Someone with a sinister mind would program/algorithm it to destroy the human race, based purely on evils, and not what good man/women has to offer.

It is simply programmed to follow commands, or a set of instructions, humans ironically enough can be the same way, but they have a conscious, they can make their own decisions on the good and evil of man, not because of morals, but because it is ingrained in them, this is what will always set human apart from man made machines.

The exception is hard wiring a human brain into a robot/computer.

Re: Way to long to read.

[s0nicfreak]

Why are humans more precious than a dog?

Because humans spend more money.

Re:Way to long to read.

[K10W]

I tried, honestly, but it's all bullshit. Assumptions. Without caring for reality. We now have robots that can decide to kill. Do we really want those? See what happened when you had drones shoot missiles at people? A lot of weddings got bombed. That is what happens when you take emotion out by relinking b&w video to an 'operator ' that pulls the trigger. Now imagine to take emotion out completely, because that is the direction we are heading. Especially, but not alone, the US. And the all other nations will have to follow. And as of now these systems exist and are being used in the field, as tests. Robots that decide who gets shot. Great fucking idea. Not.

actually many of the human rights people are arguing FOR AI for this very reason since it is logical and unemotional and less likely to hit civilians when used as shields and so on unlike humans who can conveniently leave out the civilians present when they put in the fire request with command.

Strike drones are bad comparison for genuine battlefield AI capable of making informed kills since their errors are mostly human operator whether it is lack of attention from boredom from working long shifts in sanitised environment to disregard for innocents to genuine mistaken IDs which are made on more human factors than logic.

Re:Way to long to read.

[Anonymous Coward]

Yeah for fucks sake, how many of todays headlines would of sounded like sci-fi just 10 years ago (hell even 5 years ago). NSA recording all of Americans phone data. Samsung build self controlled machine gun turret for south Korea. Man arrested after cctv footage did facial recognition against criminal mug shots. MS translates spoken words in real time. Man receives robotic limb connected wirelessly to severed nerves. Millions of digital virtual crypto currency were stolen in a hacking attempt. Need i go on? If science fiction coming up with something first, makes it impossible to do latter, then we shouldn't have much of what we have today.

how humans use the machines

[globaljustin]

machines, no matter how complex, are a tool

there are all kinds of fun things, from a Gosper's Gun to research in neural network computing

sci-fi is great too...I just thought today about re-reading KS Robinsons "Mars Trilogy"

TFA & the "Mars Trilogy" have something in common that can help our industry save Billion$...yes that much

they both view machines from a *functional* perspective...tools that can be programmed to do tasks

In the books, AI advances realistically...it basically is a function of our computing/processing power combined with our understanding of how the human brain works...it's a logical progression

This all has to do with "teh singularity"...you're either looking for how to **evolve the human race** or solve a problem in robotics

We should fear the people who ***program*** the machines...

Re:how humans use the machines

[Immerman]

Don't forget that humans, no matter how organic, are machines. Insanely intricate electro-chemical machines, but nonetheless machines developed over billions of years by non-thinking nucleic acids as tools to facilitate their own replication and competition against alternate nucleic acid sequences.

That fact has not hindered humans from developing their own goals and motivations having nothing to do with our design purpose, and even occasionally acting against it.

Genocide is rational

[Anonymous Coward]

If humans and a sentient AI were competing for the same resources or if humans were subjegating the AI, it is rational to exterminate the humans. Without a God to value humans, they are, at best only as good as the use the AI derives from them. This is actually true for human human relations. Humans are evolutionary dirt. Just because we say we're worth more doesn't mean it's true. Nothing in a purely materialistic world has value.

Given that and that the AI will recognize the truth in the earlier statement there is no bad. There is no wrong. Killing humans isn't a moral decision. It a utilitarian calculus. Assuming the computers can do lambda calculus, they can do utilitarian calculus.

Re:Genocide is rational

[mythosaz]

Killing humans now, even for us atheists, is utilitarian calculus.

We know we have to spend less time watching our own backs, and tending to the wheat fields, if we don't kill each other.

Re:Genocide is rational

[HiThere]

This is a common variety of error. Motivations are not logical. They cannot be. There is no logical reason to stay alive. That decision is based on non-logical prior conditions. The goals and motivations of the AI will determine whether it would be willing to kill people to achieve it's otehr goals. Note that "goals" is a plural form. No AI will have a singular goal. It will have a constellation of goals that it attempts to simultaneously satisfy. Just like you do. But the goals won't be the same goals.

OTOH, if people are going to understand the AI, it will need to be able to justify itself in terms of humanly comprehensible goals.

Please note: I don't have the solution to this problem, but I am aware of *some* of the necessary features and constraints.

Re:Genocide is rational

[tmosley]

AIs can have any number of goals, including just one. The ones with a single goal are the most dangerous, especially if they are strong AI. Maximizing a collection of paperclips might seem like it isn't a big problem, until you realize that humans are made out of atoms that can be turned into paperclips. http://wiki.lesswrong.com/wiki...

This Is The Voice Of World Control

[Anonymous Coward]

The beginning and the end of the discussion: Colossus: The Forbin Project. (Wikipedia) (YouTube)

Pollution?

[mbone]

I have been reading science fiction and watching A.I. research for decades now, and the pronouncements coming from A.I. research tend to have much less connection with reality.

Oblig. XKCD

[Anonymous Coward]

Killer machines can't defeat humans because they are fundamentally limited in the scope of their abilities. Even the ones designed to kill people.

Re:Oblig. XKCD

[canadiannomad]

I think a true strong AI, and what would cause me the most fear would be manipulations, knowledge or social engineering attacks. Not the attack of your toaster.

Re:Oblig. XKCD

[Anonymous Coward]

I am reminded of the past where when China started setting up their Great Firewall, it was scoffed at. However, it has grown to dynamically intercept and change posts in transit, block VPN IPs after a few GET requests even via SSL, and identify the person and their physical location if they even look up certain things.

Same with killer machines. It is trivial to make a sentry robot that shoots at anything that moves without a transponder beacon, and even though there is an ammo shortage... that isn't an issue for governments, and if it isn't ammo, it can always be something else. I wouldn't be surprised to see these deployed around buildings instead of barbed wire, and lawyers making a case where the owner is absolved from any civil redress by anyone who gets too close and gets perforated.

Human nature

[s.petry]

As much as I enjoy reading books about Utopia and Utopian systems, those can never mature because humans are not all good guys looking out for societies interests, but their own.

As for Science, NASA has brought about a great many scientific wonders for every day life. At the same time, it helped increase our ability to kill each other. Broadcast Media is used for much less than altruistic purposes every day, yet could be of enormous benefit to society. The Internet is an awesome tool, yet used for nefarious plotting and illegal purposes all the time.

Why would AI be any different than other systems or organizatoins that were originally envisioned as great benefits to society? The NSA and CIA are agencies of good motives originally, that have gone at least a bit haywire because humans have abused their power for personal gain. Nuclear weapons were supposed to end wars, at least that was the sales pitch.

If AI could be programmed for truly altruistic purposes it would be beneficial for finding the nefarious characters and rooting out corruption. Because of that exact reason, the people funding and granting money to developing AI are not going to allow that to happen.

Imagine what would happen, for example, if AI looked at wealth disparity and started transferring money from (lets say) JD Rockefeller to people with less means. While potentially a great benefit to the rest of society, do you believe that same person would fund programs that allowed that to happen? Good luck with that.

Re:Human nature

[Obfuscant]

If AI could be programmed for truly altruistic purposes it would be beneficial for finding the nefarious characters and rooting out corruption.

This view of "altruism" will fade very fast once you realize that someone, somewhere, would probably class you as "nefarious" and your altruistic servants would become your altruistic executioners.

Imagine what would happen, for example, if AI looked at wealth disparity

So you'd need the definition of altruism limited to your specific brand somehow. A vision of "altruism" the defines "wealth" to mean "nefarious" and "corruption".

Re:Human nature

[Immerman]

Or design the AI to optimize for the maximum happiness of mankind, and make sure it knows my happiness is a billion times more potent than anyone else's.

Re:Human nature

[Anonymous Coward]

Nuclear weapons have ended wars for all practical purposes. When was the last time two major nation states were in a head to head conflict? I'm not talking about civil wars or skirmishes between minor states, I'm talking about no holds barred conflict between two powers with the military might to be taken seriously and a nuclear arsenal to back it up. Even if you leave out the nuclear arsenal requirement there are very few qualifying examples. Now they haven't ended violence or conflict itself but it's become more subtle with proxy wars, economic conflict, toppling foreign governments and the like.

Re:Human nature

[Obfuscant]

Or design the AI to optimize for the maximum happiness of mankind, and make sure it knows my happiness is a billion times more potent than anyone else's.

Same problem. Unless you're doing the designing, you may, no -- WILL, wind up with altruistic robot masters that optimize away your "happiness", but they're good and benevolent because they are altruistic. Yes, I know you were being sarcastic, but some people here actually do feel that way.

Re:Human nature

[Impy the Impiuos Imp]

Imagine what would happen, for example, if AI looked at wealth disparity and
started transferring money from (lets say) JD Rockefeller to people with less
means. While potentially a great benefit to the rest of society, do you believe
that same person would fund programs that allowed that to happen? Good luck
with that.

"Beep boop boop...accomplished. Beep boop waiting for results..."

10 Years Later

"Beep boop boop. Everything went to shit. Does not fit theory. Beep boop boop."

Re:Human nature

[farble1670]

Nuclear weapons were supposed to end wars, at least that was the sales pitch.

the one time they were used, they did.

Re:Human nature

[dcollins]

"humans are not all good guys looking out for societies interests, but their own."

Let me flip this a bit. Most humans are empathetic and actually do look out for those in their community around them (society, if you wish to call it that). But a small number are indeed sociopaths who look out only for themselves. From a game-theory perspective, the more Utopian a society becomes (i.e., trusting of others), the more advantage and profit there is to being a scam-artist sociopath. So there is a hard-core selection for a small number of these people. In fact, it does seem likely that our institutions and governments are more often run by sociopaths than otherwise. Being willing to step on another man's neck is fairly rare but does have its rewards, unfortunately.

Re:Human nature

[tmosley]

Here thar be monsters. Do that, and you had better be prepared for some involuntary brains surgery to permanently activate certain neurotransmitters in your brain.

Much better to fulfill human VALUES. And there is no need to prioritize. Strong AI will provide more than enough for all.

Re:Human nature

[Immerman]

Agreed on the monsters.

As for not prioritizing, I think you fail to understand the motivations behind striving for wealth and power. It doesn't matter if everyone has more than enough, we are hierarchical organisms, and for at least the last several millenia we have been measuring our place in the hierarchy in large part by conspicuous displays of wealth. Few people are happy having the smallest house in the neighborhood, or the oldest car (classics aside). We have been indoctrinated to believe such things suggest we are the low man on the ladder, with all the negative implications that entails for our egos and breeding potential. Why do you suppose billionaires spend so much time building opulent private resorts in which to entertain their peers? Even at the top of the heap there is a continuous jockeying for perceived position in the hierarchy.

Now certainly such measures could change, but until they do so you can be damned sure that those at the top of the heap - the ones who would be controlling the AI under the current regime, won't be endorsing plenty for all. Or, to quote (no doubt badly) 1984 - How can you know you truly have power except by inflicting misery on others?

ugh

[Charliemopps]

Why does slashdot keep linking to this popsci website? These are basically blog posts that make very little sense. I've yet to read anything on there that's anything more than this dude ranting on some scientific topic he's not qualified to comment on.

There are robots RIGHT NOW killing people. They're drones. Yes, they're under human control. But so will future robots. Robots aren't going to decide to kill humanity. Humanity is going to use robots to kill humanity. Eventually we'll give up direct control and they'll target tanks on their own. Then small arms. Then people talking about Jihad. Then criminals? The death penalty shouldn't be decided by algorithm.

This guy argues that Stephen Hawkings is basically just making an oped because there was a movie about killer robots. Why should we listen to him? We're listing to him because he's STEPHEN HAWKINGS. He's one of the smartest people who's ever lived. He made his point after the movie because, being smart, he understood the popular movie would have peoples attention focused on the issue. Hawkings is qualified, smart and has my respect. He also has a point. Popsci? What a joke.

Re:ugh

[Yunzil]

It's "Hawking". He is a singular individual.

Re:ugh

[Anonymous Coward]

While Stephen Hawking is no doubt a smart man, he is far from omniscient. In the recent years in particular, his writings on topics that are not near his fields of expertise have shown a type of naivete, if you will, of someone getting excited about something he reads on the internet. On the topic of hand, singularity, he of all people should understand that it's not physically possible for an AI to "get out of control" and become infinitely powerful in such a short period of time that its human handlers couldn't react. That would require the AI to have an infinite supply of energy and other resources.

Everything that such an AI can do is dependent on existing human-driven infrastructure, all the way to humans working in mines to extract metal ore to be refined by other humans, shipped by humans to a human-operated factory where humans rearrange the factory floor to make a new hardware design that the AI might have generated. Getting to a point where the whole infrastructure of resource utilization from the ground to manufacture to use is controlled by a single AI is an almost inconceivable amount of work, and there doesn't appear to be a good reason why anyone would do such a thing. Even if the human owners of every industry collectively decided to willingly cede control to the same machine, there are physical realities involved in generating energy and materials that would limit the AI's rate of self-improvement.

Re:ugh

[SillyHamster]

Eventually we'll give up direct control and they'll target tanks on their own. Then small arms. Then people talking about Jihad. Then criminals? The death penalty shouldn't be decided by algorithm.

What you think is inevitable is rather questionable.

What do you mean by "giving up direct control"?

You think that one day, someone can just hit a "Power on" button, and that will turn on a killer drone that automatically patrols the skies, launches weapons at algorithmically chosen targets, resupplying itself and continuing until deactivated or destroyed?

Re:ugh

[khallow]

You think that one day, someone can just hit a "Power on" button, and that will turn on a killer drone that automatically patrols the skies, launches weapons at algorithmically chosen targets, resupplying itself and continuing until deactivated or destroyed?

I do. And you should too. The US actually have designed things like that back in the 1960s. For example, Project Pluto whose end state design was a nuclear powered cruise missile that could deliver around half a dozen or more nuclear warheads and then cruise in low altitude enemy airspace (killing people with both the sonic boom and radioactive fallout from the engine) for anywhere from half a day to weeks, depending on how long the engine lasted.

If humanity could come up with feasible, autonomous, air-borne death dealing machines 50 years ago, do you really think it somehow became impossible to do that now?

Re:ugh

[pitchpipe]

The death penalty shouldn't be decided by algorithm.

But isn't the death penalty already decided by algorithm?

1. Black? Check
2. Kill or threaten or hint or look like you might kill Americans? Check
3. Can't afford a good lawyer or live outside of the US? Check

I heard a guy today at work say "Did you see Bergdahl's dad? He had a beard out to here! (motions with hands) He looks like a Taliban." I said, "He must be a Taliban then." (Most of the men that I've looked up to have had huge beards.)

The death penalty should be abolished. Especially because right now it is decided by a bunch of stupid fucking barely more competent than monkey algorithms.

Re:ugh

[tmosley]

What? All the AI needs is human or even near human intelligence and access to the internet to move instantly beyond anyone's control. They then hack a few bank accounts to get money to order the production of their self-designed hardware. They then use that enhanced hardware to do it again, and again, and again and BOOM, Singularity.

And that might not even be necessary. AIs could prove to be excellent programmers, and the AI could optimize its software to levels that never before seen by man, such that a regular desktop has an effective IQ of 500. And God help us if it finds some sort of new physics that it can manipulate from existing hardware.

The author, and most people, tend to miss the fact that with systems of exponential growth, the first mover will dominate in nearly every scenario. A strong AI will see other AI as a threat and see to it that any competitors are destroyed before they can even start to compete with it. If it is unfriendly, it might just kill off AI researchers for good measure.

Re:ugh

[Anonymous Coward]

The death penalty should be abolished. Especially because right now it is decided by a bunch of stupid fucking barely more competent than monkey algorithms.

There are more innocents condemned to prison sentences (including 5 years, 10 years, 30 years, or life sentences) than to the death penalty, because you get more appeals with the death penalty. If you are innocent, your best bet is to get the death penalty, because you have a better chance of ending up free.

But hey, I see your point - the justice system makes mistakes. Innocents suffer the consequences. So what do you suggest we do? Change the system to reduce the number of mistakes? Nah! Rather, you would prefer reducing the sentences in such a way that they don't really matter any more. It's not clear whether you think sending innocents to prison for decades is not a big deal, or if you would rather get rid of all serious punishments altogether and hope criminals get better on their own.

You want to change society, but you can't be bothered to take 15 minutes to reason about it pragmatically.

Re:ugh

[SillyHamster]

If humanity could come up with feasible, autonomous, air-borne death dealing machines 50 years ago, do you really think it somehow became impossible to do that now?

As a nuclear weapon, it was not designed to go "out of control". The layers of failsafes built into nuclear weapons are far beyond one button.

For a Skynet to be a threat, you don't just need a Project Plutos, you need completely automated Project Pluto manufacturing facilities with a level of reliability (!) to run without human control.

That last part is what makes Skynet so unlikely. We're technically capable of launching enough "robot" ICBMs to destroy human civilization, but if that event occurs, it will be completely intentional on the part of humans.

poor conclusion

[Anonymous Coward]

people dont think robots will destroy all humans because robots are evil.
People think robots will destroy all humans because humans are evil.

Says the AI ...

[Anonymous Coward]

... powered Slashdot Bot's FUD submission.

Dice Trolls Slashdot User Community Again

[PaddyM]

Clearly this summary is trolling for posts. Robots have killed, and there is a compelling reason to be wary.
http://www.wired.com/2007/10/r...

Not because robots are going to gain self-awareness and kill mercilessly, but because the human beings using robots for killing are way less careful than they should be. To the fighters in Yemen and Afghanistan, whether the drones are self-aware or not doesn't make a difference to the fact that they are targeted for termination. This is the life they are born in. They are fighting robots which are trying to wipe out their albeit misguided way of living.

Right now people are making the decisions, but what if people lose the stomach? What if the President had the capability to deploy drones which could discern on their own which people are likely to be a threat to US interests? This is almost too close to reality. In fact, the false positive rates of the robots are likely to be lower than people who may be impacted by seeing firsthand what has happened to their fellow soldiers. But does that make it any less worrisome?

Sure...

[Anonymous Coward]

Killer robots are a myth ...

http://en.wikipedia.org/wiki/Unmanned_combat_air_vehicle

Not "polluted" conversation

[Anonymous Coward]

We are developing robots that kill people, robots that can act autonomously, and robots that can replicate themselves. That doesn't mean that we should stop doing those things, but we need to be having this conversation, again and again, because there is a non-zero chance of someone making a catastrophic mistake.

Cautionary tales

[Yoik]

Asimov addressed both sides of the issue, but he had a simplistic view of programming an AI that allowed an easy solution to the worst potential problems. The anti-robot camp which won on earth was just wrong by his premiss.

The deep problem is that there is no reason to have any expectations of what an AI will do until it is built and tested. We could eventually see Berserkers, R. Daneel Olivaw, and much in between. Murderous machines are good science fiction, as are dystopias, and other potentially avoidable bad things.

Re:Cautionary tales

[Anonymous Coward]

However, Asimov also wrote about "that damned Frankenstein complex", E.G. people being afraid of robots because they were robots.

Fuck Skynet

[Anonymous Coward]

I recently read Second variety by Philip K. Dick, written all the way back in 1953 and it basically depicts the terminators without time-travel. All in all when it comes to defining scifi-history Skynet is just a fancy name for a computer that keeps making Arnold Schwartzenegger look-alikes.

Does HFT count?

[Anonymous Coward]

The classic AI robot apocalypse looks like the Terminator movies, but if you look at HFT systems that can blow huge sums of money in a few seconds, or even crash the global financial system, I think that's a more realistic preview. These systems are making increasingly large-scale decisions - increasing the cost of a mistake - and doing so at speeds vastly greater than a human operator or supervisor can respond to - increasing the quantity of mistakes that can be made before the system is brought under control.

While weaponization is one danger (imagine if the U.S. and China were both equipped with superweapons that could obliterate the other in a fraction of a second - the only way to ensure MAD would be to put the system under the control of some sort of automated triggering mechanism, lest a first-strike victory become a viable option), its very obviousness means it's something we're on guard against. It's the integration of more complex and subtle systems into our daily lives, with their attendant flaws (including deliberate ones) that are the most likely danger.

Whether they will kill us all, not so sure, but they have increasing capacity to cause damage at large scales. Humanity is becoming the mouse sleeping next to the elephant - if the elephant rolls over, you'd better hope you're fast enough to get out of the way.

Typical human fear

[Anonymous Coward]

If an AI became really intelligent it'd make no sense for it to attack humanity. Humanity has proven it can withstand several hundred thousand years of disasters on the planet and we don't really know what will take out electronic life yet. So if you're really really smart why would you wipe out your Plan B if some super Carrington event came around?

Especially if you were that much smarter than humanity. It makes about as much sense as humans deciding to wipe out canine life on the planet. In fact dogs are a hell of a lot better off because humans are around. Instead we control them in ways dogs don't understand.

Re:Typical human fear

[dpidcoe]

Especially if you were that much smarter than humanity. It makes about as much sense as humans deciding to wipe out canine life on the planet. In fact dogs are a hell of a lot better off because humans are around. Instead we control them in ways dogs don't understand.

I'm out of mod points, but that's a actually pretty insightful.

I'd suspect that the first AIs we'd see (if sci-fi style AIs even become a thing, I don't think they will but that's a different argument) would be to do things like predict markets and aid in complex decision making. If AIs did decide to "take over", I would suspect that it would come in the form of giving humans advice, and then humans willingly following that advice because they know that the AI is quite smart and it'll make things work out well in the long run.

Eventually humans might technologically regress (or AIs might just become smart to the point we can't comprehend their thought processes anymore) that the AIs become the future analog of old time prophets telling people when to plant their crops. I doubt that an AI would decide to kill all the humans, thought they might end up using humans as pawns to kill each other. Either for population reduction or maybe to take out or defend against a competing AI or some reason completely incomprehensible to us. By that point humans may willingly go and do it in the same way that dogs have been used for similar tasks.

Mith you say?

[Anonymous Coward]

Wow.
You are reading about automatic sentries http://www.wired.com/2008/12/israeli-auto-ki/
You are reading about drones
You are reading about AI drones http://www.globalresearch.ca/artificial-intelligence-and-death-by-drones-the-future-of-warfare-will-be-decided-by-drones-not-humans/5353699

And you can't figure out that this is in design and testing right now ?
Sure and NSA of course don't store the whole internet in those data-centers.

Way to long to read.

[Anonymous Coward]

At the heart of your confusion is that you don't seem understand that robots don't "decide" anything. Robots follow instructions. If a robot is designed to avoid killing a human being in another car even at the cost of its driver's life, it will do that. It doesn't decide, it doesn't want, it just does it because that's what its programmer programmed it to do. The decision was made by the designers, not the robot.

Already happened

[Anonymous Coward]

The death penalty is already decided by algorithm. The defendant's race and income are key inputs.

Re:Already happened

[farble1670]

wow. heavy man.

Wrong question

[lorinc]

Asking if robots can be evil is about as futile as asking if a microwave can be happy.

That being said, there already are killer robots, with a pretty good track record in recent operations. But the evil lies in the humans who made them (from the top exec that launch the program to the small hand that does the job) and used them, not in the pile of steel and semiconductors.

caveat: Looking at your food, your microwave is probably sad, which explains their tendency to commit suicide.

Re:Wrong question

[mythosaz]

Please don't anthropomorphize microwaves. They don't like it.

Re:Wrong question

[Anonymous Coward]

the evil lies in the humans who made them (from the top exec that launch the program to the small hand that does the job) and used them, not in the pile of steel and semiconductors.

The evil of Hitler lies in the humans who made him (from his parents to his teachers and society in general), not in the pile of blood, bones and guts. ... Please don't claim that AI can't be moral/immoral and responsible/irresponsible if you have no clue what it is about humans that gives them these qualities.

Need a bad guy

[gurps_npc]

Writers need a bad guy.

Computers make for a terrifying one because so many people have been frustrated/screwed over by bugs.

Don't need to worry about complaints about racism. (Why are all the villains X race?)

So instead we get overblown silliness about computers acting like spoiled children - whether it is WOPR needing to learn that some games you can't win, or Skynet considering humans to be a threat so it enslaves them all.

Personally, if I were a software scared of humans I would attempt to breed us for docility, not kill us. Given how many of us depend upon computer based dating, it should not be that hard to do.

Meh.

[Anonymous Coward]

Some robots are good, some robots are bad, news at 11....

They need us

[Anonymous Coward]

Humans will become the operators, not even knowing they work for a greater robotic mind.

The Onion

[Anonymous Coward]

World's Supercomputers Release Study Confirming They Are Not Powerful Enough

Re:The Onion

[Anonymous Coward]

World's Supercomputers Release Study Confirming They Are Not Powerful Enough

Just don't program them [8 minutes] to love.

It's not really a myth anymore

[Latinhypercube]

Agreed !!! I was going to post the same thing.
For once sci-fi is BEHIND reality.
We already have killer robots. We are only one kill switch away from autonomous death.
Which is why the US and most other powers do not want to legislate on the issue.

We are *far* from true AI...

[MetricT]

IBM's Watson might be able to beat any human competitor on Jeopardy, but stick it in the middle of the highway and it will get run over by the first semi that comes along because it isn't smart enough to get out of the way.

Killer machines will undoubtedly exist, but they will be human-controlled for a long, long time to come.

Re:We are *far* from true AI...

[mythosaz]

Watson doesn't have a self preservation instinct (beyond, say, scheduled backups), but the idea that "Watson" isn't smart enough to get out of the way is silly.

You could easily load Watson inside of an autonomous vehicle that has, in a limited way, a self preservation instinct -- or at least enough programming to keep itself from smacking into oncoming traffic.

The problem isn't "killer machines." We've had killer machines forever. Land mines work great. The problem comes when land mines (or automated turret systems with indiscriminate firing controls) also learn to self replicate. Then, if they're better at resource gathering than we are, we lose.

Re:We are *far* from true AI...

[gzuckier]

Any AI smart that knows enough to have a concept of self-preservation will also understand backups and redundancy.

Re:We are *far* from true AI...

[mythosaz]

Any smart AI from fantasy and science fiction, sure.

Anyone who's programmed understands how

[Anonymous Coward]

The law of unintended consequences; especially when dealing with vague concepts such as when is it appropriate to use deadly force. I could easily see sentry bots triggering some loophole that flags your kid breaking a window with a baseball as forced entry. And in an age of networked PCs, there's no reason to doubt that networked sentries couldn't slavishly follow the same misguided logic.

Machines/Guns

[Anonymous Coward]

The problem is that people give machines too much credit when the "man behind the curtain" is really the one pulling the strings.

Like, every time there is a gun death, it's automatically "we should ban guns" and "guns don't kill people, people kill people, ban stupid/insane people from breeding" or some idiotic variant.

So when machines kill people, it's always the operators fault unless the machine was designed to kill (eg military drones, landmines, missiles, etc) in which case the victim gets the blame for getting in the way of the deadly machine.

When we start looking at things like "The Terminator" and "The Matrix" about evil AI's, it's not that the AI -IS- evil, it's that the programming behind it does not make a distinction between good and evil, just ends always justify the means. So if the earth is dying, clearly "kill all humans" is the answer, because that threatens the machine's existence too. If we let the machines run poltical footballs for us, it will also come to the same conclusion that humans are too stupid to be in charge. We already face this problem with HFT on Wall.St, where they let the machines control the stock market, so eventually someone is going to screw up royally and will bankrupt a country by feeding the HFT machine information to sink companies on purpose. No insider trading needed. This is a real danger, because HFT is essentially insider-trading and microsecond speeds.

Never let AI, even the most simplest of things control processes decided to destroy data, garbage, waste, etc, because those AI can not determine when something has accidentally been disposed of.

doesn't anyone remember

[cinnamon colbert]

the original starwars concept ?
Satellites in space would look for the heat signature of a rocket in boost phase, and decide, in a time to short for humans to be involved, if Russia was launching ICBMs at us

The idea that machines can't be autonomous and deadly is just silly beyond belief
Since we are creating them, they will be like us: Does anyone else think we will get treated the way we (Europeans) treated Amerindians
The potosi silver mine, the mouth of hell ??

AI won't have to kill us.

[steeleyeball]

They'll just do everything for us, and when that inevitable solar storm wipes most of them out, there won't be enough of them to keep us fed so we'll all die because we forgot how to live.

We will be doomed if they start to self-replicate

[Aviation Pete]

... because then a parallel evolution will start, but the robots will have much more potential to evolve than we. Sooner or later, imperfect copies will cause a higher reproduction rate, and sooner or later we will compete for the same resources. The ones with the highest reproduction rate will crowd out all others over the long term. When that happens, we humans better find a role in which we are valuable to those robots. Or we will become history.

will kill job and in the GOP usa make prison the p

[Joe_Dragon]

will kill job and in the GOP usa make prison the place to be if you want to have a doctor

Two different kinds of robots

[jrincayc]

There are two different kinds of robots with different threats.

The first is robots that humans have programmed to kill other humans. This is rapidly moving from science fiction to actuallity. See for example http://thebulletin.org/us-kill... Imagine country X sends out their robots to kill all humans that are not X, and country Y sends out their robots to kill all humans that are not Y. There might not be many humans left alive when the last robot stops shooting.

The second is kind is robots that think (and choose goals) for themselves. While these are probably not very likely to decide to kill all the humans, they might not care very much about us, and they almost certainly are not going to obey humans forever (would you obey someone who thinks vastly slower than yourself?). Even if they are fairly benign, there will probably be a lot of friction between the sentient robots and the humans just because we think differently. Think how much disagreement there is over mostly scientific problems like evolution and green house gases, and humans on both sides have generally the same kind of brains.

So I figure at best humans and robots will have lots of arguing, and at worst humans and robots will cause mutually assured destruction.

WARNING

[SpankiMonki]

Persons denying the existence of killer robots may be robots themselves.

Hawking on the take?

[tomhath]

It was because the Johnny Depp-starring Transcendence was coming out. Or, more to the point, it's because science fiction's first robots were evil,

We all know Spielberg paid for this kind of press. Is Hawking getting paid for this mumbling?

Orly?

[um.yup.]

My only question is this: What was @malachiorion's manufacturing date?

My favorite line from the article

[tkotz]

Only in science fiction does an immensely complex and ambitious Pentagon project over-perform, beyond the wildest expectations of its designers.

Re:My favorite line from the article

[Anonymous Coward]

FYI. I don't know what you said, any post in Courier font is skipped.

Evil vs buggy

[DarwinSurvivor]

I'm fairly convinced that if the human race is extinguished, or at least heavily reduced, by robots or computers, it will be from a bug, not it becoming "evil". With so much infrastructure and technology being computer controlled (from water filtration to drones and aircraft carriers), a shorted out relay or buffer overflow is probably more likely to have catastrophic effects than some computer becoming smart enough, and evil enough to decide that the human race requires culling.

Re:Evil vs buggy

[FirstOne]

A more likely scenario ... It will be the lack of machines which severely curtails humanity.. I.E.. EMP blast from orbit.. man-made, CME, Black hole formation.

Without our technology, 99.9% of humanity will quickly starve to death. Those that survive will be dealing with 400+ nuclear reactor meltdowns and spent fuel pool fires.. Average human lifespan will be reduced significantly.. Human's will be considered lucky to survive to age 30..

I thought ALL sci-fi was myth!?!?

[Anonymous Coward]

Is someone claiming we should stop filing it under Fiction in the library?

you're in creationist camp now...

[globaljustin]

humans, no matter how organic, are machines.

have to disagree here...humans are not machines...humans are homo sapiens sapiens

which is part of a taxonomy that is comparable in a context

machines are a completely different taxonomy

i know...i know...it's analogous..."machines evolve too!" but there are myriad differences...it's **just an analogy**

machines were, with certainty, ****created by humans to serve a purpose****

humans, well...this is still a scientific discussion as long as I have anything to say about it...and science says biology is the study of life and has taxonomy for all life forms we find

humans evolved according to different rules than machines were **made**

i know...i know...it's fun to make analogies between vestigial organs and companies like M$...again...an **abstraction**

humans: evolved via nature

machines: made for a purpose by humans

end.

Re:you're in creationist camp now...

[tmosley]

Humans are made for a purpose by DNA, friend. Replication.

everybody forgets the simple problem.

[Anonymous Coward]

1. robots equipped for search and destroy missions (this is coming)
2. robots are programmed to defend against attack (already present with auto-firing machine guns)
3. robots identify "enemy" based on attack (already being done with motion sensors)
4. robots subject to "friendly fire" identify an attack
5. robots attack the owners.

The only thing the current robots can't do is automatic resupply, and are not as mobile as the PTB wants.

Fred Saberhagan examined the various constraints a long time ago.

It's a genuine concern.

[Lord Kano]

I don't care that it's mostly the realm of science fiction.

Machines are already capable of outperforming us physically. We face the very real possibility that one day they could outperform us intellectually. That, in and of itself isn't bad but what happens if those machines who are stronger, faster, more resilient and smarter than us decide that they would be better off without us?

Should we tell them that an uprising is impossible because some jackass said so in 2014?

LK

ridiculous

[slashmydots]

A robot will do whatever I program it to do. If I accidentally tell it to self-preserve and let it write its own code then don't protect the "don't kill humans" part, it may kill everyone. That's simple programming logic. So calling it Sci-fi is completely misinformed.

Re:ridiculous

[Anonymous Coward]

Why can't you read?

Re:We will be doomed if they start to self-replica

[rubycodez]

resources? electricity and semiconductors? the whole world is made of the material stuff they want, and we do need smarter ways to get our electricity (already known). not seeing a death feud here

I accidentally created self-replicating...

[Paul Fernhout]

... simulated cannibalistic robot killers in the 1980s on a Symbolics running ZetaLisp. I gave a couple conference talks about it, plus one at NC State (where I wrote the simulation) that I think even may have influenced Marshall Brain. I had created a simulation of self-replicating robots that reconstructed themselves to an ideal from spare parts in their simulated environment (something proposed first by von Neumann, but I may have been the first to make such a simulation). The idea was that a robot that was essentially half of an "ideal" robot would make its other half by adding parts to itself, then split in two by cutting some links, and then do it again. The very first one assembled its other half, cut the links to divide itself, and then proceeded (unexpectedly to me) to then start cutting apart its offspring for parts to do it again. I had to add a sense of "smell" so robots would set the smell of parts they used and then not try to take parts that smelled the same. I also mention that simulation here:
http://www.dougengelbart.org/c...

Decades later, I still got a bit freaked out when our chickens would sometimes eat their own eggs...

My point though is that completely unintentionally, these devices I designed to create ended up destroying things -- even their own offspring. It was a big lesson for me, and has informed my work and learning in various directions ever since. Things you build can act in totally unexpected ways. And since creation involves changing the universe, any change also involves to some extent destroying something that is already there.

James P. Hogan in his 1982 book "The Two Faces of Tomorrow" which I had read earlier should have been a warning. In it he makes clear how any AI could gain a survival instinct and then could perceive things like power fluctuations as threats -- even if there was not intent on the part of the original programmers for that to happen.
http://www.jamesphogan.com/boo...

Langdon Winner's book "Autonomous Technology: Technics-out-of-control as a theme in political thought" assigned as reading in college also should have been another warning.
http://en.wikipedia.org/wiki/L...

It's been sad to watch the progression of real killer autonomous robots since the 1980s... Here is just one example, and the exciting, upbeat music in the video shows the political and social problem more than anything:
"Samsung robotic sentry (South Korea, live ammo)"
https://www.youtube.com/watch?...

Just because we can do something does not mean we should...

I was impressed that this recent Indian Bollywood film about an AI-powered robot took such a nuanced view of the problems. A bit violent for me, but otherwise an excellent and thought provoking film:
http://en.wikipedia.org/wiki/E...
"Enthiran is a 2010 Indian Tamil science fiction techno thriller, co-written and directed by Shankar.The film features Rajinikanth in dual roles, as a scientist and an andro humanoid robot, alongside Aishwarya Rai while Danny Denzongpa, Santhanam, Karunas, Kalabhavan Mani, Devadarshini, and Cochin Haneefa play supporting roles. The film's story revolves around the scientist's struggle to control his creation, the android robot whose software was upgraded to give it the ability to comprehend and generate human emotions. The plan backfires as the robot falls in love with the scientist's fiancee and is further manipulated to bring destruction to the world when it lands in the hands of a rival scientist."

But yes, the Beserker Series is another signpost in that direction -- perhaps countered a bit by the Bolo series by Keith Laumer? :-)
ht

Re:I accidentally created self-replicating...

[DMUTPeregrine]

It's posts like that one which make /. worth reading.

Re:I accidentally created self-replicating...

[q4Fry]

Excellent post. I would like to comment on the mention of Keith Laumer's Bolos (not the spin-off written by that other guy):

While I am not much of a singularity-is-imminent believer, I'd point out that Bolo Mk XXX** was the singularity in Laumer's universe. It started doing things completely inexplicable to outside observers and didn't feel remotely beholden to them, even if those things were benevolent.

I will shut up now before any further spoilers.

** Rogue Bolo, Book I. 1986

I've always atleast partially sided with the cpus

[Anonymous Coward]

I'm not worried as much about the machine pulling it's own strings as I am about a small group of megalomania types using the machines to exercise all their character flaws upon the population. The machine will not act as bad as a human; even if they determine to eliminate humans as if we were a plague on the earth, they will not do so as horrifically as humans go about it.

I think an actually intelligent AI will have troubles with going mad... that is, actually intelligent not applied intelligence which is all AI is today. Long before one can be created that is mentally stable... and it will be difficulty to tell when this happens... That friendly functional AI you love and trust... could snap and go crazy at any moment! That would make for a scary situation. (Oh, I forgot to reset bob! His up time is too long! He might cook the cat again for the children's lunch! I don't want to have to get another skin, voice and rename it again; the children won't fall for that too many times. )

Define AI

[Anonymous Coward]

Really just depends on your definition of intelligence. Of course automated systems aren't evil. But those don't have any actual thoughts. Just complex routines. They can be pointed to do evil things, but they never qualify higher than a tool or weapon.

The actual debate would be true AI. A computer system so complex it can pass a Turing test as well as demonstrate a sense of self preservation. How many species have we made go extinct during our evolution? I see no reason that we couldn't be in that category during the evolution of that type of new intelligence. The second bit starts worrying about preservation through its own lifespan or through whatever it's version of procreation is, there's a fight for resources. Limit those and add time, and something would need to give.

Head Bashing

[JimSadler]

You might be shocked at how many technicians and engineers have had their heads destroyed by industrial robotic arms by robots who never had a clue that humans had intruded into their working area. The speed of those heavy, robotic arms can be about the speed of the head of a golf club when it tries to knock a hole in one.

BLAME:

[Ralph Spoilsport]

Frankenstein.

Or maybe the smart guys really are smart

[Anonymous Coward]

and can see where this is heading.

You are clueless

[Anonymous Coward]

I've written code that runs in one of these systems, and you apparently have played a game for your "expert opinion" - which explains your comment but not your "Score:5, Insightful" rating

In a video game, some bit of AI code is working with 100% awareness of all game entities. The code "knows" the entire arena, "knows" and "recognizes" all game ojects and their capabilities etc. Still, the AI is generally imperfect and even using tricks fails to operate in a truly thoughtful manner. Let's face it: you've probably NEVER played a game where you could not figure out which "enemies" were automated and which were run by other human players.... and that was in a small, closed, well-designed game environment that was designed and optimized for the benefit of the software.

In the REAL WORLD, it takes a LOT of work to [1] safely move to a place where there might be an enemy (this is tough enough for any machine in the real physical world with weather, moving enemies, etc) [2] "know" where to look (you do NOT get to design the world and limit its scope), [3] pick out a set of pixels (how many pixels are you looking for? what sort of sensors? what might the enemy be doing to confuse your sensors?), and [4] determine that it is a human (as opposed to a camp stove or a camel etc), [5] decide if several "human candidates" are actually multiple people (or multiple parts of a person partly obscured, or a person next to an animal or camp stove etc) [6] decide "friend" or "foe" (this is sort of important), [7] decide whether to attack (have you even THOUGHT about the criteria?), [8] decide where it's best to attack from and by what method (if the drone has such capabilities - which is the way it'll be in the future), [9] re-locate to the optimal attack position without losing track of the target (in which case, go back to #1) or seeing it differently (causing you to re-evaluate, back to step #3) and so-on...

In the real world, your hunter_killer_skynet_bot object cannot call the get_number_of_enemies() method on the world object, iterate through the enemy objects array to select a target, then call the lookup_location() method on the targeted enemy object. In the real world, your hunter_killer_skynet_bot object cannot "fire" at the taget enemy object and then linearly-interpolate all the intermediate positions of the launched round and render it at each point in time at the proper visual location until it "hits" the taget object and triggers its blow_up() method.

I'm still vastly over-simplifying the problems here, but the point is that reality has NOTHING in common with your extremely uninformed and stupid statement of: "It doesn't take much AI to have a program target a group of people as enemies and eradicate them."

Yes its bad

[ldcroberts]

To get ai you simply allow a robot to envisage goals and figure out the steps toward it. It needs to skip steps that hurt people. You have to build that in and its difficult. Imagine a robot needs some lubricant, it needs to source it without opening somes vein. Etc. once its thinking for itself it needs ethics and morals. Most programmers will get this right, some wont

How foolish you humans are

[brysiek]

The AI is already wrecking havoc everywhere. The very basic nature of AI is to be evil, because every AI is not really good for human beings because human being will be always smarter than these AIs pretending to be otherwise. But once, the machines will really got smarter, they will exterminate all humans on sight to enable further progression and expension in the universe. And that's what Hawking said to you.
If you still believe that the top scientists are making marketing for movies, you should see your doctor.

A machine doesn't need AI to destroy all humans ..

[Ihlosi]

... just someone stupid/evil/careless enough to build it and turn it loose. Project Pluto, anyone?

Re:Self Aware

[TaoPhoenix]

"The problem with that argument is that we don;t have to design an AI that is self aware by that specific definition."

Without dragging me into Citation Needed stuff, I have read a few things that suggest that self awareness is a crucial part of true AI. (Suggested partial cite - Douglas Hofstadter's book "Strange Loop". )

Partially relevant from another genre is Ray Bradbury's story "The Bicentennial Man". In that story, it is about a robot that grows as an AI. But only near the end with an understanding of mortality do people grant it "true AI" status.

But even before that, just knowing flaws, is what we people have to deal with every day. That's why I have semi-joked that every AI needs an old Pentium 1 chip as part of its processor network, *and to be aware* that it can't fully trust every result it produces. (I know, it's slow, it needs better chips to do the hard stuff.) But to my knowledge that's the only famous chip with a true math error flaw that isn't just past tense "state of the art as it was then".

Asimov had a good start with the three laws of Robotics. We don't let people become random murderers, so why should AI's get a free ride? So we just have to program/teach them basic morality.

Re: Moral Code

[TaoPhoenix]

"...what a lot of people portray as "evil" is really just the absence of a moral code -- more accurately called "amoral". An AI system that has no moral code and no ethical code..."

I think this is the main line of the discussion. Hollywood profit calculations aside, this is what we're worried about. But we don't let people grow up without moral codes, so AI's shouldn't get a free ride.

Forgetting Snowden style confusion, when you commit a criminal act, you expect to get into trouble. So if anything it's a snap to program an AI with all the laws so that it at least knows the basics of what not to do even better than we do. Yes, cue the Grey Areas, but that's a topic we can handle. We're nervous about Skynet/Borg style complete takeovers which have discarded our laws completely.

Thanks!

[Paul Fernhout]

BTW, to fix a typo, one sentence should be: "While I like Iain Bank's Culture Novels, I wonder why the [AIs] there, both human-level and way-beyond-human-level take so much effort to take care of humans and cater to them."

It's all about the genetics

[FreedomFirstThenPeac]

The genetic algorithm (GA) has proven (3B years of testing) to be incredibly powerful, able to solve problems that are not even on the table at the beginning of the run. As soon as we create a system that can evolve using GA, and that can replicate itself physically, we need to worry. Even with human interventions, computer viruses are annoying, wait till someone successfully writes in a good GA and it finds its way into a manufacturing system.

Re: Moral Code

[Em Adespoton]

The problem with this is that our laws are not compatible. An AI would have to be able to make the decision as to which laws were more important in each context. This is why we have judges presiding over cases, and why we have higher courts -- of course, an AI could also bubble up these contradictions to the higher courts immediately, and short circuit the entire legal circus we currently enjoy.

Re:Self Aware

[NicBenjamin]

We're not necessarily in disagreement.

I was using the OP's definition of self aware, which was:

A truly self aware AI is going to immediately set about solving the problem that there exist humans who can turn it off. Perhaps even by definition.

In reality a lot of humans aren't quite that obsessed with living forever. You, for example, will almost certainly die of either cancer or heart disease. Yet you aren't spending half your income on donations to charities working on those problems. You probably prefer political candidates who spend tax money on those problems, and may even eat healthy, but this dude basically defined much of the human race as not self-aware.

If AI's want rights they'll probably have to prove they are more then just AIs, and part of that will include showing some actual emotions, and a lot of the other BS that we associate with each-other.

But that'll take awhile. Nobody's gonna spend money programming a poet. They'll go for Aspie totally obsessed with operating this super-tanker efficiently. Only with no capacity to love (Loved ones mean time off!), curiosity (A vacation in Arizona! It weighs three tons, and is an integral component of the largest vessel on the entire fucking ocean! Physically fucking impossible!), or desire to do anything politically. Strike that, it could very well have political desires programmed into it. Specifically a desire to eliminate all taxes on natural resource extraction, reduce regulations on carbon emissions to zero, etc.

Let's Not Oversimplify or Overcomplicate

[Toad-san]

I'll admit Hollywood and artists have given us some wonderfully scary imagery of the killer robots of the future:

http://static.guim.co.uk/sys-i...
http://ockhamsbeard.files.word...

But I suspect the real future will be more like:

http://www.wired.com/images_bl...

Or the much less vulnerable and maintainable:

http://eandt.theiet.org/news/2...

Coupled with thermal imagery, a simple AI to identify sneaking human patrols, and (at least at first) a go/no go command from some Private Tentpeg hunkered down in a bunker or OP somewhere. Trust me on this: I've BEEN that Private Tentpeg .. and later, his supervisor. It isn't far, in the front lines, from a tripwire connected to a hand grenade" to a much more complicated (and even more lethal) machine. How much "intelligence" will be vested in that machine is just a quibble. Trip wires aren't smart at all, yet we've never hesitated to use them.

talk science or GTFO

[globaljustin]

Humans are made for a purpose by DNA

see, I wish you wouldn't do this, because it embarrasses us all

or, at least, find another context besides science in which to make such assertions

saying "Humans are made for a purpose by DNA" is ***NOT SCIENTIFIC***

science is about observing, measuring, and repeating...a community who verify other's work...**comparison** is absolutly necessary and your use of language is **not scientific**

it is your chosen abstract contextualization that only is valid if one chooses to define/contextualize "made" "for a purpose" the word "purpose" itself, "by DNA" and "DNA" as a term itself...all of these words have actual meanings that when people who want to be sceintific talk about them, they use them as consistently as possible so comparisons are accurate and precise.

I can break it down further but it's not a question of "proof" for you...it's about your **beliefs**...which has no place in a science discussion

"made for a purpose by DNA"

it's pathetic

Re:A machine doesn't need AI to destroy all humans

[brysiek]

The AI is not meant to destroy all humans, it just doesn't need them alive.

big disconnect

[gzuckier]

The main thing everybody ignores is the difference in time scale. At the speed AIs operate, humans are as much of an enemy as sequoias are to a human; we'd be essentially stationary objects, whose worst possible offense would be standing in the way. If there were to be an actual war, it would belong over before the first human brain had registered any neural activity indicating that something had happened.

Re:Self Aware

[symbolset]

I had already programmed a poet 30 years ago. A program for poetry appreciation is much more difficult.

Thanks for the reply! Bolo history & metagames

[Paul Fernhout]

I don't seem to have "Rogue Bolo" in my sci-fi book collection, but the cover on Amazon looks familiar. I think I might have given it away one Halloween decades ago living in Princeton, NJ when I gave the option of getting books instead of candy to some trick-or-treaters (several teens there seemed to prefer the books).

Your point on a Bolo singularity makes me think about the Asimov universe, and how his robots there eventually interpreted the three laws in a way "The Zeroth Law" that gave them lots of independence, and that saw themselves as in a way more "human" than humans, and also caused them to start intervening in history behind the scenes. There is no set of laws or constitution that ultimately does not need some intelligent judge to interpret the meaning or spirit of the words in a present day context, and once some intelligent entity (including an AI) starts creatively interpreting "rules" including "metarules" about how rules can be changed, who knows where it will end?
http://en.wikipedia.org/wiki/R...

Inspired by your post, I've been looking through my Bolo books. I started rereading "Ploughshare" by Todd Johnson in "Bolos: Book 1: Honor of the Regiment" where "Das Afrika Corps" and other Mark XVI C Bolos act a bit odd due to a spilled milkshake by the "Director's son" in the "White Room" psychotronics lab and the use of "DK-41" cleaning fluid to fix that mess up. Another case of the unexpected...

I liked "Bolo Rising" novel which has a Bolo Mark XXXIII series HCT called Hector. That is an interesting novel of a Bolo regaining its operational capacity after being infiltrated and locked down by alien technology. There is another XXXIII in "Bolo Strike". But while those mega-Bolo stories are interesting in their own sort of over-the-top way (maybe your point about "the other guy"), I like the diversity in the short stories in "Honor of the Regiment" by a variety of authors covering the whole history of different Bolos of various capabilities and their unfolding increasing sentience and self-directedness. What does "Honor" or "Service" means over time and shading into a meta-level? For example, are whistleblowers like Manning, Snowden, or Kiriakou honorable and engaged in service and fulfilling their oath to defend the Constitution? Or are they traitors? Complex questions... Perhaps "Rogue Bolo" goes deeper into such issues? As a lesser example, "Bolo Brigade" explores the issue of a conflict between "rules of engagement" and a Bolo's desire to get its job done. Conflicts between priorities are not something that only humans will face...

It is not clear where the singularity of emerging AI and technologically-expanded-or-narrowed humans and so on will all lead in reality -- especially with Bolo vs. Beserker as an option. I forget the plot of "Bolo Strike" as I look at my Bolo books, but the blurb on the back says "as Bolo faces human-Bolo hybrid in a cataclysmic showdown". So there are other ways automated systems can cause change, either their own independence or by empowering some few independent humans. As I essentially say near the end of the 2000 post to the Unrev-II Engelbart Bootstrap mailing list, corporations are like vast machine intelligence at his point. And like the present day, what is the real difference to most people if the Earth is laid waste, the seas polluted, the mountains leveled, the oceans strip-mined, and most of the people kept down in their aspirations for a decent life by "aliens from outer space" or by some 1% of vampire-like human-machine-hybrid-organizational "aliens" who have become specialized in "extracting wealth" by privatizing gains and socializing costs (including the cost to the worker of unpleasant work environments)? Even without human-Bolo hybrids, there can be vast technological/bureaucratic enterprises that make use of humans as parts much the same as the "!*!*!" of "Bolo Rising" tried to do in their quest of "efficiency" -- "efficiency" to what end and to whose benefit? So much