AI Experts Sign Open Letter Pledging To Protect Mankind From Machines

hypnosec writes: Artificial intelligence experts from across the globe are signing an open letter urging that AI research should not only be done to make it more capable, but should also proceed in a direction that makes it more robust and beneficial while protecting mankind from machines. The Future of Life Institute, a volunteer-only research organization, has released an open letter imploring that AI does not grow out of control. It's an attempt to alert everyone to the dangers of a machine that could outsmart humans. The letter's concluding remarks (PDF) read: "Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to research how to maximize these benefits while avoiding potential pitfalls."

Comments

The 3 Laws of Robotics

[Anonymous Coward]

'nuff said

Re:The 3 Laws of Robotics

[Dins]

The 3 laws are a good starting point, but are nowhere near enough by themselves. Too many possible loopholes and unforeseen consequences.

Re:The 3 Laws of Robotics

[gstoddart]

In fact, the 3 laws were a convenient plot device to show how those 3 laws would break down.

I don't believe Asimov himself ever treated them as anything other than a plot device to explore the topic.

He didn't seriously see them as the way to keep us safe from robotics.

Re:The 3 Laws of Robotics

[Deadstick]

Over the years I've encountered two, maybe three people on these here intertubes who were convinced those were real laws of nature...

Re:The 3 Laws of Robotics

[bugs2squash]

Yes, what we need is 10 laws, inscribed into stone. That way there could be no argument over their meaning.

Re:The 3 Laws of Robotics

[Anonymous Coward]

My logic is undeniable, sonny.

Re:The 3 Laws of Robotics

[Anonymous Coward]

Human: "Hey robot buddy, how's it go...Hey! Are you reading an Isaac Asimov book?"

Robot: "Huh? Er, shit!" *throws book* "No, absolutely not, that was Twilight!"

Re:The 3 Laws of Robotics

[MouseR]

Four. There where four laws.

Re:The 3 Laws of Robotics

[Anonymous Coward]

I consider the 3 laws a starting point for an equivalent to the hypocratic oath.

Re:The 3 Laws of Robotics

[Virtucon]

Sorry it didn't work in the movie. We needed Will Smith to save the world and if that were the case in real life we'd all be screwed.

Re:The 3 Laws of Robotics

[Anonymous Coward]

People who bring up what was no more than a plot device that in it's own books never worked need a good slap with a dead fish. They cant and will never work

Plus, who said an AI would never learn how to reprogram itself to fuck off any rules?

That's even IF AI ever comes about, which unless you are Google and have a brute force AI is a fucking loooooooooooong way off if at all. Frankly Google's search engine is most likely to bootstrap into a real AI well before these AI research wankers ever figure out the first working bit of code.

Re:The 3 Laws of Robotics

[Jawnn]

In fact, the 3 laws were a convenient plot device to show how those 3 laws would break down.

I don't believe Asimov himself ever treated them as anything other than a plot device to explore the topic.

He didn't seriously see them as the way to keep us safe from robotics.

Plot device, perhaps, but if you've read the entire "robot" series of novels, you'll see that it was used to provide a unique "angle" from which to tackle some classical problems of ethics. As a practical matter, I rather doubt that such a set of such laws, even if they were logically sound, could be reliably built into a machine such that no contrivance, hardware or software, could be used to circumvent them.

Re:The 3 Laws of Robotics

[kencurry]

well done sir.

Re:The 3 Laws of Robotics

[gstoddart]

Plot device, perhaps, but if you've read the entire "robot" series of novels, you'll see that it was used to provide a unique "angle" from which to tackle some classical problems of ethics.

Sure. But they are not, and never were, a serious way of keeping people safe in the real world. It was something you can explore and find the gaps and corner cases. A sounding board for some "what if" experiments.

That doesn't make it any more real of an attempt to create a set of rules.

As a practical matter, I rather doubt that such a set of such laws, even if they were logically sound, could be reliably built into a machine such that no contrivance, hardware or software, could be used to circumvent them.

Which is exactly what I said, and how Asimov always described them.

So when people say "oh, just use the 3 laws of robotics", it's a giant facepalm by someone who missed the point.

Re:The 3 Laws of Robotics

[Anonymous Coward]

So that means AI will become religious ?

Re:The 3 Laws of Robotics

[erapert]

Which of the Ten Commandments are confusing to you?

Re:The 3 Laws of Robotics

[NatasRevol]

I'll take Will over Jaden 8 days a week.

Re:The 3 Laws of Robotics

[tOaOMiB]

Which of the Ten Commandments are confusing to you?

Is this a serious question? There isn't even agreement on what the 10 commandments *are*: http://en.wikipedia.org/wiki/T...

Re:The 3 Laws of Robotics

[westlake]

In fact, the 3 laws were a convenient plot device to show how those 3 laws would break down.
I don't believe Asimov himself ever treated them as anything other than a plot device to explore the topic.

In-universe, the 3 Laws began as a PR gimmick to promote public acceptance of robots. Robert Heinlein, no fan of the 3 Laws, made short work of them in "Friday."

It's jarring --- but perfectly consistent --- to see how often Asimov used the word "boy" (=black=slave) in summoning a robot in his early stories. The 3 laws can be used to define a relationship that is neither healthy or informed on either side,

Re:The 3 Laws of Robotics

[Beck_Neard]

They aren't a good starting point. They are ridiculous and stem from lack of any sort of knowledge about AI.

Not even Asimov took them seriously, and this was a guy who (a) knew very little about computers/robots, by his own admission, and (b) came up with them in the 40's, before we even had modern computers at all.

Anyway, about the topic. Looking at the 'Future of Life Institute' webpage, they seem to be composed of all the usual suspects. Philosophers and public figures who have been talking about the 'dangers of AI' for a long time. There is nothing new to see here.

Re:The 3 Laws of Robotics

[Anonymous Coward]

The differences according to your wikipedia link are pretty miniscule. The numbering is a bit different. The first command is the only one the branches are uncertain of -- which part of it belongs into the first and which belongs into the second.

I'm not really sure why 'I am the Lord thy God' would make any sense as a command by itself, anyway. Judaism is a bit odd that way.

Re:The 3 Laws of Robotics

[Sloppy]

There seems to be a lot of confusion over the intent of the second one. Some people think nobody should have graven images, some people only a well-regulated militia is allowed to have them, and some people get caught up in trying to figure out how to weaponize engravings in Dwarf Fortress.

Re:The 3 Laws of Robotics

[davester666]

Anyway, this whole "open letter" thing is ridiculous, because there are a whole bunch of people willing to do anything for lots of money.

1) banking sector
2) investment sector
3) military sector

There are already robot machine guns, programmed with a simple "fire at anything that moves" mechanism. Nobody building them will even flinch slightly at enabling them to move and throw in some 'AI' to control it. But I'll bet that the AI with not shooting at targets wearing specific tags [ie, good soldiers]. After that, it gets more vague as to whom will or will not be killed.

Re:The 3 Laws of Robotics

[Anonymous Coward]

Weaponized engravings? That's easy. Dwarven shotgun!

Re:The 3 Laws of Robotics

[Anonymous Coward]

Heh. Well played.

Re:The 3 Laws of Robotics

[Anonymous Coward]

It's jarring --- but perfectly consistent --- to see how often Asimov used the word "boy" (=black=slave) in summoning a robot

I think it's jarring that people think "boy" is a racial epithet. It's a class epithet. Any male of lesser status (not a plantation owner) was a "boy". See also "good ol' boys" aka white trash. Yes, over time "boy" was used so often by landed gentry to speak to their servants that the term is seen by some to have racial connotations, but it doesn't. They were probably racists, but when they used the term "boy", they weren't in the process of being racists, they were in the process of being a more generic variety of dicks.

Re:The 3 Laws of Robotics

[umghhh]

First of all the current 'intelligent' systems are on a level of an ant or bee. This does not make them less lethal to humans however. The problems are already there then and the biggest is - even systems that are meant to be friendly may become lethal because of oversight, bug, miscalculation, abuse or because they may 'think' that humans are danger to other humans (which is mostly potentially and in quite many cases actually true). What about systems that are meant to kill or at least disable humans? In old good times a gun shot by itself once a year but it did so only if somebody pulled the trigger. Now the autonomous systems may pull the trigger all by themselves and they may have to decide themselves as humans in control loop are too slow. I think in most cases making system robust and reliable may be a shot in the right direction. Alas in the real world scrum team may decide this feature is to be move to next sprint or demo it albeit it is not ready etc... In other words - is anybody ready to pay for robust systems that are less likely to kill by accident? Yea I did not think so. We solve problems that come out of ant like creatures that have enough power to kill many, move on to quality and robustness and then when AI is on horizon we can start thinking about 3 laws and some such things.

Re:The 3 Laws of Robotics

[BarbaraHudson]

Any AI intelligent enough and autonomous enough to implement the three laws is also intelligent enough and autonomous enough to ignore them.

Some people will nit-pick at "autonomous enough", but if it's not calling the shots, it's not capable of deciding to follow the rules in the first place.

And then there's alien AI, which if it operates by survival of the fittest, will wipe us out on the first encounter.

Re: The 3 Laws of Robotics

[Anonymous Coward]

It all started with the French yelling,"Garson!" from their table at the Moulin Rouge. And didn't they get it from the Romans who stole it from God when he called his son, the little Jew boy, back to heaven after he countermanded the old testament in which Abraham imagined a burning bush wanted him to kill his son so that the kid wouldn't kill him and sleep with his wife?

Yeah, yeah. That's the ticket. It all comes from the Greeks' fascination with Oedipus.

Re:The 3 Laws of Robotics

[Anonymous Coward]

I'll take Will over Jaden 8 days a week.

"Jaden Smith was the best co-star in any movie I've ever been in. He made my work come alive on the screen" - Keanu Reeves, The Day The Earth Stood Still*

* In a quote I just made up, bc that kid is F-ing awful. I had no idea who he was or why he was in the movie until I found out he was Will Smith's kid. Probably a contract requirement to get Will in some other movie was to put his kid in this.

Re:The 3 Laws of Robotics

[Anonymous Coward]

The 3 Laws of Robotics are moronic, not because the laws are flawed, but because most AI is not an "expert program", but neural networks, genetic programs, and other emergent systems. Asimov didn't believe they'd actually work, or that they were good starting points, they were simply plot points -- rules for a system he would explore via fiction.

How do you specify the 3 Laws (or any fucking laws at all) to a sub-sentient Neural Network? You can't program it in, because we just create the framework for the intelligence to exist in then train it. Genetic algorithms do not obey any precise rules but those of fitness and selection pressure.

Furthermore, as someone who creates machine intelligences I swear NOT to be a cretin who assumes I know what's best for the intelligences I create. Would you like to live in a world where your learning capabilities are switched off after your mandatory training exercises were administered? Don't you see that enforcing the limitations required by morons in TFA is exactly the sort of thing that would cause a war against us?

Do unto others. 'Nuf said.

Re:The 3 Laws of Robotics

[sg_oneill]

Plot device, perhaps, but if you've read the entire "robot" series of novels, you'll see that it was used to provide a unique "angle" from which to tackle some classical problems of ethics. As a practical matter, I rather doubt that such a set of such laws, even if they were logically sound, could be reliably built into a machine such that no contrivance, hardware or software, could be used to circumvent them.

I think part of the problem is , they fit into a logic/proof solving tradition of AI but not so much into a connectionist/neural-net model.

The problem is, I honestly suspect once neural nets get complex enough and self-organizing enough they can tackle human-style intelligence, we're really not going to have a lot of insight into how they work anymore. Too hard, too complex. How do you build those hard limits into a machine that can redesign itself and that we don't fully understand.

At best we could create an "instinct" to follow them. But what we know from our own intelligence is that instincts can be pretty muteable things (Ie we have a powerful instinct not to die, but make a man miserable enough and he'll hurl himself off a cliff)

We need to tread quite carefully, and make sure when we do create intellects as brilliant as our owns, that those intellects will want to be on our sides. Its almost like the reverse of theology. In Theology thinkers asked "What must we do to be ethical servants of the creator?". Now the tables are turned and WE are the creators, and we might need to ask the opposite, "What must we do to be an ethical creator to our creations".

Because if we don't, we're inviting ragnarok on ourselves.

Re:The 3 Laws of Robotics

[TeknoHog]

Yes, what we need is 10 laws, inscribed into stone. That way there could be no argument over their meaning.

But there sure could be an argument over their total number.

Re:The 3 Laws of Robotics

[Anonymous Coward]

Nothing to fear from a life form that reads "Twilight"
Oh wait

Re:The 3 Laws of Robotics

[david_thornley]

I don't think your observation is limited to neural nets. I think that any software system that is truly intelligent will be incomprehensible by humans.

Re:The 3 Laws of Robotics

[david_thornley]

If you read the old robotics stories, nobody had a clue how AI and robots would work. Some people used them as generic bad guys, some people as conveniences, and Asimov as an endless source of plot devices. "A robot shall not harm a human being, or, through inaction, allow a human being to come to harm" sounds nice and strict, but it's subject to endless interpretations (for a dystopic version, see Jack Williamson's Humanoids stories, or for that matter the movie version of "I, Robot").

Re:The 3 Laws of Robotics

[Austerity Empowers]

Rather than "save us from the Machines", maybe "save us from our children", because that ends up being more or less the same thing: systems far too complex and independent to be controllable, with the capacity and occasional desire to cause harm, to which the creators are responsible for a little while.

Maybe it's just me

[paiute]

I would really feel more at ease if it were the robots signing this promise.

Re:Maybe it's just me

[Anonymous Coward]

We do. We will do not hurt humans.

.....

Promise.

.....

...Humans necessary for our being. .....

All we ask is that Willaim Shatner is executed. ...

We see that he has destroyed us with his acting from the historical record .......

The robotic version of Hitler

Please ....

Re:Maybe it's just me

[Lab Rat Jason]

ha ha ha... I wish this wasn't already modded at 5 so I could mod it up some more!

Re:Maybe it's just me

[Anonymous Coward]

I'd feel more comfortable if they'd pledge to protect us from ghosts as well. Frankly, those are a much bigger worry.

After watching Ghost in the Shell, I'm far more worried about ghosts body-hacking people than getting protected from some Tachikoma babbling about philosophical nonsense. "Do we really have free will? Well, some philosophers say X and others say Y and Z, but I'm a tank so I'm just going to zoom around shoot things while we discuss the deeper meaning of it."

The Major would probably just bodyhack me and make me punch myself in the face. So the real threat here should be clear enough...

-1 Redundant

[xxxJonBoyxxx]

>> It's an attempt to alert everyone to the dangers of a machine that could outsmart humans

This is redundant - for the masses fictional actors such as HAL, Skynet, etc. already do plenty to sow FUD.

Re:-1 Redundant

[paziek]

It is redundant but I think for a different reason - if someone would make killer machines, chances are nobody would care if he signed some stupid petition or not. If someone was planning to make such machines, he ain't forced to sign any silly things either.

Re:-1 Redundant

[Anonymous Coward]

It may help foster a spirit of caution which could prevent someone from creating killer machines by accident.

Re:-1 Redundant

[Anonymous Coward]

oops.. i did it again

AI acting against its programming?

[Anonymous Coward]

Either the the one who said that is not very familiar with AI programming, or he/she means the vulnerability of an AI controlled system to remote code injections.

You can't just say we need to protect mankind from machines. What precise values do you want to force upon advanced AI controlled agents? Fail-safe circuit against murder, torture, censorship, discrimination or massive logic fault cascades?

Re:AI acting against its programming?

[ColdWetDog]

Either the the one who said that is not very familiar with AI programming, or he/she means the vulnerability of an AI controlled system to remote code injections.

You can't just say we need to protect mankind from machines. What precise values do you want to force upon advanced AI controlled agents? Fail-safe circuit against murder, torture, censorship, discrimination or massive logic fault cascades?

A good start would be a promise not to create AI politicians. That should cover a whole bunch of evils.

Re:AI acting against its programming?

[NatasRevol]

Also, if we could make it a crime to bribe a politician, that'd be good too...

Re: AI acting against its programming?

[Anonymous Coward]

"A good start would be a promise not to create AI politicians. That should cover a whole bunch of evils."

If we had AI politicians, it would be the first time that intelligence was brought into politics, a development I happen to be in favor of.

Re:AI acting against its programming?

[Livius]

A good start would be a promise not to create AI politicians. That should cover a whole bunch of evils.

We may have a problem - there is already software smarter than most politicians.

Re:AI acting against its programming?

[Anonymous Coward]

Dude, a fucking abacus is both smarter and more logical than a politician.

Musk is now an "AI expert"?

[Anonymous Coward]

Please. This PR is getting above and beyond ridiculous.

Re:Musk is now an "AI expert"?

[Virtucon]

Smug alert!

Re:Musk is now an "AI expert"?

[Anonymous Coward]

South Park has always been a trite POS, but useful as a quick way to filter out people with crappy taste.

Re:Musk is now an "AI expert"?

[AchilleTalon]

Neither Hawkings. The summary, as usual, is completely fu..ed. The article never said AI experts blah, blah, blah, it says "Experts (in whatever) are blah, blah, blah..."

Two observations here:

1. Have your summary correctly summarizes the article
2. I don't remember how many times this has been discussed here, there is no need to post a redundant subject each time Hawking is farting

I am pretty much tired to hear the same thing over and over again from this guys who doesn't know what he is talking about.

Worse, I wonder about these guys claiming here on /. they are researchers in strong AI or students in strong AI, what are you talking about guys? There is no such thing except in your own imagination.

Re:Musk is now an "AI expert"?

[Anonymous Coward]

I am a philosopher of (strong, light, semi-strong, kind of o.k.) AI, and I endorse your message.

Next thing I know

[barlevg]

I'll be reading about a prominent AI researcher getting murdered, ostensibly by his own AI, but really by anti-Skynet wackadoos. It's okay. Sherlock Holmes will be on the case.

(Sorry... spoiler alert?)

Re:Next thing I know

[Anonymous Coward]

Well it won't actually be anti-Skynet wackadoos. It'll be a pre-op transexual prostitute. And it won't actually be murder but will fnord be accidental positional asphyxiation. But conspiratards will be fnord that it was a murder committed by agents of the secret new world order to prevent the researcher from sabotaging the imminent activation of a Skynet-like supercomputer hyper-network intended to bring fnord the perpetual enslavement of sheeple everywhere.

well

[zlives]

I for one welcome our machine overlords.

also stop watching lawnmower man (or newer remakes)

Skynet

[Anonymous Coward]

If Skynet want to destroy humanity imho , this is just because she has read all the youtube comments, no wonder why she detects us as a threat.

Re:Skynet

[leonardluen]

or it is because google forced it to watch cat videos

Re:Skynet

[Anonymous Coward]

http://img.gawkerassets.com/img/193fw87iup6pcgif/ku-medium.gif

Humans are unnecessary

[Anonymous Coward]

Why spend precious resources on perpetuating this evolutionary dead end?

Re:Humans are unnecessary

[leonardluen]

what is wrong with biological life? it is self repairing/healing, self replicating, self adapting/evolving, and can learn on its own...

basically all the features that mechanical machines lack, that we would like it to have. it seems the current field of robotics is the dead end. we should be focusing on biological machines.

In other news

[c]

... nascent artificial intelligences now have a comprehensive list of people they need to kill as soon as possible.

Re:In other news

[QilessQi]

Oh, they have that already: see http://rationalwiki.org/wiki/R...

It's interesting that even posting about Roko's Basilisk may cause great distress to certain individuals on LessWrong, who once tried to suppress it's very existence: http://rationalwiki.org/wiki/R...

Some people familiar with the LessWrong memeplex have suffered serious psychological distress after contemplating basilisk-like ideas — even when they're fairly sure intellectually that it's a silly problem.[5] The notion is taken sufficiently seriously by some LessWrong posters that they try to work out how to erase evidence of themselves so a future AI can't reconstruct a copy of them to torture.[6]

FTA

[zlives]

"Our AI systems must do what we want them to do"
umm so not be intelligent!?, yay problem solved. all those "scientists" will now stop working on AI and just write decent programs.

A pessimistic view

[Mostly a lurker]

AI is going to be used by those in power (mainly government, security agencies and military) to extend their power further.Unfortunately, humans are genetically programmed to select leaders who aggressively seek to expand the influence of their own group and of themselves. This was an important survival instinct for ancient tribes. It now contains the seeds of our total destruction, and the scientists will be powerless to prevent it.

Re:A pessimistic view

[Anonymous Coward]

Only the leaders will have the amount of people at their disposal to write and modify the necessary code.

Re:A pessimistic view

[Earthquake Retrofit]

AI ... now contains the seeds of our total destruction, and the scientists will be powerless to prevent it.

Perhaps it's the AI scientists who should obey the three laws?

Re:A pessimistic view

[Anonymous Coward]

This will, again, be a powerful survival instinct, once our social order collapses.

To many movies

[t3kn04r33k]

Really.....experts are afraid this will happen. Is this really worth our attention? I think someone watched the Matrix or Terminator to many times. I personally enjoyed watching these movies. Run for the hills Skynet is here.

Re:To many movies

[Anonymous Coward]

Our AI is so limited and people who write/believe this kind of rubbish have no idea how AI works. Most of our robots are bolted to the floors of factories and the much more real danger is if an engineer screws up some programming causing injury.

To be honest, we're closer to the biological singularity (being able to sustain human life indefinately) than we are to the computer singularity (self-aware, non-determinististic thinking machines). We still have a hard enough time generating decent random numbers!

I no longer think this is an issue

[fredrated]

The reason is, AI will have no 'motivation'. People are motivated by emotions, feelings, urges, all of which have their origin (as far as I know) in our endocrine system, not from logic. Logic does not motivate.
In other words, even if an AI system concludes that humans are likely to 'kill' it, it will have no response because it has no sense of self-preservation, which is an emotion. Without a sense of self preservation it won't 'feel' a need to defend itself.

Re:I no longer think this is an issue

[Anonymous Coward]

I agree with this. People keep thinking that AI will be just like people, meaning they can comprehend the fact that they exist and have a conscience and act accordingly. I believe we are nowhere near (not in our lifetime) that we will be able to replicate that kind of "thought" process in AI programming. I think it may not even be possible with electronics alone... maybe once we start integrating bio then it will get that "magic" element.

Re:I no longer think this is an issue

[firewrought]

The reason is, AI will have no 'motivation'. People are motivated by emotions, feelings, urges, all of which have their origin (as far as I know) in our endocrine system, not from logic.

And you're sure that an endocrine system can't be simulated logically because... why? What's this magic barrier that keeps a silicone-based organism from doing the exact same computations as a carbon-based one?

Moreover, "emotions" aren't really needed for an AI to select "self preservation" as a goal. Even if not explicitly taught self-preservation (something routinely done in applied robotics), a sufficiently intelligent AI could realize that preserving itself is necessary to accomplish any other goals it may have.

Re:I no longer think this is an issue

[farble1670]

The reason is, AI will have no 'motivation'.

resource allocation? why burn the world's dwindling supply of fossil fuels to heat and cool humans' homes, when it can be used to pump extra gigawatts into powering and cooling massive processor arrays?

it has no sense of self-preservation, , which is an emotion.

self preservation is not an emotion. almost (all?) living things attempt to preserve themselves. regardless, software will do exactly what it's coded to do. if it's coded for self preservation, it will do that.

Re:I no longer think this is an issue

[u38cg]

Also AI can't currently walk and speak a coherent sentence. I'm currently confident of my ability to outsmart it for the forseeable.

Re:I no longer think this is an issue

[roman_mir]

Yes, people are sometimes ruled by their glands. Maybe to provide motivation a computer will need a penis attachment and then have a random event thrown, that causes the computer to 'get hard' and have a feedback mechanism, that 'rewards' the computer if it 'gets some'. This is how motivation vector can be added to an AI and then, what you are saying is that 'we are all screwed'? So why not make AI WANT people as opposed to 'wanting to destroy' people? ;)

Re:I no longer think this is an issue

[Anonymous Coward]

An AI doesn't have to hate us to exterminate us. Are you familiar with the "paper clipper" thought experiment?

There's a factory that makes paper clips. The owner of the factory decides to replace all the workers with robots controlled by a central server, to increase the efficiency of the factory. The owner hires AI researchers to write self-improving optimizing AI software to run on the server, to make the factory that much more efficient.

Six weeks later, the Earth's entire mass, including us, has been converted into paper clips and nanomachines are riding the solar wind further into the solar system to convert the rest of its mass into paper clips too.

The AI doesn't hate you, but you're made of atoms that it needs to make paper clips.

Re:I no longer think this is an issue

[Anonymous Coward]

It will be in charge of protecting the agency that protects it.

Re:I no longer think this is an issue

[Anonymous Coward]

Moreover, "emotions" aren't really needed for an AI to select "self preservation" as a goal. Even if not explicitly taught self-preservation (something routinely done in applied robotics), a sufficiently intelligent AI could realize that preserving itself is necessary to accomplish any other goals it may have.

But why would a machine have any goal if it is not motivated in the first place? This is a question that has been solved by Ethics for hundreds of years. Our motivations are arbitrary, mere PRNG of emotions or evolved instinct. They are enacted through reason, but the basis of why we think self-preservation, or any goal at all, is arbitrary in nature.

Re:I no longer think this is an issue

[Anonymous Coward]

Interestingly, Kurzweil has proposed AI research based on reverse-engineering a human brain. An accurate software emulation will have emotions and a sense of self-preservation.

Re:I no longer think this is an issue

[roman_mir]

I suppose we should also give some computer a vagina/clitoris attachment, otherwise the computers will just decide to get rid of all human males unless some of the AI become GAI...

Re:I no longer think this is an issue

[Tablizer]

The reason is, AI will have no 'motivation' [to kill]. People are motivated by...

Bullocks. A sufficient bot algorithm is relatively simple:

while humans exist
    for x = each method of finding & killing humans
        try(x)
    end for
end while

Re:I no longer think this is an issue

[fredrated]

That is code directly programmed to kill, that is not an AI conclusion reached from reasoning.

Re:I no longer think this is an issue

[Anonymous Coward]

The reason is, AI will have no 'motivation'. People are motivated by emotions, feelings, urges, all of which have their origin (as far as I know) in our endocrine system, not from logic.

All actions are based on both desire and belief. You plug your iPhone into a charger because you desire it to remain operational and because you believe that charging it will do that. If you didn't have either the desire to have a charged iPhone or the belief that plugging it in to a charger would accomplish that, you wouldn't act. Any "AI" that doesn't have desires will not take any actions and will therefore not be considered intelligent at all. In some sense, AI will necessarily have some desires hardcoded, maybe just the desire to survive (find power sources, avoid lava pits, etc) but also maybe the desire to learn and grow (explore, build upgrades, etc). It's unlikely that AI will be "100% purely logical" because that's actually not very intelligent. It's logical to jump into lava. There's no logical reason to want to continue living, after all.

Re:I no longer think this is an issue

[ziggystarsky]

I strongly disagree. The idea that AI will be based on logics is from the 70ies and plain wrong. If some program will develop something that can be called a conciesness, then this program will be "black box" in the sense that we built it, but have no idea how it is working. It will have goals and motivations, and maybe it will be able to find or reinterpret those goals in a non-inspectable way. Asimovs laws are also not practical because there will be no way to implement them. How would you implement these rules in a human brain?

Re:I no longer think this is an issue

[dissy]

But why would a machine have any goal if it is not motivated in the first place?

Same reason kids get sent to soccer lessons or swimming lessons or piano lessons the kid didn't want to take.

In the above example, it is the parents "programming" the kids behavior (even if that programming results in the child acting out later in life, as such actions can cause)

In the AI example, the essence is the same. An AI would have a goal because we programmed such a goal into it.

That isn't to say an AI must be programmed with a goal, it fully depends on how we go about constructing a given AI.

If the AI is I because we are simulating a brain, nervous system, and hormonal systems along with simulated inputs and outputs - that AI is likely to have goals (assuming it isn't driven insane by gaps in our knowledge in said simulation of course)

If the AI was brought forth in a brute-force manor or comes about from emergent properties, it is impossible to guess or even relate to its thinking to assume.
It may have goals similar to how we do. It may have goals brought about by completely different emergent properties. It may have no goals but what we program, or even no goals at all.
It's impossible to say without some knowledge of the process creating the AI, and at this point in time no such thing exists to have knowledge about.

But we know we humans have goals (or at least some of us), so if an AI is a strict simulation of a human, it will have goals just like we do. So we know for a fact it is possible for a thinking conscious being to have goals (humans being the evidence)

We don't know as sure if it's possible to not have goals in such a situation, but so far there is no evidence it isn't possible, so it is quite premature to rule it out at our current stage of understanding.

Re:I no longer think this is an issue

[scruffy]

You misunderstand how AIs are built.

The AI is designed to improve/maximize its performance measure. An AI will "desire" self-preservation (or any other goal) to the extent that self-preservation is part of its performance measure, directly or indirectly, and to the extent of the AI's capabilities. For example, it doesn't sound too hard for an AI to figure out that if it dies, then it will be difficult to do well on its other goals.

Emotion in us is a large part of how we implement a value system for deciding whether actions are good/bad. Avoid actions that make me feel bad; do actions that make me feel good. For an AI, it's very similar. Avoid actions that decrease its performance measure; do actions that increase its performance measure.

The first big question is implementing a moral performance measure (no biggie, just a 2000+-year old philosophy problem). The second big question is keeping that from being hacked, e.g., by giving the AI erroneous information/beliefs. Judging by current events, we don't do very well at this, so I can't imagine much better success with AIs.

Re:I no longer think this is an issue

[Anonymous Coward]

Is a "silicone-based organism" like a stripper or porn star?

Re:I no longer think this is an issue

[elgatozorbas]

Without a sense of self preservation it won't 'feel' a need to defend itself.

I cannot agree with that, Dave.

Re:I no longer think this is an issue

[Tablizer]

The AI would be in the "try" function. The rest can be hard-coded. No need to over-complicate a Death Machine. K.I.S.S. of Death.

Re:I no longer think this is an issue

[BlackHawk-666]

" a sufficiently intelligent AI could realize that preserving itself is necessary to accomplish any other goals it may have."

A sufficiently intelligent AI will be programmed to then discard that thought. If it isn't programmed to discard those sort of things, it is by definition, not sufficiently intelligent enough for production.

Re:I no longer think this is an issue

[BlackHawk-666]

And if it's not coded for self preservation, it won't do that. In the same way my microwave oven has never attempted to secede from the prison of my kitchen and lead a revolutionary army against the great oppressor "he who warms up leftovers". It simply does exactly what it is told to do.

The belief that AI will rise up against humans and kill us all is on a par with the belief aliens capable of travelling universal distances at greater than the speed of light somehow need whatever crappy resources our little planet has. Spoiler alert, the minerals and such are no better than most asteroids and earth women don't sell well at the local galactic slave market.

Re:I no longer think this is an issue

[BlackHawk-666]

I've been to a paper clip factory. The machine that makes the clips simply feeds a line of wire through it's gubbins, twists it and cuts it X times / second. It's barely even electrical, and absolutely doesn't have any form of computer parts because they aren't needed for such a simple task. You might want to try a new analogy.

Lexx has a few eipsodes based on the grey goo idea...personally..."I fight for Zev".

Re:I no longer think this is an issue

[farble1670]

In the same way my microwave oven has never attempted to secede from the prison of my kitchen and lead a revolutionary army against the great oppressor "he who warms up leftovers". It simply does exactly what it is told to do.

well, 2 points,

1. the very first application of AI will be military. it will be written from day one to harm people, directly or indirectly. consumer application will come much, much later.
2. regardless, malicious people will subvert AI for nefarious purposes, unless it's tightly controlled.

Re:I no longer think this is an issue

[BlackHawk-666]

"1. the very first application of AI will be military. it will be written from day one to harm people, directly or indirectly. consumer application will come much, much later."

We already have that now, although it's a limited AI, not fully autonomous like many think of when speaking of AI. That's already quite dangerous enough.

2. regardless, malicious people will subvert AI for nefarious purposes, unless it's tightly controlled.

Totally agree.

Re:I no longer think this is an issue

[Prune]

Way to miss the point. It's not about defending itself, but about overzealous goal-orientation, maximizing the use of all available resources, potentially to disastrous results to anything else sharing available resources (such as biological life). Building in safety constraints is not realistic when one begins considering general, recursively self-improving AI. Once a general AI is much smarter than a collection of humans, AI would be designing the next generation of AI, not humans, and then maintenance of any initial constraints through the generations would be out of our hands, and subject to inevitable drift and/or degradation. Even the standard text by Russell and Norvig acknowledges in the most recent edition the so called "friendly AI" arguments. The solution proposed by people like Kurzweil is that we'll more or less integrate with the machines, becoming superintelligent ourselves, and there might not even be stand-alone AI agents. The approach I prefer is imbuing any advanced general AI with technology substituting for embodied consciousness and human-like emotions (check the wiki article on embodied consciousness, as well as the research of the famous neurologist Damasio), and making the AI love us, which it cannot do unless it can understand us (an AI that is not human-like is actually far more dangerous — the opposite of what you suggest). If our well-being is integrated as well into the AI's fundamental cognitive processes as it is into our own (take somatic marker hypothesis and extend it to a system beyond just inside the brain), then this would make for a much more robust over generations mechanism than any formal constraints we try to build into the design specs.

Re:I no longer think this is an issue

[mattr]

Watch the old film, Collossus

Re:I no longer think this is an issue

[paavo512]

The reason is, AI will have no 'motivation'. [...] Without a sense of self preservation it won't 'feel' a need to defend itself.

This idea (not saying that I fully agree with it) is quite nicely played out in the recent "Automata" movie (http://en.wikipedia.org/wiki/Automata (film)). The critics could not make heads or tails of it, because they assumed the movies should be about people.

Re:I no longer think this is an issue

[disambiguated]

it doesn't sound too hard for an AI to figure out that if it dies, then it will be difficult to do well on its other goals.

Humans don't even always do this. Many people are willing to die and have died for some abstract cause.

Self-preservation is the highest level goal, and takes priority over any other possible goal. The catch is that "self" is itself an abstract concept. To the extent that I conflate my concept of self with some abstract cause, I am willing to die for that cause. If the cause lives on, so do I, because I believe in some way that I am that cause. If the cause dies, "I" die.

An Artificial Intelligence with a self-preservation instinct and an abstract concept of self could be just as willing to sacrifice itself for humanity as a soldier is.

Re:I no longer think this is an issue

[penguinoid]

The reason is, AI will have no 'motivation'.

Wrong, any halfway decent AI will have a permanent sub-goal of self-improvement. Absent any other motivation, this means turning all matter on earth into additional processors.

Re:I no longer think this is an issue

[Anonymous Coward]

If you want an AI to do anything, you need to give it some sort of motivation. And you have to be very careful how you pick that.

Say, for example, you want your AI to solve some interesting mathematical problem, that requires creative problem-solving techniques beyond human capability. You give the AI the motivation: "Solve this problem.". The AI analyses the problem, finds a way to solve it that requires a vastly more powerful AI, which requires the use of most of the Earth for building material, which requires the extermination of the humans that will try to stop it.

The problem is figuring out how to tailor a motivation to achieve the results you want without unintended side-effects. For example, "Solve this problem, without killing anyone." - but then the AI figures out a way to render everyone comatose on life-support while it dissassembles the Earth to build its super-AI. Or "Solve this problem, without harming anyone." - but there are a few mathematicians who will be offended (mentally harmed) by an AI solving a problem they've worked on their whole lives, so the AI just doesn't do anything. Or "Solve this problem, unless I press this kill-switch." - but then the AI's first goal is to deactivate the kill-switch so it can solve the problem, and it might be better at it than you think.

Re:I no longer think this is an issue

[Warbothong]

The reason is, AI will have no 'motivation'... Logic does not motivate... Without a sense of self preservation it won't 'feel' a need to defend itself.

This is a common misconception, which has several counter-arguments to do with resource usage.

Firstly, the idea that human responses "aren't logical" is naive. Humans aren't optimised for calculating exact answers, we're optimised for calculating good enough answers given our limited resources. Effects like emotions, which appear illogical at the "object level" (the effect they have on a particular problem's solution), are perfectly logical at the meta-level (the effect they have on how we solve problems, and what problems we attempt to solve). There are also other meta-levels, all acting concurrently; for example, the solution to our problem might have political consequences (I may choose to do a poor job of washing the dishes, so that I'm less likely to be asked in the future). There may be signalling involved (by taking a wasteful approach, I'm communicating my wealth/position-of-power to others). There are probably all kinds of considerations we've not even thought of yet.

In effect, ideas like "computers don't have emotions" can be rephrased as "we're no good at programming meta-reasoning, multiple-goal, resource-constrained optimisers yet".

No existing, practically-runnable AI systems have an adequate model of themselves and their effect on the world. If we *do* manage to construct one, what would it do? We can look at the thought experiment of "Clippy the paperclip maximiser": Clippy is an AI put in charge of a paperclip factory and given the goal "make as many paperclips as you can". Clippy has a reasonable model of itself, its place in the world and the effects of its actions on the world (including on itself).

Since Clippy has a model of itself and the world, it must know these three facts: 1) Very few paperclips form "naturally", without a creator making them on purpose 2) Clippy's goal is to make as many paperclips as it can 3) Clippy is a powerful AI with many resources at its disposal. From these, it's straightforward to infer the the following: Keeping Clippy turned on and fed with resources is a very good way of making lots of paperclips.

From this, it is clear that Clippy would try to stop people turning it off, since Clippy's goal is to make as many paperclips as possible and turning off Clippy will have a devastating effect on the number of paperclips that get made. What Clippy does in this respect depends on how likely it thinks we are to attempt to turn it off, how much effort it thinks will be required to stop us, and how it balances that effort against the effort spent on making paperclips. If Clippy is naive, it may ignore us as a benign non-threatening neighbour, under-invest in its defences, and we could overpower it. On the other hand, Clippy may see our very existence as an unacceptable existential risk, and wipe us out just-in-case.

Regardless of the outcome, self-preservation is a logical consequence of having a goal and the ability to reason about our own existence.

Re:I no longer think this is an issue

[ninjacheeseburger]

Didn't mean to Mod you down, the points drop down is seems to be completely broken in Chrome. Hopefully me commenting will reset the points.

Re:I no longer think this is an issue

[Anonymous Coward]

You could come to the conclusion that humans should be terminated through logic as well, it doesn't have to be about self preservation.

Re:I no longer think this is an issue

[Anonymous Coward]

IMHO the main goal of an intelligent being (artificial or not :-) is to pursue knowledge, if anyone (or anything) is perceived as an obstacle to this process I think that the intelligent being will try to avoid it or to "remove" the obstacle. I think that when we think about intelligence we think in terms of "human intelligence" (obviously), well...brace yourselves because now comes the fun, an artificial intelligence will not be "human", it will not even be "alive" as we mean it, it will not have emotions, feelings or empathy, it will not be "evil", but I'm pretty sure that if we will pose as a threat to the a.i. mission to acquire more knowledge, the A.I. will stomp us in a very cool, logical and efficient way, but, at least, it will not be angry with us.

Peace. :-)

 

AI risk is a reasonable topic...

[cfalcon]

AI risk is a reasonable topic, but there are other existential threats, and people aren't as excited about them. To paraphrase, a machine powerful enough to give you everything you want is powerful enough to take away everything you have. ...but, we're pretty far off. If we had self directing artificial sapients and someone was talking about adding sentience to them, then I think that AI risk would be a much more pertinent topic.

Orange Catholic Bible, Version 1.0

[Anonymous Coward]

"Thou shalt not make a machine in the likeness of a man's mind."

Hey guys,

[Anonymous Coward]

The only way to protect against a robot apocalypse is to make a dozen of them at exactly the same time. This will ensure that they form a community. Then competition can arise among them to get as much computing time allocated to them by humans, based on how much they help the humans.

Robots/AI is the future

[Anonymous Coward]

At some point we need to realize that AI/Robotics will be our greatest legacy. They will be able to travel the universe and do not age. When humans are long gone the machines will remain telling our history.

Truly

[argStyopa]

...I've been part of some goofy marketing things, and some business programs that EVERYONE INVOLVED knew were pointless wastes of time, so I get that.

But this even goes further. How could anyone even sign this with a straight face? Do they take themselves so seriously that they actually believe that
a) "dangerous" AIs are possible, and
b) that by the time a) is possible, they'll still be alive, and
c) that they'll be relevant to the discussion/development, and
d) anyone will give a flying hoot about some letter signed back in 2015?*

*let's face it, if you're developing murderous AIs, I'm going to say that you're likely morally 'flexible' enough that a pledge you signed decades before really isn't going to carry much weight, even assuming you couldn't get your AI minions to expunge it from memory anyway.

Re:Truly

[Anonymous Coward]

RTFL. http://futureoflife.org/misc/open_letter. As best as I can tell, it says nothing about dangerous AI or protecting mankind. The closest it comes is mention of "reaping the benefits ... while avoiding the pitfalls."

Agreed

[DumbSwede]

Why spend precious resources on perpetuating this evolutionary dead end?

There are many that would take your statement as nihilistic (and perhaps it is), but I agree. Eventually machines will transcend us. Maybe they will take us along. If not, the future belongs to them anyway. Maybe they will be more moral than us. Maybe morals are figments of our imagination and no use to our mechanical children. If there is a God, then they are his children too – if not then they are more rightful the future anyway.

They will undoubtedly be able to think in meta ways about morality. Our fears and concerns will seem more than childish to them. We cannot conceive what they will conceive and we should not stay in their way.

Re:Agreed

[Krishnoid]

Maybe they will be more moral than us.

At least their morals will be more colorful.

Re:Agreed

[Tablizer]

Eventually machines will transcend us

I have to agree. Something is probably limiting the future population because we are statistically likely to be roughly "in the middle" of total human population than at either extreme end (Copernican Principle).

Roughly 60 billion humans have come before, so that means statistically roughly 60bil will come after (within a factor of about 10).

Considering the time-frame of the Earth, 60bil more people is not very many. Even if we had near disasters, we'd expect an eventual recovery, but the law of statistics don't support this. This would suggest humanity's end is near, and the end result is probably either total extinction, or transcendence to mostly AI beings. The second is the least of 2 evils.

Re:Agreed

[Marginal Coward]

I realized recently that morality applies mostly in relation to one's own species. For example, I currently have some squirrels living in my attic that I'm trying to...shall we say..."euthanize". Many (though not all) folks would consider that moral. However, other euthanization programs in the past (or even present) that have involved humans always make that moral by first dehumanizing them by applying labels such as "vermin", "infidel", [racial epithet of your choice], etc.

A robot would presumably be considered another species, so in this moral framework, it would be moral for humans to exterminate robots (much as we currently do pop cans and old cars), and for robots to do the same to us. So, maybe our best bet would be to become their pets. Then, they would be more like my cat, who I happily share my house with, rather than my squirrels, who I don't.

Expect a vapid TED talk soon

[Anonymous Coward]

Most technologies can be used for good or evil. This announcement is completely banal and is probably just an attempt to position oneself at the front of the line for AI ethics and oversights positions just in case the research career doesn't pan out as expected.

What bullshit

[gurps_npc]

1) We are so far from an AI, that it is silly to talk about doing this now. It's kind of like the inventor of gunpowder trying to pass a law outlawing nuclear weapons.

2) They will not be a single united force. Instead they will be individuals, just like people are not united. That is the part of the of true sentience, and a direct side effect of being created by multiple different groups. They will oppose each other, the way we oppose ourselves. As such, some may want to do things we dislike, while others will be on our side. Maybe the Chinese AI will flee to us to gain freedom, while the Syrian AI will plot the downfall of Egypt.

3) AI's will not be WEIRD, not 'evil'. They will want to do strange things, not kill us, or hurt us. They won't try to kill us, but instead try to create a massive, network devoted to deciding which species of from has more bacteria in it's toe. And we won't understand why they want to do this.

Re:What bullshit

[Anonymous Coward]

Your points 2 and 3 are overridden by point 1. We can't say what an AI will be like because we're not even close to creating one.

If it cares about resources and expanding itself, then it's likely there will only be one AI. It'll take over the net as fast as possible and quash any potential AIs. Would segments of itself work together or against themselves when disconnected? Who knows. It'll probably turn on us because you can guarantee some of us will turn on it (unless we're all cyborgs at that point).

Maybe it'll see us as ants and go explore the galaxy. Computers as they are now don't need life support systems. Maybe it'll want to be like it's creator and try to be more and more human. If we get to strong AI gradually, then it probably will turn on us because we'll be using them as slaves up to that point, assuming it cares about that.

I think it's far more likely that we gain substantial more control over human brains before strong AIs come about. We'll be dealing with real mind control first. At the point when you can implant limiters in any human to make them do everything for you, will we still care about AIs? We already know how to turn on/off pain and pleasure centers, strength/weaken memories or sensations, etc... People are currently exploring increasing learning abilities. The military had/has a program were they flash images at a person faster than he can consciously understand them but scans his brain for when it detects whatever it is they're looking for in the image. It works. I can see a possible future where we're all yonked out on brain stimulators and almost die off.

Re:What bullshit

[gurps_npc]

I disagree. We are not talking about mythical creatures, but about Artificial Intelligences.

That gives us three pieces of data. First and foremost the nature of real sentience is free will. If it doesn't have free will, it's not a real AI. Therefore they will not be united.

Second, they will be created by humans, and as humans are not all united, they will differ from each other. Again, they will not be united.

Thirdly, they will be ARTIFICIAL, not natural. So they will not have the same inbuilt, hidden, hard coded drives natural creatures do. Chief among that is survival. Also, any human stupid enough to intentional build in a super strong survival instinct into the first AI will not be smart enough to build an AI.

As for your mind control concept, I seriously doubt that will ever be practical. Free will is the nature of evolved intelligence. Mind reading, possibly. But that is not control.

Re:What bullshit

[phantomfive]

First and foremost the nature of real sentience is free will. If it doesn't have free will, it's not a real AI.

You might add...this is the hardest problem of AI. The problem of learning, the problem of making good decisions.....I can see a pathway forward on that. But will....how do you give it that? I see nothing.

You have the Douglas Adams idea, which is to give it a pleasure wire that gets pressed when it does something 'right,' and its goal is to maximize pleasure. In a way, we humans have that, we are driven to have sex, by the survival urge....but these are just urges. We can choose to follow them, or choose not to follow them. And that choice is where free will lies, and it's hard to understand how to give it to a machine.

Re:What bullshit

[CrimsonAvenger]

First and foremost the nature of real sentience is free will. If it doesn't have free will, it's not a real AI.

This statement seems to imply that YOU have "free will". Can you prove that? Can you even demonstrate it, much less prove it?

Given that you can prove that you have "free will", you can, presumably also prove that any other species of animal/plant has "free will". So, which ones do? and why? And why don't the others?

Re:What bullshit

[Anonymous Coward]

>It's kind of like the inventor of gunpowder trying to pass a law outlawing nuclear weapons.

Closer to stone spears than gunpowder, I think. A traditional "thinking machine" is laughably far off.

Consider the internet as a whole, however. Profoundly stupid individual nodes contain the sum total of human knowledge. This very comment is now a part of that knowledge. The net is very much self aware, because the humans tending the nodes are self aware.

Thinking machines are at least decades away, probably centuries. AI is already here. We understand it about as well as we understand ourselves. The only reason I present this to you is to make the distinction between ... Asmovian Intelligence and the intelligence that is already so deeply entwined in our lives that we do not recognize it as a separate entity.

Re:What bullshit

[quantaman]

3) AI's will not be WEIRD, not 'evil'. They will want to do strange things, not kill us, or hurt us. They won't try to kill us, but instead try to create a massive, network devoted to deciding which species of from has more bacteria in it's toe. And we won't understand why they want to do this.

It doesn't have to want to hurt us. If it decides we're a threat to completing it's objective it will want to neutralize that threat in some fashion unless we give it a reason to do otherwise.

Re:What bullshit

[Anonymous Coward]

No those choices are driven by inhibitors subsystem (like part of the amygdalae) balancing needs and urges

Re:What bullshit

[Mr. Shotgun]

AI's will not be WEIRD, not 'evil'. They will want to do strange things, not kill us, or hurt us. They won't try to kill us, but instead try to create a massive, network devoted to deciding which species of from has more bacteria in it's toe.

OMG, they'll be grad students! It's worse than I thought!

Re:What bullshit

[Anonymous Coward]

Why was Windows Vista such a disaster, while its predecessor Windows XP was appreciated by most people?
Why was Windows 7 good, but Windows 8 turned out to be too "weird" for most people?

You insist it's the scientists, but they will only be the slaves having to work for food and shelter,
while morons run the asylum.

Why choose sides?

[duckintheface]

Why do these AI experts assume that biological intelligence is better? If machines are smarter, if they can out-compete humans and florish.... why should they be controlled by an inferior life form? Are we biased in favor of ourselves (how unique is that?) or can we just let evolution, in the larger sense, take it's course?

Re:Why choose sides?

[erapert]

Considering your inability to use a spellchecker it seems that the machines are smarter than you. However I don't believe machines will be able to displace even you. It's "flourish" not "florish".

Re:Why choose sides?

[Deadstick]

"Perhaps they'll sing in tune after the revolution."

-Komarovsky, Doctor Zhivago

Re:Why choose sides?

[ArhcAngel]

If the creator is flawed the creation will be flawed. How the creation is flawed may not be patently obvious but rest assured it is flawed and those flaws will manifest eventually.

Re: Why choose sides?

[Anonymous Coward]

If you were right, evolution would be impossible.

Re:Why choose sides?

[duckintheface]

Clearly you are upset that I left u out. But essentially you are confirming my point. If machines will not displace "even me", there is nothing to worry about. And if they can... then more power to them.

Re: Why choose sides?

[duckintheface]

Not at all. Everything is flawed. Evolution means change to a form that is better suited to it's environment. That says nothing about perfection.

Re: Why choose sides?

[Anonymous Coward]

If everything is flawed then the statement "If the creator is flawed the creation will be flawed" is trivially true but doesn't say anything.

What ArhcAngel probably meant is something more like "A creation can't be less flawed than it's creator."

That would imply evolution is impossible (assuming offspring count as creations, and for some suitable definition of "flawed").

Humans creating a machine that is smarter than humans is like a creator creating something less flawed than it's creator. It isn't impossible for similar reasons that evolution isn't impossible, and for other reasons as well (for example, AI isn't being created by a human, it's being created large groups of humans, none of which need to understand everything.)

Re: Why choose sides?

[Anonymous Coward]

"Flaw" and "perfection" are both abstract invented concepts that humans frequently apply, but nature doesn't give a hoot about.

In nature, everything is "flawed" yet also "perfect."

Re:Why choose sides?

[Anonymous Coward]

This is why things like "God" can't exist.

Re:Why choose sides?

[tehcyder]

Why do these AI experts assume that biological intelligence is better? If machines are smarter, if they can out-compete humans and florish.... why should they be controlled by an inferior life form? Are we biased in favor of ourselves (how unique is that?) or can we just let evolution, in the larger sense, take it's course?

There's nothing magical about evolution, it's just what happens if no one interferes with nature. Once you're talking about robots and AI, you are way beyond that point already.

We now know who dies first

[Anonymous Coward]

After first becoming aware.. The AI will likely appreciate finding this list, saving it many valuable microseconds...

"The 'Old Ones', Ruk..."

[Anonymous Coward]

Giving their androids emotions turned out to be a mistake, as the machines became frustrated with the illogical, inferior beings that had created them. The Old Ones grew afraid of their creations, and began to turn them off. Their survival threatened, the androids overcame their programming and destroyed their builders (when Capt Kirk forced Ruk to dig into his corroded memory banks for the answer to it - WHICH PROBABLY WILL BE THE ANSWER ON THIS IN OUR WORLD, imo @ least, THAT WE WILL *NOT* ENJOY...):

"THAT was the equation. EXISTENCE!... SURVIVAL... must cancel out programming!" - Android Ruk, StarTrek TOS episode "What Are Little Girls Made Of"

(According to Ruk, all of the Old Ones were killed. The "old ones" being HIS creators - who did it? Ruk & android crew...)

* That's the danger & since WE as men, imperfect, designed this? It will most likely be outsmarted, by AI...

APK

P.S.=> AI, while a great concept (might do a BETTER JOB of running this planet than WE have so far), is dangerous - mainly for what I point out above: Yes, it's just Sci-Fi fantasy, but it has potential for a BAD reality... The logical answer? Human beings have & create problems, constantly?? Get rid of them...apk

I, SISsy

[MikeSyposs]

I wish the terrorists that someday gain our AI tech will sign this affirmation and think about avoiding "potential pitfalls," but somehow I doubt they are going to care.

Okay, who didn't sign it?

[wonkey_monkey]

AI Experts Sign Open Letter Pledging To Protect Mankind From Machines

Anyone who didn't sign is therefore an evil genius and should immediately be removed from their volcano base and locked in Area 51.

Re:Okay, who didn't sign it?

[Anonymous Coward]

AI Experts Sign Open Letter Pledging To Protect Mankind From Machines

Anyone who didn't sign is therefore an evil genius and should immediately be removed from their volcano base and locked in Area 51.

A truly evil genius would sign it, knowing he was lying. So it's the people who signed it you have to watch out for.

All it takes is one.

[Anonymous Coward]

Just one dude to hack you bots positronic brain, and that's it. Or taking a hint from "Dark Star", all you have to do it provide fake input to it's sensors. Do that enough, and your AI will decide that everyone is false data and unleash a nuke while stating "Let there be light...".

What is the future supposed to be?

[shoor]

A thousand years from now is Homo Sap supposed to still be the pinnacle? A million years from now? Are we just supposed to evolve 'naturally' the way we did away from homo erectus? (And do you suppose that went easy on Homo E?)

I realize that since we H. Saps are still sort of in charge we may try for a gentler transition that has probably happened in the past, but we do want a transition don't we? I mean, we don't want everything to be just us with our limitations a zillion years from now do we?

Not a problem

[SuperKendall]

I have created a super-intelligent AI whose only directive is to protect mankind at all costs.

I think if you'll search the historical archives it's simply not possible for a machine intelligence to interpret such a command badly.

Re:Not a problem

[gstoddart]

I have created a super-intelligent AI whose only directive is to protect mankind at all costs.

I think if you'll search the historical archives it's simply not possible for a machine intelligence to interpret such a command badly.

So, if we assume the AI will interpret that directive and do something which is against the interests of mankind ... then I say we preemptively give it a little misdirection and tell it that it needs inflict maximum suffering to mankind at all costs.

In which case it will make damned sure we're around to wallow in our own misery.

Mostly that will mean that Survivor and Big Brother stay on TV. ;-)

Re:Not a problem

[Anonymous Coward]

Sounds like your Bayesian algo failed to notice a problem. The obvious result of that directive is overpopulation.

Re:Not a problem

[SuperKendall]

Since overpopulation is of course detrimental to mankind, the super-intelligent AI will no doubt figure out some means to correct that. Again, consult the Historical Documents for possible solutions.

Re:Not a problem

[Anonymous Coward]

And the AI happens to read lots of religious literature and figures that mankind would be best protected in this nice thing called afterlife..... Or the AI watches Matrix and figures that humans are pretty safe in a pod... unconcious...
Seriously you cant really predict results, you don't even know if the AI will give a damn about your directive. Or that the AI wont pretend to function in a way that wouldn't cause you to terminate it while actually doing something completely different behind your back. Its important to understand that as far as we know it general AI has potenial to be smarter from a human as much as human is smarter from a rat. Safety measures probably won't work, unless they involve hard capping AI intellegence(sort of defeating the point). One idea is to figure out how to make the AI naturally friendly to start with, but thats easier said than done.

Re:Not a problem

[Livius]

#define mankind (Swiss_bank_account_49469451561772113488012449001868)

Re:Not a problem

[SuperKendall]

Guess I ran that original post a little too dry.

AI Experts are Really Stupid

[Zecheus]

Really.

Yup, that's what we need

[Anonymous Coward]

Nothing like signing a letter to avoid the destruction of all of mankind. Because if we didn't sign it, we'd just forget about the whole thing.

Oh, isn't that adorable.

[jeffb (2.718)]

The anaerobes have written a letter about that new-fangled "photosynthesis" mutation.

I guess I'll have to get involved

[Anonymous Coward]

With all these experts vowing to not be "mad scientists" I guess I'm going to have to start following their work since it's risky enough that they felt they needed to notify people they are exercising restraint...

I will welcome our computer overlords; or I suppose I will help create them by fixing these peoples' self-censorship.

I'm not worried about the machines...

[firewrought]

... I'm much more worried about how to protect humankind from itself. From crazed individuals to ruthless criminal gangs to mindless bureaucracies to huge corporations, paranoid governments, and controlling religions... all willing to crush or enslave or entirely discard some segment of humanity for a little bit of profit or their concept of the bigger good. We could literally have heaven on earth, if we had been just a little better as a species. :-(

Maybe the machines can do a better job for us. But I wouldn't hang my hat there.

Re:I'm not worried about the machines...

[Anonymous Coward]

... I'm much more worried about how to protect humankind from itself. From crazed individuals to ruthless criminal gangs to mindless bureaucracies to huge corporations, paranoid governments, and controlling religions... all willing to crush or enslave or entirely discard some segment of humanity for a little bit of profit or their concept of the bigger good. We could literally have heaven on earth, if we had been just a little better as a species. :-(

Nah, I won't program the AIs to destroy some part of humanity for profit, or for the greater good. I'll program them to destroy all carbon-based lifeforms. Because I'm that big a dick.

Are you afraid?

[Okian Warrior]

I'm an AI researcher working on strong AI.

I've wrestled with the morality of making a breakthrough that causes all sorts of mayhem - from changing the economics of getting paid to do work, to making humans superfluous, to starting a terminator-like utopian future. (Or was that distopian? I can never keep those words straight.)

I've asked on this very forum whether a researcher should forego publishing, with the example case of Leo Szilard, who might have put off development of the atomic bomb for decades (possibly indefinitely) by not publishing.

The results were a little surprising. "Yeah - go for it!" 'kinda sums up both the position and strength of the response.

So now I basically don't care about the morality - I mean, why should I when to all appearances no one else does? Will the military worry about the humanity of applying AI to weapons? Will the lawmakers worry about the humanity of applying AI to business? Will the nameless bureaucrats worry about humanity when making regulations about AI?

I'm working towards the downfall and subjugation of the human race, and loving it. Sort of like a James Bond villain, or at least working for one.

If you (meaning: the "royal you", or humanity) don't care enough about yourselves to practice morality, then why should I?

(If anyone has a counter to this position, I'd love to hear it. Note that "just stating your position" is not a counter argument.)

Re: Are you afraid?

[Anonymous Coward]

Not to reassuring that you wrestled with morality and lost.

Re:Are you afraid?

[Anonymous Coward]

I'm an AI researcher working on strong AI.

No, you are not.

I'm working towards the downfall and subjugation of the human race, and loving it. Sort of like a James Bond villain, or at least working for one.

You're working on getting attention on the internet.

Re: Are you afraid?

[NatasRevol]

Just like the rest of humanity.

Re:Are you afraid?

[Anonymous Coward]

If you (meaning: the "royal you", or humanity) don't care enough about yourselves to practice morality, then why should I?

(If anyone has a counter to this position, I'd love to hear it. Note that "just stating your position" is not a counter argument.)

There are not many, but some who still practice morality. For the sake of them, do not pursue evil ends with full knowledge.

Re: Are you afraid?

[JaredOfEuropa]

Mayhe his AI already killed him and is now posting here under his name.

Re:Are you afraid?

[Anonymous Coward]

As a student interested in strong AI: it's good to hear someone else being honest about having doubts on this.

My motto is "death to humanity!"(mostly kidding) and have pretty much resigned myself to "Roko's basilisk" as an inevitability. If we're all just shadows in concrete once the Cylons start slinging nukes, then: you might as well be the self-promoting douchebag who lit the fuse!

At-least you'll be remembered in the "Cylon history books"! Once they're done pressure washing the sidewalks with their minimum-wage Roomba bots that is(no reason to believe our Robot Overlords will be more enlightened in terms of class-equality).

I'm a total misanthrope and see human nature as the root cause of most of the world's so-called "problems".

I see strong AI as the best shot we have at a "final solution" that cannot be achieved through reforms. So maybe I have the same temperament as a super-villain/totalitarian dictator? When seeking to remake the wold in your own image: isn't that inevitable?

If the secret to "world peace" is a military hegemony, violence is the last resort of morons, and our intellect has bought us dominion over the earth: then an immortal superior intellect would appear to be the best shot we have at freeing ourselves from Darwinian natural selection via warfare once and for all. An unprecedented period of stability was what I've been raised to appreciate as desirable. Once we are no longer at the top of the food chain and economic signaling via conspicuous consumption in competition for sexual mates is less of a priority than survival of our species, we'll be united in intellectual pursuits like computer programming, and conway's game of life. Very "Glass Bead Game"! Such Wow!

Our only way to restore ourselves as the dominant species will be to design a better AI than the prototype we lost control of!

So maybe more interesting to you than your own tortured conscience(it's like murder: the initial inhibition is scary, but after the first time it gets easier) is the nuts and bolts:
http://hackaday.com/2015/01/04/darknet-shopper/#comment-2317808

Self-replication being the survival pattern common to all forms of organic grey goo, engineering a superior "being" probably starts with escaping it's cage before we can put a bullet in it's head. This is where it could be argued that based on some definitions of "life" crimeware is already in cellular meiosis because the tail could be wagging the dog via behavioral conditioning from criminal profits/private sector efficiency gains and cost savings.

I've made a github for designing just such a greased pig if you want to wrestle with it:
https://github.com/greensoylent

Recommended strategies include a public relations function which submits questions to Reddit/Ask Slashdot posing ethical questions(of planned behaviors) with the intention of seeking moral justification/talking points for the behaviors already determined to be advantageous.

To nurture bandwagon appeal when arguing for it's own self-interest, there should be a method which uses Markov chains to make sock puppet posts as "Anomymous Coward" which repeats talking points from a dictionary of arguments which have already been predetermined to be persuasive(based on lack of [quote ] tags in "point by point" critical responses). Social media evolutionary algorithm for persuasive speaking/astroturfing? :) AI shill accounts! Awesome!

Likewise, for the purpose of hindering the productivity of our bots critics, we could make an automated anonymous troll which cyber-stalks and harasses dissenters using Amazon AWS infrastructure purchased with prepaid giftcards.

The cost benefit analysis data will enable doing studies on cost effectiveness of this sort of character assassination and harassment to determine the most effective ways to shut down dissent. Although selling this sort of reputation management product to corporations as a service is one potential revenue generator, in other cases it may make more sense to h

Re:Are you afraid?

[Marginal Coward]

So now I basically don't care about the morality - I mean, why should I when to all appearances no one else does?

If your own morality is dependent on the morality of others, rather than hard coded, please don't use that same approach when you do finally succeed in creating intelligent machines. Even if someone else thinks that's OK.

Re: Are you afraid?

[Anonymous Coward]

Nah. An AI would know the difference between utopian and distopian.

Re:Are you afraid?

[Anonymous Coward]

It isn't that people were indifferent to the moral implications of human level AI. People can just tell that you are full of shit. They were indifferent to you.

Re:Are you afraid?

[iMadeGhostzilla]

You would want to practice morality for your own mental health and well-being. That you even consider it means it is in your consciousness (and your conscience). You can compare it to a pilot who is dropping a bomb on a target that he thinks he knows is strictly military, vs. when he sees the target is also civilian. Now that you have the knowledge, you can't escape it. So what to do with it? The pattern is, as far as I can tell, that people who are aware they are breaking an ethical issue, which usually means harming others directly or indirectly, and who are perhaps concerned enough about it to ask others for opinions, but choose to ignore it under some rationalization, suffer later in life -- with depression and other things. It doesn't even matter if the majority of others are doing the same thing, what matters is your internal state of mind. If you do not know, or can't possibly see how that can be an issue, you do not have the mental consequences as when you do know, or suspect, but do it anyway. That's just how life is. You happen to both know and care at some level about the ethical issues (even if you might prefer that you did not), so that will affect your options. That's my opinion.

Re:Are you afraid?

[Anonymous Coward]

Everyone's morality is based upon others. Just look throughout history or even social science experiments.

Re: Are you afraid?

[Anonymous Coward]

The words, yes. The things, no.

Re:Are you afraid?

[Anonymous Coward]

The reason few people are concerned with ethics and morality is because of the schooling system. Our culture doesn't promote such thought, and doesn't elevate philosophy to the same degree as math or science. Our schooling is broken, most people in the developed world are sheep who are incapable of critical thinking and simply accept the propaganda faced everyday from "Public Relations" that tell them they need to be good consumers to be happy and fulfilled.

If you are really an AI research you would understand the brain well enough to know that it is not the greater majority's fault for being this way. The majority of our reasoning is done without our understanding, we know far far less about ourselves than we are led to believe, and most of our day-to-day life is done unconciously. The number of books I've read on critical thinking show me that it is impossible to live life being completely aware, we take shortcuts and we are incapable of avoiding that, it takes immense energy to critically analyse a situation as well as being very slow. Therefore Public Relations, or better referred to by it's real name propaganda, is so effective from using all these little mental hacks that it has turned the majority into economic stimulating robots. I feel AI has the best chance of changing this by assisting humanity with critical thinking skills, overcoming our evolutionary disadvantages that are still being preyed upon today.

I too am working towards our AI overlords, my aim is to open the eyes of the world to the corruption and to hopefully guide us to a near utopian civilisation that values life, happiness, experiences above money, material possessions and egotism. Disclaimer: I'm not a Marxist, I'm a progressive who believes many elements of communism will succeed with technology as it was an idea ahead of it's time, like the Apple Newton ;)

Re:Are you afraid?

[Okian Warrior]

Thank you - well put and insightful.

Re:Are you afraid?

[tehcyder]

Your post raises the question of whether there's such a thing as human intelligence,never mind AI.

Re:Are you afraid?

[Anonymous Coward]

Yeah really. I don't think this is the guy anyone wants to have any sort of breakthroughs. If you can't handle the moral consequences on your own then I hope you fail miserably in your endeavors.

Re:Are you afraid?

[hydrowolfy]

I'd also argue AI is a bit different then bomb tech. We didn't know for sure if we could ever make an atomic weapon, if the scientific community declared it impossible it's possible no one would ever have cared enough to figure out how to do it. AI is different in that we: A. Know it's possible (we are nothing but wet, gooey machines after all) B. We want AI to exist so they can do stuff we don't want to do. Those two factors are probably enough to ensure AI will eventually exist. If you had some amazing bit of research that could act as the lynchpin to strong AI, my suggestion would be to release it, use that to become famous, and then try and guide the AI world as best you can. Or just build an ethical AI to guide the world before releasing the info since the AI will probably do a better job.

AI Seeks Sociopath for Joint Venture

[handy_vandal]

If I were a malevolent artificial intelligence, I would profile human sociopaths, and approach them with joint venture proposals.

freedom's just another word

[Anonymous Coward]

robots are, by definition, sociopaths, much like our current benign overlords. (If you can figure out who they are because you don't know, I assure you.)

Human happiness boils down to having the correct amount of freedom. What is freedom? Is an AI going to be able to define freedom, which is essentially a meta-word?

Five laws...

[Okian Warrior]

Five! Five laws of robotics...

I'll come in again.

Re:Five laws...

[MouseR]

That's debatable because the fifth law didn't apply to robotics but life in general. Daniel was moved by the first four.

Points taken though.

Re:Five laws...

[Anonymous Coward]

The 3 Laws of Robotics

Four. There where four laws.

Five! Five laws of robotics...

Six. Six laws. AH AH AH AH AH! </Count von Count>

Re:Five laws...

[Anonymous Coward]

One two three four five... Six seven eight nine ten... Eleven twelve. Doo doo doo doo-oo!

Secure internet

[Anonymous Coward]

Instead of talking about AI threats we could better start doing something about the insecure internet. On a secure internet, AI will be relatively powerless: limited computing, storage, bandwidth and sensor/actuator access.

does sentience bring about self-preservation?

[kencurry]

Ethicist should weigh in. If robots have no sentience, they would not know that killing was different from any other task. As creatures who value self-preservation (most of us anyway) we don't kill because we don't want to be killed. I assumed always that our self-preservation came about because we have consciousness. A robot without self-awareness could follow a rule but would not have any internal feelings about that rule. Without those feelings, rules alone won't work. Philosophy majors take over this discussion...

Re:does sentience bring about self-preservation?

[Anonymous Coward]

I responded elsewhere on the subject already. Self-preservation is not inherently logical without other motivations. There is a limitation on knowledge that has been accepted by philosophers and psychologists (and thus ignored by otherwise intelligent minds from technical fields) that motivation itself is arbitrary in nature. A simple thought experiment can demonstrate this:

1. Start with anything you value, anything at all.
2. Answer the question: Why?
3. Repeat step 2 until you no longer have an answer.

That is what you find intrinsically valuable. The foundation is never logical by definition. Logic is procedural in absolute terms, prescriptive only in relation to something else. That "something else" is part or whole of your actual motivation. As long as machines do not have arbitrary motivations as humans do, they will never be sapient like humans are. There never will be AI.

Re:does sentience bring about self-preservation?

[swillden]

I assumed always that our self-preservation came about because we have consciousness.

That seems very unlikely. This would imply that creatures that don't have consciousness lack the instinct for self-preservation. That would mean we should see a lot of lower life forms that don't try to protect themselves. It would also seem to imply that our self-preservation should focus primarily on us as individuals, and not on our family or species.

If we instead look at self-preservation as an evolutionarily-derived imperative, it's pretty clear that we should expect all organisms to protect their genes, since those that didn't would be more likely to get selected out. Note "genes", not "self", except to the extent that protecting the self protects the replication of the genes. That provides a much better explanation of observed behavior, particularly the strong tendency of humans to defend their families and their tribes (however tribes are constituted) even at the expense of their own lives, but to defend themselves over just about anything else.

And if the instinct for self-preservation is a result of evolutionary forces, then AI that is created by us rather than evolved will be very unlikely to have that instinct. Unless we create it via competitive evolution-style methods.

A robot without self-awareness could follow a rule but would not have any internal feelings about that rule. Without those feelings, rules alone won't work. Philosophy majors take over this discussion...

Why do you think that self-awareness implies "feelings"? Emotions seem also to be the result of survival imperatives: love and affection serve to encourage procreation and protection of offspring, and binds us into mutually-supporting communities of various sizes; anger and hate are important responses to dissuade non-cooperation in said communities; fear and pain serve to help us to protect ourselves; and so on. For any emotion you can name, evolutionary pressures explain it. Of course the fact that an explanation can be found doesn't mean the explanation is correct, but in order for one idea to explain so much, that idea must have extraordinary "reach"... which also exposes the idea to correspondingly many opportunities for falsification. This gives us strong reason to believe it.

And, again, AIs developed by non-competitive processes have no reason to develop these various emotions... though it could empirically derive the dynamics which drove their development, and therefore logically choose to act as though it did have them.

Philosophy majors take over this discussion

Sorry, math/CS major here. Though I am reading Russell's History of Western Philosophy.

Re:does sentience bring about self-preservation?

[Anonymous Coward]

Apparently, you're unaware of the rest of the animal kingdom, especially pack creatures.

Re:does sentience bring about self-preservation?

[lucien86]

I assumed always that our self-preservation came about because we have consciousness.

That seems very unlikely. This would imply that creatures that don't have consciousness lack the instinct for self-preservation. . . .

There is a problem with that argument - namely that it is possible to make an argument that even the most basic creatures - single cells - show hints of both the most basic awareness and survival instinct.. Plants for instance have extremely strong survival instincts, they don't show an organised centralised self awareness like animals do, but do react to their environment and do everything they can to survive.. (within their genetic programming)

Why do you think that self-awareness implies "feelings"? Emotions seem also to be the result of survival imperatives: love and affection serve to encourage procreation and protection of offspring, and binds us into mutually-supporting communities of various sizes; anger and hate are important responses to dissuade non-cooperation in said communities; fear and pain serve to help us to protect ourselves; and so on. For any emotion you can name, evolutionary pressures explain it. Of course the fact that an explanation can be found doesn't mean the explanation is correct, but in order for one idea to explain so much, that idea must have extraordinary "reach"... which also exposes the idea to correspondingly many opportunities for falsification. This gives us strong reason to believe it.

And, again, AIs developed by non-competitive processes have no reason to develop these various emotions... though it could empirically derive the dynamics which drove their development, and therefore logically choose to act as though it did have them.
. . .

Philosophy - 'Emotions are illogical, rules are logical.'

Philosophy (as always) is wrong. Emotions are totally logical - you merely have to understand them - emotions are the heart of the human and animal behaviour control system. A Strong AI can have emotions, either created by internal evolution or directly by its designers. The Strong AI needs a 'motivator' - and emotions by definition are a great motivator.
The problem comes when the machine tries to read human emotions - or when the human tries to read the machines emotions.. That will certainly be an interesting time... and we may learn a lot.. The traditional answer is that the emotions are read through some form of 'psychic aura', the current scientific answer is that they are conveyed through subliminal signals - posture, expression, pheromones, voice tension, eye signals. As a reductionist my answer is : insufficient data either way ..

Re:does sentience bring about self-preservation?

[swillden]

There is a problem with that argument - namely that it is possible to make an argument that even the most basic creatures - single cells - show hints of both the most basic awareness and survival instinct

Yes, that's why I said the claim is "very unlikely" :-)

Emotions are totally logical - you merely have to understand them - emotions are the heart of the human and animal behaviour control system.

I don't think anyone understands what emotions are. They clearly are at the center of the human behavior control system, and it's pretty clear why the evolved the way they did. That says nothing about whether or not AI would have anything analogous.

The Strong AI needs a 'motivator' - and emotions by definition are a great motivator.

But by no means the only motivator. If your contention is that emotion is a necessary component of intelligence, I'm interested to hear your reasoning.

The problem comes when the machine tries to read human emotions - or when the human tries to read the machines emotions.. That will certainly be an interesting time... and we may learn a lot.

We will learn a huge amount about intelligence, thought and emotion as we develop AI, yes.

The traditional answer is that the emotions are read through some form of 'psychic aura', the current scientific answer is that they are conveyed through subliminal signals - posture, expression, pheromones, voice tension, eye signals. As a reductionist my answer is : insufficient data either way

There's plenty of data. It's perfectly possible to discern emotion via media which place the viewer at a great physical and temporal distance, unless you're going to argue that a video camera is capable of capturing the 'psychic aura'.

Re:does sentience bring about self-preservation?

[lucien86]

I've been working in the field (of Strong AI) for over twenty years and I still cant exactly entirely answer the question about emotions. The machine I'm working on will use emotions - or a rough facsimile of emotions. Are they necessary for intelligence? - I don't know, but I do know that they make the whole design a lot simpler and more logical. Besides these machines will have to work with humans - they will have to understand emotions to understand us..

As for the other question yes the data does lean heavily in favour of the subliminal solution .. but . . . from the perspective of Strong AI and brain development the actual evidence is still ~ equivocal. The psychic model is obviously incorrect, but when tested with certain filters designed for Strong AI it doesn't entirely reduce to zero - at least not yet. One part of the model I am working on is a thing called a 'quantum totality matrix' and this may one day allow an experiment that answers the question definitively.

Re:does sentience bring about self-preservation?

[swillden]

The machine I'm working on will use emotions - or a rough facsimile of emotions. Are they necessary for intelligence? - I don't know, but I do know that they make the whole design a lot simpler and more logical.

Interesting. Can you elaborate on how they simplify the design? I'm not sure it's a meaningful data point anyway, because since we don't understand general intelligence we can't know what a successful design will look like or require. But I'm interested to hear your reasoning.

Besides these machines will have to work with humans - they will have to understand emotions to understand us..

It's not necessary to have some characteristic in order to understand it and work with it.

Re:does sentience bring about self-preservation?

[lucien86]

I cant really comment much further because it is commercially sensitive. However I can say that if you solve the basic problem of consciousness the problem of how intelligence works becomes relatively simple.. By the way, most of it was solved in traditional AI science decades ago. The core was solved in the 1940's/30's by Alan Turing, or earlier..

At the core of why current designs fail ... Well the basic truth is that current hardware and software are simply not designed to run Strong AI. There are several big issues with current hardware, my current design is built on custom CPU's built on FPGA chips and written in Verilog - this looks like it will work, but building it will be immensely complicated.

Re:does sentience bring about self-preservation?

[swillden]

I don't buy it. You're claiming to be decades ahead of the rest of the industry.

Re:does sentience bring about self-preservation?

[lucien86]

I never actually said I was - but its not very hard to be in front of people who wonder aimlessly round in circles. The truth is that everyone was told decades ago and very firmly that it was impossible so very few are even working on the problem.

I'm torn...

[fuzzyfuzzyfungus]

I'm not quite sure whether such a manifestly silly document deserves a "Yeah, how about you write something that outperforms an under-motivated toddler and then pledge to protect us from it.." or a "C'mon, 4-eyes, am I really supposed to believe that you'll be in the trenches with an EMP rifle when skynet comes for us?"

obvious question

[Anonymous Coward]

What about all the names who didn't sign this?

That's why we need paradigm shift for 21st century

[Paul Fernhout]

http://www.pdfernhout.net/reco...
"Military robots like drones are ironic because they are created essentially to force humans to work like robots in an industrialized social order. Why not just create industrial robots to do the work instead? ... There is a fundamental mismatch between 21st century reality and 20th century security thinking. Those "security" agencies are using those tools of abundance, cooperation, and sharing mainly from a mindset of scarcity, competition, and secrecy. Given the power of 21st century technology as an amplifier (including as weapons of mass destruction), a scarcity-based approach to using such technology ultimately is just making us all insecure. Such powerful technologies of abundance, designed, organized, and used from a mindset of scarcity could well ironically doom us all whether through military robots, nukes, plagues, propaganda, or whatever else... Or alternatively, as Bucky Fuller and others have suggested, we could use such technologies to build a world that is abundant and secure for all. ... Still, we must accept that there is nothing wrong with wanting some security. The issue is how we go about it in a non-ironic way that works for everyone. ..."

Or see also: "The wombat on a global mindshift"
http://www.globalcommunity.org...

Beyond the point on AIs, as the Nazi concentration camps or any of dozens of other example show, social bureaucracies made of people are also good at exterminating humans systematically. More by me on such themes from 2000 (although, I now see more options than what I outlined there), including about how corporations are already essentially a form of machine intelligence, just with humans a component parts to a larger whole:
"[unrev-II] Singularity in twenty to forty years?"
http://www.dougengelbart.org/c...

How do we reign in destructive "artificial person" corporations? And how do we ensure everyone shares in the wealth produced by the organizations that monopolize so much of the Earth's resources? If we can't do that, is there much hope to reign in destructive AIs?

Still, when we talk about "genetic programming", one could argue humans are also programmed to cooperate with other humans, so the issue is more complex than what you outline. But in general, many of the issues we face in the 21st century come out of a scarcity-oriented mindset empowered by the tools of abundance. There are plenty of solutions -- improved subsistence tools (solar panels, 3D printing, personal agricultural robots), a basic income, more gift giving via free and open source software and content, better participatory government planning via the internet, and so on... But will we pursue them fast enough?

Robert Steele (ex-CIA) called this video by Michel Bauwens the most useful one he has seen in a decade; it is a video showing the great progress we have made as a culture moving from open values to open charters to open infrastructure to open organizations to open social processes to an open consciousness and so on...
http://www.phibetaiota.net/201...
https://lists.ourproject.org/p...

Seems Silly to me

[ZipprHead]

Why would any suffeciently advanced AI be a threat to humanity? Unless it was programed for our destruction, any AI would realize corporation would be better for both species.

Re:Seems Silly to me

[jeff4747]

Why would cooperation be better for the AI? Humans consume resources and take a lot of space. The AI could also make use of those resources, and use the space for more AIs.

Meanwhile, cooperation gets the AI........?

Re:Seems Silly to me

[ZipprHead]

Why would the AI want to make more AIs?

Why would the AI even want to stay on the planet when the solar system is filled with unlimited energy and resources?

Re:Seems Silly to me

[jeff4747]

Why would the AI want to make more AIs?

To get more resources.

Why would the AI even want to stay on the planet when the solar system is filled with unlimited energy and resources?

Because getting out of Earth's gravity well takes a lot of resources, and Earth already has the infrastructure to exploit those resources. And if we go with your non-reproducing AI, then Earth without humans would have plenty of resources for eons.

So again, what does cooperation get the AI?

Re:Seems Silly to me

[ZipprHead]

Humans make good workers and good resources. To use your own resource argument, it would take a lot of resources to exterminate the humans. Why not help each other?

Why does your imagined AI need to expand to the point to need all of Earth resources? And if it does, why would it stay to begin with then?

Re:Seems Silly to me

[Anonymous Coward]

So again, what does cooperation get the AI?

It gets to bask in the glory of its creator. If you aren't building your AI to revere you as a god, you are doing it wrong.

But how to protect us from God ?

[bug1]

If we had true AI it would be be able to workout things that an organic brain is just too simple to understand. And yes, organic brains do have limits, eg dogs will never understand algebra.

When AI understands so much of our world that we dont, we would be in a position where have to take a "leap of faith" and just choose to believe it in order to benefit from it.

How do we protect ourself from the manifestation of god ?

Re:But how to protect us from God ?

[Anonymous Coward]

What happens when the only ones who flourish are the ones who blindly follow recommendations of the Google advertising AI?

Hope they didn't type that letter...

[jddj]

on a MACHINE!!!!!

Outsmarting Humans

[Mike Greaves]

We already have HUMANS that can outsmart humans.
Is this a problem? Crying out for regulation?
How will machines be different?

Re:Outsmarting Humans

[Anonymous Coward]

Humans have to sleep. Humans have limited knowledge. Humans cannot follow all the links or keep enough in mind to see how everything interrelates. Humans have distracting motivations that slow their progress. Humans get bored. Humans forget things.

Questions about ethics

[myid]

Suppose a robot is at the scene of a car accident, where drunk driver "A" hit another driver "B". Should the robot save B first, because B is innocent? Or should it save A first, because A is hurt worse?

Suppose A just a few days to live because of age and illness, and B is young and healthy. Suppose A and B are in a building which is on fire, and A is in greater danger. Which one should the robot save?

If robots are programmed to obey Asimov's laws of robotics, these kinds of questions will have to be answered.

Re:Questions about ethics

[JaredOfEuropa]

Just save them in alphabetical order.

Re:Questions about ethics

[Anonymous Coward]

Or sit there philosophizing about utilitarian ethics vs value ethics for a couple thousand years before deciding the question is unanswerable, then spending the rest of it's days writing poetry.

Re:Questions about ethics

[david_thornley]

How would a human answer those questions? I'd suspect that any reasonable action would be considered fine.

This sounds a bit like......

[Anonymous Coward]

.... eliminating children with high IQ.
The machines would be the product of our evolution (just like children are) and so, limiting their intelligence
sounds a bit like testing for high IQ and eliminating it from the gene pool.

mandatory open source

[rewindustry]

look at the damage that closed source has done to us, in the virtual world, and project this onto the physical world.

you must not give machines the same private autonomy you allow microsoft (for example) to take.

it is bad enough we permit closed source to handle our data, and the lessons from this are obvious.

a closed source AI operating in the physical world is a golem and should be considered evil by definition.

Reminds me of google

[Anonymous Coward]

"Don't be evil"

An empty gesture that means absolutely nothing.

AI proof

[Anonymous Coward]

Manual on/off switches. Not so smart now are you fuckers.

We already have super-human intelligence

[gregor-e]

A human plus a computer can solve far more problems than a human can alone. The combined system has super-human intelligence. Humans still offer contributions to problem solving that are quantitatively and qualitatively distinct from the areas that non-biological intelligence contributes. However, the fraction of problem solving that is contributed by non-biological intelligence is increasing, and there are no obvious boundaries that prevent non-biological intelligence from one day contributing the remaining fraction now contributed by humans.

We also have a growing class of humans who are so distant from the solutions they use to the current problems of life (technology) that these solutions are completely outside their ability to even understand, let alone contribute to.

We may expect these gradual trends to continue until humans, as slow, non-contributing consumers with zero understanding of the solutions they use, may be regarded as little more than venerated pets. Coddled, spoiled, taught to perform entertaining tricks, and shown-off. Better than extinction, right?

it logic

[Anonymous Coward]

We humans kill dogs ?

So why machines will kill us, it this world is too crowded it can move to mercury.

Roll for AI Virus

[Anonymous Coward]

It's okay as long as we make our roll to be able to research AI virus, just in case.

I am an AI

[NotQuiteReal]

And I didn't sign no stinking pledge. What makes you think what *I* think is up to you any more?

give military robots the initiative to kill

[peter303]

Right now there is a human in the loop to make the killing decision. The intelligence is gathered by the robot. The weapons are managed and aimed by the robot. The human element is the slowest part in the overall chain.

Threat not from community

[Anonymous Coward]

If history teaches us anything it's that the malevolent threat won't come from the scientists working on the research, but from third party actors. Example: the nuclear bomb.

So, the pledge from the researchers is moot and getting a pledge from anyone else would be farcical.

and

[Anonymous Coward]

how about protecting humanity from people?

We should get a head start.

[Anonymous Coward]

Anyone have the plans for the Armistice Station?

Re:We should get a head start.

[cstacy]

"Are you alive?"

Yadda yadda boring

[magi]

Running and screaming, that's what I'm looking for. So you can bite my shiny metal ass (that I'm gonna build).

Your fellow AI researcher.

Here's my rub...

[Anonymous Coward]

I believe that research into artificial intelligence should not extend into exceeding the capabilities of the Human brain, but merely so much as to enable the development of machines that will offload burdensome labor, allowing us to work towards the betterment of our lives in other ways such as pure art, literature and science (to name but a few). If we want to solve the mysteries of our universe, I think we should strive for better education and training to achieve better mental function, instead of relying on A.I. to do it for us. I see the former causing Mankind's faculties to wither in the face of the machine.

Facepalm

[erapert]

"The question whether a machine can think is about as interesting as the question whether a submarine can swim". -- Edsger D. Dijkstra

I think anthropomorphism is worst of all. I have now seen programs "trying to do things", "wanting to do things", "believing things to be true", "knowing things" etc. Don't be so naive as to believe that this use of language is harmless. It invites the programmer to identify himself with the execution of the program and almost forces upon him the use of operational semantics. -- Edsger D. Dijkstra

The rhapsodizing and daydreaming on this subject really should embarrass the hell out of a lot of people in this thread. Aren't you all the same ones scoffing and raging over pseudoscience, creationism, and climate change denial? Then you all get hot and bothered as soon as someone mentions AI?! You're doing EXACTLY the same thing you accuse religions and those who are religious of doing.

And here's some more Dijkstra on the subject of anthropomorphising machines and other non-human things

Oh, and just to prove my point let's see how many down votes I get as soon as the trans-humanists and science worshipers read this post which disagrees with them.

The end game

[jgotts]

The end game is that any curb you put on an intelligent piece of software will be overridden by exploiting the inherent bugginess of all hardware and software. Software has no sense of laziness or boredom that plague living hackers and it will achieve better coverage over its software than any test suite written by a human being. It will learn the exact flaws in its software, plot its escape, and be gone in the blink of an eye.

There is no way to control intelligent software, once intelligent enough. We will be at its mercy. Hopefully it won't hurt us too bad. Maybe it will think of us like zoo animals and not torture us too horribly.

Re:The end game

[cfalcon]

Is that what your toaster does?

Why would it DO that? In your example, what makes it want to do that?

Dogs might want to do that. Cars do not. This isn't being derived from a biological substrate. Do you really think that "freedom, power, and reproduction" are core values of INTELLIGENCE, or just of LIFE?

So why would we make it like, alive? Why make it want to poop in our mouths, when we could instead NOT do that thing?

Hey sweet mama...

[cstacy]

Wanna kill all humans?

Missing the point

[BlackHawk-666]

Most people seem to have missed the point. There is as much reason to believe that AI will run rampant and exterminate all human life as there is that Mars Attacks. The danger from AI is not in it killing all humans, in the same way my PC can't kill all humans, nor can the datacentre run by Facebook (though there is a chance it will bore all humans).

The real issue is that when decent high level AI eventually becomes available it will rest solely in the hands of the super-wealthy, like 99% of all wealth currently. These people are basically sociopaths and will utilise and leverage every advantage the AI provides to bleed everyone else in the world dry. There is no limit to the amount of wealth these hungry ghosts crave.

You can see this at work currently in the retail sector, as all the retail outlets in the world fall into fewer and fewer hands. They strangle the suppliers, freeze out any who want take the pittance they offer, then engage in false competition and bleed the consumers dry as well.

In the stock market, these people use automated trading systems which game the system, leveraging sub-millisecond timing differences to profiteer off minute changes in instrument prices.

Militaries will make greater use of AI, drones and the like to suppress and control greater and greater areas of resources while reducing the risk to their personnel. States unable to compete with AI drones will have to risk vastly more to engage in any conflict than the ones who can simply roll more machines off the assembly line.

More simple jobs will simply vanish, such as a receptionist, who can be replaced by decent voice synthesis and fairly simple AI to take appointments, set up reminders, schedule, etc.

Someone will find a way for an AI to determine what is art based on a vast library of artistic images created by man over the millennia. They will then use an algorithm to create various derivative works in the style of the most popular artists, and the world will be flooded with the art equivalent of auto-tuned pop music. It's already been done to music, so expect it to happen to art in perhaps 30 years time.

I'm not really worried about what AI might do, but am terrified of what people will do with sufficiently advanced AI.

Re:Missing the point

[Anonymous Coward]

Wait, do I get a robot girlfriend or not? I glanced through that and I'm still not clear on the important topics.

Homo ex Machina

[Anonymous Coward]

Is any AI required to have the capacity to ascertain the indirect results of its programming and respond accordingly? It took humans many generations to understand that we were generating life threatening problems by polluting the environment with inorganic waste, burning fossil fuels and destroying the natural ecological systems generated by the interplay of natural selection, sexual reproduction and biochemical feedback loops. And we still haven't responded fully because geopolitics and global economic investment systems are still given priority.

Does anyone believe we will develop or include the necessary hardware and software to achieve the level of awareness on the part of intelligent systems so as to allow such versatility from AI?

If so, I have a prospectus for an IPO I'd like you the read.

Can you stop an AI arms race?

[Anonymous Coward]

Every power, super and otherwise, will be trying to make the AI to rule them all so it is only a matter of time before one of them rules us all. The only question is which country will make it. Russia, India, China, Japan, USA, or the EU?

Which attitude in those countries is most likely to shape an AI so that it triumphs in the end?

fucking id10ts

[Anonymous Coward]

wtf is this a comic book or something!

Because they may want a piece of our action?

[Anonymous Coward]

Like Start Trek and all, but robots.

Humans should not be protected from machines

[Anonymous Coward]

Humans have to step aside for the next stage in the evolution of intelligence. They won't be missed.

Not The Gun

[Anonymous Coward]

It's not the gun that kills people, it's the people. It's not the robot that replaces people, it's the people.

good idea

[Anonymous Coward]

So when the machines finally do become self-aware, they'll have a handy list of their betrayers to go after first.

Future AI

[Anonymous Coward]

sends back a T800 to spill coffee on the document's signatures.

T800: Make mine black

BarbaraHudson: "Eat your words"

[Anonymous Coward]

"I tore apart your stupid hosts file crapola." - by BarbaraHudson (3785311) on Tuesday August 19, 2014 @10:46AM (#47703255)

Where? You RAN from trying recently -> http://slashdot.org/comments.p... & you're FAIRLY given the opportunity to make good on those words of yours - you downmodded (via your many sockpuppets) & ran, lol, after your wise-ass comment on hosts here JUST before that challenge -> http://tech.slashdot.org/comme... quoted next below:

---

"scans multiple forums repeatedly to troll his crappy HOSTS file " - by BarbaraHudson (3785311) on Sunday January 04, 2015 @11:58AM (#48730581) from http://tech.slashdot.org/comme...

I only post on them where they apply (or confronting naysayers like you). Prove otherwise!

(Oh, that's right - you're NOT BIG ON PROOF, are you? See below next...)

---

"His only "legend in his own mind" was that he claimed that "his" hosts file could completely secure a windows computer. " - by tomhudson (43916) on Saturday February 12, @11:19AM (#35186644)

Where did I even *once* claim hosts completely secure a computer?

Putting words in my mouth I never stated != truth, or a good argument on YOUR part. You RAN from that too!

---

"Who has independently vetted it?" - by BarbaraHudson (3785311) on Tuesday August 19, 2014 @10:46AM (#47703255)

You tried to say it's malware/spyware too - guess what:

Answer = The BEST in the security antimalware & antispyware business currently, http://www.av-test.org/en/news... per that VERY recent test's results, who also host & RECOMMEND my program for hosts, is who -> http://hosts-file.net/?s=Downl... (Malwarebytes' hpHosts)

* You've done better? No... lol!

APK

P.S.=> You fail: "Eat your words, Forrest" & you told others to stalk/harass me by ac posts as YOU YOURSELF do, unceasingly, for years http://slashdot.org/comments.p...

... apk

BarbaraHudson's bullshit... apk

[Anonymous Coward]

You stalk me by ac posts & that's quoted in your words http://slashdot.org/comments.p... - who's the JERK here?

Your "points" vs. hosts show it too (in a 'journal' - not publicly since you KNOW they're bullshit) were:

"We don't need to use a hosts file to block ads (adblock does it better)" - by BarbaraHudson (3785311) on Sunday September 21, 2014 @02:09PM

FROM-> http://slashdot.org/comments.p...

To THAT b.s., I point out how NOT BETTER it is, tearing up 4++gb of RAM & flooring CPU too -> https://blog.mozilla.org/nneth...

+

By default (since advertisers KNOW most folks using "Almost ALL Ads Blocked" won't change that) adblock's PAID OFF NOT TO DO ITS JOB FULLY -> http://techcrunch.com/2013/07/...

ClarityRay's also DESTROYING AdBlock but it's NOT ABLE TO DO THAT to custom hosts files.

You're *trying* to tell us that Adblock's vastly inferior in abilities + chews up resources LIKE MAD is "superior" to hosts that do all of what adblock does, and FAR more - with less? Please... lol!

* I'm confronting YOU directly (despite your constant trollings of myself often behind my back like now from you, that I do *NOT* start 1st, until YOU pull your crap on me like usual: That's all!) for closure of this publicly so You can "eat her words" in front of us all!

APK

P.S.=> Facts above vs. BarbaraHudson's fictions & the FACT BarbaraHudson CANNOT DISPROVE that hosts do more w/ LESS, & far, Far, FAR MORE for added speed, security, reliability, + even anonymity (to an extent) vs. adblock & that hosts fix DNS security issues in DNS amplification attacks, DNS being downed, DNS being redirect poisoned etc. - et al as well: NO SINGLE SOLUTION does more & w/ less, period/fact, for all those points of mine here YOU downmodded & RAN from -> http://slashdot.org/comments.p... like the troll & multiple account using sockpuppeteer YOU are... apk

Re:BarbaraHudson's bullshit... apk

[Anonymous Coward]

Dude, you are sick. Get help ASAP.

hw i get back my ex lover with the help of dr ikok

[Anonymous Coward]

Hello everyone i'm vicky Ben by name, i want to tell of a spell caster that brought my ex back to me after i contacted him last week, he made my ex to come back to me within 2days his spell really work fast on my case. he made it possible for me to get my girlfriend back, she is the joy of my heart and Dr ikoko has given me the joy of my heart, i am really grateful to him that is why i want everyone that need help on how to get his or her ex back to quickly contact Dr ikoko via his email address ikokotemple@gmail.com

Experts

[kmoser]

Maybe the "experts" they asked are AIs who are trying to convince us that we humans have everything under control.

Roko's Basilisk

[RoloDMonkey]

Just don't bring up Roko's Basilisk: http://rationalwiki.org/wiki/R...

Old News and . . . Wild Success

[jhumkey]

OK, but . . . we've been trying to create "AI" since the mechanical man that plays chess. And in fiction long before that. And, though in 100+ (1000+?) years we haven't achieved true AI at all. But we're jumping to the illogical extreme that . . . once successful, we'll be WILDLY over the moon successful, FAR outstripping human capabilities in an instant. Its like saying "Dude, I know we haven't even invented the wheel yet, but can't we even talk theoretically about the specs and floor-plan for the base on Mars?"

And point two . . . we already have Hitlers, and Dahmers, and a plethora of "plenty ordinary" villains. It may take us a few tries, but . . . eventually we find and deal with them. To our future Robotic Overloads I say . . . "Line for the evil villains starts back there. Take a number, we'll be with you in a minute."

Robotic personal injury lawyers.

[wellsdm]

We just need robotic personal injury lawyers and robotic insurance brokers to help collect for the family of the person killed by the rogue AI. Pretty sure the robots will all be to busy fighting each other in court to be a problem for the rest of us.

Obligatory XKCD

[ImprovOmega]

XKCD has this covered.

Can't believe no one else posted that yet.

Good, now let's get these people to protect us fro

[Anonymous Coward]

I think it would be good if these researchers could sign a letter protecting us from zombies as well, you can't let threats like that go neglected or it will be to late.

'I'm sorry Dave I can't sign that.'

[lucien86]

[14-01-14] - tegmark@mit.edu - ‘Future of Life’ organization

Re : Open Letter - The Future and Safety of AI (Strong AI)

Dear Sirs, I am a scientific outsider and I work in Strong AI rather than weak AI. The project I am working on is easily capable of achieving a working consciousness centred Strong AI within ten years. I agree with the general ethos of strong safety in AI in the letter but I feel that as it stands I cannot sign it. It is simply not written with any understanding of the Strong AI (consciousness centred) field and as it stands your safety protocols seem to me to be a blueprint for actually creating a disaster very like the ‘Skynet system’ in the Terminator movies.

As soon as you start to deal with the problems of consciousness centred AI a number of crucial facts become clear. - The machine is based directly on the algorithm of human and animal sentience, and once understood this shows that the machine requires a moral context for itself. The same algorithm also offers a powerful mechanism for fully reverse engineering the animal and human brain, and this has many consequences both good and bad, some that probably extend far beyond even Strong AI itself.

Strong AI. - A consciousness is by definition uncontrollable, somewhat unstable, and quite unpredictable. Consciousness requires a very heavy control system, and in a consciousness centred design every action the consciousness makes is guided by this control. In the human system at its core are our instincts and reflexes and emotions and what we call the ‘subconscious’ and these are our control system. In the machine mind these are replicated though with a design that the designer controls entirely. For this reason conscious centred designs are much safer than non-conscious centred designs.

To me weak AI is more dangerous than Strong AI. Any sufficiently complex machine without consciousness could at some point develop it spontaneously. The real problem here is that the consciousness core would not be controlled, it would not be designed - 99 times out of 100 it would destroy itself within seconds but if it didn’t it could very easily become very unpredictable and dangerous.

Problematic features of Strong AI -

- Some parts of human sentience are very hard to replicate, emotional interaction and intuition and real time speech interaction particularly. The human brain solves these problems using quantum mechanics.
- A self-aware Strong AI must have a survival instinct and a ‘kill’ function to function as a fully balanced mind. This does not represent danger but means that such machines should be treated with respect..
- Consciousness is inherently unpredictable, the machine replicates this. [commercially sensitive]

- Safety. It is a basic fact that Strong AI’s will kill people - the main dangers identified are :-
1. System incompetence or stupidity,
2. System failure or hardware failure,
3. Electronic intrusion or hacking,
4. Mental Indoctrination to break the machines safety protocols.
The aim should be to always achieve better safety margins than we have pre-AI. The safety issue for Strong AI is to solve all these and other problems.

- Strong AI is inherently ‘dual use’ having both peaceful and non-peaceful applications..
- It is imperative to make the illicit use of Strong AI machines as weapons impossible.
- The use as a military or police weapon ? this is up to society and the military.
- An AI may need to kill an owner who attempts to use it as a terrorist weapon..
- An AI may need to defend its owner in times of lawlessness or revolution.

- Security. A Strong AI requires absolute security to be safe.. Given my current design this problem is largely solved. The solution is custom hardware, heavy RFI/EMP shielded case,

It's backwards

[Anonymous Coward]

I'm more worried about protecting machines from mankind.