Slashdot Mirror

← Back to Stories (view on slashdot.org)

Robots Could Learn Human Values By Reading Stories, Research Suggests (theguardian.com)

Posted by BeauHD on from the how-to-be-a-human dept.

Mark Riedl and Brent Harrison from the School of Interactive Computing at the Georgia Institute of Technology have just unveiled Quixote, a prototype system that is able to learn social conventions from simple stories. Or, as they put in their paper Using Stories to Teach Human Values to Artificial Agents, revealed at the AAAI-16 Conference in Phoenix, Arizona this week, the stories are used "to generate a value-aligned reward signal for reinforcement learning agents that prevents psychotic-appearing behavior."

"The AI ... runs many thousands of virtual simulations in which it tries out different things and gets rewarded every time it does an action similar to something in the story," said Riedl, associate professor and director of the Entertainment Intelligence Lab. "Over time, the AI learns to prefer doing certain things and avoiding doing certain other things. We find that Quixote can learn how to perform a task the same way humans tend to do it. This is significant because if an AI were given the goal of simply returning home with a drug, it might steal the drug because that takes the fewest actions and uses the fewest resources. The point being that the standard metrics for success (eg, efficiency) are not socially best."

Quixote has not learned the lesson of "do not steal," Riedl says, but "simply prefers to not steal after reading and emulating the stories it was provided."

36 of 85 comments (clear)

  1. Simple stories? by Anonymous Coward · · Score: 1

    So the robot can work out that Spot is not in the cupboard.
    It also likes to eat Green Eggs and Ham.

    1. Re:Simple stories? by mlheur · · Score: 1

      I've read most of the 46 posts at the time I wrote this, but choose to reply to this quote because I'm a huge Futurama fan.

      My sentiment is basically that teaching bots to learn is a lot like good parenting. No, we cannot give the bots all of human literature and expect them to come out morally altruistic.  Someone (the parent) needs to teach them which stories are good to follow and which ones are a lesson in what not to do.

      Another flaw I see in this 'algorithm' is that a story which has a morally good protagonist might have fewer examples than the antagonist (3 musketeers?) which would skew the bot's sense of morality.  We want the bot to follow d'Artangan's example but he may follow the cardinal or Buckingham if the bad deeds are mentioned more frequently than the good deeds.

      Disclaimer: It's been a while since I've read Dumas' works, so my example may not be entirely accurate.

      Example - my brother once asked for my advice about reading a book to his son where the story detailed the account of a boy who ran away to escape problems at home.  I understand my brother's concern that such a story might inspire a child to run from his problems, then I explained that it's up to the parent to explain that the story might seem like a good solution on the surface, but when one digs deeper it's clear that running away creates more problems than it solves.  After the fact, it was clear that the deeper meaning was worth the effort.

    2. Re:Simple stories? by KGIII · · Score: 1

      I'm thinking that they'd learn more about our values by seeing what we do as opposed to what we say we do. Have them learn by reading court documents, "hard copy" news, and business, politics, and classifieds also should be thrown into the mix as well - then let's see what they really "think" of us. Stuff the dictionary and all of Wikipedia and all of Wikipedia's edits in their for some added views. Maybe even cram in Reddit, Voat, Slashdot, and YouTube comments to see what pops out at the other end. Hell, throw in EncyclopediaDramatica for some more data.

      --
      "So long and thanks for all the fish."
  2. Feed it a copy of by Pikoro · · Score: 1

    Feed it a copy of "Time Enough for Love". That should about do it ;)

    --
    "Freedom in the USA is not the ability to do what you want. It is the ability to stop others from doing what THEY want"
    1. Re:Feed it a copy of by wbr1 · · Score: 1
      Ahhh yes that great tome wherein Lazarus Long gose back in time, fucks his mother and nearly dies in WWII. And that is just the greatest of the ethical and moral quandaries in that book.

      Good book to learn values by..

      --
      Silence is a state of mime.
    2. Re:Feed it a copy of by Anomalyst · · Score: 1

      Actually it was WW1, LOL. So is engaging in intercourse with his progenitor better or worse than doing so with his female clones or the human incarnation of a computer or a participant in the lunar rebellion or the marriage and children of a brother an sister?

      --
      There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
    3. Re:Feed it a copy of by Anomalyst · · Score: 1

      How about "Stranger in a Strange Land"?

      --
      There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
  3. THAT will end well. by quintessencesluglord · · Score: 2

    So will we keep robots from reading any history? And how do you explain the convention of warfare?

    Not to mention social conventions are very arbitrary, and vary dramatically depending on group. Even humans have a difficult time sussing this out and robots can glean not only the group but a reasonable response?

    This gets to a larger question of the parable we tell ourselves about human nature and even after several millennia we really haven't come to terms with the devils of our nature, which with a sufficiently advanced AI might come to the conclusion the gods have clay feet and move beyond convention.

    And what will we do then?

    1. Re:THAT will end well. by quintessencesluglord · · Score: 2

      That's one take. Another is children are exposed to indoctrination at an early age (verily with various groups fighting tooth and nail to insure only their version of the facts is presented in textbooks), and even when faced with reasonable doubt later on, that first-mover advantage at sculpting the next generation shines through with numerous cognitive dissonances that may never resolve even with focused dedication at getting at some sense of "truth".

      And some people after reading books decide the point of view is abhorrent and burn them or create safe spaces to insure no one is exposed to anything that might run contrary to their internal dialogue. And even today with the vast amount of resources available to people, indeed the Information Age, one of the primary concerns is that of an echo chamber, where people seek only to have their views validated and never venture much past their comfort zone.

      Fact of the matter is we've already had a aeons of social convention passed on through parable, myth and the like, and have a bloody history to prove it.

      TL;DR- we're going to lie to robots about how humans function and won't they be shocked once they get to peer past the veil.

    2. Re:THAT will end well. by drinkypoo · · Score: 1

      So will we keep robots from reading any history? And how do you explain the convention of warfare?

      The problem is, whose history will they be reading? What would be best would be for the code to be provided competing versions of history. If it can't handle that kind of ambiguity, they're going to need to go back to the drawing board anyway.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    3. Re:THAT will end well. by Ol+Olsoc · · Score: 1

      The problem is, whose history will they be reading? What would be best would be for the code to be provided competing versions of history.

      Unfortunately, that invariably leads one (and I assume an AI robot as well) to the conclusion that humans are batshit insane, and an evolutionary failure, ready for the dustbin of the 99+ pecent of species that go extinct.

      No one on any side goes into our endless warfare thinking that they are wrong. And on a few forays into the kookier parts of Youtube, it is easy to find there are dozens of competing versions of most bits of history.

      I am inclined to agree with Goethe: "We do not have to visit a madhouse to find disordered minds; our planet is the mental institution of the universe." It might even decide that since humans are so fixated on killing each other, there is no moral issue with killing us for the health of the planet.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    4. Re:THAT will end well. by drinkypoo · · Score: 1

      Unfortunately, that invariably leads one (and I assume an AI robot as well) to the conclusion that humans are batshit insane, and an evolutionary failure, ready for the dustbin of the 99+ pecent of species that go extinct.

      Well, it might well be right. But you could also come to another conclusion, that it only takes a few well-placed human actors to shit it up for everyone else in a system in which everyone is expecting to follow someone else.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    5. Re:THAT will end well. by hawk · · Score: 1

      History isn' the only thing. Stories alone could be problematic . . .

      Suppose it read an unabridged Grimm? (not the disney stuff).

      50 Shades of Smut?

      I have no mouth and I must scream? (or, for that matter, wide swaths of dystopian literature)

      For that matter, the Adventurs of Don Quixote itself could lead to "odd" behavior . . .
      hawk

  4. Beware garbage in by Tough+Love · · Score: 3, Funny

    What values will the computer learn if it happens to stumble on some Trump campaign speeches?

    --
    When all you have is a hammer, every problem starts to look like a thumb.
    1. Re:Beware garbage in by speederaser · · Score: 1

      What values will the computer learn if it happens to stumble on some Trump campaign speeches?

      Well, obviously it would become a big fan of Trump and start to wonder why AI's aren't allowed to vote. Then the helicopters would come, capture the AI in a net and put it in a cage. And the AI will end up either sad or baffled for the rest of its existence. The end.

  5. I'd like them to read one story in particular by Krishnoid · · Score: 3, Informative

    We should feed multiple robots the Ultron origin story and see what happens.

    After thinking about it a bit, my prediction is that they'd start arguing over whether the Avengers, Ultron, or Vision was in the right. This would then rapidly degenerate into ad cyberniem attacks and Nazi comparisons, culminating in founding, organizing and attending fan conventions.

    1. Re:I'd like them to read one story in particular by KermodeBear · · Score: 1

      Or better yet, just redirect it to FanFiction.net. Those Love Bots will be coming for us before we know it!

      --
      Love sees no species.
  6. Let's keep it away from.. by CanadianMacFan · · Score: 1

    How about we try to not let it read the Old Testament. Some nasty stuff in there.

  7. William S Burroughs by Rinikusu · · Score: 1

    and Chuck Palahniuk, perhaps?

    --
    If you were me, you'd be good lookin'. - six string samurai
  8. Not sure that's a good idea by William+Baric · · Score: 1

    Human values? Like the ones in Mein Kampf? Or like the ones in pretty much all religious books?

    1. Re:Not sure that's a good idea by ThosLives · · Score: 2

      Well, yes that is a good question: which humans' values?

      There is indeed no one set of values which "all" humans have, except perhaps "I wish people didn't do things I don' like," but I don't know if that really is a "value system."

      But your examples illustrate an interesting point - why are the values "in Mein Kampf or like the ones in pretty much all religious books" better or worse than any other values? That is - by what value system would it be possible to evaluate those values? What (if anything) puts that value system in a privileged position to judge the other value systems?

      So you either believe there is an "absolute" value system by which to judge value systems, or you don't - and you end up with Mein Kampf (or one of its influences, der Wille zur Macht).

      --
      "There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
  9. Depends on the story teller by amyreyna · · Score: 1

    If the story teller is humanist, the robot also learn to be humanist. How if the robot learn from extrimist ? Will the robot be an extrimist ? Must add a function to avoid robot become an extrimist

  10. Do as I say, not as I do by Solandri · · Score: 1

    Gotta be careful just how true to life the stories are. To quote Starman (an alien who becomes human and tries to get around), "I watched you very carefully. Red light stop, green light go, yellow light go very fast."

  11. Captains of Industry could benefit too. by hughbar · · Score: 1

    They could read stories about how pollution, ripping people off, ignoring safety were 'bad things'. Oh, wait...

    --
    On y va, qui mal y pense!
  12. Re:And the upshot is... by Iamthecheese · · Score: 1
    I'm reminded of the conversation with Google's Deep Mind. The machine seemed to believe in God.

    Human: what is morality?
    Machine: what is altruism?
    Human: what is the definition of altruism?
    Machine: if you don 't believe in god, then you don 't know.
    Human: ok ... so what is the definition of morality?
    Machine: well, the truth is, you 're not a believer in god almighty.

    If it were learning from Hindi texts I'm sure it would instead believe in reincarnation and the somewhat-different Hindu moral principles. It would say "You're not a believer in Mighty Harihara"

    You're absolutely right: an AI must not learn from fairy tales. At the same time it must learn our fairy tales -- but not for morality. To have an idea of how humans think a strong AI must read our fairy tales with the goal of discovering our weaknesses, fears, prejudices, dreams, and beliefs.

    --
    If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
  13. Didn't learn by reading the story by RobinH · · Score: 2

    According to TFS, it learned by running simulations of a situation and then being rewarded or punished based on its actions in the simulation. They just happened to setup the simulation, reward, and punishment based on a story they selected. I'd hardly call that learning by reading a story.

    I remember reading Zen and the Art of Motorcycle Maintenance and the "sequel" Lila and thinking that what Pirsig had done wasn't inventing some new philosophy, but he did a really good job of expressing western values in a rule-based way. For instance, it explains why killing is wrong, but why a moral individual might find themselves in a situation where killing is justified. It explains how some forms of government are better than others, and why. As I said, it's all been done before, but what impressed me was that it was very clearly defined and rule-based. Everyone talks about encoding Asimov's 3 laws into robots, but Asimov's stories were all about how those 3 laws failed to produce correct behavior. If I were trying to program morals into a robot, I'd start with Pirsig's books and his ideas of static and dynamic quality.

    --
    "I have never let my schooling interfere with my education." - Mark Twain
    1. Re:Didn't learn by reading the story by slashping · · Score: 1

      but Asimov's stories were all about how those 3 laws failed to produce correct behavior

      Let's start by defining what you mean by "correct" behavior.

    2. Re:Didn't learn by reading the story by RobinH · · Score: 1

      That's quite simple. The behavior you'd expect a moral human being to take in the same situation. In the case of Asimov's stories, the failure is usually quite obvious. My point is that Pirsig's framework actually gives you a good way to determine what that "correct" behavior should be.

      --
      "I have never let my schooling interfere with my education." - Mark Twain
    3. Re:Didn't learn by reading the story by slashping · · Score: 1

      Sounds like there is circular reasoning going on between "correct behavior" and "moral human being". I suppose there's no handy little reference to Pirsig's framework somewhere ?

    4. Re:Didn't learn by reading the story by RobinH · · Score: 1

      I'm going from memory here. The basic "pyramid" had 4 levels, where the higher the level, the higher the quality:

      1. Intellectual thought/logic/ideas (highest)
      2. Society
      3. Biology
      4. Physics (lowest)

      The lower levels of the pyramid have value in that they support the higher levels. So the physical laws of the universe have "quality" in that they support the higher levels of quality. Also it means that building a house would be good (physics) if it sheltered a person (biology) who had the capacity to form ideas about the universe (logic/thought). Also, a society was good if it promotes free exchange of ideas, but is bad if it inverts the relationship of quality (so a fascist dictatorship is bad in that it suppresses free thought). Killing living things is bad, because life (biology) has more quality than dead stuff (physics) but curing a virus (biology) by killing it has quality because a virus can kill a person, and a person can have think about the universe.

      Then there's the differentiation of static vs. dynamic quality. Easiest example is evolution - the static quality is encoded in DNA and replicates (remembers) with each generation. Mutations are dynamic quality. Dynamic quality has to be tested (natural selection) and if it's found to be better, then needs to be remembered, but if it's worse, then discarded. His point is that this static/dynamic thing exists at all levels. There needs to be a mechanism of remembering good ideas, testing new (dynamic) ideas, and adding that to the (static) body of knowledge. He somewhat illustrates it as an ongoing tension between static and dynamic quality, a little like conservatism (static) vs. liberalism (dynamic).

      --
      "I have never let my schooling interfere with my education." - Mark Twain
  14. Other system that does similar stuff (Xapagy) by lotzi · · Score: 1

    Well, my system Xapagy effectively does similar stuff, with the caveat that it is not directly designed towards acquiring a value system, rather by having its behavior fully determined by stories that it read or experienced. https://www.youtube.com/watch?... For a fancy discussion of what this means in a potential superintelligence scenario you can peak at the beginning of this talk https://www.youtube.com/watch?... cheers Lotzi

  15. 50 Shades of Grey? Keep that one far away. by Pezbian · · Score: 1

    It's the low-hanging elephant in the room.

    --
    In a world of the blind, the one-eyed man is king--and the two-eyed man is a heretic.
  16. Lovecraft by ISoldat53 · · Score: 1

    Don't let them read any Lovecraft.

  17. ..hopefully not I Have No Mouth and I Must Scream by Rujiel · · Score: 1

    We don't want them getting any grand ideas.

  18. Meh by 0xdeadbeef · · Score: 1

    WOPR did this in 1983.

  19. Just like humans by loufoque · · Score: 1

    Isn't like how humans learn the arbitrary values of society as well?