Hilarious (and Terrifying?) Ways Algorithms Have Outsmarted Their Creators (popularmechanics.com)

"Robot brains will challenge the fundamental assumptions of how we humans do things," argues Popular Mechanics, noting that age-old truism "that computers will always do literally, exactly what you tell them to." A paper recently published to ArXiv highlights just a handful of incredible and slightly terrifying ways that algorithms think... An AI project which pit programs against each other in games of five-in-a-row Tic-Tac-Toe on an infinitely expansive board surfaced the extremely successful method of requesting moves involving extremely long memory addresses which would crash the opponent's computer and award a win by default...

These amusing stories also reflect the potential for evolutionary algorithms or neural networks to stumble upon solutions to problems that are outside-the-box in dangerous ways. They're a funnier version of the classic AI nightmare where computers tasked with creating peace on Earth decide the most efficient solution is to exterminate the human race. The solution, the paper suggests, is not fear but careful experimentation.
The paper (available as a free download) contains 27 anecdotes, which its authors describe as a "crowd-sourced product of researchers in the fields of artificial life and evolutionary computation. Popular Science adds that "the most amusing examples are clearly ones where algorithms abused bugs in their simulations -- essentially glitches in the Matrix that gave them superpowers."

Comments

A well asked question ...

[petes_PoV]

... is already half-answered

And most of the situations described in the reference article describe poorly framed problems. I understand that it is supposed to be a jokey, light, non-serious, read. However it illustrates the problem with people asking the wrong question, or making incorrect assumptions.

Many years ago the multi-billion $$$$ utility company I was working for had a team from [ name removed to protect the stupid ] a well-known consultancy outfit. One of their conclusions was that some of our servers were running with too much idle time - under utilised in their opinion. All they had done was collect %idle data from sar (Unix systems from Sun, IBM and HP). and their junior idiot looked at that and decided it was a "problem"

When I was asked about this by the CIO and the "consultants", my response was that I could easily increase the utilitisation figure to whatever the CIO desired, or that the consultants recommended - how high would he like it to be? Since he knew me, and saw the smile, he saw the trap. I explained that "idle" time and user response time were tightly linked: that reducing one would increase the other. This was news to the "consultants" once I explained the maths and Queuing Theory behind it.

Re:A well asked question ...

[tlhIngan]

... is already half-answered

And most of the situations described in the reference article describe poorly framed problems. I understand that it is supposed to be a jokey, light, non-serious, read. However it illustrates the problem with people asking the wrong question, or making incorrect assumptions.

  Many years ago the multi-billion $$$$ utility company I was working for had a team from [ name removed to protect the stupid ] a well-known consultancy outfit. One of their conclusions was that some of our servers were running with too much idle time - under utilised in their opinion. All they had done was collect %idle data from sar (Unix systems from Sun, IBM and HP). and their junior idiot looked at that and decided it was a "problem"

When I was asked about this by the CIO and the "consultants", my response was that I could easily increase the utilitisation figure to whatever the CIO desired, or that the consultants recommended - how high would he like it to be? Since he knew me, and saw the smile, he saw the trap. I explained that "idle" time and user response time were tightly linked: that reducing one would increase the other. This was news to the "consultants" once I explained the maths and Queuing Theory behind it.

Or more like AI simply did the real human thing and figured out the weakness in the measurement system in use and exploited it.

In other words, the AI simply did what a human would eventually figure out and do - cheat the system.

All the examples in there are basically how the AI figured out a way of cheating the calculations, something humans would figure out as well.

And the reason we have to cheat is often the "measurement" item cannot be measured. One popular goal setting thing in use is "SMART" (specific, measurable, achievable, realistic, time-bound), but there are a lot of things that can translate into that easily. For example, productivity. Since time immemorial, people have wanted a way to measure programmer productivity, and the most obvious measurement was well, lines of code. Which did nothing but bloat the codebase up with needless lines of code. Then people tried bug counts ("I'm going to write myself a new Ferrari"' from Dilbert). And to this end, there's no way to measure "productivity" than by a proxy measure (proxy measure is something me can measure that hopefully relates to the actual quantity we wish we could measure directly), we implement those measurements. But then people find shortcuts - ways to increase the thing the proxy measures, but without increasing actual expended effort.

Take another example - say my goal is to make my blog more popular. Well, how do I measure popularity? Visitors per month? Comments per month? A little sensational click-bait bit of fake news will boost both numbers easily enough. But did I accomplish the goal, or did I simply game the system?

All AI has done is exposed these limitations in our proxy measurements and simply exploited them. In short, AI simply figured out the limitations of the system and exploited them.

Outsmarting Mother Nature

[mentil]

If an evolutionary algorithm is pitted against real life, and 'outsmarts' it, that's one measure of evolutionary progress. The real issue is the same as in 'teaching to the test', or even the 'kobayashi maru solution': the metrics are gamed once the one being tested realizes what they are, and then the metrics no longer hold meaning.
Replace 'metrics' with 'simulation parameters' and it's the same thing. The simulation has to be as intelligent as the uncontrolled agents operating inside of it, or else these types of things will happen. Self-modifying simulations perhaps?

Computers still do exactly what we ask

[Anonymous Coward]

We just need to know how to ask them to do what we really want.
If the simulations are inaccurate representations of the problems we want to solve, the answers given by the AI will be inaccurate.
Hitchhiker's Guide to the Galaxy already touched on this problem.
If you don't understand the question, the answer will be meaningless.

Re:Stupid local minima

[uvatbc]

But a lot of the time it does something stupid.

Much like evolution: The algorithms that survive are useful.

Re:Stupid local minima

[Kjella]

So it sent itself off the virtual edge of the simulation area, ending the run and minimizing it's negative score as best as possible. By accident someone created a suicidal bot, yay! (...) But a lot of the time it does something stupid.

Who did something "stupid"? The bot achieved its goal, but the programmed goal completely failed to achieve the intended goal. This is basically "The code did what I said, not what I meant" taken to a new level. The problem is that you can't easily inspect a neural network's logic in human terms the way you trace through code, it's more like another person. I think this is a cat, you think this is cat, the AI thinks this is a cat but we can't exactly quantify exactly what makes this a cat or non-cat which means the model can break down unexpectedly in ways you can't possibly predict, like you show it a one-eyed cat and suddenly the AI thinks it's a cyclops. And that's going to be a problem as we start relying on AI, like this self driving car thinks you're a pedestrian until one day for some inexplicable reason you don't qualify.