Slashdot Mirror
A Big Problem With AI: Even Its Creators Can't Explain How It Works (technologyreview.com)
Last year an experimental vehicle, developed by researchers at the chip maker Nvidia was unlike anything demonstrated by Google, Tesla, or General Motors. The car didn't follow a single instruction provided by an engineer or programmer. Instead, it relied entirely on an algorithm that had taught itself to drive by watching a human do it. Getting a car to drive this way was an impressive feat. But it's also a bit unsettling, since it isn't completely clear how the car makes its decisions, argues an article on MIT Technology Review. From the article: The mysterious mind of this vehicle points to a looming issue with artificial intelligence. The car's underlying AI technology, known as deep learning, has proved very powerful at solving problems in recent years, and it has been widely deployed for tasks like image captioning, voice recognition, and language translation. There is now hope that the same techniques will be able to diagnose deadly diseases, make million-dollar trading decisions, and do countless other things to transform whole industries. But this won't happen -- or shouldn't happen -- unless we find ways of making techniques like deep learning more understandable to their creators and accountable to their users. Otherwise it will be hard to predict when failures might occur -- and it's inevitable they will. That's one reason Nvidia's car is still experimental.
How do humans work? Not knowing how genius humans arrive at their conclusions doesn't seem to be a huge stumbling block for society to use their output.
How many scientists really know how "creativity" works?
I just don't have any faith in a system that is not fully understood.
But intelligence and consciousness are not fully understood, and may not even be understandable. And I say that not to invoke some kind of mysticism, but because our decision making processes are lots of overlapping heuristics that are selected by yet other fuzzy heuristics. We have this expectation from sci-fi that a general purpose AI is going to be just like us except way faster and always right, but an awful lot of our intelligent behavior relies on making the best guess at the time with incomplete information. Rustling in bushes -> maybe a tiger -> run -> oh it was just a rabbit. Heuristics work until they don't.
It may be that an AI must be fallible, because to err is (to think like a) human. But forgiveness only extends to humans. When the human account representative at your bank mishears you you politely repeat yourself. When the automated system mishears you you curse all machines and demand to speak to a "real person." The real person may not be much better but it doesn't make you as angry when they mishear you. With automobile pilots we tolerate faulty humans whose decision-making processes we absolutely don't understand such that car crashes don't even make the news, but every car AI pilot fender bender will "raise deep questions about the suitability of robots to drive cars."
We don't have a state-run media we have a media-run state.
Uh, it's simple .... No, it's not easy. But it's absolutely knowable and testable.
I agree that it's completely doable, but the poster I replied to was stating that the programmer who wrote the algorithm must understand how it's making decisions and that only the less skilled maintenance coders would be confused. That's simply not true. I know people who could write a neural net from a reasonable spec but doing the steps you described above would blow their minds. I'd also argue that a NN with even a few layers of nodes can get complex fast enough that what you're proposing would result in a document the size of a novel and still not capture all the nuances.
I really appreciate your point that
Getting any useful info out of that will be an issue though. You may find out that somewhere deep in your neural net it's looking for a seemingly random pattern of contrast or checking against some strange distance/angle.
If the net is using some seemingly random pattern that's where you can get some bizarre (to human thinking) failures. We tend to understand when something goes wrong in a way we can comprehend. If the seemingly random pattern the computer finds happens to call a slightly obscured "stop sign" a "no u-turn" sign that would be incomprehensible to a human, but might make perfect sense to the NN.
This all isn't to say that you can't reduce the odds of this sort of problem to such a small number that it's meaningless especially in comparison to human error. Still, when crap like this happens it makes the news and gets blown all out of proportion, so expect "the sky is falling" stories to follow any uncertainty AI behavior.