Slashdot Mirror
Artificial Intelligence Pioneer Says We Need To Start Over (axios.com)
Steve LeVine, writing for Axios: In 1986, Geoffrey Hinton co-authored a paper that, four decades later, is central to the explosion of artificial intelligence. But Hinton says his breakthrough method should be dispensed with, and a new path to AI found. Speaking with Axios on the sidelines of an AI conference in Toronto on Wednesday, Hinton, a professor emeritus at the University of Toronto and a Google researcher, said he is now "deeply suspicious" of back-propagation, the workhorse method that underlies most of the advances we are seeing in the AI field today, including the capacity to sort through photos and talk to Siri. "My view is throw it all away and start again," he said. Other scientists at the conference said back-propagation still has a core role in AI's future. But Hinton said that, to push materially ahead, entirely new methods will probably have to be invented. "Max Planck said, 'Science progresses one funeral at a time.' The future depends on some graduate student who is deeply suspicious of everything I have said."
Expert systems aren't AI, and pattern-matching algorithms aren't AI. AI is something that can creatively solve problems based on unreliable inputs and abstracting specific experience to general cases.
The problem there is we don't even understand how that works in theory, so modeling and developing an actually AI based on that model is impressively difficult.
Personally, I think we'll get there (understanding intelligence) faster by trying to replicate a mammalian brain in silicon that we will trying to bash out new algorithms.
Also however important back-propagation is, it is hardly the entire foundation of AI. From my perspective AI is proceeding apace. There are many AI methods. Yes some core algorithms should be reexamined, as should anything in science or industry. We see some stuff that seems to lag in how much improvement we expected (general intelligence), and yet others that are leaping ahead of where we thought they would be like machine learning and pattern recognition. Eventually all the threads will start to come together, but progress will remain hard to predict.
Letter To Iran
The half-assed junk they keep trotting out to us and over-hyping is a dead end. We won't create machine intelligences that actually 'think' the way we're doing it, and if we're going to have anything that actually understands us in any significant way, that's what we really need.
Likely he is not right either, because AI beyond statistical classification ("weak AI") may well be impossible, but trying new things is at the core of actual research. Although other approaches have been used in other fields and have failed to produce any hint of intelligence as well. For example automated theorem proving found that it cannot really be used to _find_ theorems, because the universe is a bit too small and short-lived to build the machinery for that. It is a very good tools in verifying tools though that humans have found, even if humans need to help it along with that too.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
The future of AI is a dirty word "stereotyping".
The brain works by making associations, and then drawing stereotypes from them. Every time I've seen a dog or hooded man in a dark alley, it has attacked me. I stereotype dogs and hooded men in dark alleys as being scary and run from them. But then one day, I meet a green hooded man with a bow in the alley, and he saves me from the dog. I have to 'learn' by reshaping my stereotype to include men in green hoods.
Stereotypes get a bad name due to people the refuse to update or rely on bad ones, but they are actually a very useful tool humans have developed to deal with the world.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
They are not going to develop an AI unless they can say what "intelligence" really is. And that's a problem about which thousands of years of philosophy, psychology, and general inquiry has thus far failed to yield any meaningful and objective results. - Unless, of course, you're prepared to say that a 4-function calculator is intelligent in the same sense that you may think you are.
I might add, that God alone knows what we really are. And I doubt very much that AI research is going to yield an abstraction which is able to represent a decision-making process (much less a functional implementation).
Backpropogation is a form of supervised heuristic learning where you have to know the desired output and so it works backwards. In that context it's about perfect. We don't have any algorithmic techniques in an unsupervised learning context that are as good. Expectation maximization and blind signal separation algorithms all generally suck balls. The goal is unsupervised learning that works as efficiently as backpropogation. I suspect this is what he is saying but since this article and his language aren't technical or sufficiently detailed that's guess work on my part. For what it does there is nothing about backpropogation that needs improving, but it only works in specific use cases and it is not AI. It's just a "gradient descent optimization algorithm" (had to google the formal name for what it is).
Algorithm development at this point is just obscenely hard and is going to be obscenely rarely seen. The easy stuff has all been done and the stuff that is still being worked on may quite literally be impossible.
AI is a joke. There has been no real progress in AI since the 60s. What you see now is parlor tricks and a byproduct of Moores Law. Now that Moores Law is over, we need to find some other way to do computing. We will never have AI with digital computing.
The AI community needs to be much more cautious and circumspect. They have been promising the sky, and otherwise hyping things, for decades now, and, as result, they have become something of a laughing stock in academic circles. And do not say it is the press - luminaries like Minsky and others couldn't wait to come out with ever more outlandish forecasts, that were then just disseminated by the press. The final straw is when these days they are still trying to sell ridiculous gimmicks like Alexa, Google Home, Siri, etc. as AI wonders, when the only wonderful thing is how notably inept and limited in their capabilities they are: good for grins and giggles, party games, but little more. Stop the nonsense.
Neural nets using back propagation will likely remain a valuable tool forever, just like Newtonian mechanics. Will they be the only go to solution for all similar AI learning going forward? Of course not, they already aren't. When we do achieve strong AI it will likely be from a system incorporating thousands of different algorithms, of which Dr. Hinton's contributions will be just one.
Tell me why this is important to me?
"four decades later" would be 2026.
I've been working in Optimization research for years myself, and what we currently call A.I. is such an incredibly limited subset of what is possible. It is still useful, and a great marketing success for what Maths can do, but completely over-hyped and over-focused on neural nets and biology-based nomenclature.
we may be into the fourth decade since 1986 but it's been 31 years since not 40+.
he must be a "fart-official-in-smelly-gents"
If biology is the working model we have why are we not pursuing intelligence genetically? Is a modified animal intelligent enough to enslave any different that a robot with the same mental capabilities?
Consider a nueron will not fire 1 time in 10. To simulate forgetting.
The future may be table-oriented AI (TOAI).
It uses tools and/or conventions more typical of a regular office and thus allows AI problems to be split up and analyzed in a modular team-oriented fashion. Tables are easier to relate to than traditional neural nets (without a lot of training, at least). TOAI allows compartmentalizing AI tasks to distribute to staff (tasks, sub-tasks, etc.), and encourages a kit-oriented approach (modularization).
For example, you may have 3 sub-teams: 1) pattern/test makers, 2) results examiners, and 3) coordinators.
The first group focuses on making the individual tests (patterns or "factors"), the second group focuses on determining which tests are most or least useful for given situations, and the 3rd group be the coordinators who split up the problems and/or AI decision flow between groups and/or template sets (filters).
Table-ized A.I.
Deep learning and other related machine learning techniques are proving very useful for a wide range of tasks. We don't need to "start over" to advance useful machine learning techniques.
Hinton seems to mean to get "strong AI". Yes, I read TFA, but the strength of Axios articles is that they are very short, but that is also their weakness. Very little is actually said in TFA.
We are a long, long way from anything that emulates a natural neural system at any level.
Consider Caenorhabditis elegans. Every cell in this simple worm has been mapped, also the development of every cell from a single cell has been mapped (male worms have 1031 cells). We know every cell in its nervous system (there are 302), and every cell that each cell is connected to, and we know the type of connections for all. What's more we have completely sequenced its genome. We know more about this little multi-cell organism than any other multi-cell animal on the planet.
Since we know every cell in its nervous system, and every connection between every cell, we must be able to emulate this worm's "brain"! Heck we must be able to "upload" the worm's brain to a computer! Right? Right?
No.
We are still working on understanding the functioning and capabilities of a single neuron in its brain. That has proven so complex as to defy characterization thus far. We are essentially nowhere in understanding how this 302 cell brain works despite decades of effort.
Meanwhile Kurzweil has changed his prediction of "when computers will have human-level intelligence" from 2020 to 2029. I guess believing it was going to happen in the next 26 and a half months was cutting it a little too close. I have been reading about his predictions about AI for a couple of decades now and have yet to see any explanation of how he imagines this is going to happen - other than his expectations about hardware capabilities, and that there is still an unspecified "software issue" that needs to be solved. Indeed.
Second class citizen of the New Gilded Age
I'm not ready for 2026 yet!
Downmodding is the refuge of the weak. Don't downmod, make a better argument!
Thanks to Demis Hassabis.
Why Google's DeepMind Is Clueless About How Best to Achieve AGI
Back propagation is used setup a translation between features and results with the least cost. The problem is some features have more importance than other features, this is where the optimization in the learning process can be done. During the learning process if features that have more importance are given higher importance then the learning will be faster and require less resources. This is where cluster analysis comes in, by optimizing the clusters to achieve the desired results self learning can be achieved.
Hassabis believes that sensory learning in the cortex is supervised and based on backpropagation. Not even wrong. Is this the man who is going to solve AGI?
From his recent Cell paper, Neuroscience-Inspired Artificial Intelligence:
A different class of local learning rule has been shown to allow hierarchical supervised networks to generate high-level invariances characteristic of biological systems, including mirror-symmetric tuning to physically symmetric stimuli, such as faces (Leibo et al., 2017). Taken together, recent AI research offers the promise of discovering mechanisms by which the brain may implement algorithms with the functionality of backpropagation. Moreover, these developments illustrate the potential for synergistic interactions between AI and neuroscience: research aimed to develop biologically plausible forms of backpropagation have also been motivated by the search for alternative learning algorithms.
What?
Just imagine what the human mind's distributed representation of the "intelligence" concept would look like. Clever animate entities (and most associations therewith) are way off in their own private corner of vector space compared to just about everything else.
When the gap is this large, the enormous void in between somehow becomes a non-object (to superficial cognition) and so people just begin to presume that we need to jump the gap, rather than slowly filling the gap in.
It's almost like the travelling moon illusion when you're driving in a car and the moon is low in the sky, off to the side (which children find amazing, but adults have learned to ignore).
I was thinking about the sun this morning and about relative illumination at different latitudes. The correct physical model is parallel rays, which immediately suggests that for a perfect sphere, the poles get no direct radiation at all during equinox, the eternal kiss of sunrise=sunset.
Then I looked outside through the window, and realized that the human brain—which knows the sun is far away—still doesn't think it's as far away as the earth is wide (very wide, if you believe in a flat earth model) or even a few multiples (but it's actually thousands), and so the intuition from our eyes never says parallel rays.
We've been nibbling away at the giant AI void quite successfully, but the travelling moon illusion still makes us think we need to jump.
The reason we keep reclassifying our victories as "not really AI" is because we know for a moon fact that the void never actually changes size. But it does, and it has, and it will continue to shrink, and I really don't think we're going to spring generalized intelligence all at once out of scary clown box.
First we must learn to perceive the void as a continuum of many way points, mapped out by many generations of technical improvement, like Vancouver and Cook or Lewis and Clark.
For me, recent results with LSTMs have made the void seem just a little bit smaller than it was before. I'm now at the very beginning of an ability to perceive the moon as being at a great, yet finite distance.
_____
With something so thoroughly hived off in its own corner of distributed-representation hyperspace as intelligence, what's to define, anyway? Definitions are street signs erected in conurbia, which one resorts to after Toronto and Hamilton and Niagara Falls have all become built-structure indistinguishable as you skirt the horseshoe.
There are many conurbations in distributed-representation vector space where definitions are the last gasp at forestalling cognitive Gangs of New York. Definitions are less important under open skies of Boise, Idaho or Butte, Montana; even less important still when you've wandered out into the green grid-lines of the entirely unpainted Matrix.
Here's a quick test: if your frontier town's "population #,### sign" (there is only one sign) and it has at most one comma, definitions are premature.
Because I Solved Diffie-Helman Exchange For Catalytic Conversion: https://pastebin.com/ZVvLYYiV
moo
Hebb showed us the way forward right from the start, yet we still managed to get stuck with backpropagation and perceptrons time and fucking time again.
Obligatory reference to the much older history of back-propagation: http://people.idsia.ch/~juergen/who-invented-backpropagation.html
TL;DR: in 1986 BP was already well known. Hinton made it popular in the cognitive science community, and the rest is history.
And yet we still can't get a robot to walk, move or dance like a human. That's suppose to be the easy mechanical stuff.
We understand that. But we also understand that the beginning of the big AI learning curve is also the end of it because of how much faster silicon will be able to evolve itself to supremacy. Modelling the mammalian brain is the only way to get it done but it's also the way the system will have the independence to turn on us.
"The future depends on some graduate student who is deeply suspicious of everything I have said."
...he will be an aggressively creative male. Oh, wait a minute, he couldn't get a seat. Well, never mind.
E Proelio Veritas.
But, as Vernor Vinge pointed out in one of his stories (True Names - 1981), who says it needs to run in real time?
Maybe we're aiming to high right now. We want to simulate what we're capable of doing, at the same speed that we can do it. Why?
We talk about mapping the neurons in a worm, and replacing the worms brain with silicon to see if it can still act like a worm. Simulate the rest of the darn worm, and it's environment, and see what happens instead.
If it takes weeks or months of processing to give a second or two of worm thought, why would that be a bad thing? Processors and memory will continue to improve, and we'll still have learned something interesting.
A neural network is, at the end of the day, glorified curve-fitting. But it beat all the other methods (and there are many, with very diverse underlying math) at a few tasks, like machine translation, image and voice recognition, etc. It does well even for robot object manipulation apparently, a totally different application. But it looks likely that it is way too simple a model to solve intelligence. It could still be useful, perhaps in combination with some other methods or models. For example, AlphaGo uses Monte Carlo tree search in combination with neural networks. Maybe the final architecture will be several low-level modules based on neural networks, dealing with things we find hard to express explicitly, eg, vision, voice, locomotion, pattern recognition, and on top of that higher level models that deal with the world at the level of concepts, also models for reasoning and inference, massive databases of real-world knowledge, and extensive training over decades using reinforcement learning and many human teachers in parallel.
So what happened to his own theory about how backprop works in the spike-train networks of the human brain?
I am the most deeply suspicious. Fuck all this group think noise.
It's been said -- 90% seriously -- that AI is computers doing things that require human intelligence, that computers can't do yet.
Once computers can do it, it's "pattern recognition" or "heuristic blah-blah-blah" or whatever.
The recently developed specific ability or method has a name, so it's no longer considered to be "artificial intelligence" when a computer does it.
There's no time like the present. Well, the past used to be.
The AI community needs to be much more cautious and circumspect.
It's not the fault of the AI community. They are always very cautious about the claims they make.
The misconceptions about AI on the part of the laity is all down to the PR and marketing peeps making claims about things they know nothing about.
My experience with G. Hinton, is that he will refute anyone who conflicts with his ideas, not applying a logical/mathematical argument, but by calling them "a CRACKPOT".
Soo,... looks like he's now calling himself a CRACKPOT !
By the way, wonder how those large corporations (Google, Nvidia, etc..) that invested 100s of $Millions into his "Deep Learning" brand of NN's, feel about Hinton's sudden change of heart.
Hope they are A-OK with an investment, where the inventor is now publicly claiming that "it should all be thrown away"... HA !