Slashdot Mirror
Implementing Artificial Neural Networks
Floydian Slip wrote to us with the updated story about a company called Axeon that is aiming to use the concept of artificial neural networks in a processor called the "Learning Processor." It's an array of 256 8-bit [RISC] chips in parallel. The company is aiming in a lot of places - mobile communications, inertial navigation and image analysis. The article also gives some of the background of the "neural chips".
Well, there's TD Gammon, which plays a pretty mean game of Backgammon using a neural network (and is rated as a master level player, no less!).. The author, Gerald Tesauro, has written a paper on the subject at:
http://web.cps.msu.edu/rlr/pub/Tesauro2.html
from the abstract:
"TD Gammon is a neural network that is able to teach itself to play backgammon soley by playing against itself and learning from the results, based on the TD(Lambda) reinforcment learning algorithm (Sutton, 1988). Despite starting from random initial weights (and hence random initial strategy), TD Gammon achieves a surprisingly strong level of play. With zero knowledge built in at the start of learning (i.e. given only a "raw" description of the board state), the network learns to play at a strong intermediate level. Furthermore, when a set of hand crafted features is added to the network's input representation, the result is a truly staggering level of performance: the latest version of TD Gammon is now estimated to play at a strong master level that is extremely close to the world's best human players. "
The folks at Cyberlife would have also used neural networks at the core of their 'Creatures' games and other applications. I like Creatures as an example, just because the neural networks clearly show their strengths and weaknesses in that context. The networks quickly reach an uncanny level of 'intelligence', but that 'intelligence' vanishes rapidly if training continues for too long. (In the Creatures fanbase, this loss of intelligence is called the 'One Hour Stupidity Syndrome', since the Norns typically start showing signs of it about an hour after being hatched)
Cyberlife can be found at:
http://www.cyberlife.co.uk/
or
http://www.creatures2.com/
There's been a huge amount of hype around neural nets (NNs) some time ago, thankfully most of it has died down.
Basically, NNs are highly tunable and very flexible nonlinear statistical models. That's it. Once you understand that you are dealing with statistical modeling and not bionic miracles, a lot of things fall into place and life generally becomes much easier. Note that many popular NN types are mathematically equivalent to well-known statistical models -- only the names are different. For example, projection pursuit regression is basically a direct match for a three-layer feed-forward neural net.
Because NNs are statistical models, you have to deal with all the classic statistical problems. You still have the problems of input selection, of overfitting, of regime switches in the process that you are trying to model, etc. etc. Basically, NNs are very useful and quite complicated statistical tools. To use them correctly demands considerable sophistication and more than a passing acquaintance with statistics.
For the interested, there is a very good FAQ on neural nets maintained by Warren Searle. It is technically the FAQ of alt.ai.neural-nets (I am not sure about the name of the newsgroup) and can be found in the usual places.
Kaa
Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.
Okay, so basically what you're saying is:
If we model a computer after a brain, it will make mistakes.
The whole "flying into the sun" thing could be averted by installing three such nets, and have them try to agree. Then, one of three things would happen:
1. One would think, "hey, let's fly into the sun." It tells the other two, because the ship is programmed to get instructions from two nets, and the other two say, "wowe, buddy, that's a square, not a planet. Were you off the day they said planets were shaped like spheres?" Or vice versa, and into the sun we go.
2. One would think, "hey, let's fly into the sun." The other two say no, so it takes over and flies into the sun while the other two desperately try to save themselves somehow.
3. One would think, "hey, let's fly into the sun." The second one would say no, and the third would say Idunno. The whole net freezes, and the ship keeps going straight until it runs into Alpha Centauri.
This is sort of like putting three people in charge of flying the ship. I would still trust this more than putting a normal computer in charge, because if something happens that the programmers haven't thought of, you are bound to die.
The Lord DebtAngel, Lord and Sacred Prince of all you owe
Is this post not nifty? Sluggy Freelance. Worshi
products:
- most OCR programs, such as the ones that now come free with your scanners,
use neural networks for at least some of the steps to recognize
characters. See, for example, Caere OmniPage and Ligature,
which uses them in its "ocr-on-a-chip" that goes into its
handheld "Quicktionary" pen.
- data mining programs used NNW's to analyse
transactions for unusual patterns, e.g. credit card fraud. This is
now a big time business. See, for example, HNC Software, co-founded
by Robert Hecht-Nielson, a famous NNW guru at Univ. of San Diego.
- Sensory Inc. uses them in its voice recognition chips.
They've sold millions of such chips, which recognize just a few words
but with speaker independence, high background noise, and for low cost.
See the recent article at EE Times: "Toys that talk..."
- Synaptics , co-founded by Carver Mead, uses analog hardware
neural network techniques in its Touchpad that is used in many notebooks.
Have I convinced you yet? Most of these applications are at the
infrastructure level and don't get much PR, often for proprietary
reasons. Calera for example, was using NNW's in its OCR already
in the late 80's but didn't say anything about them until Caere started
bragging in ads in 1992 that it was using NNW's.
The only thing I ever heard a neural net doing was simple character recognition. Is there any other successful neural network appliction? In my humble experiments with neural net, I could never get anything interesting to work.
There are a fairly large set of questions to be asked about this sort of project, especially in which AI is being utilised as the sole method for optimising signals or navigation. The primary one is an age old question which is just going to get harder: Is it a feature or a bug?
Take a neural network as the sole navigational utility. Sure its been trained through 100,000 generations to work out the opimal path in realtime to fly out to saturn and back, but when it finally coes down to it, do you trust it? There is no algorithm you can check, there is no definite way of predicting what it may do if it encounters some previously unthought of situation.
Imagine, you turn on your ship, give your target, ship starts flying there no problem, then a meteor flies past you which happens to look remarkably like a square. Neural network gets a flashback to its initial training when simple squares were used to indicate planets because it was simpler, and it makes a massive erroneous gavitational adjustment and starts flying towards the sun.
Thats bad enough, but you can't even tell whether its idea of flying towards the sun is a good idea it has suddenly had about a slingshot that it could do to get you to your destination faster, or whether it has just gone nuts and is trying to get you killed.
The same, although less extreme cases apply to most things, if the AI is the only thing doing signal adjustment on your cellphone, maybe it'll flip out for no reason that is discernable. What do you do then? you can't "fix" the bug, its buried deep in such a complex neural network that it'd be like trying to figure out why a mute human with no body language drew a picture of a frog when told to draw a picture of an apple.
At least to start with, I think we are going to find that neural networks will only be good for tweaking certain aspects of a standard algorithmic system, and while these limitations are in place, they won't be able to show such huge advantages in signal tracking etc that is proclaimed for them. It will be some time yet before we can figure out ways of making AI safe.
You can't win a fight.
Cool. Very very cool. But they have failed to mention one possible application that I see as the most promising use for artificial neural nets (a-nets? I just coined a word!): Artificial Intelligence. Those four applications mentioned above all involve learning and AI to some degree, but I see much broader implications for their technology. Think about our brain for a moment. It handles:
Obviously, some systems such as muscular control and "critical systems" are better-handled by traditional processors; they involve no skill, only timing. However, other systems are best controlled by an a-net (ding!).
For example, speech recognition could be handled extremely easily by the Learning Processor. It deals primarily with fuzzy logic (i.e. what the word sounds most like), which is handled very poorly by traditional processors. Sensory input deals with conditioned responses -- butt sensors have low alert levels for pressure because we're used to sitting on chairs; however, extreme levels of heat should (hopefully) attract attention. This, too, is best handled by a-nets.
I purposely have left the one biggie for last. It is... (drumroll)... learning and memory. This may seem too ambiguous to be useful; one might argue that the 4 applications mentioned in the quote all employ some form of AI. I am talking about something completely different: An AI with no practical value whatsoever! An AI that is designed to be a true intelligence, not just a limited intelligence with a clear purpose (translation, image analysis, etc). An AI that tries to come as close as possible to sentience. It could be outfitted with a robotic body or whatever. Remember COG?
This goal would not be suitable for Axeon; they are a commercial company, and they have to make money somehow. However, once the Learning Processor becomes available to the public (or at least to research institutions), I can foresee such a machine coming into being.
BTW, whatever happened to COG? Is that project still going on? I haven't heard anything about it recently.
Friends don't let friends misuse the subjunctive.
We did alot of neural network work at the Media Lab (using them with HMMs are really popular now in "intelligence" systems).
I can see this as being particularly useful for some applications, like the cellular network example the article had. However, there are several problems with Neural Nets that don't make them a panacea, or a wiz-bang solution to duplicating the human brain.
The last point is the biggest hinderance to neural net usage - we don't really know how to apply it to generalized (or even many specific) problem areas.
This is not to belittle to accomplishment. There are quite a few well-defined areas that neural nets are extremely useful, and we should find more as time progresses and our knowledge increases.
Just don't expect any kind of general intelligence system to be coming soon. It won't.
-Erik
There are always four sides to every story: your side, their side, the truth, and what really happened.
Ironically, I can think of only one use for a chip like this right now - traffic controllers. The very same reason the original 8086 was developed by intel.
--
Yes, neural nets are useful in some applications, but the hard fact of the matter is that they have been completely oversold. If you have a messy problem to solve that you have no clue how to solve, chance are that a neural net won't do it. You can waste weeks of CPU time trying to train the thing, and you will never be able to get the error down to anything usable.
I suppose the main point to keep in mind is that, NNETs are a special kind of AI. I've always preferred to think of them as... artificial stupidity programs. Yes, they can learn (if you know what you are doing), but, damn, will they take a long time, even if you DO know what you're doing. Of course, perhaps someday better supervised training algorithms will come along, but right now the bread and butter of the trade is the back-propagation error correction scheme. It is highly effective for some problems, but it always takes a long time for the network to converge to something useful.
As far as the processor in the article, it sounds very interesting. If it can be made for cheap, great. Though I don't think that it is going to show up in handhelds and other small devices as much as the article seems to imply. Training is what takes so horribly long with neural nets. But the great thing about them is that once they're trained, they will spit out answers in no time at all, because are usually feed-forward, and hence not iterative. For many applications on handhelds, offline training will be fine.
Analog neural nets work best in special purpose application -- where what you want to do is to take a stream of input vectors, feed it into a MLP, and generate output vectors, ideally at the maximum rate that the analog circuits work. Or, if you're into power instead of speed, push down the power used by the net until you're just barely making the timing. This works the best when there's integrated sensors (like photoreceptors or a silicon cochlea) or actuators (micromachined stuff) on the chip, so you dispense with an A/D or a D/A by going analog as well.
There are a few products out there that use analog nets just in this way -- there's one in your Logitech Marble Trackball, computing the motion vectors of a pseudo-random dot pattern on the ball in IR.
But most problems out there actually don't fit in that nice little niche -- instead, the neural net is part of a signal processing chain, and the adjacent steps don't fit well in analog, so you need a DSP anyways. And once you have the DSP, its usually is a lose to put an analog neural net on the die, instead of just adding another ALU to the DSP, since the rest of the system is digital anyways.
One hope is this networking protocol, the address-event representation, that lets analog neural net chips communicate with each other in a very digital way, while being very efficient to implement in analog. See this paper for details ...
The question, which others have commented on here, is what neural networks are especially good at. Certainly narrow things like character recognition, face recognition and so forth seem to be a natural. Picking data out of noise also is promising - IIRC IBM was trying to use neural nets for one of their ultra-huge storage technology, where the signal-to-noise ratio on the magnetic heads is way too low for traditional encoding. Communication technology has essentially the same problem and will benefit.
Now, will you trust a neural network to pilot a Boeing you're on? Arguably, you already do - witness the recent disaster with a Korean jet which was ruled a pilot suicide. I doubt that we'll see such a "general AI" application in the field soon. Using dozens of small neural networks in sharply defined functions on an airplane, or on a car, will be more useful and done early... Mercedes' A-Class car's electronic suspension already uses serious heuristics to stabilize the car in dangerous situations, this sort of thing will probably be one of the first applications of neural networks.
Personally I think the term is somewhat of a misnomer. It's based on an early and too reductionistic view of how neurons were hypothesized to work in the brain. Sort of like the steam-era metaphors in Freud's work are being superseded by information-age metaphors in psychology. And there's the possibility that neurons aren't the basic "neural" building blocks at all... Roger Penrose in his books - "The Emperor's New Mind" and "Shadows of the Mind", proposes that each neuron's behaviour is actually defined by millions or more of tiny quantum switches in each neuron's microtubuli. His theory is very well argued but still controversial among orthodox researchers... I like it personally. If he's right, we'd need first to build a million-element quantum processor building block and then build useful neural networks out of millions such blocks - and even a single block wouldn't be mathematically simulatable by a conventional processor, even at very low speeds!
Frankly, I believe any "strong AI" applications for a neural network chip are out for the next decade or so. All this talk about pattern recognition is nice, but as soon as you get into symbolic processing - meaning as soon as "meaning" is involved" - you get into uncharted territory. "Meaning" is an emergent, bottom-up quality rather than a top-down macrofunction in the human brain (or any animal brain for that matter).