Slashdot Mirror
UK Researchers Make Neural Networks Smarter
Small Hairy Troll writes: "EDTN is running this story concerning a researcher in the UK who has come up with a method for getting those pesky Neural Nets to teach themselves to see. Called the 'Product of Experts,' the Neural Net is built using 'Experts.' If "you had one expert that preferred furry animals, whereas another expert preferred domesticated animals and another preferred small animals, their votes ... would light up dogs and cats very nicely." And an Edinburgh professor is quoted in the story as calling it "the first neural-network architecture that is both sensibly implementable and worth implementing."
> Question: Seeing the words 'biologically valid' conjures up an image of scientists pursuing pure science rather than concentrating on the applications of it. Is the goal of NN today more theoretical (we want to get something to behave more like a smart being) than practical (we want something that will specifically put names to faces/discriminate balloons from weapons/identify handwriting like an expert).
Both.
Cognitive scientists are using NN technology as a 'biologically valid' model for cognition. (Though only a fool would remain unaware of the enormous gap between our NN toys and the real thing, and of the enormous simplification that goes into our toys.)
Others just look at NN as a technology to be exploited without reference to biology.
> I suspect that this field has narrowed in the last decade (but I may be wrong), and so I fear that it may be getting wayyy esoteric.
Wayyyyy. Like any other branch of science, especially CS, this field is rapidly "narrowing" in the sense of getting deeper, but also "broadening" in the sense of developing more branches and more connections to other fields. (E.g., lots of parallels have been shown between NN and physics, and between NN and statistics.)
> As a practical engineer who needs solutions today, should I devote more energies to this or less? What is happening elsewhere in the field?
It's no longer possible even for NN researchers to stay on top of everything that's going on in the field, so don't even think about investing that much time in it.
Beyond that, what's your field of application in engineering? Do your journals ever cover relevant NN technology? If not, you might be able to start a SIG, so that the effort of keeping an ear to the ground and filtering out the uninteresting material could be spread among the members, rather than going it solo.
--
Sheesh, evil *and* a jerk. -- Jade
This appears to be similar to the technique used by Hopfield's Mus Silicium neural net speech recognition contest. The solution ended up being that recognition occurs when a large number of neurons connected to the same output neuron 'synchronize' and fire at about the same time. The big difference between these approaches seems to be that Hopfield is using spiking neurons and these guys are using some form of back propagation to train smaller networks that have to agree on what some data set represents in order to return a positive result.
Well, this is partially true. At least the MLP neural network is just a non-linear function approximator, that can, in theory, learn any mapping from the inputs to the outputs. The network is trained using points from the input space together with the desired output. Statisticians would probably call MLP non-linear regression. I also know some statistics profs who have a rather high regard for neural networks. The name has a lot of hype, but the methodology works. About correct extrapolations: I would like to see anything correctly extrapolate in a general case given only a few random observed points and desired output. The performance of these methods depends on how well the assumptions of the method correspond to the way the observed data really behaves and you cannot tell that with certainty from a few random points.
The training is usually done using two separate sets of data: a training set and a test set. The training set is used to train the network and the test set is used to test the performance. When the performance of the network starts to degrade instead of improving, then the network is starting to overfit and loses it's ability to generalize.
Basically you can just think of the MLP neural network as a classifier (when your goal is to classify, which is usually the case with neural network) that draws arbitrary boundaries in n dimensional space. Somehow I always think of those blobby objects in computer graphics, when I think of the classification boundaries, but this is of course not strictly correct. I guess it still helps.
Hi,
;)
I'm an AI researcher, and I'll tell you that you're patently wrong
One thing first: Hinton (the inventor of PoE) was one of the people re-popularizing NN's way back in the early 80's while at U of T.
Now, others have tried combining experts before. But Hinton's approach beats them empirically. The reason is that the experts are trained together in PoE rather than being trained seperately and considered jointly for evaluation, as in previous approaches...
Another thing: the NN's here are _very_ different from backprop NN's. The entire topology is different. Backprop neural networks are a special case of Bayesian nets, but PoE is based upon random fields. While there is current research being done on training random fields from bayes nets, there are in fact things you can represent with a random field that can't be represented with bayes net (some of Minsky and Papert's proofs for perceptrions in fact be extended to prove limitations on backprop nets and bayes nets). The converse is also true: there are distributions that can be represented in Bayes nets and not in random fields. (But of course, both classes of distributions can be represented as factor graphs...)
> The article was rather light on details, but this doesn't look like much of a breakthrough.
All the more so, since the notion of combining NN experts is already quite old. Haykin mentions it in the 1994 edition of his textbook.
Notice that that's 10% of the way back to the invention of electronic computers, and about 43% of the way back to when the backpropagation algorithm rescued neural networks from obscurity.
--
Sheesh, evil *and* a jerk. -- Jade
> As described to me, neural nets are _huge_ (every datapoint is in) underconstrained matricies with an infinite number of equally valid solutions.
Your criticisms are cogent, but in practice NNs can be excellent problem solvers.
For example, I suppose it is possible to solve the pole balancing problem with statistical methods, but I have never seen it done. With neural networks the problem and its more demanding variants have become so trivial that people are losing interest in it as a benchmark. (This is mostly as a result of advances in the art & science of NNs over the last 5 years or so.)
--
Sheesh, evil *and* a jerk. -- Jade
You can think of a NN as a non-linear function which is modelled after nueral connections. The inputs are weight vectors determining the strength of the connections between nodes, and a test vector with length equal to the number of nodes in the first layer. The output is a vector of length equal to the number of nodes in the final layer. Backpropagation is a technique that will optimize the link weights in order to minimize the error function - usually one half the sum of the square of the difference between network output and intended output (makes the derivative look nicer).
I don't think you can call a NN any dumber than a curve you fit to a graph, since you are just optimizing a function to minimize the error between the observed output and the output of the fit function. A traditional function may give you slightly more insight into a problem, but they can also easily mislead you (eg every function looks like a parabola around a min or max).
My point is that NNs are tools, sometimes effective, sometimes not. They aren't anything close to the AI you read about in scifi books, but nothing is. More interesting actually are support vector machines (svms). There are several papers on the web about them, they were devloped at bell labs by some russian dude whose name escapes me at the moment. They can be more effective than NNs, but the math is a bit harder to understand.
Scuttlemonkey is a troll