Slashdot Mirror
Studying Intelligence Thru Entropy?
"A case in point. Neural networks are weighted switches. They store their 'weights' in the neuron. The storage of these weights determines the networks ability to perform an intellectual task. Therefore studying the 'entropy' of these weights and what and how they change and the effects of these changes is to study the networks 'intelligence' directly?
Another case in point. Genetic algorithms can search a solution landscape and then select the 'best' solution as a seed to the next iteration. This 'best current solution' will have an entropy or measure of order or disorder. So, in these terms, the system is measuring the level of chaos in the system according to some rules and selecting the solution that produces the least chaos (most entropy)
Is this striking any cords with anyone?"
Negative changes in entropy are information, not necessarily intelligence. Intelligence has connotations that information does not. Therefore, while several tons of algae are able to produce more information (by converting CO2, H2O and energy into carbohydrates) than a similar mass of humans, I suspect that the humans possess more intelligence. I think in order to have an intelligent discussion on the subject, an effective definition of intelligence would be needed. Not being a cognitive psychologist, I'll leave that to the experts and go on growing tons of algae.
Only studying the predictable change in entropy would follow from your suppositions. ie.
HOW predicatble entropic changes in any given system is the study of intelligence itself...
-- The morphemes of your disquisition are ascertainable, but they have eschewed an ambit of transpicuous exposition.
> Given that any force that changes the entropy of any system in a predictable way is an 'intelligent' force.
The second law of thermodynamics is pretty predictable, but it has nothing to do with intelligence. Unless you consider randomly colliding molecules to be functionally intelligent.
No flame intended, but have you by any chance been listening to the proponents of "intelligent design theory", the latest reincarnation of creation 'science'?
> A case in point. Neural networks are weighted switches. They store their 'weights' in the neuron. The storage of these weights determines the networks ability to perform an intellectual task. Therefore studying the 'entropy' of these weights and what and how they change and the effects of these changes is to study the networks 'intelligence' directly?
You seem to be confusing the training of the network with its operations after it has been trained.
> Another case in point. Genetic algorithms can search a solution landscape and then select the 'best' solution as a seed to the next iteration. This 'best current solution' will have an entropy or measure of order or disorder. So, in these terms, the system is measuring the level of chaos in the system according to some rules and selecting the solution that produces the least chaos (most entropy)
Actually, depending what problem the GA is working working on and what exactly you measure for the entropy calculations, the entropy may either increase or decrease as it progresses. (I know this for a fact, because I've done it.)
> Is this striking any cords with anyone?
Yeah, the same kind Lister strikes when he plays his guitar on Red Dwarf.
There is certainly room for applications of entropy to the study of these things, but you don't seem to be off to a good start. For some basic applications of information theory to neural networks, see Haykin's textbook. There's surely lots more literature out there, if you care to track it down.
Sheesh, evil *and* a jerk. -- Jade
Three points.
1) Yes, chaotic systems do have an entropy rate, production of information in bits per second.
This is the Kolmogorov-Sinai entropy, and when you partition your state space into a discrete form correctly this K-S entropy will equal the Shannon informational entropy rate.
The ideas of entropy can be applied to deterministic dynamical systems when discretized which induces effective 'probabilistic' laws even without assuming any "fundamental" probablism in the laws of motion (which I don't believe anyway).
2) The overall topic seems really silly and is reversing cause and effect.
All reduction in entropy is hardly intelligence.
A complete reduction of configurational entropy is going to a fixed point: e.g. dead.
3) the quoted professor's hysterical denial of any relation between Shannon's "informational entropy" and physical entropy is exaggerated.
It is true that physical entropy does refer to entropy in the specific physically realistic space of the probabilities of particles' degrees of freedom, at least classically, their positions and momenta.
If you apply the theory informational entropy to that particular space, and add in the laws of motion (namely chaos that brings you to physical equilibrium) you get regular physical entropy and thermodynamics.
Information theory is a mathematical theory (and it is great), which can be applied in some circumstances to real physics.