Studying Intelligence Thru Entropy?

An Anonymous Coward asks: "Given that entropy is the measure of order or disorder. Given that any force that changes the entropy of any system in a predictable way is an 'intelligent' force. Is it true that the study of HOW entropy changes in any given system is the study of intelligence itself, in that given system? I is it true that producing systems whose sole purpose it is to capture and synthesise changes in entropy is the production of intelligent systems?"

"A case in point. Neural networks are weighted switches. They store their 'weights' in the neuron. The storage of these weights determines the networks ability to perform an intellectual task. Therefore studying the 'entropy' of these weights and what and how they change and the effects of these changes is to study the networks 'intelligence' directly?

Another case in point. Genetic algorithms can search a solution landscape and then select the 'best' solution as a seed to the next iteration. This 'best current solution' will have an entropy or measure of order or disorder. So, in these terms, the system is measuring the level of chaos in the system according to some rules and selecting the solution that produces the least chaos (most entropy)

Is this striking any cords with anyone?"

Comments

Nah.

[Black Parrot]

> Given that any force that changes the entropy of any system in a predictable way is an 'intelligent' force.

The second law of thermodynamics is pretty predictable, but it has nothing to do with intelligence. Unless you consider randomly colliding molecules to be functionally intelligent.

No flame intended, but have you by any chance been listening to the proponents of "intelligent design theory", the latest reincarnation of creation 'science'?

> A case in point. Neural networks are weighted switches. They store their 'weights' in the neuron. The storage of these weights determines the networks ability to perform an intellectual task. Therefore studying the 'entropy' of these weights and what and how they change and the effects of these changes is to study the networks 'intelligence' directly?

You seem to be confusing the training of the network with its operations after it has been trained.

> Another case in point. Genetic algorithms can search a solution landscape and then select the 'best' solution as a seed to the next iteration. This 'best current solution' will have an entropy or measure of order or disorder. So, in these terms, the system is measuring the level of chaos in the system according to some rules and selecting the solution that produces the least chaos (most entropy)

Actually, depending what problem the GA is working working on and what exactly you measure for the entropy calculations, the entropy may either increase or decrease as it progresses. (I know this for a fact, because I've done it.)

> Is this striking any cords with anyone?

Yeah, the same kind Lister strikes when he plays his guitar on Red Dwarf.

There is certainly room for applications of entropy to the study of these things, but you don't seem to be off to a good start. For some basic applications of information theory to neural networks, see Haykin's textbook. There's surely lots more literature out there, if you care to track it down.

Re:Information "entropy" is not entropy.

[grammar nazi]

I wouldn't be the grammar nazi if I didn't take point out that: "Is this striking any cords with anyone?" (line from original story) is incorrect. The proper homonym is chord, not cord; "to strike a chord"

The first paragraph makes the following hypothesis: "...the study of HOW entropy changes in any given system is the study of intelligence itself...?" That seems to be the main question in this ask slashdot.

In the second paragraph, he shows us how little he knows about neural networks (or grammar). It is not the storage of the weights in a neural network that determines the networks "intellectual" ability, but rather the value of the weights (I would argue it's the training method, network structure, rather than the actual weights). Furthermore, this sentence about the storage of the weights does not lead to the next sentence: "Therefore, studying the "entropy" of these weights... is to study intelligence". One point does not imply the other point. I'm not sure what he/she is trying to say here. As a side point, there is little entropy to the neural network nodes anyways. Similar networks, performing similar but different functions, have similar node weights (I know this from working developing an OCR system for 2 years at Lockheed Martin).

The author's third paragraph is to study the entropic nature of Genetic Algorithms. However, the entropy derived in each generation of a genetic algorithm is directly derived from a random number generator. All GA entropy is derived from randomness in the selection. To study each generation is equivalent to studying random number generation, nothing more.

Finally, I wouldn't be doing my Mathematics degree justice if I didn't point out that CHAOS IS NOT ENTROPY. Chaos is marked by having a complicated, seemingly random, system described by a simple structure or order (Period 3 implies Chaos, York, Lee, et al). Entropy is a random system having NO simple structure or order.

Does the original author still have a valid question? Probably. Little understanding of intelligence is to be gained from studying randomness and entropy in GAs and Neural Networks. Perhaps, the time would be better spent studying entropy XOR GAs and NNs. Or learning a little bit more about any of these things and then reposing the question.