Reading Guide To AI Design & Neural Networks?

Raistlin84 writes "I'm a PhD student in theoretical physics who's recently gotten quite interested in AI design. During my high school days, I spent most of my spare time coding various stuff, so I have a good working knowledge of some application programming languages (C/C++, Pascal/Delphi, Assembler) and how a computer works internally. Recently, I was given the book On Intelligence, where Jeff Hawkins describes numerous interesting ideas on how one would actually design a brain. As I have no formal background in computer science, I would like to broaden my knowledge in the direction of neural networks, pattern recognition, etc., but don't really know where to start reading. Due to my background, I figure that the 'abstract' theory would be mostly suited for me, so I would like to ask for a few book suggestions or other directions."

Comments

PDP

[kahizonaki]

Parallel Distributed Processing (both books) by Rumelhart, McClelland, and the PDP research group, 1986. "THE" classic neural network resource--and still somewhat relevant.

Re:PDP

[agravier]

For a somewhat more up-to-date and maybe complementary book, I advise you Computational Explorations in Cognitive Neuroscience by Randall C. O'Reilly and Yuko Munakata (The MIT Press). The simulator intends to extend and replace PDP++ and is quite pleasant to use. It is on http://grey.colorado.edu/emergent/index.php/Main_Page

Re:PDP

[babbs]

I prefer James Anderson's "An Introduction to Neural Networks". I think it is better suited for someone coming from the physical, mathematical, or neuro- sciences.

Re:PDP

[kahizonaki]

The great thing about the PDP books is that they make almost NO assumption as to what the reader's background is. There's no code, a bunch of pictures, and something in there for everyone. Each chapter is written with a specific goal in mind, and by leaders in the field--there are chapters on the mathematics of the networks, the dynamical properties of them (i.e. how they can be thought of as boltzmann's machines), as well as lots of ideas for applications and specific studies of how real experiments worked. In addition, of course, there is the chapters which actually introduce the different types of networks--and there are equations (and appendices of equations--in case one likes them even more) which can be ignored if one wishes. Overall, in addition to an interesting read in general, by offering the opportunity to just pick-and-choose what one's interested in after reading the initial bit, these books are extremely dynamic and I recommend them strongly. Not to mention you can buy the full set in hardback used (off of amazon or whatever) for ten dollars (what a deal!).

Re:PDP

[Schwarzchild]

Cosma Shalizi is also a Physicist. I don't think he is actually doing research in machine learning or AI but he likes to read a lot and he tends to have fairly extensive reading lists.

Machine Learning

AI

You may also want to get familiar with Geoffrey Hinton's current work in neural networks.

The Resistance

[Anonymous Coward]

Due to the possibility of a robot army rising up, I refuse to help.

AIMA

[omuls are tasty]

Artificial Intelligence: A Modern Approach by Rusell and Norvig is more or less the standard AI textbook and the book I'd suggest to get an overview of AI and its different methodologies. Mind you, it's over 1000 pages, but a very interesting read.

Re:AIMA

[xtracto]

I must second that, Russel and Norvig book is one of the most important books.

I would also recommend:

Artificial Intelligence: A new Synthesis from Nills J. Nilson, who is considered one of the founders of A.I.

Re:AIMA

[Anonymous Coward]

I'd like to add to this. AIMA gives you a very broad and moderately deep overview of the state of AI ten years ago. As such, it is a truly excellent introduction introduction to the subject.

If you want a more recent, much more thorough and narrow introduction to neural networks in particular and machine learning in general, I'd recommend Chris Bishop's book: Pattern Recognition and Machine Learning (http://research.microsoft.com/~cmbishop/prml/), which focuses on learning rather than searching and planning. An outstanding more broad, shallow and dated book on machine learning is Tom Mitchell's book, Machine Learning (http://www.cs.cmu.edu/~tom/mlbook.html)

(Posting AC for the obvious reason that I can't be bothered to create an account)

Re:AIMA

[Yvanhoe]

Agreed. All the basic knowledge about the field is in this book. Part of these are available freely online. You can be judge : http://aima.cs.berkeley.edu/

Re:AIMA

[six11]

Also seconded. Russel & Norvig. Artificial Intelligence: A Modern Approach is a good book, well illustrated, and generally lacks the undecipherable academia-speak that pervades lots of AI literature.

Here's an article that was particularly influential on me and some of my friends: Brooks, R. 1991. Intelligence Without Reason. MIT AI Memo num 1292. Even though it is 'just' a tech report, it is frequently cited. He had another one, Intelligence without Representation, which is also good.

Somebody else mentioned the McClelland and Rumelhart PDP (neural networks) book, and it is also still quite good in spite of its age.

The interesting thing about AI (to me) is the funny mix of domain expertise. You have philosophers, sociologists, cognitive scientists, psychologists, computer scientists, and mathematicians. That's not a complete list---I'm in human-computer interaction and design research.

But because of the motley crew of domains you have a hundred people speaking a hundred different dialects. Some people put everything in really mathy terms, and their journal articles look (to me) like they are written in Klingon. Then you have others who write in beautiful prose but don't give any specifics on how to implement things. Still others express everything in code or predicate logic.

The oldest school of AI holds that you can reduce intelligence to a series of rules that can operate on any input and make some deterministic and intelligent sense of it. That works to a degree, but it falls apart at some point partly because of the computational complexity (e.g. the algorithm works if you have a million years to wait for the answer). Another reason it falls apart is because there are some kinds of intelligence that can't be reduced to rational computation (e.g. I love my wife because of that thing she does...).

There's a newer kind of AI that is based on having relatively simple computational structures that eat lots of data, "learn" rules based on that data, and are capable of giving fairly convincing illusions of smartness when given additional data from the wild. Neural nets fall into this category.

A third kind of AI brings these two schools together in the belief that there are fundamental computational structures like Bayesian Networks that can model intelligence* but those structures by themselves are insufficient and must be able to adapt based on exposure to real data. So instead of having a static BN whose topology is defined at the start and remains the same throughout the life of the robot, we can have a dynamic BN whose structure changes based on the environment.

I remember reading a recent article by John McCarthy arguing that all this statistical business is hogwash, and that the old school positivist, reductionist approach will eventually win. He's a smart guy, inventor of LISP and a Turing Award recipient. It seems his view is in the minority, but I'm not one to say he's wrong. However, my inclination is that the third hybrid group is probably going to be the one to make the most progress in the years to come.

The reason for my preference to the hybrid school could probably be best explained by Lucy Suchman's Plans and Situated Actions . I can't really do her thesis justice in a few sentences, but the short version of her argument is that there are plans (the sequence of steps that we think we are about to carry out before performing some task) and actions, which is the set of things we actually do. In my mind, a plan corresponds roughly with the underlying computational mechanism, but the actions correspond with how that mechanism executes and what happens when the underlying structure is insufficient, wrong, misleading, or fails.

Hope that helps.

Gabe

* None of this is to say that computational structures that we implement with software/hardware ar

Re:AIMA

[hoofinasia]

Nope. Its got neural networks. (section 20.5) Try walking into any cog sci / AI faculty office without seeing this book. Don't let anyone tell you it's dry (its got math! gasp!). It's accessible and thorough.

Also:
Statistics!

...learn it, love it. Thats mostly what AI is under all the gloss. That sound is a thousand Cog Sci students screaming in terror, ignore them.

AI != design brain

[Kupfernigk]

There is a very big difference between AI - which is based on guesses about how "intelligence" works, and studies of brain function. I'm going to make a totally unjustified sweeping generalisation and suggest that one reason that AI has generally been a failure is because we have had quite wrong ideas about how the brain actually works. That's to say, the focus has been on how the brain seems to be like a distributed computer (neurons and the axons that relay their output) because up till now nobody has really understood how the brain stores and organises memory in parallel- which seems to be the key to it all, and is all about the software.

So my feeling is that the first people really to get anywhere with AI will either work for Google or be the neurobiologists who finally crack what is actually going on in there. If I wasn't close to retirement, and wanted to build a career in AI, I'd be looking at how mapreduce works, and the work being done building on that, rather than robotics. I'd also be looking as seriously parallel processing.

So my initial suggestion is nothing to do with conventional AI at all - look at Programming Erlang, and anything you can find about how Google does its stuff.

Re:AI != design brain

[Anonymous Coward]

The human brain does not use anything that even remotely resembles software. The brain is hardwired.

Software in brains... that a paddlin'

Re:AI != design brain

[Dan East]

http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html

As both educators and researchers, we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications. MapReduce may be a good idea for writing certain types of general-purpose computations, but to the database community, it is:

      1. A giant step backward in the programming paradigm for large-scale data intensive applications

      2. A sub-optimal implementation, in that it uses brute force instead of indexing

      3. Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago

      4. Missing most of the features that are routinely included in current DBMS

      5. Incompatible with all of the tools DBMS users have come to depend on

Re:AI != design brain

[dmbasso]

The universe is software, the brain workings are just a tiny side-effect, but can still be considered software.

From universe.c:

int main()
{
      [...]
      return 42;
}

Re:AI != design brain

[TapeCutter]

"I'd also be looking as seriously parallel processing."

If you haven't seen this it might interest you. Note that it's a simulation for use in studying the physiology of the mammalian brain, not an AI experiment. Any ghost in the machine would have to emerge by itself in pretty much the same way mind emerges from brain function.

Heard of AGI?

[QuantumG]

http://www.opencog.org/wiki/OpenCogPrime:WikiBook

Some interesting stuff.

Russell & Norvig

[Gazzonyx]

In my AI class, last semester, we used Stuart Russell and Peter Norvig's Artificial Intelligence A Modern Approach, 2nd Ed.. It's fairly dry, but good for theory nonetheless. If you're a physics geek, it should be right up your alley; they approach everything from a mathematical angle and then have a bit of commentary on the theory, but never seem to get to the practical uses for the theory.

If you're in the US, send me an email and I'll send you my copy. They charge an arm and a leg for these books and then buy them back for 1/10 the price. I usually don't even bother selling them back.

machine learning resources

[Anonymous Coward]

Following Books are must have for machine learning enthusiasts:

Christopher Bishop
http://research.microsoft.com/~cmbishop/prml/

Richard Duda
http://rii.ricoh.com/~stork/DHS.html

There you will get an insight how machine learning methods (like neural networks, SVM, boosting, bayes classificator) work

for general AI (not so much in direction of statistical learning as the books above, but more into higher level learning like inference rules) I can recommend published work done by

Drew McDermott
http://cs-www.cs.yale.edu/homes/dvm/

Re:machine learning resources

[DocDJ]

+1 for the book by Bishop (don't know about the others). In addition, have a look at Information Theory by David Mackay which I found stunningly good. There is a free on-line version available, but you should buy it: http://www.inference.phy.cam.ac.uk/itprnn/book.html

Re:machine learning resources

[Beezlebub33]

I'll second Duda and Hart, though I guess it's Duda, Hart, and Stork now.

It's probably the most widely used pattern classification book that I've seen, and covers most of the techniques that you'll find. The coverage of neural networks is limited to Backprop though, so you'll need to look elsewhere for more in-depth on those.

Ask an Eliza

[MosesJones]

Question: Where can I find a Reading Guide to AI Design & Neural Networks

Answer: Why do you want to AI design & Neural Networks?

Question: Because I want to learn.

Answer: Will learn AI design & neural networks make you happy

Question: Yes

There you go. Now the question is whether Slashdot beats the Turing test on this one.

choose your subjects wisely

[Gearoid_Murphy]

be careful before committing to a large scale neural network project. Aside from the intuition that the brain is a massively interconnected network, no one is really sure what aspect of neural network functionality is necessary for intelligence. My advice to you is to spend time coming to terms with the abstract nature of intelligence rather than coding up elaborate projects. This link is a philosophical discussion on directed behaviour which I found quite interesting (if a bit vague, which is the mark of philosophy). Also, as you become familiar with the literature, you will see many examples of algorithms which claim to model certain aspects of intelligence. These algorithms work because they have a reliable and unambiguous artificial environment from which they draw their sensory information. The problem with practical artificial intelligence is that the real world is extremely ambiguous and noisy (in the signal sense). Therefore the problem is not creating an algorithm which can emulate intelligent behaviour but solving the problem of taking the empirical information of the sensory input and producing from that data a reliable abstract representation which is easily processed by the AI algorithms (whatever they may be, neural networks, genetic programming, decision trees etc) Good luck.

Cognitive Psychology

[tgv]

I would strongly recommend starting with a text book on Cognitive Psychology, or reading it in parallel. AI tends to overlook the fact that intelligence is a human trait, not the most efficient algorithm for solving a logic puzzle. Anderson's book can be recommended: http://bcs.worthpublishers.com/anderson6e/default.asp?s=&n=&i=&v=&o=&ns=0&uid=0&rau=0.

Reinforcement and Machine Learning

[jacksonpauls]

These might seem a little old, but are still a couple of my favorites:
Reinforcement Learning by Sutton & Barto
Machine Learning by Tom Mitchell

Theres nothing magical about parallel computation

[Viol8]

.. as applied to normal computers. In this case its simply speeded up serial computation - ie the algorithm could be run serially so Programming Erlang is irrelevant. With the brain , parallel computation is *vital* to how it works - it couldn't work serially - some things MUST happen at the same time - eg different inputs to the same neuron, so studying parallel computation in ordinary computers is a complete waste of time if you want to learn how biological brains work. Its comparing apples and oranges.

No it isn't

[Kupfernigk]

You've just reinforced my point by not understanding how the brain works. Neuron inputs and outputs are known to be pulse coded, and as you would imagine with chemical based transmitters, the pulse frequency is low (it evolved, it didn't get designed from first principles!) So it is perfectly possible to represent a neuron by a time-slicing parallel system, because it is extremely low bandwidth, and its output varies very slowly, but is NEVER DC. As a result, the output of the neuron does not need to be continuously available and it never needs to be polled. Your statement that "some things must happen at the same time" is just incorrect, quite irrespective of a theoretical physicist telling you that it is impossible. It is exactly the same principle by which you can send multiple audio channels over a digital RF channel.

However, to make this work you need a very efficient inter-process messaging prototcol that allows multiple virtual neurons to send messages to another virtual neuron. Languages like Erlang are optimised for doing this.

If I wanted to replicate the "brain" of a sea slug, which has (I believe) about 26 neurons, it would be much easier and cheaper to do this on a standard computer running 26 pseudo-parallel processes, than on 26 computers each imitating a single neuron, with a huiige number of potential interconnects.

As to what those pseudo-parallel processes look like, they have to respond every time a message is received (equivalent to a pulse from another neuron) by doing a calculation based on state history and then deciding when next to send an output to the destination process. For small numbers of neurons this is a manageable programming task; for large numbers, like brains with billions of neurons it is not.

Re:No it isn't

[Viol8]

And you've missed my point. Parallel computing on a von neumann computer raises issues of race conditions, deadlocking etc. These are the sort of things you have to worry about with parallel silicon systems. None of these issues apply to brains (as far as we know) so what is the use in learning about them? You're talking about simulating a neural system which is not the same thing - a simulation of anything can be done serially given enough time, never mind in parallel. But it will never be an exact representation of the real physical process and in the case of brains , seems to have given little insight into how they actually work anyway beyond the most basic I/O of neurons.

Also neurons are not just affected by signals from other neurons - they respond to chemicals in their enviroment and not forgetting that 90% of the brain consists of glials cells - and their full functionality is far from being understood.

Hawkins is misguided

[joeyblades]

I read "On Intelligence", too. While Hawkins has some interesting thoughts, I was less than inspired. Probably because I read John Searle's "Rediscovery of the Mind" first. Actually, most of Searle's work, as well as the work of Roger Penrose has led me to the conclusion that the Strong AI tract is missing the boat. The Strong AI proponents, like Hawkins, believe that if we build a sufficiently complex artificial neural network we will necessarily get intelligence. Searle and Penrose have very convincing arguments to suggest that this is not the right path to artificial intelligence.

Realistically, how could one build an artificial brain without first understanding how the real one works? And I don't mean how neural networks function; I mean how the configuration of neural networks in the brain (and whatever other relevant structures and processes that might be necessary) accomplish the feat of intelligence. We still do not have a scientific theory for what causes intelligence. Without that, anything we build will just be a bigger artificial neural network.

Also, the thing that Strong AI'ers always seem to forget... An artificial neural net only exhibits intelligence by virtue of some human brain that interprets the inputs and outputs of the system to decide whether the results match expectation (i.e. it takes "real" intelligence to determine when artificial intelligence has occured). Contrast this with the way your brain works and how you recognize intelligence from within, then you'll realize just how far from producing artificial brains we really are...

I'm not saying that artificial intelligence is impossible, and neither is Searle (Penrose is still on the fence). I'm just saying, don't think you can slap a bunch of artificial neurons together and expect intelligence to happen.