Slashdot Mirror

← Back to Stories (view on slashdot.org)

Reading Guide To AI Design & Neural Networks?

Posted by kdawson on from the open-the-library-doors-hal dept.

Raistlin84 writes "I'm a PhD student in theoretical physics who's recently gotten quite interested in AI design. During my high school days, I spent most of my spare time coding various stuff, so I have a good working knowledge of some application programming languages (C/C++, Pascal/Delphi, Assembler) and how a computer works internally. Recently, I was given the book On Intelligence, where Jeff Hawkins describes numerous interesting ideas on how one would actually design a brain. As I have no formal background in computer science, I would like to broaden my knowledge in the direction of neural networks, pattern recognition, etc., but don't really know where to start reading. Due to my background, I figure that the 'abstract' theory would be mostly suited for me, so I would like to ask for a few book suggestions or other directions."

19 of 266 comments (clear)

  1. PDP by kahizonaki · · Score: 5, Informative

    Parallel Distributed Processing (both books) by Rumelhart, McClelland, and the PDP research group, 1986. "THE" classic neural network resource--and still somewhat relevant.

    1. Re:PDP by agravier · · Score: 3, Informative

      For a somewhat more up-to-date and maybe complementary book, I advise you Computational Explorations in Cognitive Neuroscience by Randall C. O'Reilly and Yuko Munakata (The MIT Press). The simulator intends to extend and replace PDP++ and is quite pleasant to use. It is on http://grey.colorado.edu/emergent/index.php/Main_Page

    2. Re:PDP by Schwarzchild · · Score: 2, Informative

      Cosma Shalizi is also a Physicist. I don't think he is actually doing research in machine learning or AI but he likes to read a lot and he tends to have fairly extensive reading lists.

      Machine Learning

      AI

      You may also want to get familiar with Geoffrey Hinton's current work in neural networks.

      --

      "sweet dreams are made of this..."

  2. AIMA by omuls+are+tasty · · Score: 5, Informative

    Artificial Intelligence: A Modern Approach by Rusell and Norvig is more or less the standard AI textbook and the book I'd suggest to get an overview of AI and its different methodologies. Mind you, it's over 1000 pages, but a very interesting read.

    1. Re:AIMA by Anonymous Coward · · Score: 2, Informative

      I'd like to add to this. AIMA gives you a very broad and moderately deep overview of the state of AI ten years ago. As such, it is a truly excellent introduction introduction to the subject.

      If you want a more recent, much more thorough and narrow introduction to neural networks in particular and machine learning in general, I'd recommend Chris Bishop's book: Pattern Recognition and Machine Learning (http://research.microsoft.com/~cmbishop/prml/), which focuses on learning rather than searching and planning. An outstanding more broad, shallow and dated book on machine learning is Tom Mitchell's book, Machine Learning (http://www.cs.cmu.edu/~tom/mlbook.html)

      (Posting AC for the obvious reason that I can't be bothered to create an account)

    2. Re:AIMA by Yvanhoe · · Score: 2, Informative

      Agreed. All the basic knowledge about the field is in this book. Part of these are available freely online. You can be judge : http://aima.cs.berkeley.edu/

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    3. Re:AIMA by six11 · · Score: 3, Informative

      Also seconded. Russel & Norvig. Artificial Intelligence: A Modern Approach is a good book, well illustrated, and generally lacks the undecipherable academia-speak that pervades lots of AI literature.

      Here's an article that was particularly influential on me and some of my friends: Brooks, R. 1991. Intelligence Without Reason. MIT AI Memo num 1292. Even though it is 'just' a tech report, it is frequently cited. He had another one, Intelligence without Representation, which is also good.

      Somebody else mentioned the McClelland and Rumelhart PDP (neural networks) book, and it is also still quite good in spite of its age.

      The interesting thing about AI (to me) is the funny mix of domain expertise. You have philosophers, sociologists, cognitive scientists, psychologists, computer scientists, and mathematicians. That's not a complete list---I'm in human-computer interaction and design research.

      But because of the motley crew of domains you have a hundred people speaking a hundred different dialects. Some people put everything in really mathy terms, and their journal articles look (to me) like they are written in Klingon. Then you have others who write in beautiful prose but don't give any specifics on how to implement things. Still others express everything in code or predicate logic.

      The oldest school of AI holds that you can reduce intelligence to a series of rules that can operate on any input and make some deterministic and intelligent sense of it. That works to a degree, but it falls apart at some point partly because of the computational complexity (e.g. the algorithm works if you have a million years to wait for the answer). Another reason it falls apart is because there are some kinds of intelligence that can't be reduced to rational computation (e.g. I love my wife because of that thing she does...).

      There's a newer kind of AI that is based on having relatively simple computational structures that eat lots of data, "learn" rules based on that data, and are capable of giving fairly convincing illusions of smartness when given additional data from the wild. Neural nets fall into this category.

      A third kind of AI brings these two schools together in the belief that there are fundamental computational structures like Bayesian Networks that can model intelligence* but those structures by themselves are insufficient and must be able to adapt based on exposure to real data. So instead of having a static BN whose topology is defined at the start and remains the same throughout the life of the robot, we can have a dynamic BN whose structure changes based on the environment.

      I remember reading a recent article by John McCarthy arguing that all this statistical business is hogwash, and that the old school positivist, reductionist approach will eventually win. He's a smart guy, inventor of LISP and a Turing Award recipient. It seems his view is in the minority, but I'm not one to say he's wrong. However, my inclination is that the third hybrid group is probably going to be the one to make the most progress in the years to come.

      The reason for my preference to the hybrid school could probably be best explained by Lucy Suchman's Plans and Situated Actions . I can't really do her thesis justice in a few sentences, but the short version of her argument is that there are plans (the sequence of steps that we think we are about to carry out before performing some task) and actions, which is the set of things we actually do. In my mind, a plan corresponds roughly with the underlying computational mechanism, but the actions correspond with how that mechanism executes and what happens when the underlying structure is insufficient, wrong, misleading, or fails.

      Hope that helps.

      Gabe

      * None of this is to say that computational structures that we implement with software/hardware ar

    4. Re:AIMA by hoofinasia · · Score: 3, Informative

      Nope. Its got neural networks. (section 20.5) Try walking into any cog sci / AI faculty office without seeing this book. Don't let anyone tell you it's dry (its got math! gasp!). It's accessible and thorough.

      Also:
      Statistics!

      ...learn it, love it. Thats mostly what AI is under all the gloss. That sound is a thousand Cog Sci students screaming in terror, ignore them.

  3. Heard of AGI? by QuantumG · · Score: 2, Informative

    http://www.opencog.org/wiki/OpenCogPrime:WikiBook

    Some interesting stuff.

    --
    How we know is more important than what we know.
  4. machine learning resources by Anonymous Coward · · Score: 4, Informative

    Following Books are must have for machine learning enthusiasts:

    Christopher Bishop
    http://research.microsoft.com/~cmbishop/prml/

    Richard Duda
    http://rii.ricoh.com/~stork/DHS.html

    There you will get an insight how machine learning methods (like neural networks, SVM, boosting, bayes classificator) work

    for general AI (not so much in direction of statistical learning as the books above, but more into higher level learning like inference rules) I can recommend published work done by

    Drew McDermott
    http://cs-www.cs.yale.edu/homes/dvm/

    1. Re:machine learning resources by DocDJ · · Score: 2, Informative

      +1 for the book by Bishop (don't know about the others). In addition, have a look at Information Theory by David Mackay which I found stunningly good. There is a free on-line version available, but you should buy it: http://www.inference.phy.cam.ac.uk/itprnn/book.html

    2. Re:machine learning resources by Beezlebub33 · · Score: 2, Informative

      I'll second Duda and Hart, though I guess it's Duda, Hart, and Stork now.

      It's probably the most widely used pattern classification book that I've seen, and covers most of the techniques that you'll find. The coverage of neural networks is limited to Backprop though, so you'll need to look elsewhere for more in-depth on those.

      --
      The more people I meet, the better I like my dog.
  5. Re:AI != design brain by Dan+East · · Score: 2, Informative

    http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html

    As both educators and researchers, we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications. MapReduce may be a good idea for writing certain types of general-purpose computations, but to the database community, it is:

          1. A giant step backward in the programming paradigm for large-scale data intensive applications

          2. A sub-optimal implementation, in that it uses brute force instead of indexing

          3. Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago

          4. Missing most of the features that are routinely included in current DBMS

          5. Incompatible with all of the tools DBMS users have come to depend on

    --
    Better known as 318230.
  6. Reinforcement and Machine Learning by jacksonpauls · · Score: 2, Informative

    These might seem a little old, but are still a couple of my favorites:
    Reinforcement Learning by Sutton & Barto
    Machine Learning by Tom Mitchell

  7. Re:If AI Design was any Good by CnlPepper · · Score: 1, Informative

    Sorry, no. Genetic algorithms are optimisation algorithms that use a parallel, quasi-historical method to explore parameter space. They can not an artificial intelligence.

  8. Re:AI != design brain by TapeCutter · · Score: 3, Informative

    "I'd also be looking as seriously parallel processing."

    If you haven't seen this it might interest you. Note that it's a simulation for use in studying the physiology of the mammalian brain, not an AI experiment. Any ghost in the machine would have to emerge by itself in pretty much the same way mind emerges from brain function.

    --
    And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
  9. Machine Learning, Tom Mitchell, McGraw Hill, 1997. by Anonymous Coward · · Score: 1, Informative

    Me degree is in AI, so I've come across quite a few books on the subject. I have to say that I didn't find Rusell and Norvig all that useful. For pattern recognition using statistical methods or multi layered perceptrons (neural networks) Machine Learning by Tom Mitchell is probably better. I would also recommend An Introduction to Genetic Algorithms (Complex Adaptive Systems) by M Mitchell for an interesting approach to neural network training.

  10. AI Books but it's not really AI by Anonymous Coward · · Score: 1, Informative

    Like the other people here mentioned, Stuart Russell and Peter Norvig's Artificial Intelligence A Modern Approach, is the text book most intro AI classes use. Another great book is Machine Learning, Tom Mitchell, which is used at a few of the top universities. That's really heavy on the theory. and finally there is The Elements of Statistical Learning, Hastie, Tibshirani, Friedman. I've run across these two books multiple times in the class room and outside in the industry. I've also seen some professors recommend the bishop book above, and duda.

    but i'd have to agree with some other people here in that the book On Intelligence is really a different form of AI, in that it tries to model the brain very differently. Traditional AI and neural networks are *vastly* different than what the Hawkins presents. Neural Networks are usually said to be _inspired_ by the brain and is nothing like how it really works. As a few of the other people have mentioned, this book is probably closer to cognitive science and there is a whole different field of research in how the brain works and how to possibly model it.

    If you're interested more in this book, I believe that the author had at one point created a small company around implementing it's ideas.

  11. Re:Russell & Norvig by Anonymous Coward · · Score: 1, Informative

    Here's a copy: http://black2d.com/upload/dl/Russell%20S.,%20Norvig%20P.%20Artificial%20intelligence-%20a%20modern%20approach%202ed.pdf