Interesting reaction. Sorry for being vague and bogus, however I didn't think a quick post to this discussion required a full exposition of the details of the method. That is why I gave a URL for our papers. However, since you obviously 'get' the math, you probably also know that *all* artificial neural networks are instantiated in matrix algebra. So why do you grumble? I really don't see this as a valid criticism. As you saw from our papers we use a linear learning method, singular value decomposition, the general method from which principal components analysis comes. Using this method and some pretty hefty computers we can use training data that are similar in size to the amount of information a person would have learned from. Non-linear learning methods would never settle when presented with 10-20 million running words of data. Vector space models of knowledge representation are an 'old' idea, traceable back to the 'semantic differential' of Charles Osgood in the '50s. I inherited the name Latent Semantic Analysis from its originators, Landauer, Dumais,Furnas, Deerwester, and Harshman of Bellcore from their first paper/patent on using the technique for information retrieval 10 years ago. (BTW, Tom Landauer is my PhD advisor, my partner and co-inventor of the IEA.) It is not only a method for information retrieval, it is also one of several respected psychological models of memory that are neurally plausible. I suppose they could have called it 'just another application of algebra (JAAA)', but then it would be confused with all the other technology out there. I also don't see anything wrong with using an old idea to help solve an old problem. It wasn't until the last few years that computers got powerful enough to do a matrix decomposition on matrices of this size. *It is not just keyword matching.* It is primarily the dimension reduction step that makes this method significantly different from simple keyword matching. Check out the Landauer & Dumais 1994 Psychological Review paper for details if interested.
... and I would like to take this opportunity to fill in a few details about it. My trusty lead programmer here at Knowledge Analysis Technologies alerted me that the slashdot community was chatting about the IEA and after reading some of the posts I thought I should join in (I'll try to read them all over the weekend).
First, let me give you a little history (the lingua franca article referenced in the story is a good read for a bigger history).
I am a cognitive psychologist, as are my partners Tom Landauer and Peter Foltz. I am also trained in educational psychology with an emphasis in measurement. The intelligent essay assessor started as an experiment testing our computational model of human knowledge representation. The model, called Latent Semantic Analysis (LSA), is similar to other artificial neural networks, but it learns its representations from extremely large bodies of text (it is limited to text right now). If you are interested in the underlying technology, please go to our academic website lsa.colorado.edu and download some of our journal articles.
We have found in our research that LSA makes judgments about text that closely mimics human judgments on a number of standard psychological experiments (e.g. categorization and sorting tasks). We wondered if the model could judge the quality of content in student essays similar to trained readers. I have spent the last several years testing this and we have found that the IEA consistently agrees with human judges *as well as they agree with each other* when trained properly. This tends to be an inter-rater reliability of around a.7 -.8 correlation. Given a set of reader scores, you cannot tell which is the human and which the computer.
When we presented our research we got so many people wanting to use it that we decided to apply for a patent on the method and to form a company to market it.
There have been a lot of misconceptions/disinformation about what it can do and how it should be used. The current form of the IEA is appropriate to use for short answer essays (aka constructed response items) for directed prompts (aka focussed questions). It is meant as a replacement for multiple choice questions on content driven material, not as a replacement for English lit and creative writing teachers. It should be used in support of the 'Writing across the curriculum' movement so that students get more of an opportunity to write (rather than just fill in bubble sheets). It is not appropriate for 'term paper' type of essays where each student response should be unique. By using short essays to assess content knowledge rather than multiple choice questions, you encourage the student to learn the material at a deeper level. It is much more difficult to *recall* the correct answer and present it than it is to *recognize* the correct answer and circle it.
We currently want the IEA to be used as an interactive tutoring system for writing -- if you go to our website you will see some demonstrations of its use. We are interested primarily in formative assessment allowing revision rather than summative assessment to rank the students. Our goal is to help students learn. Our latest demonstration ties the technology to specific textbooks. You can have a list of essay questions at the end of each chapter of a textbook. After reading the text you choose a question then write an answer. The feedback will tell you whether or not you learned the information that the author of the textbook thought was important and where in the textbook you can find that information.
We honestly think that this system will help students learn and communicate. The press, to their discredit, has focussed on 'cheating teachers' implying that this system is a way for them to get out of their jobs. This is absurd. Look at any professor in college with several hundred students in a class or any teacher in K-12 with over-burdened resources and you will see that they rarely can afford the time to assign essay questions, so students never get the opportunity to write. This system gives students that opportunity. In some ways it is better than teachers (speed of feedback, objectivity, consistency) and in many ways it is worse than teachers (limited capabilities of understanding novel approaches, needs to be specifically trained for each domain/question), but we never wanted to see it as a replacement for teachers, rather as another tool for them to use in the daunting task of education.
I do appreciate the intelligent (for the most part) conversation you have brought to this subject. I look forward to continuing this discussion.
Slashdot Mirror
Interesting reaction. Sorry for being vague and bogus, however I didn't think a quick post to this discussion required a full exposition of the details of the method. That is why I gave a URL for our papers. However, since you obviously 'get' the math, you probably also know that *all* artificial neural networks are instantiated in matrix algebra. So why do you grumble? I really don't see this as a valid criticism. As you saw from our papers we use a linear learning method, singular value decomposition, the general method from which principal components analysis comes. Using this method and some pretty hefty computers we can use training data that are similar in size to the amount of information a person would have learned from. Non-linear learning methods would never settle when presented with 10-20 million running words of data. Vector space models of knowledge representation are an 'old' idea, traceable back to the 'semantic differential' of Charles Osgood in the '50s. I inherited the name Latent Semantic Analysis from its originators, Landauer, Dumais,Furnas, Deerwester, and Harshman of Bellcore from their first paper/patent on using the technique for information retrieval 10 years ago. (BTW, Tom Landauer is my PhD advisor, my partner and co-inventor of the IEA.) It is not only a method for information retrieval, it is also one of several respected psychological models of memory that are neurally plausible. I suppose they could have called it 'just another application of algebra (JAAA)', but then it would be confused with all the other technology out there. I also don't see anything wrong with using an old idea to help solve an old problem. It wasn't until the last few years that computers got powerful enough to do a matrix decomposition on matrices of this size. *It is not just keyword matching.* It is primarily the dimension reduction step that makes this method significantly different from simple keyword matching. Check out the Landauer & Dumais 1994 Psychological Review paper for details if interested.
... and I would like to take this opportunity to fill in a few details about it. My trusty lead programmer here at Knowledge Analysis Technologies alerted me that the slashdot community was chatting about the IEA and after reading some of the posts I thought I should join in (I'll try to read them all over the weekend).
.7 - .8 correlation. Given a set of reader scores, you cannot tell which is the human and which the computer.
First, let me give you a little history (the lingua franca article referenced in the story is a good read for a bigger history).
I am a cognitive psychologist, as are my partners Tom Landauer and Peter Foltz. I am also trained in educational psychology with an emphasis in measurement. The intelligent essay assessor started as an experiment testing our computational model of human knowledge representation. The model, called Latent Semantic Analysis (LSA), is similar to other artificial neural networks, but it learns its representations from extremely large bodies of text (it is limited to text right now). If you are interested in the underlying technology, please go to our academic website lsa.colorado.edu and download some of our journal articles.
We have found in our research that LSA makes judgments about text that closely mimics human judgments on a number of standard psychological experiments (e.g. categorization and sorting tasks). We wondered if the model could judge the quality of content in student essays similar to trained readers. I have spent the last several years testing this and we have found that the IEA consistently agrees with human judges *as well as they agree with each other* when trained properly. This tends to be an inter-rater reliability of around a
When we presented our research we got so many people wanting to use it that we decided to apply for a patent on the method and to form a company to market it.
There have been a lot of misconceptions/disinformation about what it can do and how it should be used. The current form of the IEA is appropriate to use for short answer essays (aka constructed response items) for directed prompts (aka focussed questions). It is meant as a replacement for multiple choice questions on content driven material, not as a replacement for English lit and creative writing teachers. It should be used in support of the 'Writing across the curriculum' movement so that students get more of an opportunity to write (rather than just fill in bubble sheets). It is not appropriate for 'term paper' type of essays where each student response should be unique. By using short essays to assess content knowledge rather than multiple choice questions, you encourage the student to learn the material at a deeper level. It is much more difficult to *recall* the correct answer and present it than it is to *recognize* the correct answer and circle it.
We currently want the IEA to be used as an interactive tutoring system for writing -- if you go to our website you will see some demonstrations of its use. We are interested primarily in formative assessment allowing revision rather than summative assessment to rank the students. Our goal is to help students learn. Our latest demonstration ties the technology to specific textbooks. You can have a list of essay questions at the end of each chapter of a textbook. After reading the text you choose a question then write an answer. The feedback will tell you whether or not you learned the information that the author of the textbook thought was important and where in the textbook you can find that information.
We honestly think that this system will help students learn and communicate. The press, to their discredit, has focussed on 'cheating teachers' implying that this system is a way for them to get out of their jobs. This is absurd. Look at any professor in college with several hundred students in a class or any teacher in K-12 with over-burdened resources and you will see that they rarely can afford the time to assign essay questions, so students never get the opportunity to write. This system gives students that opportunity. In some ways it is better than teachers (speed of feedback, objectivity, consistency) and in many ways it is worse than teachers (limited capabilities of understanding novel approaches, needs to be specifically trained for each domain/question), but we never wanted to see it as a replacement for teachers, rather as another tool for them to use in the daunting task of education.
I do appreciate the intelligent (for the most part) conversation you have brought to this subject. I look forward to continuing this discussion.
Cheers, Darrell
dlaham@knowledge-technologies.com