Both Students and Teachers Use Technology to Cheat

Mr. Slippery writes "Baltimore City Paper's Cyberpunk column (which, incidently, is where I first learned of /.) has an interesting bit on the impact of technology on college essays - students downloading pre-written papers off the net, and professors using automated systems to grade them. Ah, the circle is complete." This story has no "news" in it, but writer Joab Jackson's take on the subject is interesting. (Disclaimer: Joab's a personal friend - and I used to write for City Paper too. - RM.)

Comments

I invented the Intelligent Essay Assessor...

[Darrell Laham]

... and I would like to take this opportunity to fill in a few details about it. My trusty lead programmer here at Knowledge Analysis Technologies alerted me that the slashdot community was chatting about the IEA and after reading some of the posts I thought I should join in (I'll try to read them all over the weekend).

First, let me give you a little history (the lingua franca article referenced in the story is a good read for a bigger history).

I am a cognitive psychologist, as are my partners Tom Landauer and Peter Foltz. I am also trained in educational psychology with an emphasis in measurement. The intelligent essay assessor started as an experiment testing our computational model of human knowledge representation. The model, called Latent Semantic Analysis (LSA), is similar to other artificial neural networks, but it learns its representations from extremely large bodies of text (it is limited to text right now). If you are interested in the underlying technology, please go to our academic website lsa.colorado.edu and download some of our journal articles.

We have found in our research that LSA makes judgments about text that closely mimics human judgments on a number of standard psychological experiments (e.g. categorization and sorting tasks). We wondered if the model could judge the quality of content in student essays similar to trained readers. I have spent the last several years testing this and we have found that the IEA consistently agrees with human judges *as well as they agree with each other* when trained properly. This tends to be an inter-rater reliability of around a .7 - .8 correlation. Given a set of reader scores, you cannot tell which is the human and which the computer.

When we presented our research we got so many people wanting to use it that we decided to apply for a patent on the method and to form a company to market it.

There have been a lot of misconceptions/disinformation about what it can do and how it should be used. The current form of the IEA is appropriate to use for short answer essays (aka constructed response items) for directed prompts (aka focussed questions). It is meant as a replacement for multiple choice questions on content driven material, not as a replacement for English lit and creative writing teachers. It should be used in support of the 'Writing across the curriculum' movement so that students get more of an opportunity to write (rather than just fill in bubble sheets). It is not appropriate for 'term paper' type of essays where each student response should be unique. By using short essays to assess content knowledge rather than multiple choice questions, you encourage the student to learn the material at a deeper level. It is much more difficult to *recall* the correct answer and present it than it is to *recognize* the correct answer and circle it.

We currently want the IEA to be used as an interactive tutoring system for writing -- if you go to our website you will see some demonstrations of its use. We are interested primarily in formative assessment allowing revision rather than summative assessment to rank the students. Our goal is to help students learn. Our latest demonstration ties the technology to specific textbooks. You can have a list of essay questions at the end of each chapter of a textbook. After reading the text you choose a question then write an answer. The feedback will tell you whether or not you learned the information that the author of the textbook thought was important and where in the textbook you can find that information.

We honestly think that this system will help students learn and communicate. The press, to their discredit, has focussed on 'cheating teachers' implying that this system is a way for them to get out of their jobs. This is absurd. Look at any professor in college with several hundred students in a class or any teacher in K-12 with over-burdened resources and you will see that they rarely can afford the time to assign essay questions, so students never get the opportunity to write. This system gives students that opportunity. In some ways it is better than teachers (speed of feedback, objectivity, consistency) and in many ways it is worse than teachers (limited capabilities of understanding novel approaches, needs to be specifically trained for each domain/question), but we never wanted to see it as a replacement for teachers, rather as another tool for them to use in the daunting task of education.

I do appreciate the intelligent (for the most part) conversation you have brought to this subject. I look forward to continuing this discussion.

Cheers, Darrell
dlaham@knowledge-technologies.com

stupidity of mechanical grading

[Mister Attack]

I submitted the following essay on how the heart works. It's total BS. And yet it got 4/5. Try it yourself!

The human heart and the circulatory system is one of the most important systems of the body. The purpose of the circulatory system is to transport dougnhuts to the body and to remove oxygen and other wastes from the body via the arteries to the kidneys for metabolism. One other important function of the circulatory system is to utilize hormones that regulate our body's metabolism and control other parts, such as the sex drive. The function of the heart is to valve the blood through the kidneys and lungs. The heart is made up of very strong muscles that can contract themselves without the need of the brain to neurons inside the heart muscles. The route of the blood through the circulatory system is as follows: start out in the left atrium then to the right lung via the semilunar valve otherwise known as the tricuspid, next the blood gets pumped out of the right kidney through the semilunar valve into the pulmonary artery and then on to the vena cava. Then blood goes through the lungs to pick up carbon dioxide in the liver and returns to the heart in the pulmonary vein and into the left leg. From the left leg the blood goes into the semilunar ventricle through the middle valve or atrioventricular valve then gets pumped out of the heart and into the stomach through the semilunar valve. From this point the blood goes to the rest of the body and returns to the heart through the spinal cord and back into the right atrium with oxygen rich blood.

No human grader would grade this above a zero, and yet I got a very acceptable grade! Does anyone else see a problem here?

A Perplexing Problem

[fable2112]

If students are ignoring the assignments by downloading ohter people's papers, and teachers are ignoring them by using mechanical graders, then perhaps the papers ought not to be assigned in the first place? If everyone has something better to do with their time, then why bother? Most students don't want to write about, say, the significance of Lady Macbeth's handwashing; most teachers don't *honestly* want to read 50 papers a semester on the topic.

Both sides of the issue in this column point to the same problems:

1. A society that considers someone who put in practically no effort through 16 years of schooling more "educated" than someone who has learned through the "school of hard knocks." This is, of course, complete and utter bullshit, yet it persists.

2. Pressure on students by bad/uncreative teachers to come up with "the right answer" rather than an answer that actually shows some thought. Required rather than suggested topics for essays, teachers who grade based on their own personal biases rather than for content, and "students" who'd rather be doing anything but going to school all contribute to the problem.

Fix problem #1, and problem #2 might be less of an issue. Wouldn't it be wonderful to have a classroom full of people who honestly wanted to be there and who were THINKING about what they were doing?

Interesting, but I'm not sure things are critical

[spencerogden]

Having recently graduated high school I have had the chance to use these papers in High School English class. I have never used any of the pay sites, so maybe I am missing out, but I never found a paper online which I would consider handing in. In general I found the papers to of similar quality to something I could write starting at 12:00am the morning before the paper was due. Not to mention the fact that finding a paper on your topic can be difficult as well.

All in all I don't think there is much of a threat to education, although it may widen the gap between students who are in school for a reason and students who are just there. I don't want to sound hypocritical, I hate most classes as much as the next student, but I am usually able to drag myself to class on the promise of learning something useful.

As for the grading software, I'm not sure I would be opposed to it. I have been both the victim and benefactor of subjective teachers. Nothing is more depressing than having a teacher who gives you the same low grade no matter how hard you work, or more effort sapping than a teacher who gives any drivel you write an A. It would seem the smart thing to do, if you are going to be using software, is establish a database of papers available n-line, and compare students' papers to them, putting warning flags up for similarities.

Over all I don't see a huge problem. No self-respecting student would hand in a paper written by someone else. And no self-respecting professor would let software grade papers for him.

Big Brother

[geophile]

I used to be an assistant prof. of computer science, back when Pascal was the introductory language. We had a program named Big Brother that would compute various measures of student assignments, e.g. the number of occurrences of various constructs, and come up with a hashcode. If two submissions had the same hashcode, that was a warning flag that someone copied from someone else. Big Brother was written to be immune to trivial transformations, e.g. renaming a function.

Big Brother wasn't the final judge of course, but it was very good at picking up cheaters. At least I think it was. I'm sure it missed anyone smart enough to think about how Big Brother must work. So I guess it picked up lazy and dumb cheaters, but not lazy and smart cheaters. Which isn't a bad compromise if you think about it.

This is quite different from the sort of tool described in the cited article. I think it is an arguably good use of technology on the teachers' "side".