Indiana First With Computerized Grading

Mz6 writes "Computerized grading has been talked about previously, however, the New York Times reports that Indiana has become the first state to grade high school English essays by computer. The computerized grading process, called 'e-rater', uses a 6-point rating scale and uses artificial intelligence to 'mimic the grading process of human readers'. The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. The big question is, will other states begin to emulate Indiana by tossing human grading?"

Comments

I smell lawsuits, how about you?

[Allen Zadr]

... and produced results virtually identical to those of trained readers.

Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."

I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).

Re:I smell lawsuits, how about you?

[m0rph3us0]

I think you've taken one to many collage level english classes my friend.

Re:I smell lawsuits, how about you?

[m0rph3us0]

s/to/too/. I feel stupid now.

Re:I smell lawsuits, how about you?

[magefile]

extra points to a particularly good paper from a stupid student or take away points for a particularly bad paper from a brilliant student.

A "stupid" student doesn't deserve more points just for improving than a non-"stupid" student, and vice versa.

Re:I smell lawsuits, how about you?

[Shalda]

I disagree. I see lawsuits as no more likely. Furthermore, any process where you're subjectively evaluting something there has to be quality controls and an appeals process. My wife once held a part time job grading essay questions on a high school exit exam. Every few hours of grading exams, she would have to take and pass a "calibration battery" of 10 exams. Quality control is fundamental to the process.

What I see as being problematic is kids learning to beat the system. Typically these systems are predicated on gramatical analysis (use of punctuation and sentence compeleteness) and evidence of citing the text the question is based off. I'd bet its a real easy system to beat.

Re:I smell lawsuits, how about you?

[DrEldarion]

Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

This is why essays SHOULD be subjectively graded instead of objectively graded. You need to take into account the writing abilities of the student and determine if it's a good or a bad paper based on what they're capable of in addition to technical aspects of the paper.

Isn't this why there are remedial and accelerated english classes? To take into account the different levels of intelligence in students? If you took an 'A' paper from a remedial class, it's quite likely that it would be a 'D' in the accelerated class.

Re:I smell lawsuits, how about you?

[Bombcar]

I wonder if it has bounds checking on input..... perhaps you could submit /proc/kcore and cause a buffer overflow and get r00t on the gr4d3b00k!

Re:I smell lawsuits, how about you?

[nine-times]

The way I read it was "Makes sense. With some of the stupid teachers I had in public school, a chimpanzee with a box of crayons could produce results that are virtually identical."

Re:I smell lawsuits, how about you?

[king-manic]

Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

Why punish those with ability, why reward those who are not talented. Surely they won't be punished or rewarded in the same manner in real life. If the talented one was cruising and not expending any effort, thats his perogative. If he can be a productive member of society without effort thats fine, even if he has the potential to eb the next great mind, it's still his choice. The rewarding of those who work hard is important but if they work hard to only measure up to the minimium standard then they belong to that minimium standard regaurdless of how much effort they put in.

Re:I smell lawsuits, how about you?

[Anonymous Custard]

The real answer is to adjust your teaching methods per student based on subjective analysis. A low objective mark on a paper or test would indicate that, ideally, the teacher needs to pay closer attention to the needs of that student, and teach him in a way he can learn. (VERY few teachers have the time and skill required for this, unfortunately).

You can't grade subjectively because those grades will be compared objectively down the line. You can't say "this is pretty good for kevin, I'll give him an A", but then say "josh's paper is way better than kevin's paper, but josh is a bright kid, so I'm giving him a C". Kevin will think he's mastered the english language while Josh will go insane trying to achieve perfection.

Grading, when used for anything other than helping the teacher learn about each students, just plains sucks, and is only used for competition.

Re:I smell lawsuits, how about you?

[cloudmaster]

Darn straight. I'm all for making dumb people feel not so bad about things they can't control - like being dumb - but not at the expense of making the non-dumb feel dumb. There's a lot to be said for working smarter, not harder, and a lot to be said for those who have figured that out on their own. English provides several correct ways to do something and some incorrect ways of communicating. An "A" paper is an "A paper, regardless of the source and effort put in by said source.

Re:I smell lawsuits, how about you?

[clifyt]

Actually, I've been a part of writing software like this for their competitors and have worked with this software in the past as part of my duties as manager of development at the IUPUI Testing Center (thats Indiana University Purdue University Indianapolis). We've worked on this shit for about 10 years now.

One of my tasks in the past was to push this type of software onto the local schools. We've used it for rating in class essays.

The idea is that everyone knows that the only way you get better writing is to write more and get some feedback on it. It doesn't matter if you are an educator throwing the papers back at the student or a computer algorithm. It all forces the student to find the mistakes and not make them the next time.

The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day. So this is where this type of software comes into play -- you assign 2x the work you can normally handle, and let the computers handle half of it. You don't tell the students which assignments will be computer rated. Thus the students grades got better. Not much better, but they were better than the students not using the system in the same types of classes.

One of our smaller studies actually had us installing this software locally for instant feedback. It was a small percentage, but the students work was even better than before.

Yeah, you 'steal a copy' if you can't seem to get one given to you, and run it through until it likes it. How is this cheating or anyway underhanded. One of the better and far more dedicated educators I know actually allows students to hand in papers and have them marked up as many times as they want until the paper is due. His students final works are generally light years ahead of other educators in his facility that don't have the dedication (and for $26k a year, do you really want to give up your nights and weekends???).

Same thing here.

Shit, even using the grammar checker under work will force you to learn to write better (up to a point). You learn what its looking for and you avoid it. I'm not a good speller and I know the spell checkers help me learn after I hit the same error over and over again.

All these tools work for you in the learning process as long as you are willing to not just put this stuff on autopilot.

As for the title of this thread -- Lawsuit? The only lawsuits will come from idiots. None of the high stakes testing does purely computer rating. They all put humans into the equation. You will most likely get better rating because instead of having three or four humans look at your paper for 30 seconds each before moving on, you will now have one that is able to devote some serious time to it. All these humans will still be working just as many hours as before, but studies have shown that the eyeballs on the paper are there longer with this type of software than without (sorry, but these studies belong to the bigger testing companies or I'd post links...I just get paid to crunch the data).

Secondly, 5 years ago when I was working on this stuff full time, the software had a human agreeance of around 62% with a rater pool of 3 raters. Meaning that if you asked 3 people what they thought, took an average of this, and then asked a fourth, 62% of the time, the human agreed with the others. This was on a 12 point scale. The application, however, actually rated between 70 - 80% of the time depending on the model used.

In both cases, the raters were all trained together with the same things to look for, and the models were designed around this rater pool -- in a sense trying to simply guess at what the others would pick. The computer:human agreeance was higher than the human:human agreeance.

Back to the parent post, beating the system only means you beat learning to write.

BTW -- My post is not indicative of my writing skills outside of a conversational and informal setting, sans spell checker and proof re

Re:I smell lawsuits, how about you?

[slam smith]

It does seem unfair. After all at least the roofers are doing something useful.

Re:I smell lawsuits, how about you?

[neurojab]

>Using a computer to measure ____ seems like an improvement, as a computer won't mark you down for race, religion, opinion, etc - things that tend to drive humans to have lower opinions of each other (and consequently can bias grading).

Yes, a computer has no bias. It also cannot comprehend what is written AT ALL. All it can do is process a few heuristics. This means that a child that's incapable of forming an intelligent thought will get rewarded for simply putting random words in a sentence structure.

Sorry, but that's not an improvement. We already have spelling and grammar checkers that students can use at home. What we need are intelligent people grading essays to give feedback on things like coherency, strength, voice, etc.

The point of writing is to communicate. How can you test communication when the target of that communication cannot understand what is written? To give you an example: Suppose I want to test your ability to speak french. I hire a Russian who can't speak a word of French, and ask him to count the words you speak and determine if your accent sounds French. You could be speaking Swahili with a French accent and he would never know.

Using a computer to test essays is even more ridiculous because not only does it not know the language, but it's incapable of even intelligently guessing at the meaning of an expression.

OSS?

[Karamchand]

Is this program available under an open source licence? It sounds really interesting!

Re:OSS?

[twistedcubic]

Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.

Re:OSS?

[Glonoinha]

Slashdot First With Computerized Moderation.

Now with Computerized Moderation the famous Slashdot message site can pre-emptively down-mod 'Redundant' posts long before they are actually 'Redundant.' The computerized modding process, called 'e-modder', uses a 6-point rating scale and uses artificial intelligence to 'mimic the modding process of human readers - including doing stupid shit like modding the first instance of a concept as Redundant'.

Re:OSS?

[borkus]

Imagine high school students re-writing their essays ...
Actually, anything that would encourage students to re-write their papers and improve their writing would be pretty amazing. Most students jot something down, run a spell checker and turn in their work. If they could pre-grade their work, they might be better motivated to put out more effort and improve their writing.

Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.

Re:OSS?

[Mad Marlin]

My guess is that they are talking only about things like this. I used to use a similar program back when I was taking English classes, in order to bring my papers down to an 8th-grade reading level.

These are encredibly easy to mess around with. For example, the fog index is:

Fog Index = 0.4*(words/sentences+100*((words >= 3 syllables)/words))

Which is roughly equal to the school grade reading level required for the essay. If I remember correctly, Associated Press articles are written to a 4th-grade reading level, which is why all of the paragraphs are only one sentence long.

Re:OSS?

[Mad Marlin]

For example, my post above (with the formula removed):

readability grades:
Kincaid: 6.4
ARI: 6.6
Coleman-Liau: 9.2
Flesch Index: 77.8
Fog Index: 8.5
Lix: 35.8 = school year 5
SMOG-Grading: 8.0
sentence info:
408 characters
96 words, average length 4.25 characters = 1.33 syllables
6 sentences, average length 16.0 words
50% (3) short sentences (at most 11 words)
33% (2) long sentences (at least 26 words)
3 paragraphs, average length 2.0 sentences
0% (0) questions
100% (6) passive sentences
longest sent 28 wds at sent 2; shortest sent 6 wds at sent 4
word usage:
verb types:
to be (9) auxiliary (0)
types as % of total:
conjunctions 1(1) pronouns 9(9) prepositions 14(13)
nominalizations 1(1)
sentence beginnings:
pronoun (3) interrogative pronoun (0) article (0)
subordinating conjunction (1) conjunction (0) preposition (0)

Re:OSS?

[arose]

The thing is that a human can actually understand what you are writing. Clever arguments? A nice flow? Humor? Why would I concern myself with things that make reading interesting, if nobody is ever going to read it. Who cares if its dull, just get the punctuation right! Nevermind that it doesn't make much sense, just get a dictionary and show the computer your rich vocabulary...

I would have loved this is a kid

[cheezus]

it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system.

Re:I would have loved this is a kid

[jeffy124]

an opinion paper on why dyhydrogen monoxide must be banned would perhaps do the trick?

Re:I would have loved this is a kid

[Kiriwas]

In the florida school system this is a writting test called the Florida Writes. Its a standardized test, doesn't count for much, but we all had to take it. This is basically exactly what we had to do. Write the most horrible peice of trash you could think of, but make it adhere to a few preset guidelines and BAM instant grade. My first time taking it I wrote a rather good peice of work if I do say so myself. Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

Re:I would have loved this is a kid

[LnxAddct]

Well I've read almost every post under this story and noone has mentioned how it actually works. From the e-rater site "E-rater learns to score essays on a particular topic by processing a significant number of essays on the topic, each of which has been scored by two or more faculty readers. While e-rater is a powerful scoring engine, it is not meant to replace a teacher whose judgment is essential to helping students improve their writing ability." This means that its essentialy a bayesian filter that instead of being fed spam and ham and told which is right from wrong, it is fed good papers and bad papers and told which is right from wrong. You could literally reproduce this with something like SpamBayes but instead of feeding it spam, feed it essays. Anyway, you can probably beat the system but the fact that it is based on preprocessed papers means that a) it will be much harder to game the system without knowing what papers were scanned, and b) all future grading is based solely on the fact that previous generations have written papers and the futre grades will be based on them. The good thing is that unless the teachers are willing to write 20 different papers all on the same subject and scan them in, then the computer won't expect anything better then what was previosuly written. This also means, assuming the teacher won't write essays but rather will use ones from students from prior years, that the essay topics can't be changed, otherwise the teacher would have nothing to scan from prior years and thus nothing to base the grades on. As a result of the essay topic not being able to be changed that simply means that you can find someone from a year above you and ask them to save their papers for you, or in a worst case scenario, everyone must be given the same topic so at least you can help each other out. Often times teachers would give everyone a different topic to avoid people "helping" each other, but with such a system that would be impossible. I sure wish I was back in highschool.
Regards,
Steve

Re:I would have loved this is a kid

[cubicledrone]

Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

It is also the exact environment of the modern workplace, as designed. Results are irrelevant. Only the process matters.

Re:I would have loved this is a kid

[Krach42]

I'll let you in on how it works, as I ported some software that does exactly what this essay grader does, as work for a professor who worked on this stuff.

It's not a baysian filter, it's Latent Semantic Analysis. LSA works by taking large amounts of text, and comparing the usage and application of the words within paragraphs. It learns very quickly what words mean, and the interesting thing is, that once it's trained far enough, it starts gaining more meaning to its words by where they're not, than by where they are.

LSA has been put through a variety of tests. And has taken tests even. LSA has been shown to produce "average" results on a synonym test. ("A Doctor is: A) Nurse, B) Practitioner, C) Politician, D) Numerologist") Producing incorrect results mainly only when one word given is more associated (strongly linked) to the word than another more suitable word. Such as in my example, it would pick "Nurse" not "Practitioner" because it's seen Nurse used more often with Doctor.

LSA has been seriously tested by the designers to see if they could write a bad essay that gets a good grade. The answer? Yes, it's possible. But you have to REALLY know the subject well, (as you'd have to produce garbage that relates words accurately between each other) and a lot of time.

The recommended the best way to cheat the system, was to do your research, know your topic, and... write a good essay. Any other way requires too much effort, and a vastly superior knowledge of the subject.

Interesting is that this system can identify plagarism, give it two papers, and it looks how closely the papers match. This gets not just exact copies, but also paraphrased plagarism. The system doesn't really care what the words are, as it looks at their similarity. It could tell that "The doctor studied the patient." is just paraphrased "The practitioner examined his customer."

If train it right, it will even do this between two languages. It's also useful as a spam detector, as it will get "Enlarge your member" from just one marked "Make your dick bigger."

(So, I was told from the professor, Apple's Mail.app is supposed to use LSA)

For any interested. The professor at New Mexico State University was Peter Foltz, and some college up in Colorado was doing a lot of work on it also.

I already want a copy of this.

[ShitPissFuckCuntTits]

I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?

Re:I already want a copy of this.

[Kozar_The_Malignant]

> I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's.

I'll bet half the people here thought this as soon as they read the headline. The normal /. response is to spend 10 hours at the computer coding up something to do a task you can do in 1 hour with a pencil. :-) Of course the high school geek who does this will make piles of money selling the output to the football players, buy a nice ride, and get a prom date with a cheerleader. Isn't that the way it works?

Re:I already want a copy of this.

[Glonoinha]

Bah, right now the program takes an essay as input and outputs a grade. Simply reverse the pipe streams, push in an A and have the programs spit out a essay worthy of that A. Magic!

Stupid

[phreak0003]

I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

Re:Stupid

[RoderickMcDougall]

I agree. Did you receive any comments from the computer or was it just an unjustified percentage? I would be most interested to know how they have managed to justify this questionable practice. Almost as someone is rushing to boast about being at the forefront of technology without the goods to back it up. Oh the children! Won't someone please think about the children!

Re:Stupid

[garcia]

I have taken plenty of essay exams where I felt I wasn't graded fairly by a human teacher/professor.

Some essays were graded out of a couple points. A paper out of 6 points carries less weight overall. If this is the only exam (ie AP tests) a 5/6 is looked at as a high score.

I don't see your point.

Re:Stupid

[Anonymous Coward]

Maybe you got docked for overuse of commas.

Re:Stupid

[Roland Piquepaille]

Not only computers may not give fair grades, but there's a deeper problem with grading using computers: to me, students working to get good grades from a computer conjures up images of sheeps going in the wool-extraction machine. While this may be fine for sheeps, how do you think the students feel about it?

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?

Re:Stupid

[garcia]

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?

You had a different school experience than I did apparently. I felt that the human's reading my papers were distant, uninteresting, and less than worthy of grading someone else's work.

Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.

On "bad papers" the comments were usually less than helpful. Don't just mark up the paper with "comma splice" or "vague". The teacher should have taught the class what a "comma splice" was or should have been following their own words of advice and kept themselves from being "vague" in their comments.

Re:Stupid

[micromoog]

They're just converting "n/6" directly into a percentage grade?! That's ridiculous. So there's no such thing as a C (going straight from 83 to 67)?

Re:Stupid

[servognome]

Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.
Your feedback was actual words!? In my day all we got was a scratch-n-sniff sticker and had to guess the meaning of getting a "watermelon" on our essay.

Re:Stupid

[ibpooks]

A computer can obviously not grade essays fairly, so it shouldn't be done.

The article states that comparisons between the computer grading and human grading revealed nearly identical scoring. By this data, I don't think it's obvious that a computer can't be as fair as a human.

I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

What does the computer have to do with it? You would have been in the same situation if a human had given you the 5/6. Perhaps what you mean to assert is that the entire test isn't a fair assessment of performance; but that sounds like sour grapes from someone who didn't get a 6. :-)

Re:Stupid

[I_Love_Pocky!]

The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.

Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?

They are not graded fairly, and they determine 10% of the final grade.

10% of the grade on the essay? Or in a particular class? 10% on the essay may actually be tolerable, because that means that at least a human actually read it to give the other 90% of the grade.

Re:Stupid

[rsadelle]

I certainly hope your essay was better written than this comment. If it wasn't, and you still got 5/6, that's pretty good evidence that the computer isn't doing accurate grading. IANAET (I am not an English Teacher), but I would grade this comment 3/6. Some comments:

Do you have any evidence or thought behind your statement that "A computer can obviously not grade essays fairly, so it shouldn't be done"? Why is that obvious? Is it obvious only because the grading of your essay was, in your opinion, not fair?

I suggest you review difference in usage between "good" and "well." Proper grammar/word choice is a large part of what makes a good essay.

How is the conversion from the computer's 6 point scale to your teacher's/school's 100 point scale the fault of the computer? It appears that what you really mean is that the implementation of the computer grading system is not fair as it doesn't use an appropriate scale.

I guess this rules out...

[Loco3KGT]

schmoozing with the teacher to get higher grades.

In unrelated news, Delicious Red Apples have suffered a terrible sales slump.

comment moderation

[Jotaigna]

for the time being, i would trust more that program to moderate my comments.



c'mon people i was only joking dont mod me down, not noooo!!

What about tricking the software?

[gtaluvit]

SPAM filters are tricked all the time depending on the text of an email. Google was f'd up not too long ago because of trackback linking in blogs screwing up their algorithms. Isn't this a similar situation? If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

Re:What about tricking the software?

[drooling-dog]

we'll have students learning to write to beat software, not form a well written essay.

Right. The software can grade things like spelling, vocabulary, grammar, and syntax, but it won't grade the quality of the ideas expressed or the depth of understanding displayed. Therefore those latter dimensions will cease to be considered important, since grading them is expensive compared to the more mechanical stuff.

So much for those essays

[Zancarius]

Perish the thought should students start writing about the dangers of artificial intelligence. They may very well fail!

Gaming the system

[ePhil_One]

While it gives identical results for now, I can easily see the coming books: How to Write an A essay! Form essays to get you into Harvard.

The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...

Not the First

[dcocos]

My alma matter graded most of my computer programs with shell scripts and I graduated in 1997. So I don't think India is the first to do that.

Re:Not the First

[gregarican]

Grading freely-written essays and structured computer programming code is two separate things *to a degree.* And India isn't the first to do the latter. Indiana is, however, according to the article.

Re:Not the First

[Sporkinum]

So are all IT jobs being outsourced to Indiana?

Those Indianans are ruining us!

[L. VeGas]

Lets just outsource all our test grading to Indiana too.

In fact,

[ShitPissFuckCuntTits]

As a student the first thing I would hand in is twenty paragraphs from refreshing this. In political science class, of course.

Trained readers...

[ari_j]

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers

I think this says more about the training that the "trained readers" are receiving than it does about the software.

Re:Google Bombing

[superpulpsicle]

Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.
Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.

Essay Result = A+

Perfect scores every time

[Rorschach1]

Ok, since you know the grading software is going to make it into the hands of the students, here's my scheme for perfect essays:

Step 1: Feed some encyclopedia articles, Wiki pages, and other random material on your subject into a Markoff chain generator.

Step 2: Use a genetic algorithm to generate variations of the text. Fitness is determined by the grade calculated.

Step 3: Repeat step 2 until desired grade is achieved. (And, of course, Profit!)

The result is totally worthless, but at first glance would probably appear legitimate even to a human reader.

Sort of like Slashdot posts.

AI

[Bugmaster]

That's actually a pretty novel way to approach the problem of creating Strong AI. Making smarter machines is hard, so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test. Damn, this is so sad.

e-Rater result?

[ackthpt]

Input: All your base are belong to us!

Output: A+

identical results to those of trained readers...

[GGardner]

Maybe these says more about the readers than the computer program?

This is both good and bad

[n1ywb]

Good: The computer probably won't grade you down for writing an anti-Bush essay, and it probably won't get fired for it. Good: Computers won't play favorites, and you can't kiss up to a computer. Bad: The computer really can't grade you up for expressing original ideas. Bad: It's probably possible to fool the computer somehow.

I can't wait

[proverbialcow]

...until some wiseass figures out a way spoof the grader, probably by sliding under the radar of whatever probabilistic models they've got that pass for spell- and grammar-checkers.

For example:

Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

Grade: B+ Your use of double-negatives continues to haunt you, but I'm glad you've gotten over hanging participles.

At least the parent proves something...

[Ieshan]

At least the parent post proves one simple truth: human english teachers can be replaced by simple shell scripts.

In Other News

[ThisIsFred]

Indiana parents are the first to buy (en masse) licenses for Essay Constructor Pro v2.0. The software produces essays that are indistinguishable from those written by real students, using the latest screen-scrape-from-Internet 'n' plagiarism-from-non-credible-sources techniques.

Indiana Director of State Board of Ed comments: "Isn't it wonderful how technology is improving education?"

This says more about "trained readers"...

[Phurd Phlegm]

than it does about the software, methinks. I imagine it gives helpful hints like the ones I always turn off in Microsoft Turd. Any construction that deviates from the norm in a boring business document apparently triggers the "grammotron" or whatever they call it. A human reader has some appreciation for style and may actually accept something a little different for the sake of variety and sparkle.

Not that there's anything in this post that serves as an example. I guess that's because I was graded by humans. Seriously, I don't recall getting any encouragement in writing back in the '70s in high school, and not much in college. I guess it wouldn't have been any worse if the Grade-O-Vac was inspecting my papers instead of my mostly-marginally-literate teachers. There were several exceptions, but they focused much more on reading than on writing. I suspect they had a lot greater effect that way--I know they had a great effect on me.

Students

[Gettinglucky]

Now all the students need is e-writer so that they can just type in the subject and the score they want to achieve and then e-reader will grade it accordingly!!!

Can students get this program?

[Theovon]

If I were a student, I'd want to get a copy of this software and use it to pre-grade my papers so that I could find out what's wrong and fix it before I turned it in.

That's ok

[paranode]

schmoozing with the teacher to get higher grades

This works better for the Slashdot crowd. They are much better at romancing computers than people to get what they want.

India vs. Indiana

[the eric conspiracy]

Wouldn't it just be cheaper to grade the tests at call centers in India? What are those Indians doing when there are no incomming calls? Just slacking off??? They could be grading tests.

Completely offtopic papers

[fulldecent]

...and how would one ever devise software capable detecting papers that are

-1 Offtopic

while completely gramatically correct?

Don't blame the computer

[Scott Richter]

I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

Sure, but that's the fault of the humans implementing the grade system, who don't understand the difference between Gaussian and uniform distributions. Don't blame the computers.

Not that computers are a great idea here - they can only grade at the shallowest level, and if they were grading like real teachers, then those "real" teachers weren't doing their jobs.

But this specific problem that you mention is entirely human based.

Standard Grading

[Tony]

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

So it gives its favorite students 'A's without reading, least favorite students 'F's, and the rest arbitrary grades somewhere in between to mimic a bell curve?

Excellent!

"Artificial Intelligence is easy. It's artificial stupidity that impresses me." -- Arthur Oscar

Missing feedback

[Hamster Of Death]

How are these system supposed to scribble in the margins and tell you your ideas don't fit together?

How do they judge the content? What if you submit an excellent paper on middle ages history but the assignment was on socialism?

Human feedback is required in order to learn how to write well, you can't just expect a machine to tell you how to improve your writing. Grammar perhaps, but not ideas and how to let them flow coherently.

In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?

Seems like they are trying to solve the wrong problem with this system, or a problem that dosen't exist. (Are there really so many papers to mark you need a machine to do it?)

Re:Missing feedback

[RzUpAnmsCwrds]

"How are these system supposed to scribble in the margins and tell you your ideas don't fit together?"

They can't. But on a standardized test, you don't get any feedback anyway.

"In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?"

Because it takes too long.

"Are there really so many papers to mark you need a machine to do it?"

Yes. Human graders for standardized tests get about 1-3 minutes per paper. Human graders don't have time to really read your essay, so they grade you on the same kinds of criteria as this software does (grammar, spelling, a clear layout, etc.). Thus, it's not hard to create a computerized system which performs the same tasks as well as human graders.

"How do they judge the content? What if you submit an excellent paper on middle ages history but the assignment was on socialism?"

As I said, human graders don't have time to evaluate this anyway. The computer systems actually tend to be better at this.

Look, this system can and should not be used to relpace English teachers grading papers. A good English teacher will spend at least 30-60 minutes on a paper, and will write lots of comments.

What this system *is* good for is standardized tests. When everyone takes a test, you have (in many cases) a million tests to grade. This system can blast through the data and can actually perform better than an underpaid, overworked employee who has 1-2 minutes to grade a paper (and who grades hundreds of papers per day).

Antidisestablishmentarianism!

[Bluesman]

By creating a vernacular consisting of elongated words and sophisticated verbiage, obviously indifferent to definition but simultaneously observing grammar regulations while eschewing colloquialisms, perhaps students may increase individual chances of achieving substantial academic acclaim.

If this works anything like the writing level indexes you find on word processors, it should be easy to fool.

Patriot Act Junior

[Kaenneth]

I wonder how long until the FBI is linked to this system?

Grammar, 90%
Spelling, 95%
Patriotism, 80%

also:

I'd love to see famous writings graded by this system.

Too Uniform

[Roger_Wilco]

My writing style is somewhat peculiar, though I can't exactly say how (or even approximately how). Partially as a result of this, my marks in English class over the years of high school ranged from C to A, depending not on me, but on who the teacher was. If the teacher happened to like my style, I got a good mark.

This is annoying, but at least each year there was a different teacher, who may like my style. If the marking is computerised, it will not change; if your writing doesn't fit what the computer likes, you're screwed; likewise, if it does like it, you might never learn to express yourself more creatively (ie you'll be punished for trying to write in a manner different from what you usually do).

There are possibilities in this technology, but I suspect that it will be a long while before the eccentric aren't labeled as poor writers.

Babelfish

[Waffle Iron]

I will not trust computer grading until I see a computer-translated document that isn't laughable.

To illustrate my point, I'll restate it. [English -> German -> English]:

I do not trust the computer, which arranges, until I see a computer-translated document of this laughable isn't.

That's about how well a computer "comprehends" language today.

Whoever approved this should be fired

[Retired Replicant]

A computer can check spelling and even grammar to a certain extent. However, it cannot evaluate factual accuracy, strength of argument. Even with spelling, the computer is not likely to catch improper use of homonyms. I can guarantee you that it will be possible to create a piece of writing that is utter crap that would get an A+ using this or any other possible computerized grading system. Unfortunately, there are probably many teachers out there who make poorer graders than this system does. The answer to the problem of poor-quality teaching is not replacing teachers with computers; the answer is a combination of better teacher pay and putting higher standards in place for our teachers via competency testing.

Essay grading is harder than science grading

[sustik]

I believe that (English) essay grading is harder than grading science exams based on problem solving (no bubbles please), at least if essays are about content and not just grammatical correct sentences.

I say this because there is an objective criteria for grading the solution to a physics or math problem: correctness. For essays I do not beleive that we (and the current state of AI) can come up with an exact criteria like that. You might determine whether an essay is too different from essays which were written by experts, but cannot a very different essay to be just as good?

To my knowledge the AI programs can solve physics problems which are limited to some well defined domain (for example: http://www.cs.utexas.edu/users/novak/cgi/isaacdemo .cgi though that is from 1977...) I am not aware of programs grading physics problem solutions.

I will accept an essay grading program after they grade solutions to math and physics problems.

I conjecture that some writers would feel offended if their essay did well according to the program: they might think it means they are too conformist and conservative and not novel in their approach...

Matyas

From the web site.

[Bluesman]

>If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site.

Submit "and" essay? I guess they haven't run the software on themselves.

F.

the triumph of mediocrity

[KMonk]

Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.

Re:the triumph of mediocrity

[whyde]

This was the point I was going to make, too.

Nobody wants to read 5-paragraph themes all day long, even if they do get the point across. They are just a means to an end.

One of the best English teachers I've ever had would point to the use of alliteration, clever turns of phrase, humor, novel word choice (not just synonym-madness), and other completely subjective facets of writing as some of what makes the written word worth reading.

As someone procrastinating grading right now...

[Dr. Spork]

I don't think that automatized, high-school-level grading is an all-bad thing. We can call it unfair if we like, but as someone who grades a lot of stuff I can tell you that I'm nothing like fair. I don't always even know how to distinguish a B- from a C+, and I just go with my gut, which, as far as I can tell, is much like flipping an internal coin. If we looked at human grade assigners as an algorithm, we would find a whole lot of stuff wrong, even among those of us who try hard to be fair (cover author names, compare close grades for consistency, keep a constant mood, that sort of stuff).

But I think that if a computer grading program which is no worse than humans could be devised, it would be a great learning tool. A lot of people make it to college as borderline illiterates. I'm not kidding. I read a lot of their crap. That's because their HS teachers were too overworked to grade their writing, so they didn't assign much. If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

Sure, the first grading applications are going to make a few serious errors. This is the first stage of every application when a computer is asked to interpret rich data. Early voice recognition sucked. Now it sucks much less, and it will just keep getting better. Same with OCR, chess software, machine translation, etc. So the right debate to have is about when this will be good enough for school use, and not whether. I'm prepared to admit that the answer to the right question is "not yet" (I'm sure how deep the current problems go), but I fully support working on this system until it works right.

Re:As someone procrastinating grading right now...

[winwar]

"That's because their HS teachers were too overworked to grade their writing, so they didn't assign much."

With all due respect, that is a failing of the teacher, or in other words, the difference between an excellent teacher and one that is average. My best high school teachers had no problem assigning an essay a week. Yes, they had a lot of grading, but they realized it was important and the best way for the students to learn. Any teacher unwilling to do the same is not worthy of being considered a good or excellent teacher.

Granted, a computer grading program could make it easier. In the hands of a good teacher, I don't have a problem. It certainly won't be any more biased than a person. But if the teacher wasn't willing to put forth the effort in the first place, how is this program going to make it better? Are they going to check all of the papers to make sure the program didn't screw up, use the extra time to emphasize advanced writing skills, etc.? Or are they going to treat it like an assembly line? I know which one I would place money on...

Re:As someone procrastinating grading right now...

[Idarubicin]

If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

Unfortunately, the system described here doesn't return any such useful feedback. The Indiana system returns a grade from a six point scale. No comments, no criticism, no hint that the evaluation is meaningful.

Incidentally, what's this about "teaching instead of tedium"? Grading essays by evaluating construction, insight, and creativity should be part of the teaching process. Perhaps this is something that should be addressed earlier in the education of these students - if they're reaching college as "borderline illiterates" there is a problem - but grading in general is a part of teaching. If I were a student, I'd want to know that a human being - at some point - had bothered to look at the work that I did.

I took this test

[tundog]

I live in Indiana (no, NOT India) and took this test. Being a techie, I figured I'd try to fake out the system. This test works out to be 10% of the final grade and since I had a 98 going into the test, I figured I could afford to gamble a little, figuring if it back-fired I could blame it on a computer error since every one would figure the kid with a 98 MUST be telling the truth.

I almost wimped out. I wrote about 80 percent of the essay (about influence of pop-culture on society - and silly me I always thought society influences pop-culture but anyway). I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion. I reoreded the paragraphs, copied the one I felt was the best written and pasted it into the body 3 times.

Guess what I got.....6/6 (six point grading scale which is pretty messed up because a 5/6 is an 83%). Hopefully they won't audit mine....

Re:I took this test

[miyako]

The 5 paragraph essay is the cookie-cutter bottom line least common denominator for education these days.
I would estimate 90% of the essays given in the US educational system today are 5-paragraph essays.>br? Essentially, the 5 paragraph essay is a mold consisting of an introduction, 3 body paragraphs each focusing on one supporting fact, and a conclusion, and the teachers give students the ingredients in the form of a topic (definitely something that won't require too much thinking),and a style, The students generally mix these ingredients in their heads for about 30 seconds, vomit them into the 5 paragraph mold, and are graded on how aggressively the spell and grammar checked the paper.

Re:No way this is sound

[IWannaBeAnAC]

Edsger Dijkstra would no doubt have something profoundly and humerously offensive to say to the writers of this software ;-) There doesn't seem to be anyone to take over his mantle, which is presumably why the software industry is going pot at an ever increasing rate. sigh.

Computers can't grade "interesting"!

[Luscious868]

A computer can not replicate certain aspects of the grading process. Sure they can grade spelling and grammar and probably certain aspects of your writing style but there is plenty of important aspects of writing that they can not grade you on.

For instance, does your essay really grab the reader? Anyone here who reads technical documents knows what I'm talking about. There are some writers that, no matter how dull the subject, can make their work interesting and fun to read. A computer can not possibly grade one on that. I have a good friend who's a high school English teacher and occasionally I'll read some of the things written by his students. I've come across plenty of papers that are grammatically correct, have perfect spelling and are fairly well written from a syntactic and stylistic point of view, but are just plain boring to read. Then I'll move on to another paper, about the same subject, which is interesting and actually fun to read.

That's just one example of something a computer can not possible take into account when grading an essay. The bottom line is that a computer will never be able to grade you on certain subjective things, which although they are subjective and therefore open to a certain amount of interpretation depending on the person doing the grading, are nevertheless still very important aspects of good writing.

With spelling and grammar check, almost any average student can churn out a paper that is going to be mostly correct; however it still takes a good writer to produce something interesting. In my opinion, an interesting paper with a few minor spelling, grammar or syntactic errors is just as good as a boring paper with no spelling, grammar or syntactic errors.

Computerized Story Posting

[alatesystems]

Slashdot First With Computerized Story Posting

Now with Computerized Story Posting, the artificial intelligence "seeks out" stories that have either been long archived or just posted the previous day and then posts them as new material. The program then ignores what is stated in the FAQ and disregards all emails stating that the story is a duplicate. This program is also known as "chrisd".

Other features include "mis-classification into the wrong topic", "making up stupid-titles-that-go-into-the-dept", and the most difficult, ignoring stories that should be posted.

Chris Benard

A third source: TurnItIn.com-style relationships

[Flexagon]

If it doesn't already, I would expect a service like this will eventually include plagiarism detection, due to marketing pressure if nothing else. This is something that human graders do, at least over the space of papers they grade and works they remember.

But if plagiarism detection is added, then the grading service would have to make and retain some encoding of each graded paper, a derivative work, in its database.

Once that happens, the grading service also becomes subject to all of the issues already raised with services like TurnItIn.com, already discussed here.

I also found this comment from ETS's site rather strange, to say the least:

It is important to remember that e-rater is an embedded real-time application; it is not software.

How To Write An Essay

[dpbsmith]

A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.

It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.

The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.

The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.

In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.

tricking it

[enbody]

Yes, you can trick it. From the e-rater article:

"Experienced writers, teachers, and writing assessment specialists have tested e-rater to determine the extent to which it "understands" the content of essay responses. Some of these writers have submitted essays that have tricked e-rater into giving a score even though the essay does not make any sense. The individual words in these "challenge essays" are grammatically correct, but they are strung together in such a way that they create nonsense sentences."

That observation shouldn't be surprising because earlier it says: "An e-rater score will be most beneficial to students who make a good faith effort at using it to improve their writing skills."

The program works (grossly oversimplified) by mimicking the grading of humans on essay samples.

Depends on the purpose of gtrading...

[Dr. Zowie]

If grading is intended as a motivator to encourage each student to perform his/her best, then more effort should yield a higher grade. Likewise, if grading is intended to reflect the student's ability to perform in a real-world situation, effort should probably yield a higher grade: folks who work hard tend to do better than folks who are marginally smarter but don't work hard, in real-life situations. But if grading is intended to reflect only the quality of the work that was submitted, then sure -- effort shouldn't count at all.

This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.

As a college instructor, I tend to use a strict grading protocol -- and then "bump up" a few of the students. If someone comes in to my office every week and really struggles to understand the concepts, but the computer tells me that they earned a "C+" -- they're likely to find a "B-" on their transcript. But if someone who's smart enough to get an "A" blows an exam from being hung over, that person gets little or no sympathy.

Re:Depends on the purpose of gtrading...

[hesiod]

> If grading is intended as a motivator

A grade is used to show how well you know a subject. If I knew a subject completely before starting the class & wrote the same level of paper as someone who studied his ass off, we deserve the same grade. He should not be given a better grade than me just because he waited for this particular class to learn a subject. Why am I judged differently because I took the initiative to learn the information earlier than I absolutely had to?

No, schools are not there to make you feel good about yourself (that's obvious), they are there to make sure you know the minimum information to pass a class. That'a a D. Then, if you know more, you get a better grade. The amount of work you put into it is irrelevant. In fact, if you put more work into it than you should have, it means you are not doing well, and once you get into the "real world," where you have strict deadlines, you don't have the option of getting paid more just because you worked harder for the same result. The exact OPPOSITE is true, in fact.

School should help people prepare for life. If someone is given a grade they did not deserve, they are being improperly trained how to work.

A problem, however, is with the PARENTS. Many students are C students, that's all there is to it. But they get all high & mighty towards the school if they see their child work very hard for a low grade. They figure their child isn't good enough if they don't have all As, but that they deserve them just for hard work.

If life had a payrate based on how hard you worked, vs. your productivity, I would start working as an astrophysicist. I wouldn't get anything done, since I know nothing about the work, but I would sure as hell work my ass off. Do you think anyone wants an employee like that?
(I mean the lack of knowledge -- everyone wants a hard worker, if they know enough).

I scored 5/5 on the AP English exam...

[MichaelCrawford]

...and I like to write.

I just signed up for a userid so I can take the exam online, but after submitting my info it said I may have to wait up to two days to get an account.

Curious that they can grade essays with a computer but it looks like they have to have a human pass out the user ids.

Anyway, I'll see if I can submit one of my articles to the exam, and will post here how I did. Since I have to wait for my user ID, you'll have to look back here later to see how I did.

AP Essay Rubric

[jmichaelg]

Fine and dandy if you're writing for humans who are reading for content. On the other hand, the AP Biology essay rubric assigns points based on how many keywords appear in a response. It doesn't score the essay for sense.

A friend of mine who teaches Biology said that she saw some pretty bad essays which she would have given a poor grade to because the english was atrocious but she had to follow the grading rubric and give high scores to because the keywords were present.

Let's not forget about...

[Squidbait]

The postmodern essay generator

Scan-Tron

[MrBlackBand]

Anyone remember those Scan-Tron tests we had to take every once in a while? I hated those because unless each circle was filled out perfectly it would mark it wrong. Whenever I pointed out errors in grading to the teacher he or she would just tell me to fill it out better next time. Grrr.

Of course it was fun to mess with cheaters. If I noticed someone was copying off of my work I would make a point to put down all wrong answers. Then I would pretend to check over my work. The person who was cheating off of me would usually just take their test up to the teacher right away. When they sat back down I would make a big deal out of erasing every single one of my answers and doing the whole test over.
Their reaction was always priceless.