Indiana First With Computerized Grading

Mz6 writes "Computerized grading has been talked about previously, however, the New York Times reports that Indiana has become the first state to grade high school English essays by computer. The computerized grading process, called 'e-rater', uses a 6-point rating scale and uses artificial intelligence to 'mimic the grading process of human readers'. The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. The big question is, will other states begin to emulate Indiana by tossing human grading?"

Comments

I smell lawsuits, how about you?

[Allen Zadr]

... and produced results virtually identical to those of trained readers.

Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."

I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).

Re:I smell lawsuits, how about you?

[m0rph3us0]

I think you've taken one to many collage level english classes my friend.

Re:I smell lawsuits, how about you?

[m0rph3us0]

s/to/too/. I feel stupid now.

Re:I smell lawsuits, how about you?

[clifyt]

Actually, I've been a part of writing software like this for their competitors and have worked with this software in the past as part of my duties as manager of development at the IUPUI Testing Center (thats Indiana University Purdue University Indianapolis). We've worked on this shit for about 10 years now.

One of my tasks in the past was to push this type of software onto the local schools. We've used it for rating in class essays.

The idea is that everyone knows that the only way you get better writing is to write more and get some feedback on it. It doesn't matter if you are an educator throwing the papers back at the student or a computer algorithm. It all forces the student to find the mistakes and not make them the next time.

The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day. So this is where this type of software comes into play -- you assign 2x the work you can normally handle, and let the computers handle half of it. You don't tell the students which assignments will be computer rated. Thus the students grades got better. Not much better, but they were better than the students not using the system in the same types of classes.

One of our smaller studies actually had us installing this software locally for instant feedback. It was a small percentage, but the students work was even better than before.

Yeah, you 'steal a copy' if you can't seem to get one given to you, and run it through until it likes it. How is this cheating or anyway underhanded. One of the better and far more dedicated educators I know actually allows students to hand in papers and have them marked up as many times as they want until the paper is due. His students final works are generally light years ahead of other educators in his facility that don't have the dedication (and for $26k a year, do you really want to give up your nights and weekends???).

Same thing here.

Shit, even using the grammar checker under work will force you to learn to write better (up to a point). You learn what its looking for and you avoid it. I'm not a good speller and I know the spell checkers help me learn after I hit the same error over and over again.

All these tools work for you in the learning process as long as you are willing to not just put this stuff on autopilot.

As for the title of this thread -- Lawsuit? The only lawsuits will come from idiots. None of the high stakes testing does purely computer rating. They all put humans into the equation. You will most likely get better rating because instead of having three or four humans look at your paper for 30 seconds each before moving on, you will now have one that is able to devote some serious time to it. All these humans will still be working just as many hours as before, but studies have shown that the eyeballs on the paper are there longer with this type of software than without (sorry, but these studies belong to the bigger testing companies or I'd post links...I just get paid to crunch the data).

Secondly, 5 years ago when I was working on this stuff full time, the software had a human agreeance of around 62% with a rater pool of 3 raters. Meaning that if you asked 3 people what they thought, took an average of this, and then asked a fourth, 62% of the time, the human agreed with the others. This was on a 12 point scale. The application, however, actually rated between 70 - 80% of the time depending on the model used.

In both cases, the raters were all trained together with the same things to look for, and the models were designed around this rater pool -- in a sense trying to simply guess at what the others would pick. The computer:human agreeance was higher than the human:human agreeance.

Back to the parent post, beating the system only means you beat learning to write.

BTW -- My post is not indicative of my writing skills outside of a conversational and informal setting, sans spell checker and proof re

I would have loved this is a kid

[cheezus]

it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system.

I already want a copy of this.

[ShitPissFuckCuntTits]

I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?

What about tricking the software?

[gtaluvit]

SPAM filters are tricked all the time depending on the text of an email. Google was f'd up not too long ago because of trackback linking in blogs screwing up their algorithms. Isn't this a similar situation? If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

So much for those essays

[Zancarius]

Perish the thought should students start writing about the dangers of artificial intelligence. They may very well fail!

Gaming the system

[ePhil_One]

While it gives identical results for now, I can easily see the coming books: How to Write an A essay! Form essays to get you into Harvard.

The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...

Those Indianans are ruining us!

[L. VeGas]

Lets just outsource all our test grading to Indiana too.

Perfect scores every time

[Rorschach1]

Ok, since you know the grading software is going to make it into the hands of the students, here's my scheme for perfect essays:

Step 1: Feed some encyclopedia articles, Wiki pages, and other random material on your subject into a Markoff chain generator.

Step 2: Use a genetic algorithm to generate variations of the text. Fitness is determined by the grade calculated.

Step 3: Repeat step 2 until desired grade is achieved. (And, of course, Profit!)

The result is totally worthless, but at first glance would probably appear legitimate even to a human reader.

Sort of like Slashdot posts.

AI

[Bugmaster]

That's actually a pretty novel way to approach the problem of creating Strong AI. Making smarter machines is hard, so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test. Damn, this is so sad.

In Other News

[ThisIsFred]

Indiana parents are the first to buy (en masse) licenses for Essay Constructor Pro v2.0. The software produces essays that are indistinguishable from those written by real students, using the latest screen-scrape-from-Internet 'n' plagiarism-from-non-credible-sources techniques.

Indiana Director of State Board of Ed comments: "Isn't it wonderful how technology is improving education?"

the triumph of mediocrity

[KMonk]

Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.

I took this test

[tundog]

I live in Indiana (no, NOT India) and took this test. Being a techie, I figured I'd try to fake out the system. This test works out to be 10% of the final grade and since I had a 98 going into the test, I figured I could afford to gamble a little, figuring if it back-fired I could blame it on a computer error since every one would figure the kid with a 98 MUST be telling the truth.

I almost wimped out. I wrote about 80 percent of the essay (about influence of pop-culture on society - and silly me I always thought society influences pop-culture but anyway). I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion. I reoreded the paragraphs, copied the one I felt was the best written and pasted it into the body 3 times.

Guess what I got.....6/6 (six point grading scale which is pretty messed up because a 5/6 is an 83%). Hopefully they won't audit mine....

How To Write An Essay

[dpbsmith]

A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.

It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.

The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.

The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.

In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.