How Good Are Robo-Graders?

stoolpigeon writes "With a large study showing software grades essays as well as humans, but much faster, it might seem that soon humans will be completely out of the loop when it comes to evaluating standardized tests. But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies. Mr. Perelman outlined his approach for the NY Times after he was given a month to analyze E-Rater, one of the software packages that grades essays."

Comments

More importantly

[crazyjj]

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Re:More importantly

[sglewis100]

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Spammers with poor spelling and grammar figured out combinations of gibberish to get around Bayesian spam filtering, I can only imagine relatively smart students will figure out ways to beat the software in time. But hopefully, if people implement systems like this, there will be some checks and balances. Fear of receiving a '0' for a test coupled with having essays randomly graded (smaller numbers) and reviewed / skimmed quickly (larger numbers) ought to be a good start.

Re:More importantly

[BravoZuluM]

What does it mean to game the system? The game paper, while not pertaining to the subject, is a well written paper. It is not gibberish. It would take some talent to produce the gamed paper and probably more time. Given that, why wouldn't the student just write an on topic paper?

Given the bigger picture, writing is an art form. An essay is an art form. Even a human grading the paper might miss the nuances of what is being written. Who can truly say what the author has written is incorrect, when in writing, there is no incorrect or correct. There is just a continuum from bad to good writing.

Re:More importantly

[TheRaven64]

The tube drivers in London were recently on strike over pay. Their salaries are around £40k (about $65K), but for a decade or so most of the train control has been completely automated: they're just there to press the emergency stop button if there is something wrong with the automated system (which a human will notice but another automated system won't and, for example, cut power to that segment of track). So, judging by the past, teachers that did nothing but press play on a video machine would be better paid than ones that actually taught...

Re:More importantly

[NReitzel]

Well, yes.

E-Rater (a product with which I have some familiarity) is specifically sold to improve form and grammer, and the product explicitly states that it does not grade content.

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

Well, yeah.

That's the whole point. That, and the fact that you can have a student write a short essay in 30 minutes, and give them immediate feedback on what they have done wrong, as far as sentence form and grammar are concerned.

Generally, a student may know what they want to say, and have difficulty putting it into English prose in a way that might convince the reader that they have a clue about that of which they speak.

Don't think it matters? What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

Re:More importantly

[masternerdguy]

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Re:More importantly

[anyGould]

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Problem 1: Teachers don't get to choose what classes they get - I knew an English teacher who ended up teaching Intro Computing because.. they needed a computing teacher and he was available. Especially for newer teachers - you teach what they tell you to teach.

Problem 2: Are you intending to pay all those teachers in accordance with the extra 2+ years of education you're requiring?

Problem 3: At lower levels, you have A Teacher, not A Math Teacher and An English Teacher. Do you expect your kid's grade 1 teacher to hold multiple degrees? (And see problem #2, expanded to pay for a teacher holding half a dozen post-grad degrees so you feel comfortable letting them teach your kid ABCs.)

Re:More importantly

[anyGould]

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

I think that's naive. I think one kid will figure out how to get the computer to kick out excellent grammar and form (a lot easier when you don't actually care about the content), and in short order most of the smart/cunning kids will be using that (the cunning ones because it's a cheap A; the smart ones because they'll want to concentrate on subjects where knowledge matters, as opposed to something that can be outsourced to small shell scripts).

Re:More importantly

[Half-pint HAL]

I don't see why this would be different from current auditing practices. If an external examiner finds that your students have been incorrectly marked, it's either an automatic scaling of grades for everyone, or back to the red pen and regrade everything.

Re:More importantly

[bhlowe]

A student can game the system by writing their paper, running it through one or more "grading" systems... and making changes until it comes out an "A". Obviously, you would want to do this in a way that it does this while retaining the content and expected "readability" desired.

The fact is most "jobs" that humans do will be able to be done by a robot or computer. I can easily envision a future where kids get the best personalized teaching experience from a computer "coach"... who can tailor each kid's lesson much more skillfully than the average teacher trying to teach to 120 kids of a multitude of abilities. Teacher will be left to enforce discipline, dry tears, lead group exercises (as determined by the computer) and smile and wave at the kids as they come and go.

Sorry, human intervention required

[LostCluster]

I don't think auto-graders are a good idea. Where is the information exchange between student and teachers? Teachers need to read student essays not just to assign the grade, but to exchange knowledge with their students Opinions and comments should be two-sided exchanges, if students are writing things that aren't going to be read, how does that work?

Re:Sorry, human intervention required

[Dyinobal]

Yep any essay should come back with feed back written on it in the margins/space between lines. Plus I doubt auto graders will mean anything except for kids learning to write a specific way that the auto grader is programmed to grade well.

100% A+ Perfect Reply

[MyLongNickName]

After thorough consideration of this first post and its contents, I find this I must respond in the most considerate and throughtful way possible. This first post was clearly written before the second post and well in advance of this reply. Based on this, it is only logical to assume that this first post was written before any other posts. This leads me to think that crazyjj was quicker reflexes and reading skills than his compatriots.

My research has shown that people with quick reflexes make 80% more in real dollar terms than others[1] and are more likely to lead a longer life than their slower reading friends [2]. Clearly crazyjj is at an extreme advantage compared to the rest of slashdot.

Can America survive with this type of inequality? I think not. We must institue some type of equalizer. Perhaps crazyjj should be given a keyboard with several broken keys. Or perhaps we should simply bash his fingers a few times. In the words of Abraham Lincoln, "A man who types too fast can't be trusted."[3] Abraham Lincoln saw the danger that crazyjj represents and warned us. Will we listen?

Human vs. Software

[Anti_Climax]

But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies.

Considering the fake generated paper that was published in a peer reviewed journal, I'd say that means the robo-graders are on par with human proof readers.

Robo-graders?

[Anonymous Coward]

So you're telling me we've not only solved the natural language problem, we're also wasting it on grading essays?

We're not even close. Robo-grading essays is not only cheating, it's probably the worst disservice a school could do to its students. When you grade an essay you're looking at far more than technical accuracy (spelling, word count, formatting, valid citations). You're looking for meaning, articulation and interesting points of view. Robots can't teach critical analysis, can't offer helpful critiques of writing style, and certainly can't make judgement calls on how "good" an essay is.

free graders to jusdge content

[peter303]

The rob grader can check spelling, grammar, structural style. The human grader can check for content accuracy and essay quality and creativity.