How Good Are Robo-Graders?

stoolpigeon writes "With a large study showing software grades essays as well as humans, but much faster, it might seem that soon humans will be completely out of the loop when it comes to evaluating standardized tests. But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies. Mr. Perelman outlined his approach for the NY Times after he was given a month to analyze E-Rater, one of the software packages that grades essays."

Comments

More importantly

[crazyjj]

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Re:More importantly

[sglewis100]

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Spammers with poor spelling and grammar figured out combinations of gibberish to get around Bayesian spam filtering, I can only imagine relatively smart students will figure out ways to beat the software in time. But hopefully, if people implement systems like this, there will be some checks and balances. Fear of receiving a '0' for a test coupled with having essays randomly graded (smaller numbers) and reviewed / skimmed quickly (larger numbers) ought to be a good start.

Re:More importantly

[Anonymous Coward]

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Noooooooooo.

I had to deal with a Robo grader once during an exam. Time was up and I was still writing. Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

I did immediately and it said, "Thank you for your cooperation."

Or that might have been when I was taking an art class taught by Peter Weller .... I don't remember now.

Re:More importantly

[BravoZuluM]

What does it mean to game the system? The game paper, while not pertaining to the subject, is a well written paper. It is not gibberish. It would take some talent to produce the gamed paper and probably more time. Given that, why wouldn't the student just write an on topic paper?

Given the bigger picture, writing is an art form. An essay is an art form. Even a human grading the paper might miss the nuances of what is being written. Who can truly say what the author has written is incorrect, when in writing, there is no incorrect or correct. There is just a continuum from bad to good writing.

Re:More importantly

[TheRaven64]

The tube drivers in London were recently on strike over pay. Their salaries are around £40k (about $65K), but for a decade or so most of the train control has been completely automated: they're just there to press the emergency stop button if there is something wrong with the automated system (which a human will notice but another automated system won't and, for example, cut power to that segment of track). So, judging by the past, teachers that did nothing but press play on a video machine would be better paid than ones that actually taught...

Re:More importantly

[gnick]

Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

I did immediately and it said, "Thank you for your cooperation."

You were lucky. You should see what happened to the guy in this documentary when the robo-grader didn't hear the pen hit the floor.

Re:More importantly

[NReitzel]

Well, yes.

E-Rater (a product with which I have some familiarity) is specifically sold to improve form and grammer, and the product explicitly states that it does not grade content.

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

Well, yeah.

That's the whole point. That, and the fact that you can have a student write a short essay in 30 minutes, and give them immediate feedback on what they have done wrong, as far as sentence form and grammar are concerned.

Generally, a student may know what they want to say, and have difficulty putting it into English prose in a way that might convince the reader that they have a clue about that of which they speak.

Don't think it matters? What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

Re:More importantly

[masternerdguy]

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Re:More importantly

[anyGould]

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Problem 1: Teachers don't get to choose what classes they get - I knew an English teacher who ended up teaching Intro Computing because.. they needed a computing teacher and he was available. Especially for newer teachers - you teach what they tell you to teach.

Problem 2: Are you intending to pay all those teachers in accordance with the extra 2+ years of education you're requiring?

Problem 3: At lower levels, you have A Teacher, not A Math Teacher and An English Teacher. Do you expect your kid's grade 1 teacher to hold multiple degrees? (And see problem #2, expanded to pay for a teacher holding half a dozen post-grad degrees so you feel comfortable letting them teach your kid ABCs.)

Re:More importantly

[anyGould]

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

I think that's naive. I think one kid will figure out how to get the computer to kick out excellent grammar and form (a lot easier when you don't actually care about the content), and in short order most of the smart/cunning kids will be using that (the cunning ones because it's a cheap A; the smart ones because they'll want to concentrate on subjects where knowledge matters, as opposed to something that can be outsourced to small shell scripts).

Re:More importantly

[Half-pint HAL]

I don't see why this would be different from current auditing practices. If an external examiner finds that your students have been incorrectly marked, it's either an automatic scaling of grades for everyone, or back to the red pen and regrade everything.

Re:More importantly

[bhlowe]

A student can game the system by writing their paper, running it through one or more "grading" systems... and making changes until it comes out an "A". Obviously, you would want to do this in a way that it does this while retaining the content and expected "readability" desired.

The fact is most "jobs" that humans do will be able to be done by a robot or computer. I can easily envision a future where kids get the best personalized teaching experience from a computer "coach"... who can tailor each kid's lesson much more skillfully than the average teacher trying to teach to 120 kids of a multitude of abilities. Teacher will be left to enforce discipline, dry tears, lead group exercises (as determined by the computer) and smile and wave at the kids as they come and go.

Re:More importantly

[s0nicfreak]

Did you know that often schools only teach students what is required to pass the tests, and much of that is forgotten during school vacations, not to mention after several years of being out of school?

Just the day before yesterday I was behind someone in a checkout line that didn't have enough to pay their bill on their debit card. So the cashier and the lady were trying to work out how much would be remaining after the amount on the debit card was used. After several minutes of both of then failing to figure it out, and the customer just handing the cashier some money (though not enough to cover the whole bill) they called over a manager, who showed the cashier that if she charged the debit card first it would show her the remaining amount. So then they counted how much money the customer had handed the cashier... and both tried to work out how much more was needed. After a minute the manager figured out how to type the amount into the register and be told the remaining bill.

I'm not saying cashiers don't know basic math, but quite a few of them would not be able to do their job without a register or at least a calculator.

Re:More importantly

[jc42]

What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Here in the US, we'd just elect him president.

Sorry, human intervention required

[LostCluster]

I don't think auto-graders are a good idea. Where is the information exchange between student and teachers? Teachers need to read student essays not just to assign the grade, but to exchange knowledge with their students Opinions and comments should be two-sided exchanges, if students are writing things that aren't going to be read, how does that work?

Re:Sorry, human intervention required

[Dyinobal]

Yep any essay should come back with feed back written on it in the margins/space between lines. Plus I doubt auto graders will mean anything except for kids learning to write a specific way that the auto grader is programmed to grade well.

Re:Sorry, human intervention required

[Zordak]

When I was in high school, we read A Portrait of the Artist as a Young Man. This is literally the worst alleged novel I have ever read. I actively despised it with my entire soul. So I skipped huge chunks of it wherever I figured I could get away with doing so and still pick up the threads of the mostly nonexistent plot.

When we (finally) finished the thing, we had to write a series of short essays responsive to several prompts. One of the prompts told us to describe the symbolism and significance of the "rose."

Having skipped huge portions of the book, I had never encountered this purported rose. And I certainly wasn't going to go back and pick through the dense, sophomoric prose to find it. Instead, I figured I could probably pick up some partial credit by saying some random insightful-sounding thing. So I started spewing what English teachers love. I used words like "juxtaposition" and "antithesis" and compared the rose to some other random symbolic object in the book. It was pure, unadulterated, Grade A, premium All-American BS.

I got an A on the paper. The teacher was particularly profuse in her praise of my short essay on the "rose," commented that I had captured the symbolism of the "rose" perfectly. I couldn't have agreed more.

100% A+ Perfect Reply

[MyLongNickName]

After thorough consideration of this first post and its contents, I find this I must respond in the most considerate and throughtful way possible. This first post was clearly written before the second post and well in advance of this reply. Based on this, it is only logical to assume that this first post was written before any other posts. This leads me to think that crazyjj was quicker reflexes and reading skills than his compatriots.

My research has shown that people with quick reflexes make 80% more in real dollar terms than others[1] and are more likely to lead a longer life than their slower reading friends [2]. Clearly crazyjj is at an extreme advantage compared to the rest of slashdot.

Can America survive with this type of inequality? I think not. We must institue some type of equalizer. Perhaps crazyjj should be given a keyboard with several broken keys. Or perhaps we should simply bash his fingers a few times. In the words of Abraham Lincoln, "A man who types too fast can't be trusted."[3] Abraham Lincoln saw the danger that crazyjj represents and warned us. Will we listen?

Human vs. Software

[Anti_Climax]

But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies.

Considering the fake generated paper that was published in a peer reviewed journal, I'd say that means the robo-graders are on par with human proof readers.

Robo-graders?

[Anonymous Coward]

So you're telling me we've not only solved the natural language problem, we're also wasting it on grading essays?

We're not even close. Robo-grading essays is not only cheating, it's probably the worst disservice a school could do to its students. When you grade an essay you're looking at far more than technical accuracy (spelling, word count, formatting, valid citations). You're looking for meaning, articulation and interesting points of view. Robots can't teach critical analysis, can't offer helpful critiques of writing style, and certainly can't make judgement calls on how "good" an essay is.

free graders to jusdge content

[peter303]

The rob grader can check spelling, grammar, structural style. The human grader can check for content accuracy and essay quality and creativity.

This isn't robo-grader specific...

[digitalsolo]

This problem is not specific to robo-graders. I made a solid rule of finding topics that I found interesting -and- were highly unlikely to be areas of specialty for the teacher/professor/TA grading the paper. It took slightly more effort to find the "right" topics, but it more than paid off in the long run, since the likelihood of the average test grader spending days researching every 10+ page paper they are grading is pretty low.

Obviously as your volume of large papers and required topics narrows this becomes less effective, but it's quite a good system in high school through most of undergrad studies. I guess I assumed most people did this. FWIW, I did write pretty good papers, they weren't full of B.S. (well, just average volumes of B.S.) but by getting the topic as far "out" as possible, it helped minimize criticism outside of the basic structure, citation, etc.