Slashdot Mirror

← Back to Stories (view on slashdot.org)

How Good Are Robo-Graders?

Posted by samzenpus on from the reading-robots dept.

stoolpigeon writes "With a large study showing software grades essays as well as humans, but much faster, it might seem that soon humans will be completely out of the loop when it comes to evaluating standardized tests. But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies. Mr. Perelman outlined his approach for the NY Times after he was given a month to analyze E-Rater, one of the software packages that grades essays."

5 of 157 comments (clear)

  1. More importantly by crazyjj · · Score: 5, Insightful

    How quickly will students learn to game the system to get perfect scores with perfect gibberish?

    --
    What political party do you join when you don't like Bible-thumpers *or* hippies?
    1. Re:More importantly by Anonymous Coward · · Score: 5, Funny

      How quickly will students learn to game the system to get perfect scores with perfect gibberish?

      Noooooooooo.

      I had to deal with a Robo grader once during an exam. Time was up and I was still writing. Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

      I did immediately and it said, "Thank you for your cooperation."

      Or that might have been when I was taking an art class taught by Peter Weller .... I don't remember now.

    2. Re:More importantly by NReitzel · · Score: 5, Insightful

      Well, yes.

      E-Rater (a product with which I have some familiarity) is specifically sold to improve form and grammer, and the product explicitly states that it does not grade content.

      So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

      Well, yeah.

      That's the whole point. That, and the fact that you can have a student write a short essay in 30 minutes, and give them immediate feedback on what they have done wrong, as far as sentence form and grammar are concerned.

      Generally, a student may know what they want to say, and have difficulty putting it into English prose in a way that might convince the reader that they have a clue about that of which they speak.

      Don't think it matters? What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

      Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

      --

      Don't take life too seriously; it isn't permanent.

    3. Re:More importantly by masternerdguy · · Score: 5, Insightful

      No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

      --
      To offset political mods, replace Flamebait with Insightful.
  2. Robo-graders? by Anonymous Coward · · Score: 5, Insightful

    So you're telling me we've not only solved the natural language problem, we're also wasting it on grading essays?

    We're not even close. Robo-grading essays is not only cheating, it's probably the worst disservice a school could do to its students. When you grade an essay you're looking at far more than technical accuracy (spelling, word count, formatting, valid citations). You're looking for meaning, articulation and interesting points of view. Robots can't teach critical analysis, can't offer helpful critiques of writing style, and certainly can't make judgement calls on how "good" an essay is.