Slashdot Mirror

← Back to Stories (view on slashdot.org)

Grading Software Fooled By Nonsense Essay Generator

Posted by samzenpus on from the it-is-good-report dept.

An anonymous reader writes "A former MIT instructor and students have come up with software that can write an entire essay in less than one second; just feed it up to three keywords.The essays, though grammatically correct and structurally sound, have no coherent meaning and have proved to be graded highly by automated essay-grading software. From The Chronicle of Higher Education article: 'Critics of automated essay scoring are a small but lively band, and Mr. Perelman is perhaps the most theatrical. He has claimed to be able to guess, from across a room, the scores awarded to SAT essays, judging solely on the basis of length. (It’s a skill he happily demonstrated to a New York Times reporter in 2005.) In presentations, he likes to show how the Gettysburg Address would have scored poorly on the SAT writing test. (That test is graded by human readers, but Mr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots.).'"

12 of 187 comments (clear)

  1. most schools ignore sat essay by litehacksaur111 · · Score: 3, Insightful

    I though most schools don't even care about the essay. Also the elite schools nowadays prefer the ACT and SAT II subject tests to demonstrate real knowledge. The SAT is really a dumb test, especially with all the coaching resources available now.

    1. Re:most schools ignore sat essay by Anonymous Coward · · Score: 3, Funny

      Your post tells me that you didn't score all that well on the SAT. Bad grammar, incoherent thoughts.

  2. You don't need software by Anonymous Coward · · Score: 4, Insightful

    ... because Slashdot shows that humans already make evaluations about articles without reading them.

  3. Quid pro quo by Opportunist · · Score: 3, Insightful

    When you're too lazy to read my essay to grade me and let software do it, I don't really see no moral problem with doing the same to write the essay.

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    1. Re:Quid pro quo by Anubis+IV · · Score: 4, Insightful

      As someone who graded hundreds of essays while serving as a teaching assistant for a senior-level engineering ethics course, I have to say that I find your lack of integrity rather appalling. Your moral obligation to write the essay yourself is independent of the method they use for grading it. Just because someone else is doing a lousy job does not mean that you suddenly have a license to short-change them for what you're obligated to do.

      I would guess that I graded around 300-400 essays during the three semesters I served as a TA, and that I probably averaged around 20 minutes per essay, since I was a strong believer in providing useful feedback over things the students could improve, even if they weren't necessarily incorrect. That said, other TAs spent as little as a minute or two per essay, and barely provided any feedback at all. Regardless of how much time the TAs did or didn't spend on the essays, however, the students had the same obligations, and rightfully so.

    2. Re:Quid pro quo by number17 · · Score: 3, Insightful

      Your moral obligation to write the essay yourself is independent of the method they use for grading it.

      Students pay big bucks and expect to have experts in the field teach them and grade their work. It sounds like these schools are off-shoring their marking so that they can do other work (ie Research). If the school was upfront, before paying tuition, that they were going to just send your essay to Bangladesh for marking then I would be ok with having a moral obligation to write the essay myself.

    3. Re:Quid pro quo by clifyt · · Score: 3, Interesting

      As someone that wrote software like this -- and disagreed with the subject of the story a decade ago when he tried to get us with both the Gettysburg Address as well as Kennedy's inaugural address (both of which are GREAT speeches with historical value, but shitty college entrance exams) -- you are looking at this entirely wrong.

      I can give you background of how these things are generally graded. 3 people get an essay, look at it for 30 to 45 seconds, throw a score and it and if they are all within a margin of error, they move on. If not, a senior rater comes in and and they can replace one other person and it is now within margin of error, they move on as well. If not, it is workshopped for 5 minutes.

      In 99% of the cases, you have less than 2 minutes of viewing on your essay between 3 people.

      Enter the computer...the raters are told they are going to be rated themselves. We can throw a lot more prerated essays that had been normed by a large group of raters, and train the rater. They know they are being measured and the average rater spends two or more minutes reading through these. You actually have MORE time with eyes on your essay with a computer rater involved than you do without. Having a computer rater doesn't remove humans -- it adds a safe guard. It means one person spends more time and is verified with something that is unbiased (within reason...actually was able to figure out subtle racism and otherwise that wouldn't have been detected with purely human raters...'black' or 'hispanic' names and scores go down...'asian' names and the scores go up...give the same essay with the names switched and the humans change ratings...the computer was actually more objective).

      I haven't been involved with this sort of thing in a decade, and I can only assume it is much better than when I left my project...but lazy isn't the right word. Underpaid and overworked? Yeah...but not lazy.

  4. Re:Irrelevant by smittyoneeach · · Score: 3, Funny

    Hey, Helicopter Daddy and Blackhawk Mommy dropped good boodle for that 'A', mister!
    You can just stand down from all that meritocratic whinging right now, mister.

    --
    Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
  5. Re:To generate the keywords takes knowledge by MrBigInThePants · · Score: 5, Insightful

    Not being from the USA, every article I ever read about your education system just leaves me scratching my head.

    How on earth did you guys let it get so ridiculous??

  6. Re:Irrelevant by TheMeuge · · Score: 3, Interesting

    At least helicopter daddy and blackhawk mommy give a shit about the Precious. Or do you prefer the absent daddy and welfare mommy? People DO go overboard... but I feel like the pendulum is starting to swing entirely too far the other way.

  7. Works on Slashdot posts, too! by sootman · · Score: 5, Insightful

    Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.

    --
    Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
  8. Obligatory Offal by TheRealHocusLocus · · Score: 4, Interesting

    A modern Richard Guindon cartoon that best represents this Slashdot story ... an urban legend ... [1998, archived] essay on teachers' and students' increasingly virtual role in a tech society ... a mad hunt for the original 1963 New Yorker cartoon that started it all ... and an ugly mouse squeak toy.

    --
    <blink>down the rabbit hole</blink>