Slashdot Mirror
Grading Software Fooled By Nonsense Essay Generator
An anonymous reader writes "A former MIT instructor and students have come up with software that can write an entire essay in less than one second; just feed it up to three keywords.The essays, though grammatically correct and structurally sound, have no coherent meaning and have proved to be graded highly by automated essay-grading software. From The Chronicle of Higher Education article: 'Critics of automated essay scoring are a small but lively band, and Mr. Perelman is perhaps the most theatrical. He has claimed to be able to guess, from across a room, the scores awarded to SAT essays, judging solely on the basis of length. (It’s a skill he happily demonstrated to a New York Times reporter in 2005.) In presentations, he likes to show how the Gettysburg Address would have scored poorly on the SAT writing test. (That test is graded by human readers, but Mr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots.).'"
I though most schools don't even care about the essay. Also the elite schools nowadays prefer the ACT and SAT II subject tests to demonstrate real knowledge. The SAT is really a dumb test, especially with all the coaching resources available now.
As long as Precious gets an "A', Helicopter Daddy, and Blackhawk Mommy won't try to have the school president fired for ruining Precious's permanent record.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
... because Slashdot shows that humans already make evaluations about articles without reading them.
When you're too lazy to read my essay to grade me and let software do it, I don't really see no moral problem with doing the same to write the essay.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Did you happen to read TFA? In the TFA, it is said that the College Board does not take points off for factual errors. In fact, it says that it cares not for factual errors, because errors in fact seldom subtract from the quality of the essay being graded.
WTF, right?
It sounds like the software would be perfect for writing audit reports. You hand in a phone book sized report, but all they ever read is the management summary.
But DARE to hand over just the relevant pages that you know will get read. Did you work at all, if THAT is your whole report?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Not being from the USA, every article I ever read about your education system just leaves me scratching my head.
How on earth did you guys let it get so ridiculous??
student athlete need some like this with 60 hours a week playing football they don't have time for class.
Your right you are encouraged to write long documents, but it should really be the opposite, writing is about communicating, if your document is so long that people don't bother reading it, the document has failed in its main purpose.
This standard should be applied to legal documents, such as License agreements, Insurance agreement, What your ELA is more than 100 words long, you don't expect anyone to read this do you? Agreement Invalid. If you need longer it should ensure that people understand what they are agreeing to, maybe run a 1 year course of something.
100 words yes!
I don't see a problem with automated essay graders in principle. It's just that the current essay graders are no good. Once we are able to make computer software that can actually understand essays as well as a human it will be should be perfectly competent to grade an essay.
I certainly see the motivation to have a computer grade essays. Who wants to read multitudes of mediocre essays. I might rather be put in solitary confinement. I am all for the automated essay graders, but only after they can be proven to be as competent as a human.
I have no idea how to make a such a competent essay grader, but I do know how to grade an essay grader. You have a bunch of computer graders and human graders grading the same essays. If the computer graders show a more consistent performance than the humans (i.e. are the outlier less frequently), then the computer grader is better.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 1, 2, 3, 4, and the computer scores the paper as a 9, then it means that according to most of the human graders, the computer was way off. Essays are inherently subjective. Are the humans right or is the computer right? Who cares it doesn't matter.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 4, 5, 7, 9, and the computer scores the paper as a 6, then it means that according to every human grader, the computer did better than half the humans.
If a computer can do better than the humans even by human standards, then I think it's fair to say that a computer is good enough.
Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
A modern Richard Guindon cartoon that best represents this Slashdot story ... an urban legend ... [1998, archived] essay on teachers' and students' increasingly virtual role in a tech society ... a mad hunt for the original 1963 New Yorker cartoon that started it all ... and an ugly mouse squeak toy.
<blink>down the rabbit hole</blink>
If I've been hired to build a Potemkin village, then it would be unethical of me to spend time constructing interiors for the buildings.
The English department has some nice courses on compositional writing where I can get real feedback on my progress on those skills. As far as the machine-graded essays for any other Department -- either I understood the topic before writing the essay or I didn't and if I didn't then a no-feedback essay isn't going to fix the problem.
What are you talking about? I've gotten lots of good paying jobs, and nobody once looked at my grades. Except when applying for further education, and even then, they aren't important if you test well. Where have you seen where a transcript is required for a job application? Never, that's why so many CEOs get caught lying on resumes (until they post to LinkedIn and someone recognizes them and knows they didn't get what they claim and turn them in). Even the $10,000,000 a year jobs don't look at actual grades. But no, some AC claims that grades matter. So they must, even if they don't.
Learn to love Alaska
Reference to (Babel, Tower Of).
The story is a biblical "explanation" of why humanity, despite ostensibly originating as a single tribe, uses multiple languages.
I could be wrong, but I think it's understood primarily as an allegory regarding man sinning(?) by aspiring to accomplish what only God can.
I have checked a bunch of websites and some searching and found no link to this babel generator or even a small excerpt from the submitted paper. I would have expected at least one if not both to be easily found.
Teachers are in strong unions also here in NZ. (despite anti union legislation decimating them in the last decade)
The right wingers here (and their ex-currency trader, cheesy smiled leader) have been trying desperately to beat on them but NZ has one of the best bang for buck education systems in the world. (i.e. Our teachers are not paid that high but the performance indicators are in the top grouping.)
Just wanted to mention that for the inevitable people who will read your comments and think "unions baaaad" like some ideological zombie.
Do you actually think either party has a goal other than the best schools?
Yes. I honestly believe that the Republicans want to disband public education and have a merit-based entry to private schools (parent's merit, not children's), paid for with taxpayer dollars. It's "revenge" for having forced them to educate the poor for so many years.
I've never heard anyone arguing for no public schools,
I have. Charter-schools and for-profit private schools only, and they would be banned by law from having unions and could reject children from admission for arbitrary reasons (including race and religion).
I've gone to plenty of party meetings for Libertarians and Republicans, and I've seen what some people have advocated.
Learn to love Alaska
Racism, sexism and other discrimination is quite effectively countered with anonymous grading. My university gave you a unique number before each exam and you put only that number on the sheets. Only afterwards did the administrators (not anyone involved in the course) look up and file the exam under your name. I found this helpful as a TA too because we really wanted to be fair both in grades and comments.
You can still be biased by the handwriting but we tried to counter that ourselves. If someone in my TA group recognized the handwriting of someone they knew we made sure to let someone else in the group grade that exam.