Grading Software Fooled By Nonsense Essay Generator
An anonymous reader writes "A former MIT instructor and students have come up with software that can write an entire essay in less than one second; just feed it up to three keywords.The essays, though grammatically correct and structurally sound, have no coherent meaning and have proved to be graded highly by automated essay-grading software. From The Chronicle of Higher Education article: 'Critics of automated essay scoring are a small but lively band, and Mr. Perelman is perhaps the most theatrical. He has claimed to be able to guess, from across a room, the scores awarded to SAT essays, judging solely on the basis of length. (It’s a skill he happily demonstrated to a New York Times reporter in 2005.) In presentations, he likes to show how the Gettysburg Address would have scored poorly on the SAT writing test. (That test is graded by human readers, but Mr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots.).'"
I though most schools don't even care about the essay. Also the elite schools nowadays prefer the ACT and SAT II subject tests to demonstrate real knowledge. The SAT is really a dumb test, especially with all the coaching resources available now.
While they'll eventually be able to calibrate the computers to recognize the three keyword gibberish essay, it sounds like a perfect tool for writing architectural essays and presenting projects in school. Use the following three keywords - Perhaps; Palimpsest; Paradigm - and you'll be ready to design a building!
As long as Precious gets an "A', Helicopter Daddy, and Blackhawk Mommy won't try to have the school president fired for ruining Precious's permanent record.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
Since the essays are grading subject knowledge, and it takes subject knowledge to provide the keywords, it is fairly irrelevant if the essay happens to be structured in a manner that is nonsensical.
Demonstration of deeper understanding, if it needs to be tested, can be achieved via other types of questions.
... because Slashdot shows that humans already make evaluations about articles without reading them.
When you're too lazy to read my essay to grade me and let software do it, I don't really see no moral problem with doing the same to write the essay.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Anyone else notice the article used Babel instead of babble?
"News" sites have been doing this for ages. (Including \.)
student athlete need some like this with 60 hours a week playing football they don't have time for class.
If they're using some stupid automated grader, odds are a computer-generated essay could consistently grade higher than any humans (because it can focus on scoring without worrying about content).
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
I don't see a problem with automated essay graders in principle. It's just that the current essay graders are no good. Once we are able to make computer software that can actually understand essays as well as a human it will be should be perfectly competent to grade an essay.
I certainly see the motivation to have a computer grade essays. Who wants to read multitudes of mediocre essays. I might rather be put in solitary confinement. I am all for the automated essay graders, but only after they can be proven to be as competent as a human.
I have no idea how to make a such a competent essay grader, but I do know how to grade an essay grader. You have a bunch of computer graders and human graders grading the same essays. If the computer graders show a more consistent performance than the humans (i.e. are the outlier less frequently), then the computer grader is better.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 1, 2, 3, 4, and the computer scores the paper as a 9, then it means that according to most of the human graders, the computer was way off. Essays are inherently subjective. Are the humans right or is the computer right? Who cares it doesn't matter.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 4, 5, 7, 9, and the computer scores the paper as a 6, then it means that according to every human grader, the computer did better than half the humans.
If a computer can do better than the humans even by human standards, then I think it's fair to say that a computer is good enough.
Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Whenever I had to do one of these I just copy and pasted the question or prompt a couple dozen times or until the length was good, never failed to work.
Unlike hard science or math problems, where the answer is either right or it isn't, essay grading is always subjective. And let's face it, the liberal arts majors who go on to become English teachers/professors aren't the brightest bulbs. They often fail to see the brilliance in an essay written by a student who's smarter then they are. If it doesn't appeal to their limited aesthetic, or fulfill all the checkboxes in some list in the Teacher's Guide, it gets a poor grade.
A modern Richard Guindon cartoon that best represents this Slashdot story ... an urban legend ... [1998, archived] essay on teachers' and students' increasingly virtual role in a tech society ... a mad hunt for the original 1963 New Yorker cartoon that started it all ... and an ugly mouse squeak toy.
<blink>down the rabbit hole</blink>
If your kid doesn't get the grades, he's working at Walmart or serving coffee. Who cares if he truly learned anything. He's gotta get in the door. And starting out a great GPA is the best way to go - I don't care what the self proclaimed hiring managers say here on Slashdot. Anyway, if I looked at their hiring practices, I'd find that they are fooling themselves.
If I've been hired to build a Potemkin village, then it would be unethical of me to spend time constructing interiors for the buildings.
The English department has some nice courses on compositional writing where I can get real feedback on my progress on those skills. As far as the machine-graded essays for any other Department -- either I understood the topic before writing the essay or I didn't and if I didn't then a no-feedback essay isn't going to fix the problem.
This and all of my slashdot comments were generated by an automated commenter and I always get modded up!
Each to each.
I do not think that they will sing to me....
Lawrence Person (lawrencepersonh@gmailh.com (remove all "h"s to mail)
http://www.lawrenceperson.com/
Unlike hard science or math problems, where the answer is either right or it isn't, essay grading is always subjective. And let's face it, the liberal arts majors who go on to become English teachers/professors aren't the brightest bulbs. They often fail to see the brilliance in an essay written by a student who's smarter then they are. If it doesn't appeal to their limited aesthetic, or fulfill all the checkboxes in some list in the Teacher's Guide, it gets a poor grade.
Still bitter about your poor grades in English, eh?
[nt]
File under 'M' for 'Manic ranting'
Bennett Haselton!
"The essays, though grammatically correct and structurally sound, have no coherent meaning"
Example from article: "Privateness has not been and undoubtedly never will be lauded, precarious, and decent.". There are too many comments on news sites which read like that.
The "rules" are the rules, be they right or be they wrong.
The objection you raise seems formally to be against the poster's ETHICS (... recognized rules of conduct...) rather than
"morality" (...according to an individuals' ideals and principles. ) http://www.diffen.com/difference/Ethics_vs_Morals
You have no particular charter to decide what someone else's 'moral obligations' might be. I find this apparent presumption ...rather appalling.
(This is intended to be NOT sarcastic:) The above aside, as a please accept sincere appreciation for your effort to provide useful feedback to students.
But, shockingly, intelligent students can just write any old random bullshit and get an equally good or better score. (the "bullshit" part might prove the article's point, except of course human-written bullshit works on people and auto-grading software)
Racism, sexism and other discrimination is quite effectively countered with anonymous grading. My university gave you a unique number before each exam and you put only that number on the sheets. Only afterwards did the administrators (not anyone involved in the course) look up and file the exam under your name. I found this helpful as a TA too because we really wanted to be fair both in grades and comments.
You can still be biased by the handwriting but we tried to counter that ourselves. If someone in my TA group recognized the handwriting of someone they knew we made sure to let someone else in the group grade that exam.
The solution might be to have a human sanity filter checking semantics and throwing out gibberish, and a computer grader doing the fair grading.
Automated essay-grading software is a conspiracy to allow high grades to stupid niggers who almost always write nonsense essays (if any at all). Mean white people will be getting low marks even if they essays are good. But you cannot argue a fair machine grading, can you? Everything is done to 'fight racism' and 'celebrate diversity', only to actually commit white genocide by reducing white population to below 20% in white countries. This is all anti-white racism.
You can now start your hating of what I wrote. Just remember you were informed. You could have stopped it.
Insightful, but mindless anti-racist zombies will vote for Flamebait and -1.
Finally we know where some of these Slashdot articles are generated!
Gently reply
They maybe its time they weren't taking an academic place some else could use and colleges had separate dedicated football academies.
The right wingers here (and their ex-currency trader, cheesy smiled leader) have been trying desperately to beat on them but NZ has one of the best bang for buck education systems in the world. (i.e. Our teachers are not paid that high but the performance indicators are in the top grouping.)
The key here is to realise that "best bang for buck education system in the world" is not all that impressive. Quite possibly, the education system is good despite the teacher's unions. Perhaps unions are not so powerful in NZ as they are elsewhere. Perhaps further neutering the unions and privatising education would improve the education system.
Quick! Where's the German version? I need to boost my sociology grades!
Seriously, the first thing you have to thouroughly disable when doing sociology is your brain and any sense of logic or common sense in it. The bizar bullshit that is put out in this field even at academic level is mindboggling. The blatant non-sense that's in the books and readers of this subject is unbelievable. ... I need that generator to keep my braincells from killing themselves to end the agony.
We suffer more in our imagination than in reality. - Seneca
There's no education happening there anyway. Most of the kids don't want to be there. Out of 10 kids you might have one that actually wants to learn. Most of the teachers don't want to be there. Why would anyone want to do a job everyone takes for granted, and get spit on in the process? No respect on any side be it teachers or students. Administration is more focused on how to stay out of court, and get funding for next year than they are on the educational process. It sucks for all parties involved (most of all the American taxpayer). Half the graduating students still can't even read at a third grade level. The "poor" coming out of these institiutions are still going to be poor because they didn't bother to learn anything while they were there. It's not revenge. It's called placing finite resources where they provide the most benefit instead of the least. I understand the premise of educating everybody. What actually happens though doesn't quite line up with that utopian ideal.
What they teach is the White Man lecturing the Black Man on what he "should" know. At least in the worst of the schools which is what you describe. The magnet schools are doing great. My nephews *love* to go to school. Learning Russian as a 3rd grader is fun and exciting. The program also helped organize a US-Russian hockey game with the Russian students, so they got to practice Russian in Russia. No idea how they did in the tournament.
But back to the worst performing, there is no trust by the parents of the teachers, school, or administrators. Probably because they don't have the student's best interests at heart (certainly not the adminstrators).
I think that the issue is that racists tried to make the schools for blacks worse (As they did explicitly under segregation, now done by targeting "poor" schools), and that betrayal has never been addressed. You can't teach someone that hates you.
If we simply ended truancy and mandatory jail/school, it would probably help. That and make them open for all ages at any time, with expanded night, advanced, and non-traditional classes. Need to work for a few years to support the family at age 14? Fine. Come back any time. Want to take a year off at age 12 to find yourself? Sure, so long as your parents give permission. The jail-like-ness of it prevents education. Especially when teachers can't expel students, and police are on site. It's as close to prison/slavery as we get in our society anywhere.
The problem is whenever we loosen the grip on students, they make poor choices for themselves, when they are legally incapable of making choices. So the question is, do we let uneducated adults in the world, and if so, what do we do when they can't care for themselves? Do we let people die in squalor? Or are we more "civilized" than that? I've never seen a school-lostening plan that didn't (implicitly or explicitly) sentence millions to a horrible death. How would your system handle those who make poor choices for themselves?
Learn to love Alaska
I did not see one place in the article that said what grading software they were using.
I took a class to help me with the MCAT. In the class they gave a bunch of examples of good "writing" for the essay portion. To me it seemed as though the real trick was just mention Martin Luther King or Gandhi as your counter example and you were pretty much guaranteed at least a 4 but you'd probably get a 5 or 6 even if you didn't follow the 3 requirements. (Supposedly if you didn't do the 3 requirements the most you should have been able to get was a 2. A 3 was supposed to be almost automatic if you did the requirements and 4 was good.) Suffice it to say in theory that scoring is what should have happened, in practice pretty much do that and get a good grade even if you didn't do the requirements.
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.