Slashdot Mirror
Grading Software Fooled By Nonsense Essay Generator
An anonymous reader writes "A former MIT instructor and students have come up with software that can write an entire essay in less than one second; just feed it up to three keywords.The essays, though grammatically correct and structurally sound, have no coherent meaning and have proved to be graded highly by automated essay-grading software. From The Chronicle of Higher Education article: 'Critics of automated essay scoring are a small but lively band, and Mr. Perelman is perhaps the most theatrical. He has claimed to be able to guess, from across a room, the scores awarded to SAT essays, judging solely on the basis of length. (It’s a skill he happily demonstrated to a New York Times reporter in 2005.) In presentations, he likes to show how the Gettysburg Address would have scored poorly on the SAT writing test. (That test is graded by human readers, but Mr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots.).'"
I though most schools don't even care about the essay. Also the elite schools nowadays prefer the ACT and SAT II subject tests to demonstrate real knowledge. The SAT is really a dumb test, especially with all the coaching resources available now.
As long as Precious gets an "A', Helicopter Daddy, and Blackhawk Mommy won't try to have the school president fired for ruining Precious's permanent record.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
Since the essays are grading subject knowledge, and it takes subject knowledge to provide the keywords, it is fairly irrelevant if the essay happens to be structured in a manner that is nonsensical.
Demonstration of deeper understanding, if it needs to be tested, can be achieved via other types of questions.
... because Slashdot shows that humans already make evaluations about articles without reading them.
When you're too lazy to read my essay to grade me and let software do it, I don't really see no moral problem with doing the same to write the essay.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
It sounds like the software would be perfect for writing audit reports. You hand in a phone book sized report, but all they ever read is the management summary.
But DARE to hand over just the relevant pages that you know will get read. Did you work at all, if THAT is your whole report?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
student athlete need some like this with 60 hours a week playing football they don't have time for class.
If they're using some stupid automated grader, odds are a computer-generated essay could consistently grade higher than any humans (because it can focus on scoring without worrying about content).
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
Your right you are encouraged to write long documents, but it should really be the opposite, writing is about communicating, if your document is so long that people don't bother reading it, the document has failed in its main purpose.
This standard should be applied to legal documents, such as License agreements, Insurance agreement, What your ELA is more than 100 words long, you don't expect anyone to read this do you? Agreement Invalid. If you need longer it should ensure that people understand what they are agreeing to, maybe run a 1 year course of something.
100 words yes!
I don't see a problem with automated essay graders in principle. It's just that the current essay graders are no good. Once we are able to make computer software that can actually understand essays as well as a human it will be should be perfectly competent to grade an essay.
I certainly see the motivation to have a computer grade essays. Who wants to read multitudes of mediocre essays. I might rather be put in solitary confinement. I am all for the automated essay graders, but only after they can be proven to be as competent as a human.
I have no idea how to make a such a competent essay grader, but I do know how to grade an essay grader. You have a bunch of computer graders and human graders grading the same essays. If the computer graders show a more consistent performance than the humans (i.e. are the outlier less frequently), then the computer grader is better.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 1, 2, 3, 4, and the computer scores the paper as a 9, then it means that according to most of the human graders, the computer was way off. Essays are inherently subjective. Are the humans right or is the computer right? Who cares it doesn't matter.
If a paper is scored by 4 human judges and a computer, and the humans score the paper 4, 5, 7, 9, and the computer scores the paper as a 6, then it means that according to every human grader, the computer did better than half the humans.
If a computer can do better than the humans even by human standards, then I think it's fair to say that a computer is good enough.
Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Whenever I had to do one of these I just copy and pasted the question or prompt a couple dozen times or until the length was good, never failed to work.
Reference to (Babel, Tower Of).
The story is a biblical "explanation" of why humanity, despite ostensibly originating as a single tribe, uses multiple languages.
A modern Richard Guindon cartoon that best represents this Slashdot story ... an urban legend ... [1998, archived] essay on teachers' and students' increasingly virtual role in a tech society ... a mad hunt for the original 1963 New Yorker cartoon that started it all ... and an ugly mouse squeak toy.
<blink>down the rabbit hole</blink>
If I've been hired to build a Potemkin village, then it would be unethical of me to spend time constructing interiors for the buildings.
The English department has some nice courses on compositional writing where I can get real feedback on my progress on those skills. As far as the machine-graded essays for any other Department -- either I understood the topic before writing the essay or I didn't and if I didn't then a no-feedback essay isn't going to fix the problem.
What are you talking about? I've gotten lots of good paying jobs, and nobody once looked at my grades. Except when applying for further education, and even then, they aren't important if you test well. Where have you seen where a transcript is required for a job application? Never, that's why so many CEOs get caught lying on resumes (until they post to LinkedIn and someone recognizes them and knows they didn't get what they claim and turn them in). Even the $10,000,000 a year jobs don't look at actual grades. But no, some AC claims that grades matter. So they must, even if they don't.
Learn to love Alaska
Hiring by big companies for internships and recruiting for new college hires are usually filtered by GPA before any engineer or manager sees the stack or resumes. I never had a company care about my grades, but then I was never actively recruited at college age, and so my first dev job was remarkably exploitive, not one of the good ones.
Beyond one's first full-time job, I can't imagine grades ever coming up.
Socialism: a lie told by totalitarians and believed by fools.
Reference to (Babel, Tower Of).
The story is a biblical "explanation" of why humanity, despite ostensibly originating as a single tribe, uses multiple languages.
I could be wrong, but I think it's understood primarily as an allegory regarding man sinning(?) by aspiring to accomplish what only God can.
Each to each.
I do not think that they will sing to me....
Lawrence Person (lawrencepersonh@gmailh.com (remove all "h"s to mail)
http://www.lawrenceperson.com/
Yes, but they run it through spinbot.com before it becomes official regullations.
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
[nt]
File under 'M' for 'Manic ranting'
Example from article: "Privateness has not been and undoubtedly never will be lauded, precarious, and decent.". There are too many comments on news sites which read like that.
Hiring by big companies for internships and recruiting for new college hires are usually filtered by GPA before any engineer or manager sees the stack or resumes.
That's a really dumb thing to do, since in most fields the majority of the best performers were not the ones with the highest grades in school, and vice versa.
Statistically you're usually better off taking the B or A- student than the one who got a 4.0.
Well, not this time buddy!
But, shockingly, intelligent students can just write any old random bullshit and get an equally good or better score. (the "bullshit" part might prove the article's point, except of course human-written bullshit works on people and auto-grading software)
Racism, sexism and other discrimination is quite effectively countered with anonymous grading. My university gave you a unique number before each exam and you put only that number on the sheets. Only afterwards did the administrators (not anyone involved in the course) look up and file the exam under your name. I found this helpful as a TA too because we really wanted to be fair both in grades and comments.
You can still be biased by the handwriting but we tried to counter that ourselves. If someone in my TA group recognized the handwriting of someone they knew we made sure to let someone else in the group grade that exam.
The solution might be to have a human sanity filter checking semantics and throwing out gibberish, and a computer grader doing the fair grading.
Finally we know where some of these Slashdot articles are generated!
Gently reply
This standard should be applied to legal documents, such as License agreements, Insurance agreement, What your ELA is more than 100 words long, you don't expect anyone to read this do you? Agreement Invalid. If you need longer it should ensure that people understand what they are agreeing to, maybe run a 1 year course of something.
You do realize that even the 3 clause BSD licence is more than double that with 220 words? And that one basically says "do what you like, but we take no responsibility" and if you have a license that actually tries to say anything like the GPL 3.0 it is 4632 words, not including the preamble or how to apply. How many years of your life would you like to waste? If anybody cared, we'd rather see the development of "standard terms and conditions" which would be several thousand words long but also widely deployed on most COTS software. They generally want my money, which means as long as I pay them and they deliver something I want to pay for nobody really cares what the contract says.
That usually takes one of two forms, either they want to terminate it because they don't like my use or I want to terminate it because they're not delivering as promised. Since they write the contract and most don't care, the contract says they can terminate you for anything, any time and they promise essentially nothing. Practically I figure that the courts will protect me if there's anything really unconscionable in the contract and if they need an excuse to terminate they'll find it and it won't be worth fighting it in court anyway so the actual legal text almost doesn't matter. A license doesn't have to be negotiated, if an author wants to tell me "it's the GPLv3 or not at all" is just the same as an EULA saying "it's these terms or not at all", both sides don't have to give anything.
Live today, because you never know what tomorrow brings
My automated modding bot categorizes you as "troll", so there!
Quick! Where's the German version? I need to boost my sociology grades!
Seriously, the first thing you have to thouroughly disable when doing sociology is your brain and any sense of logic or common sense in it. The bizar bullshit that is put out in this field even at academic level is mindboggling. The blatant non-sense that's in the books and readers of this subject is unbelievable. ... I need that generator to keep my braincells from killing themselves to end the agony.
We suffer more in our imagination than in reality. - Seneca
If you have enough applicants and only a few positions, you're better off taking the best performers who also got a 4.0. In the extremely competitive internships and fellowships, you can afford (stats-wise) to target only the best tail of the distribution and the outliers.
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
Stupid or not, it's how the world works. You have to filter a horde of students down to a pile of resumes small enough to go through by some criteria simple enough that an HR drone can manage it, while fulfilling a bewildering array of state and local requirements for fairness in hiring.
Heck, at the bigger shops today, managers and engineers don't even get involved in the process for interns/NCGs until the interviews start - no voice in the filtering process at all. Diversity suffers, otherwise.
Socialism: a lie told by totalitarians and believed by fools.
Do you actually have hard evidence for this, or is this prejudice? I'd really hate to think about going through the hiring process for an employer who looked down on me because I did too well in school.
If you have enough applicants and only a few positions, you're better off taking the best performers who also got a 4.0. In the extremely competitive internships and fellowships, you can afford (stats-wise) to target only the best tail of the distribution and the outliers.
You missed the point.
You don't know who the best performers are, in advance. (Remember the context here is taking people right out of school.) You only know what the grades are.
Statistically, those who got 4.0 are not the best performers in later life. Some may be, but most of them aren't. And you don't know which ones those are.
A smart employer will hire the people who give him the best statistical chances of getting the best performers. That means hiring the A- and B students.
Do you actually have hard evidence for this, or is this prejudice?
I don't have a citation on hand, but I have read past studies on this.
I just did a quick Google search and didn't find it... but that's simple because everything I searched for was drowned out by references to "predictors of academic performance" rather than academic performance as a predictor. But even where an article is about the latter, it doesn't necessarily refer to what I was saying about 4.0 students.
In any case, I will summarize:
First, there is in general a strong correlation between adolescent academic performance and later lifetime achievement. But there are 2 things to note about this: (A) that applies to adolescence, not post-secondary school, and (B) the general correlation does not necessarily hold at the extremes.
A number of studies have shown that 4.0 students often do not perform as well later in life as others who got good, but not quite stellar, grades. The reasons for this are not entirely clear.
Just as, for example, those of genius IQ do not necessarily achieve greatness. Nobody is entirely sure why. Some might be inept socially. Some might have little but contempt for the "stupid" people around them. Who knows?
What they teach is the White Man lecturing the Black Man on what he "should" know. At least in the worst of the schools which is what you describe. The magnet schools are doing great. My nephews *love* to go to school. Learning Russian as a 3rd grader is fun and exciting. The program also helped organize a US-Russian hockey game with the Russian students, so they got to practice Russian in Russia. No idea how they did in the tournament.
But back to the worst performing, there is no trust by the parents of the teachers, school, or administrators. Probably because they don't have the student's best interests at heart (certainly not the adminstrators).
I think that the issue is that racists tried to make the schools for blacks worse (As they did explicitly under segregation, now done by targeting "poor" schools), and that betrayal has never been addressed. You can't teach someone that hates you.
If we simply ended truancy and mandatory jail/school, it would probably help. That and make them open for all ages at any time, with expanded night, advanced, and non-traditional classes. Need to work for a few years to support the family at age 14? Fine. Come back any time. Want to take a year off at age 12 to find yourself? Sure, so long as your parents give permission. The jail-like-ness of it prevents education. Especially when teachers can't expel students, and police are on site. It's as close to prison/slavery as we get in our society anywhere.
The problem is whenever we loosen the grip on students, they make poor choices for themselves, when they are legally incapable of making choices. So the question is, do we let uneducated adults in the world, and if so, what do we do when they can't care for themselves? Do we let people die in squalor? Or are we more "civilized" than that? I've never seen a school-lostening plan that didn't (implicitly or explicitly) sentence millions to a horrible death. How would your system handle those who make poor choices for themselves?
Learn to love Alaska
If you have enough applicants and only a few positions, you're better off taking the best performers who also got a 4.0.
There is one thing and one thing only that does. Screens for the person who takes tests. If I was hiring person to take tests, I'd only hire those with 4.0 averages.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
You're clearly missing my point.
You're not going to exclude the 4.0ers and then randomly select from what's left, right? So apply your selection criteria to the smaller set of people with 4.0 and see if anyone makes it. When you have thousands or tens of thousands of people applying for a single fellowship, statistics say you are very likely to find someone who is awesome at everything. Why would you pass them up just because they don't fall in the center of the distribution?
These extremely competitive internships and fellowships will look for actual indicators of performance in addition to grades. The two need not be mutually exclusive.
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
You can't conceive of a person who's good a taking tests and also good at other things? How would a demonstrated ability in more than one area possibly count against an applicant?
Nobody is suggesting making the choice solely on grades. I mean, I specifically said that in the sentence you quoted.
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
You can't conceive of a person who's good a taking tests and also good at other things?
Sure, anything is possible. But I've seen too many professional test takers in my life to do anything but be suspicious when seeing those 4.0 averages.
It is not dissimilar to talent. Without some drive, talent is worse than useless. One of the most talented people I ever had work for me was also one of the laziest. Give him a task, and the first things he did was anything but the task. He'd go get a soda, then he would sniff around some of the staff assistants he wanted to lay. Then he might get a snack. I covered for so many of his missed deadlines that I got in trouble for covering his "going to be missed" deadlines.
But yeah, he was really talented. Fired his ass. His talent got him a lot of jobs he was fired for after a year or so.
While we might want to say the old chestnut "All other things being equal" - yeah, but all things are not often equal.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
You're not going to exclude the 4.0ers and then randomly select from what's left, right?
I see. Yes, I misunderstood you.
I write things for myself, which I will likely never read again. People often write things for the purpose of documenting details which will likely not, but may, be used in the future.
I took a class to help me with the MCAT. In the class they gave a bunch of examples of good "writing" for the essay portion. To me it seemed as though the real trick was just mention Martin Luther King or Gandhi as your counter example and you were pretty much guaranteed at least a 4 but you'd probably get a 5 or 6 even if you didn't follow the 3 requirements. (Supposedly if you didn't do the 3 requirements the most you should have been able to get was a 2. A 3 was supposed to be almost automatic if you did the requirements and 4 was good.) Suffice it to say in theory that scoring is what should have happened, in practice pretty much do that and get a good grade even if you didn't do the requirements.
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.