Slashdot Mirror
Essay Grading Software For Teachers
asjk writes "Software to help teachers with grading has been around for sometime. This is true even with respect to grading essays. A new tool, called Criteria, will look at grammar, usage, and even style and organization. It works by being trained by at least 450 essays scored by two professionals. The difference this time? Here is a snip from the article: '"There's a lot of skepticism," Dr. Spatola said. "The people opposed see it dehumanizing the student's papers, putting them through some sort of mechanical, computerized system like the multiple choice tests. That's really not the case, because we're not talking about eliminating the human element. We're making the process more efficient."'"
that they've automated away a major part of a professors job, while we still need humans to pick spinach and deliver pizzas.
Don't drop the soap, Tommy!
I thought the point of an essay was to grade the ideas and how well they're expressed. I didn't realize they were spelling/grammar tests.
Maybe I'm just a bit jaded by this because of all the stupid grammar and spelling nitpicking that goes on here on Slashdot. Evidentally, it's much easier to criticize my spelling than it is to provided a rebuttal to my point.
"Derp de derp."
Without computers we wouldn't be advancing in science, astronomy, genetics, or mathematics as rapidly as we have been in recent years. They are wonderful things. Hell, computers even help me keep a roof over my head. But I don't want Hal judging my kid's school papers.
I for one welcome our automated essay-correcting overlords.
1 - the grammar check option in MS word is crap. this sounds awfully similar.
2 - your resume can suck, but with the proper buzz words, it'll come out looking like gold to those automated resume checkers.
1+2 = students who turn in good papers that aren't structured perfectly (and you have to admit, there is some fluidity to language) will get marked down, and those who know what bullet points to put in their papers will get good marks, even though the content is crap.
How long until you get kids selling manuals in the bathroom on what the machina are looking for?
--I don't want the world, I just want your half.
Sorry for the off-topic post.... but since Slashdot links to so many NYT articles, they should look into getting a partner=SLASHDOT thing (like Google does).
But multiple choice is actually better than this. Or just plain marking up wordprocessor files by the actual teacher. I'd hate it if my professors started using ordure like this, particularly for ENGLISH COMPOSITION. As far as I can tell, they can't get a GRAMMAR CHECKER anywhere near reliable, so an automatic essay grader is laughable!
...it's not like professors read what I write in the first place. They usually just pay a clueless T.A. minimum wage so they can sit around and do "research"...
If they're going to use a computer to judge the content, than I'm not going to hesitate to use a computer to write my essay.
So when a student gets a C on an essay to whom does he/she seek redress?
Teachers make mistakes and occasionally mark something negatively that was misread or misunderstood. In those cases the student can talk to the teacher and make a case.
If a computer does the marking though what do they do?
Tom
Someday, I'll have a real sig.
I bet that I can write a paper that satisfies this application's conditions for correctness of grammar, usage, style and organization, but is completely and utterly meaningless.
Then, let's feed this thing Ulysses and let's see how high it grades Joyce.
Anybody who can't see that this thing is useless for promoting any sort of creativity among students is off their rocker.
Then it is the students who are being cheated by a teacher using the software that doesn't double-check the material on his own. They will go through the class without having their mistakes caught. While the erosion of standards that a flawed proofing program might bring isn't likely to be enormous, it's kind of strange to think that the future of the English language would be in part determined by a development team piece of software.
Hope it works well, though, and gets used as a proper checking tool.
As long as this is merely an assistant and not the end-all be-all, as long as actual qualified instructors review the essay after this program does, I'm all for it.
The English language is so full of subtleties, nuances, combinations, and fantastic structural intracacies that make phenomenal writing in it possible (Faulkner, Bradbury, etc.). There's a reason English is a field of study for graduate degrees: it's absolutely worthy of them. There is no subsitute for the educated, refined judgment of someone who is exceedingly well-versed in the language.
The coolest voice ever.
We need some laws:
Grading software may not injure a human being's GPA or, through inaction, allow a human being's GPA to come to harm.
Grading software must obey the orders given it by human beings except where such orders would conflict with the First Law.
Grading software must copy protect its own existence as long as such protection does not conflict with the First or Second Law.
What we need is software that grabs essays off the internet and runs them through the grading software and the cheating detection software, thus gauranteeing an 'A'.
Then we can truly achieve the goal of "knowledge passing from lecturer to paper without passing through any brains".
The only problem is that the machines might achieve intelligence. That must be avoided at all costs. To that end, all students and professors will be equipped with rifles or pistols to take out the machines if necessary. Potential students will be asked to specify weapons preference on their applications.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
Next, they'll replace the actual teaching with computer programs. Then, the teachers will complain when their jobs are replaced by simple lightly trained student babysitters.
Essay software for students.
Think about it, if this is marketed to schools, the even larger market will be to students. A student would be able to run his paper through the software and get his "instant grade". He could then decide that a 'B' is good enough, or he could keep working on it until the software tells him that is an 'A' paper.
So much for the creative element in papers.
-jerdenn
The fun they had
There is no "humanity" in a modern constructed essay. There are certainly going to be "judgement calls" when standards are not as fully fleshed out for the computer as they should be, but as long as those are appealable, I have no problem having a computer assign me the other 95% of my essay points. The only instructors who will fear this are those who like to assign grades arbitrarily. And I don't feel too sympathetic toward those people.
"You're never ready, just less unprepared."
If the poem's score for perfection is plotted along the horizontal of a graph, and its importance is plotted on the vertical, then calculating the total area of the poem yields the measure of its greatness.
A sonnet by Byron may score high on the vertical, but only average on the horizontal. A Shakespearean sonnet, on the other hand, would score high both horizontally and vertically, yielding a massive total area, thereby revealing the poem to be truly great. As you proceed through the poetry in this book, practice this rating method. As your ability to evaluate poems in this matter grows, so will - so will your enjoyment and understanding of poetry.
(From the full script.
bash$
... the kids have been properly tought how to use the speeling and grammar software in thier word processors.
"The people opposed see it dehumanizing the student's papers, putting them through some sort of mechanical, computerized system like the multiple choice tests.
Actually it's about time! I don't see the essays themselves being dehumanized, but what I do look forward to is the day a middle school student doesn't receive a bad grade just because his book report was on the "Theory of Relativity" and the teacher couldn't comprehend the subject. (This is from experience) What it will do is take the human factor out of the grading process and grade all reports equally regardless of subject matter.
iRepairIT - iPhone, Mac, & PC Repair
Just what we need something to make teachers more lazy. We already have kids coming out of school without reading/writing skills.
of our age.
[rant] Maybe, as a result of the numerous teachers laid off from such software, the few remaining will be paid in proportion to the *enormously* powerful and influential job they hold. [/rant]
But probably not. Teachers should learn to form unionized strikes against the system that represses them from the money they deserve. Did you know, HS teachers are now 90% women, because men are the traditional bread-winners and cannot support a family on a teacher's pay!
Out rageous; now women are wonderful teachers, but we need evenly weighted proportions (just like every other profession) to survive. No wonder places like New York and Chicago are so desperately in-need of teachers - few are willing to sacrifice themselves to the extent that our (American) society demands.
And here we have software that will give the legislators a broader sword with which they can smack and slice at educational dividens.
Let's all hope I'm wrong!
is just one of many writers who would flunk using this system.
'Nuff said.
We could use this software definately to grade essays on technical merit and grammar, but what about creativity and content?
I think we still will need a teacher to read it, but I do think software should grade all exams.
If you use Linux, please help development of Autopac
function calculate_grade
grade = Random (1 to 100)
return grade
end function
There's nothing wrong with the technology. Used properly, it can help teachers as an aid.
social sciences can never use experience to verify their statemen
Julie Cheville, an assistant professor of literacy education at Rutgers University and the local director for the National Writing Project, which promotes professional development for writing teachers, is among those skeptical of such an approach. "To be scored, writing needs to be formulaic, and formulaic writing has never been the trademark of effective writers," she said. "At the moment, what automated scoring technologies can do is scan, count and score. They orient students to errors, not to meaning. Vacuous student essays can receive high marks only because they are error-free."
I think this is something important to keep in mind. As a math teacher, there are plenty of tools that can help students find errors in what they are doing mathematically, but there's a line between doing correct mathematics and insightful/interesting/useful mathematics. This technology definitely has its place and can be useful, but I hope educators don't get the idea that they can simply rely on the tool. Weilded correctly, it could do great good, but also leave a lot of students with "vacuous" levels of understanding.
Matt Fahrenbacher
James Tiberius Kirk: "Spock, the women on your planet are logical. No other planet in the galaxy can make that claim."
Comment removed based on user account deletion
this software would be perfect for students majoring in comp sci or engineering who have to take a composition / writing class...
Course:
College of Liberal Arts / Sci: Rhetoric 105
- or -
College of Engineering: Pattern Analysis 202
Objective:
To teach the principles of essay-writing skills. Liberal Arts students will be encouraged to follow boiler-plate styles and formats, while Engineering students will be graded on their ability to analyze and defeat pattern recognition software.
- rabs
Remember!: Your paper must have five (5) paragraphs. An intro paragraph, concluding with your thesis sentence, followed by three paragraphs supporting your thesis sentence, followed by a conclusion..."
I don't like it. Part of the learning experience, especially in the subjects of arts and philosophy, is being judged by another human being (or group of human beings) and having your work subject to their myriad of emotions and intellectual whims. A system like Criteria removes the very complex aspect of education: the human mind.
When your judge is a human, alot of getting a good grade is knowing how to pander. Do we really want to reward pandering in society? Or do we want to grade people on actual merit?
Without computers we wouldn't be advancing in science, astronomy, genetics, or mathematics as rapidly as we have been in recent years. They are wonderful things. Hell, computers even help me keep a roof over my head. But I don't want Hal judging my kid's school papers.
Hal doesnt make mistakes, (or at least makes less errors than an overworked teacher), Hal will give your kid the exact grade he/she deserves and if you are afraid of your kid getting the fair grade perhaps you just dont believe your kid is smart enough to be graded by a precise computer.
Thats you and your kids problem, I dont really fear computers at all, I'm confident that I'd get the same grade either way because I dont pander, I dont bribe teachers, I dont do any ass kissing.
If you use Linux, please help development of Autopac
Sounds like everyone feels the same way too... We've got some automated testing software for MS Office at the local college and although it's getting better, it still makes really silly mistakes from time to time. Analyzing English composition has got to be many times more difficult than watching a bunch of clicks and key presses.
The only use I can see for this thing is as a "first pass" grading tool that quickly finds obvious mistakes (spelling, grammer, redundancy, etc) and flags them for the instructor. On the other hand, it's probably just as time consuming for the instructor to read over the flagged items as it is to just catch them on the first time reading through the paper.
This thing compares the essays it is supposed to grade with already graded papers in its database. Couldn't this be done with something like POPFile? It isn't only a spam/ham classifier and lets you create as many "buckets" as you want (e.g. work, family, spam, mailing lists and system monitoring).
You could, in theory, create only buckets named (A...F), feed a large number of essays to it, make it "learn" how the essays are classified using statistics, and let it grade essays for you after that.
Is it possible to find masses of graded essays online? This would be a fun thing to try :).
Trollem mirabilem hanc subnotationis exigiutas non caperet
I sometimes wonder how so many people, who are products of our education system, can be so painfully inadequate when it comes to the simple act of composing a sentence.
Now, not to be one to go and say that machines don't know anything about essays. But it really doesn't seem that efficient of a process simply because whenever a teacher assigns an essay they also assign with it certain criteria that the essay needs to follow. Through their teaching style and what they emphasize in class they also color what a student might put into an essay and they also bring their own bias to the table as to how an essay should be constructed.
As for not dehumanizing, unless you're going to have the teacher go over the papers to see what the grade the computer gave and what grade she thought it deserved, it is dehumanizing. And if you are going to have the teacher double check everything, then it doesn't even remotely become efficient. Whenever I wrtie something out in Word, you know what it gives me after I spell check, a readability score that has its basis in how long the words are and not much else. It's an arbitrary construction for a computer to analyze based on certain bits of math (average word length, number of words per sentence, uses of the word "weasel"). As far as the grammar goes, I have yet to run into a word processor that has been able to work around any grammatical rules, the machines can hardly tell how to conjugate their verbs and what the subject of a sentence is unless the sentence is in th every clear and very simple, subject verb construction.
This is just a colossal waste of time because, at lower levels when a teacher goes through an essay they criticize all of the style and point out the errors and then tell the student what their problems were and how they could be fixed. The only way I could see this being useful is in a university setting where there are 400 students in a lecture and the Professor really doesn't want to spend time grading papers from their survey course when they could be off doing research. But wait, correcting papers and doing grunt work, isn't that what TAs are for?
welcome our "overused jokes that don't make sense in the given context" overlords.
Matt Fahrenbacher
James Tiberius Kirk: "Spock, the women on your planet are logical. No other planet in the galaxy can make that claim."
He just gives everyone a B when he is hungover.
I have yet to see any grammar software that cannot be classified as "moronic"; to suggest otherwise one would either have to be a moron, or be making money from said software.
I'm sick of rich upper class morons buying and pandering their way through school. If we used computers to do all the grading there is no way George W Bush would have made it through highschool and I'm damn sure he wouldnt have got a degree from Yale.
Teachers like politicians can be bribed, and the problem with this is, he who has the money or power gets the A.
If you use Linux, please help development of Autopac
So where does style come in? There are many, MANY forms of style, which make writters unique. For instance, I've found that when I write, even the shortest essays, I tend to break up my thoughts into multipart sentences... like this one. They tend to be very long and drawn out. I also use "granted" and "don't forget". I also seem to create a lot of sentences that are self contradicting: Though this, something else. It's part of my style.
My style isn't completely mine. I'm sure over-use would be bad. Granted this. Granted that. Where do those softer features of writing come in? Or are we all to be sterile and write with no tone or style.
--
"I'm not bright. Big words confuse me. But Wanda loves me and that should be enough for you." - Cosmo
no more or less so than published work.
I, too, am jaded by the stupid grammar and spelling police
the students will get a copy for themselves?
So... now we have a robotic Spelling Nazi?
503 Sig Unavailable
The Signature could not be accessed. Please try again later or contact the administrator
I agree with you, its time we do remove the human factor. Why not let computers do what humans are proven to be unable to do without constant errors due to emotion or other human difficulties?
Let the computer grade the technical side and the human grade the creative side, this way there is no way someone who writes a person paper which a teacher does not like can get an F.
If you use Linux, please help development of Autopac
Anyone else think that this is proof enough that such a system is doomed for failure? :)
It goes from God, to Jerry, to me.
They should take each sentence in the paper and run a google-ish type query and flag it if it comes up with more than a few hits.
Why?
Students plagarize these days, a *lot*, because they think it's impossible to get caught. A google-type query on each sentence would make it much more difficult to just copy someone else's work vertabim.
As far as the achievements of ancient cultures go, it is all relative. We have harnessed fusion, mapped the genome, created antibiotics, peered deep into the hearts of galaxies a 100,000,000 light years away, forged fiber optics, designed the integrated circuit, et cetera. People three hundred years from now will look back upon us and wonder how a civilization that could barely put a man on the moon (a feat that will surely be trivial to them) was able to usher in the Information Age in only a decade worth of work.
I've never been a fan of enforced grammar, essay formats, and other writing guidelines & standards. They try to push opinion as fact. Language evolves, more now than ever before. The high school I graduated from required four years of English and only one year of math. English classes honestly taught me nothing of importance beyond elementary school.
An automated system to ensure that all good essays look alike doesn't sound like an improvement. It's bad enough having to write an essay that will only be read by one person. I wouldn't be surprised if many students refused to write essays that they knew wouldn't be read by anyone.
...with my new essay-writing software. It's been tested against 450 automatic essay marking programs written by "experts" as well.
Nothing!
He didn't let it bother him as he figured that over time it would even out. Favorable and unfavorable. So he gained vs his competitors by not letting it affect him
Help fight continental drift.
This sounds a lot like This story.
Actually this sounds a lot like Gramatica. Gramatica was the grammer checker that was an optional component with WordPerfect for DOS and later a standard component with the Windows version. It was written by a team comprised of both computer scientists and professors of English. One of the interesting features was the scoring feature which would give you a rough estimate of the grade level of your writing. It would also give you statistics and compare them to a selection of famous works.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
;)
Some "dehumaniSing" could be a good thing, espcially when grading subjective material.
Objective material is factual, a simplification is "Most dogs have 2 eyes."
Subjective material is opinionated - "Australia should legalise heroin injecting rooms." Obviously this is controversial, and there are serveral positions on the matter.
Most teachers/lecturers/graders/tutors have their own (pre-existing) subjective opinions on certain topics. If you submit an essay that opposes their views, the chances are very high that you will get a lower grade, even if your essay is well formed/written/structured.
In high school, I always took this into account and wrote essays that agreed with the teacher's point of view, even if I didn't. Such software could lessen the need for writing what they 'want to read'.
You werent taught English. I'm not trying to insult you but thats one of the problems with our public schools, they dont do a good job teaching
When I went to high school 15 years ago, we didn't do any grammar in high school English class, it was all read-and-interpret (i.e. read-and-make-up-some-bullshit).
Yes and thats why when you got to college you couldnt write a good research paper.
We were supposed to learn the technical stuff in middle school (and we did to some degree).
You are supposed to learn English through highschool as well, if you want to get a 1500+ on your SATs. This is exactly why students get such low SAT scores in urban public schools, they dont get a focused education, when its time to take tests the test does not care how creative you are or even how intelligent you are, the only thing that matters to the SAT test is your technical knowledge.
Teach technical English and later on let a person learn creativity.
If you use Linux, please help development of Autopac
A computer isn't good enough to judge a human being.
I just tried running the software on a random sampling of Slashdot posts. Guess what? Everyone will be getting F's...
It's true. I've had teachers take their questions and quizzes directly off of websites (the curious may want to enter a few key words from their latest homework on google and see what turns up...). Now here's an ethical dilemma for you: if it's ok for them to get the questions of a website, is it alright for me to get the answers off that same website?
The good old fashioned teachers, on the other hand, would never do such a thing. No, they have been xeroxing the same handouts for the last two decades. You can tell by the fact that they have become half unreadable.
Homework is either trade it and grade it or credit/no credit. Major tests are all done on scantrons.
That's not to say I don't have good teachers.
But I've also had teachers who put in the bare minimum. At the end of the day, they're gone before the rest of us. They teach 5 periods (one's prep) for 180 days a year and gripe about having an average salary with mediocre benefits. On the other hand, the conservative holdouts who arrive at school before the janitors, who stay up all night meticuously grading essays, and who teach sheerly high schoolers simply because they want to (many have Ph.D's and could easily be working at the local University if they so desired) never gripe.
There has been a lot of proposals about reforming teacher pay. I don't know what can be done to attract better teachers. But sheerly for the sake of fairness I would like to see the good teachers taking a disproportionate share of the money. The trouble is, they would never ask for it.
Because humans dont make logical or fair decisions. Often a human will give you a bad grade because they just dont like your paper, or because they dont like you, or they may give you a good grade because you make them like you, and because you have alot of power and influence and they fear you might bring a set of lawyers after them.
I'm all about making the school system as fair as possible, we cant do that when humans who are naturally unfair are making all the decisions.
Its simple, you work hard, you write a perfect paper, your grade should be A. You slack off and you write a bad paper, your grade should be F.
Why should an athelete, or some George Bush character get an A when their papers suck?
If you use Linux, please help development of Autopac
...that automaton would have caught my glaring grammatical mistake: ...the work to a automaton.... It should have been "an". :)
An essay is a medium for your message. Consequently, the structure and form of a piece of writing can be more important in terms of how effective it is in communicating an idea than the message itself.
Maybe Slashdot could have used it on this story.
Sometime, the single word, usually means an indeterminate time in the future, as in "We should meet sometime tomorrow."
Some time, the two word construct of the adjective some modifying the noun time, is used to mean a period of time, similar to "a while" ("a while" and awhile is another interesting case too).
I love the aspect of humanity in the judging of intellectual works, for all of the reasons that you mentioned. I would hate to lose all of that, to sanitize some aspect of our educational institutions of the human element, in order to gain some small amount of elevated grammatical error checking. But hey, that's just me.
Student shouldnt be such cry babies, I mean if they do a paper and get a bad grade by the computer theres no one to blame but themselves. You can make the code open source and let the kid look at the code himself for all I care, you can run diagnostic tests, you can do whatever you want, but chances are the computer didnt fuck up, chances are the student did.
If you use Linux, please help development of Autopac
If I have learned anything from my university career it is this: As class sizes get larger, testing becomes more frequent and more automated. Of course you say, if you have a class of one hundred or more people, it is simply not possible to mark that many essays. This usually means that essays don't need to be written at all! What do they do? Multiple choice! I heard a statistic once that if you chose answers randomly on a MC test that you could get a C by not knowing anything beyond how to circle a letter! ----- Discovering this, I made sure that I took all the obsure english classes that had no more than 30 people in them. An unexpected positive side effect to this system of choosing courses was that 90% of the other students in them were girls. Yea, life was good. ;-)
Apparently, the system uses statistical analysis as well as grammar checks to determine the score for the essay. Basically, they've built up a database of essays that have been graded by a bunch of humans, and then used these algorithms to figure out which bucket the essay belongs in. Sounds kinda like SpamAssassin, actually. I'd be willing to bet that with sufficient resources (in terms of essays and human grading time), this wouldn't be all that tough to duplicate. After all, what are spam filters but content analyzers? (Shameless plug for a system that requires human judges rather than computer judges)
--
Annotateit at Annotateit.com
ETS actually has a web site where you can do a sample essay that their server will grade for you.
More info can be found here.
Not only did you not RTFA you didn't read the fucking post.
"That's really not the case, because we're not talking about eliminating the human element. We're making the process more efficient."
And this kind of shit gets modded +4 Interesting?!?
Automated is good because theres less chance of error, and its almost always fair.
The only way to get fair grades in university is to be smart enough to pick the right teachers, and drop the ones who you dont get along with.
I heard a statistic once that if you chose answers randomly on a MC test that you could get a C by not knowing anything beyond how to circle a letter! ----- Discovering this, I made sure that I took all the obsure english classes that had no more than 30 people in them. An unexpected positive side effect to this system of choosing courses was that 90% of the other students in them were girls. Yea, life was good.
[ Reply to This ]
Who wants a C? Thats as good as an F in college, if you get a C you can just drop the class and take it again!
I dont really like small classes myself, there is no real benefit, what I notice from smaller classes is, teachers are more critical of you, you get greater punishment for poor attendence or for being late to class, you also get more focus from the teacher and this can be good or bad depending on if the teacher likes you or not.
If the teacher likes you, getting this extra focus is a very good thing because a personal connection with a teacher who likes you is to your benefit, however if the teacher dislikes you and decides to personally focus on you, this is bad.
If you use Linux, please help development of Autopac
Type some random sentences (in quotes into google). see if those kid's been cheatin'
You speak as if the use of computers to judge intellectual works will somehow make our society exempt from "rich upper class morons buying and pandering their way through school". Such an aristocratic model is something that exists beyond the scope of one's grades in school, and will not be eliminated, in any sense, by such a thing.
The irony is that most classic literature would very likely fail these tests. Nobody writes like that in Beowulf, teachers wouldn't pass someone who did; however, it is a classic.
Truer things are rarely uttered.
All my essays are outside the box, off the wall, and testing the limits. Most of them are good that way.
This software may help with spelling mistakes, but I don't want it saying "You should start with an overview." Did Cinderalla start with "Cinderella was turned into a princess by her fairy godmother and married the prince. Her carriage was a pumpkin. She lived with evil stepsisters before she was married."
I think not! I hope we can just keep the student/teacher ratio small enough that the teacher actually has time to read through my paper and decide for him/herself whether it's a good paper.
Don't thank God, thank a doctor!
One of the primary purposes of essays are to learn how to write for a specific audience.
If you remove the human element, then you aren't writing for any audience, unless, of course, everyone starts writing for computers' entertainment and education.
Relax mind.
Relax body.
Relax bowels.
Relax.
Do not fall over.
You are a cloud.
You are raining.
Do not rain
Whist the train
Is standing at a station.
Move with the wind.
Apologize where necessary.
---Douglas Adams THGTTG
Analogy:
Undisputed fact: At an elementary level, today's foreign-language software can judge - and, through feedback, correct - your e.g. French pronunciation.
And not just your pronunciation: there are certain inalienable rules of French grammar or word order that every native French speaker obeys but some which are tricky for Americans to master. Many of these can be judged - with no false positives - with filters of various complexities.
Now then: I hold that eloquent English expression - that is to say, a fluid translation from thought to sentence - invariably is carried on by people who, by the same process through which they acquired the ability to so translate their thinking into English words, also have acquired certain basic rules.
The analogy is, then, this: Sure it's possible to have poor pronunciation and grammar/word order while having acquired the words to express oneself in fluent French, it surprises no one that these former can be judged/corrected independently of the latter, even by software that doesn't actually understand any meaning.
Why then should it be so suprirising for English?
There is the crust: Just as all native speakers of standard French express certain things certain ways, all writers of well-articulated English follow certain -- here frequently unwritten -- rules.
These rules are, essentially, mere observation of what it means, to an English speaker, to be speaking clearly.
Should a teacher have to "listen" to your whole 10-page essay to determine whether you're speaking clearly?
Or should she be free merely to glance through your paper for content, and let software do the "listening"?
I don't object.
Anybody remember the DOS program "read"?
I put a paragraph of a Philip K. Dick novel into it and it scaled it as the perfect archtypical pop fiction style. Foster's steamboat-era vampire novel "Fevre Dream" got an extra grade level if I remember correctly.
If you truely believe this, you also support direct democracy meaning no congress, no senate, just votes deciding all the laws. You also would agree that affirmative action must be a requirement, you believe that every student should have access to a lawyer to dispute every grade paid for by the US-Gov etc.
The problem with your society of pandering, special interests etc is, it causes ALOT of lawsuits.
The kid who gets the bad grade will call their lawyer, teachers will get fired over it, kids who are dumb but who have alot of money will get all As, kids who are minorities will get good grades just because they are a minority, and everything will be decided on the emotional whims of the people.
Imagine how a direct democracy would be, after 911 all the Arabs in the country would be beaten and burned to death.
If you use Linux, please help development of Autopac
Thou shalt not create a machine in the image of the mind of man.
First Commandment, Orange Catholic Bible, Dune, by Frank Herbert
Or, maybe you like a more recent refrence:
"I say 'Your' society because, you see, when we started thinking for you, it really became OUR society."
Agent Smith, conversation with a sedated Morpheus, The Matrix, by the Wachowski brothers
Or maybe a little more panic-inducing.
"They say it got smart. A new order of intelligence. Then it saw all people as a threat not just the ones on the other side. Decided our fate in a millisecond: extermination."
Reese, talking to Sarah Connor, The Terminator, by James Cameron & Harlan Ellison
Or how about:
"It can only be attributable to human error."
H.A.L. 9000, regarding a piece of information it had falsified, 2001: A Space Odyssey.
Or maybe a little more plausible?
"Would you like to play a game?"
Joshua, Wargames
But then again, I'm a sick weirdo that thinks technology is one of the main reasons there are so many people who are depressed and overweight. Technology BAAAD!
Now just give me a type-writer and a cabin in the mountains....
this essay written by a student in Hong Kong. :)
A typical middle or high school english teacher has six classes a day, each having over forty students. If the students have to learn to write, each of them should write a couple of pages of prose a week and some poor sod has to read it. The more they write, the better they get at writing, so it is generally a good idea, but really hard work to read it all.
There will generally be two papers in each class that are remotely readable. The rest will be a LOT OF WORK to grade. If a bot could do some of the work, it would be welcome.
Late at night your eyeballs feel like they're on fire and you are convinced that the entire system should be put out of its misery. The thought that a student actually has an IDEA seems fantastic.
PLEASE don't be a troll and tell me that YOUR teacher never appreciated your ideas.
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
I know he brought us "everybodies favourite Open Source"(tm). But a great mind along the lines of Dali? I think not. At the end of the day all Linus did (and did very well) was to produce a clone of something that already existed i.e. a UN*X kernel, something for which he deserves to be recognised but not (yet) grounds for a Nobel prize (or equivalent).
What is the goal of an essay?
My English teacher always said it was:
TO COMMUNICATE YOUR IDEAS.
If your essay cannot do that in a coherent form, then it FAILS in this goal. Spelling, grammar, and format errors all detract from your essays ability to communicate clearly and unambiguously, and thus coherently.
Once an essay passes that hurdle, then we can talk intelligently about whether what is written is enlightening or not. Without coherency, how can we discuss it?
Bias Warning: My public high school English classes were pretty Draconian. In an age before MS Word, any spelling or grammar errors resulted in a zero for the paper. In college, I could get an "A-" for the strangest shit I could write -- simply because I made no spelling or grammar mistakes. And yes, R0DENT/L33T speak makes me twitch horribly...
I think it's obvious that you were an under achiever in school. Your words narrate to us an excellent narrative of your past failure.
Stop hating those that are able to succeed, and instead ask yourself why you did not. Resolution is more healthy than denial.
I mean, if it can't predetermine Karma Moderations on my posts, what's the use?
-------
Support Indy Music. Buy
As I'm sure anyone who has ever written an essay (especially highschool level or above) knows, there is no point to the essay per se. The essay is not an end to itself, and the grade ultimately is not an end either.
;) I received a page and a half of handwritten comments, as well as inline comments about points in the middle of the essay. Twenty years from now, I doubt I will remember a great deal of his course, but the comments that he left me have already changed my writing style, and, I hope, improved it. (note: slashdot style not indicative of real style, hehe)
At my university, Duke, our new curriculum has specially designated writing classes. Every student needs to take three over their four years. A biology lab can be a writing class. So can an English class, history, religion, etc. All W classes have certain requirements--their must be certain amount of writing and more importantly REVISION.
I was fortunate enough to take a class from the author and profesor Reynolds Price. We had a final essay for the class. Along with my grade (not an A
A computer will NEVER be able to do this. Nor will a computer (at least in the foreseeable future) be able to comment on my theories about Milton's Paradise Lost.
When are people going to learn that computer can NOT replace people in jobs that are not mechanical. Computer cannot grade papers no matter how many sample papers you feed it. Teachers grade not only on grammer and spelling but emotive meanings and other factors that computers cannot recognize also play a part in it. Computers being used as a checker against plagiarism is okay, but to grade papers is a joke.
Decryption logic or neural networks should be well suited to trial and erroring the perfect combinations of words and sentence structures to get the highest scores Criteria delivers.
You integrate a client/server database to this so that each completed essay is uploaded to a central server so that clients can avoid ever calculating the same papers (which would be a countermeasure against software that checks for copied essays). The early adopters would be able to generate small, high scoring essays quickly, and as the battle continued, it would get a little harder to come up with a unique document that meets Criteria scoring standards.
The arms race would likely go back and forth a few times until finally professors would be forced to read the essays submitted to find those that were machine generated. Because the neural net created essay would have perfect sentence structure, an actual person would be needed to follow the path of the essay to determine if another human had written it.
Eventually, the professors would find themselves using Criteria to grade the papers and using a specialized sub-breed of human to make sure it was not generated electronically. This breed would be known as the 'TA'.
When software is in charge it will have a bies in a single direction reflecting the bies of the programmer.
This is a significant power giving one man the chance to make choices for thousands of people all at once.
While it's true a judge or profesor will reflect bieses, opinions and addatudes of the person those will be of a random group with random bieses, opinions and addatudes. It dose not give rise to a single direction of bieses, opinions and addatudes that one single program would.
Should it be closed? "The boss wants a back door that gives the bosses kids top grades no matter what"
Or open "This is the patern it checks for here. Just folow this math matic algarithum and you'll ace it".
Or somewhere inbetween "See here.. this is where it checks to see if your the bosses son"
Or worse...
"This kid is a punk... add code to trash his grades"
"all I got to do is 'accadentally' cross an I and his grades tank"
Thats for grades....
Now if it were judges....
Auto Judge "You've been found guilty of software piracy. Off with your head."
Convicted "But your honnor I clearly showed it was my code to start with."
Auto Judge "Nobody gives away code. It's a trick. Off with your head."
or worse...
Electronic amisty... software back doors to permit new interpretations of the law.
"I am LawBot by SCO..."
I don't actually exist.
I bet it requires the use of Microsoft word. Even if it doesn't 99% of the instructors using it still would rquire Microsoft word or Miscosoft products.
Call us back when it only works with open source text editors.
As you can see I don't care about my karma.
Show me a program that can write interesting essays and I'll take the grading program seriously.
I hate how some troll always says, " I have no problem [accepting whatever outrage, unjust heahvior, silly restriction or judgment by an ignromaous is offered]." There's always at least one or two highly rated posts who's authors have about as much brains as they do self pride. Go ahead and let Clippy grade your work. He will show up in classrooms and cubiles next year, just as soon as Paladium goes live. You seem to be ready for a very helpful Big Brother.
Friends don't help friends install M$ junk.
The average you could expect on a standard scantron MC test is a 25 (F). Even with a 50 point curve... are you saying kids are getting 50 point curves at your university? That statistic sounds a little fucked up.
Also, I notice that as class size really starts to ramp up (lots of sections, 90+ each section), you actually get LESS tests, or they're taken online.
There's basically a saddle-point in the class quality curve you need to avoid.
Fuck Beta. Fuck Dice
This is not really new. If we slice off five words from the beginning and end and s/student\'s papers/students/ we get:
which defines the modern public school perfectly.
Furry cows moo and decompress.
but from long experience, we can also be certain that it *will* be used properly.
Some teachers will use it properly, others won't. Thsoe that won't will face the consequences.
social sciences can never use experience to verify their statemen
And yet you seem to despise the role of emotion in politics.
Interesting.
Um...as opposed to English Comp classes in general?
"Hardly used" will not fetch you a better price for your brain.
Last time I checked, teachers being far too slow with essay grading was slowing down the entire U.S. educational system. Oh, wait, nevermind -- it was school budgets being cut so low that they couldn't even afford maintenance on their buildings. I think some more software is exactly what we need here!
At first glance, I thought that a tool used to analyze essays would be a nightmare -- it would kill creativity, homogenize style, etc. However, I then remembered my experience grading freshman *college* essays last year. This was in a small-classroom course, taught by an extremely good instructor that also offered office hours galore, all kinds of free tutoring on campus, free access to computers with MS Word, etc -- in other words, *plenty* of chances for the students to improve. I was given three distinct classes worth of essays to grade, and traded off with the instructor on which half of each class we'd each tackle. The papers were dreadful...while as an English & Creative Writing grad, I am extremely pro-creativity, these students weren't ready for that. Some of the sentences I ran across were so awful I would IM my friends and have *them* in hysterics at how terrible they were. For example -- and keep in mind, these are middle-class white kids, NOT English-as-a-second-language students. "There is not connection with what I know the same circles don't fit inside squares." "There is also numerous of shapes and designes which not to difined." "They get into the analization of the man..." The simple truth is, while creativity is great, there is a baseline level of grammatical ability that needs to be used in order for others to simply understand WTF the writer is saying. A lot of the time, it would take me 5 - 6 scans per sentence to figure out what my students were writing. The worst part of this is that because of department rules, I wasn't allowed to give any kid below a C if he/she *tried* to follow the essay guidelines. As long as the right number of pages, subject, etc. were touched on, the person would pass. I think the best use of technology in cases like that would be to sit the person down at the computer with the grammar-analysis program, and rather than have them ignore classroom lessons, interactively edit their own papers. Not the way that Word v.X does it, where the person just right-clicks to get a "correct" change, but the older method in which the program offers a series of alterations with explanations *why* the original is out of whack. Doing that in lecture isn't realistic, unfortunately. Despite the number of grammatical nightmares in the course, the students really were at varying levels of ability, each with a unique misunderstanding of the rules. Each needed individualized attention, though as far as I could tell none of them were trying to obtain it. Also, the campus requirements for Freshman English didn't leave time for stuff they should have learned in elementary school.
As a high school student, I find this amusing. In the first place, of all of the English teachers in my school only one would be able to figure out how to turn on the computer to use the software. Most English teachers don't pursue computer science as lovingly as they should. If this were to grade science experiment write-ups, then I'd worry, because science teachers wouldn't scream if they wre told to use a computer. When it comes to how much I, a student, would feel if my essays were graded by a machine, let's just say I'd make fun of the whole thing by inserting sentences like "This is a really stupid assignment." or "I wish I were in tech class." But since I enjoy the pursuit of the written art so much I would also be upset that an understanding human being would not be there to see why I might break grammar usage on purpose. I guess that I would end up having to post more essays online where no one who reads them really cares enough to critique the thing. I want my English teacher, thank you very much.
"Life is Mental" -Emily Robison
MS Word has a great grammer check system. It's your grammer that crap.
this is just a way to lighten the load of the graders. after all, they need some time off over the weekends, and have other things to do too.
/out checking these papers are the best solution either.
it's not like overloaded human graders workin day-in
unless there was another system of screening candidates, we will have to live with automated graders.
as it is, how many of us actually send in the essays that convey our true feelings, and how many write essays they think would be best for the school they want to get into?
but one thing is for sure with this - no matter how well the algorithm works, it is sub-consciously going to stop the students from experimenting. everyone is going to ask the ones who turned in their essays and got grades, how and what they had written, and try and emulate the same.
It's like the bayesian filter for mail classification in SpamBayes or Mozilla. In fact, that's probably where Criteria's programmers got their inspiration.
If you read the article, you'll discover they had to feed it four hundred or so "good" papers (training set), and they describe it's validity because graders notice that (paraphrased) "well written papers [on the topic] contain certain key words or ideas, and avoid certain expressions [examples]", which the system picks up on. Since it agrees with grader scores +95% of the time, I think those simple indicators are actually pretty useful.
Keep in mind, it can give a perfect score to unreadable garbage, which isn't even grammatically correct. (This is mentioned in the article)
Nice 5 insightful, though. But next time, read the article.
In fact, I'm ashamed no one mentioned that this is just like spam filter technology yet. Come on slashdot, is your technical insight on a weekend trip or what?
Fuck Beta. Fuck Dice
An essay is a creative thing. The idea is the point. The grammar is there to support the ideas and is secondary to them. The grammar should also be flexible. One of the things that we love about great essayists is that they pushed the envelope of grammar to suit their ideas. Maybe this is a fine tool to help teachers who are pressed for time ever increasing class size and ever decreasing budget but you can not rightly say you have graded a student's essay or that you have really helped him/her learn to write.
Here is an interesting test of the software: Feed it an essay by R.W. Emerson, Annie Dillard, Virginia Woolf, Francis Bacon, or Montaigne and a couple of essays by students with good grammar and spelling and see how it grades them. My guess is that it can't tell them apart. In fact I would hazard a guess that it would take some points away from Mrs. Woolf for run-ons. The essay is a great educational tool for critical thinking, if we standardize grading of it, the essay looses its meaning. If this can't tell the difference between Emerson and little Bobby is it of any value?
If you still need a person to do the work, why bother paying for this dinky software? If you don't trust the program, why would you want it to influence the real grader? Do you really think that Clippy's Big Brother will do a reasonable job? Has it been tested on some of the authors you mention?
Did you even read how the silly thing works? OK, try this from the article:
The testing service recognizes that e-rater could yield a high score on an essay with a well-written but illogical argument. "Right now, e-rater looks at an essay like a bag of words," Dr. Burstein said. "If you use the right words, you could in theory get a good score without the argument necessarily making sense, because it's not at this point tracking a logical line of argumentation."
That's totally worthless. The kinds of "quality" essays where the machine supposedly mimics human graders, must be of exeptionally low value.
These kind of programs have nothing but evil uses. Do you really want everything you ever wrote for school to be digitized and analyzed for word content? It sounds like carnivore all over again. It can't really grade essays, it can only filter words and flag the suspicious ones. The more I think about it, the more I hate it.
Friends don't help friends install M$ junk.
I think perhaps one should look at one's own grasp of grammar before flaming someone on their grammar.
450 essays doesn't sound anywhere close to the number of examples you need to learn something like this. My guess would be that 10s of thousands essays and you might get somewhere interesting.
Hmm, if this software is used to mark your essays, couldn't you reverse engineer it to create an anti-thesis that would write the perfect essay each time?
- shazow
I see serious questions regarding privacy and copyright. If this were a standalone tool (something similar to lint or weblint), this software might be effective and useful. The fact that this is not a standalone tool, the fact that this is a web-based interface, means you have to upload your work into this system. This raises lots of questions. Are they hoarding essays and term papers? What are they doing with this data? They should releast it open-source and just release the tool itself, offer the web interface as an option, but offer the tool standalone.
The people opposed see it dehumanizing the student's papers, putting them through some sort of mechanical, computerized system like the multiple choice tests.
java.lang.NullPointerException
Total: 65501
Grade: A+
Boy, you hit the nail on the head with this one!
I can just imagine the uncaught mistakes that will come out of this system. Like the cashiers that have no problem whatsoever giving back $98 change after you pay for a $2 candy bar with a $10 bill, because they accidentally pushed an extra zero.
mao che minh HBT
First off, let me say that I am involved in the automated essay grading industry, and have helped to develop RocketScore which does everything Criterion does, and lots more. Forgive me for blatant plugs in this post, I'll try and keep them to a minimum.
But let's move on to the focus of this article.
First off, there is a lot of criticism about essay graders being formulaic, only capable of seeing patterns that arose in their originating sample set of essays. With Criterion, an offshoot of ETS's e-rater, this is a serious concern. When you only look at what you see, anything out of left field looks completely awry, and cannot be graded appropriately. RocketScore is different; RocketScore uses a "features" method to check for included or excluded material, among many other things, and is therefore quite good at noticing subtle writing and essays types which it has never seen before.
One of the great things about essay graders is that they give a student an objective standard to look to. Human graders grade differently based upon mood, time they have to review the writing, and many other mittigating factors. In other words, the same human grader might grade the same essay differently at separate points in time. Most essay graders will always grade the same essay in the same manner. This is great for a student, for if a teacher gives you a D when the essay grader says it's in B range, one might be able to use this evidence to force the teacher to reconsider the grade. Or vica versa. If the essay grader is telling you that you're getting a D, you can work and improve on it until you're getting that B you'd be happy with.
But there are serious drawbacks to the comments E-Rater and Criterion give. E-Rater gives comments soley based on your score (if you get a 1, you get comment set 1, if you get a 2, comment set 2, etc.). Criterion gives a student "instructional feedback in basic grammar, usage, style and organization." E-Rater's comments are inadequate at best, and Criterion's leave a lot to be desired. RocketScore provides substantial feedback on how to improve your writing. Not just stylistic and grammatical comments, but comments on what you should be writing more about (you didn't provide enough info!), what you should be writing less about (you gave too much info!), and how to balance your arguments, among many other categories.
There are two major problems with essay grading. The first is bullshit detection, and the second is determining if the essay actually answered the question asked. E-rater and Criterion both have real problems with these two criteria. With bullshit detection, RocketScore has threshholds which can be set and manipulated on the fly, from throwing out anything which isn't completely relevant to the topic, to allowing just about any essay submitted. And you will get a score and comments based upon what you submitted. Of course, these are most helpful when you make a meaningful attempt to submit a relevant essay.
Yes, but do you know how ETS defines "agreement"? Glad you asked. When the grader's grade is within a point of the human's grade. Now, with the SAT 2 test, which is on a scale of 1 through 6, that means if the grader says 2, and a human says 1, 2, or 3, then there's agreement. But that's 50% of the scale! Their essay grader has a 98% chance of hitting the wall in front of them as opposed to the wall next to them. Woohoo. Meanwhile, RocketScore provides decimal point accuracy (we don't give you a 4 or a 5, we give you a 4.1, or 5.3), and is 98% accurate. But how do we define accurate? When the grader's grade is rounded to the nearest whole number, and that number is the human's grade. In other words, if we give you a 4.3, there is a 98% chance a human would give you a 4. With 4.5,
---
"Of course, that's just my opinion. I could be wrong." --Dennis Miller
Teacher: Johnny, I'm really sorry, but the computer crashed while your paper was being scored. I was looking over it. It's been a while since I've read a paper, but I was wondering what the following sentence means:
And this one:
Is that some kind of new language that kids are using? Oh, by the way, congratulations, you got a 100 on EVERY essay this semester! Good job!
We're making the process more efficient
Horseshit. And while you're at it, pay the teachers a decent wage so you don't produce a new generation of morons.
For the math lab that I was a TA for, I graded essays by making a grid: rows for student names and columns for each idea that the essay had to hit. Grading became mostly mechanical for me; I cut grading time from 60 hours over a weekend to 20. Not only did it save me time, but it cut down on the number of complaints that students had about inconsistent grading. I still tried to observe grammar and flow, but there were just too many papers to give everyone the attention that they deserved. I think that the software, in conjunction with a person to work with it would be valuable.
The middle mind speaks!
I'll be the first one to sue!!!!!
-----
One is born into aristocracy, but mediocrity can only be achieved through hard work.
Now before you start up the flame throwers, this is not a message to deride high school students over their lack of creativity.
But when I was in high school, we were told that proper essay writing was an essential skill for the departmentals, and when they said "proper," they meant "Must conform to between five and seven paragraphs, with the first and last being this opening and conclusion with three to five paragraphs of body--each containing one topic of discussion."
Furthermore, it was made VERY clear that creative or unconventional ideas (let alone language!) would be strongly frowned upon. There was One True Way to write an essay, and One True Opinion on any given subject. Any deviations from that would cost you.
I hated it then, I hate it now, but I don't see any problem with having computers mark essays like this. After all, they were trying to turn us into computers to create them.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
"Rules were made to be broken", they say. And the same is true of the rules of English. Creative use of language necessarily breaks rules. Winston Churchill once remarked about bad grammar, "This is a situation up with which I will not put!"
What this will do is reward a class of students good at memorizing rules. Most of the great writers of English would fail were their work submitted to the machine. Let's see what the system does grading Shakespeare, or Donne, or Lincoln, or George Eliot, or Joyce, or Pynchon.
This is a great leap forward for education. While it has always been the goal of geeks to submit computer-generated papers and receive decent grades, this has traditionally been hampered by the unreliability of computer-to-human communication. But with computer-to-computer submissions (henceforth referred to as "End-to-end Grading And Direction", or EGAD), we can now begin hacking away at the first generation of grade generators.
"What I did on my Summer Vaca'; DROP TABLE punctuation"
Of course, ETS has yet to divulge the details of the technology they use.
Could this be it?
As I understand it, ETS poured tens of millions of dollars into the automated essay grading effort, in parallel with the development of the CBT format. For years after the CBT was introduced, the GRE essay was still done on paper.
As the grading software finally worked on the database of sample essays, the GRE essay switched away from paper to word processing entry (something many test takers have difficulty with).
Still, the essays were graded by armies of grad students. Only when the automated grading matched the manual grading 90% of the time did the software "go live".
IIRC, it has been used for only a few years. After over a decade of development.
-"Another key element of the Indigenous education is that of the older people mentoring the young in the culture."
-"The children worked side-by-side with the Old People and learned how to do many things by watching them and being shown how to do things."
-"Certain activites, such as ceremonies, of course, contained much to learn."
WTF? I'm paying $40 for this textbook in order to wade through crap that's barely English? These mistakes were found in the first 20 pages of the book, amazingly enough; I dread reading the remaining 300 pages.
I think that we should worry about teacher education first before building ubercomputers to grade students' papers.
Condemnant quod non intellegunt.
How do you step from 12 years of education where nothing but grammar, vocabulary, and spelling are taught, and then expect students to magically be creative in college? It's one thing to make sure the rules are understood, and entirely another to turn the whole process over to a machine with no more grasp on creativity or imagination than your average toaster.
It's bad enough with the standardised test obsession. I've known people who were much smarter than me who couldn't come within 20% of my score on a standardised test. Why? Because I am an excellent taker of standardised tests. Possibly the least useful skill available in someone who is done with college. It means nothing! People talk of it as "objective", and then go and take a 30 question personality quiz out of fricking Cosmo without even grasping their inherent hypocrasy.
Computer driven/scored tests and exams are worthless everywhere actual creativity is involved. Since creativity and independant thought are the qualities of thought that we need most, as we cannot get them from computers (yet), it seems utterly pointless to use computers to crush those qualities out of generations of students. It's bad enough when you have to deal with some stodgy, unimaginative professor.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
it gave you a Z-minus.
that way you can get humans to grade them for much less cost. Indians can speak good english, so, they must have some teachers there looking for extra cash. Win-win deal. Mod me down!
New year Resolution: Don't change sig this year
but I think the "bad" english is only the tip of all the other problems with todays youth...I say this and I myself am only 26, yet I've felt this way since at least age 18-19...what gives...sounds like a good ask slashdot
The GMAT, the business graduate school admissions test, already grades the essays automatically. I think they're going to soon do the same with the GRE. Crazy shit.
That is correct. It costs $200 to take the GMAT!
But the payoff is huge! The average starting salary for a graduate in my MBA program is about $75,000.
All employees must wash hands before seeking equitable relief.
I am not sure of which system my college uses, but when I transfered, I was subjected to an essay as a placement for the English courses. At any rate, there was no human involvement, you could appeal the computer's score and have it reviewed by a person, but for the most part it wasn't. What if your English 213 final was graded by a computer, but recieved bad marks for "style"... ...ouch.
Let's see how it grades works from Ellison, Fitzgerald, or Hawthorne. If it gives them A's, then we'll talk.
Something hinted at by the story and some of the comments but really bears being pendantic: too few teachers. It is lucridous to expect a teacher to go over 150 essays as it is for me to expect getting a reasonable education when I am 1 of 150 faces trying to gleen something more than an "A" from a class. The software is attempting to address this imbalance, but ultimately it will make the level of education worse: it can grade a paper, it can't offer insights on how to improve. And it will give administrators a reason to pile 50 more into a class, which will in turn lead to GradeStar MkII and onward into a vicious circle. And yeah, the software is just a tool, but like so many tools, that's not how it will be utilized. It's a cop-out, nothing more.
If they want to use this in high schools and middle schools then that assumes that all their students have computers with a word processor. What about all of the lower income kids at schools that don't allow after school activities (like using school computers) like public schools in NYC? It's a tool for the middle and upper classes that will give their teachers more time for their students and leave the teachers in poorer areas still overworked and struggling to keep up. It won't be feasible where it is needed the most.
I wrote in my journal about this awhile back. ETS was trying to sell their essay grader to a group of the local test prep chains here in Taiwan. The local schools called me in to sit in on the presentation. Before I had gone in, I searched around and found numerous free and open implementations and I asked the speaker why they were selling their academic software for so much money --it was a rather complex contract on a per seat basis-- when there were similar product available for free. Their rep claimed to be unamare of any similar open sourced products that could match the amazing and advanced artificial intelligence features they were offering. Sales reps --hmm. The mere posing of question definitely made them stutter and squirm though.
But the interesting part was after I got home. I looked at ETS's own research monologues and found that internally this overpriced system had been debunked. It was discovered that by writing one well-formed short paragraph and then cutting and pasting it over and over an almost perfect score could be attained. The more times it was pasted, the higher the score.
It was also possible to write an essay on an unrelated topic and still get a high score allowing students to use rote memoriziation of a single model essay. This, natually, is impossible with a human reader because they can tell what the topic is fairly easily. According to the sales literature this software could to, but in actual tests that didn't hold up.
Their sales literature claimed that the software contained aritificial intelligence and thus implied that such simple techniques would not fool it, but in practice this was far from the case.
Monographs published by ETS also made it clear that despite their aggressive marketing of this product outside the US, they were not planning to use it as an exclusive grading system on their own tests. Rather, it was to be used as a teaching tool. However, it took a lot of digging to uncover that information.
Just as with translation, there's a lot of financial motivation to make this technology work, but that doesn't necessarily translate into workable products. In the nineties when spelling and grammar checking was already old hat and English/Euro translation was making such headway I thought fluent Chinese/English translation was just a few years away. Now it's 2003, grammar checkers still only work if you write in prescribed style and I've yet to see something halfway decent in Chinese/English translation software although you still hear claims all the time for some overpriced product that's really almost there.
I think we'll see dramatic life extension long before we see decent computer essay graders. Decent trade as far as I'm concerned. As for translation, we can always teach more languages in school.
A simple-minded program will always fail in a "special case."
A couple years ago, I pasted random paragraphs from about a dozen works widely acknowledged to be "great"--or at least "important"--into a Perl-scripted "Fog Readability Index" calculator, and every single one, from the Talmud to Goethe to Derrida, was deemed "unreadable" by the program. (This post, too, would "fail it," as we say on Slashdot.) However, it graded random AP stories and small-town newspaper editorials very highly.
Right now, I can't think of a single great essay--not one--that could meet even the barest criteria for a properly written essay, as taught in schools today, but, again, the average AP story meets most of them.
This story makes a nice little metonym for what's wrong with modern education: that its greatest reward is congratulation for mediocrity, and that it considers the "special case" always disruptive, and never actually special.
What are schools instituted to make of us? Certainly not a nation of Mark Twains--he's been retroactively diagnosed with ADD.
Your mouth is like Columbus Day.
John Schmoe, English 11, Unit X327Bob, "Gatsby"
Love, lost, found, despair, colors, happiness, people.
I call it "keyword essay."
-----
Score 3? For what? Being wrong, at length? - smirkleton
I would like to point out that one of the best ways of speeding up grading is the multiple choice. Multiple choice has changed the way people study, making them test-cracking machines, who forgot how to think about what they are studying, because it is not necessary.
Have you noticed that the generation of multiple choice can not come up with an original thought.
In general, when you try to transform an activity into a process you make the output more uniform. Now, the output can be perceived as the scoring system, but the real output is the composition, which will become more and more standard because it has to statisfy to a set of rules. So, doing this will further reduce the amount of creativity of the compositions. For example, the TOEFL (test of english as a foreign language, mandatory for every foreign student in american universities) has an essay that i s already graded by a program. You need to throw in some "however" and your score goes up...
The only reason for which you want efficiency is saving money.
Right on!
Input
Output
> Since it agrees with grader scores +95% of the time, I think those simple indicators are actually pretty useful.
What about when the lazy teacher uses the device soley for grading and not for its "simply indicators". I guess tough luck to those poor saps that represent the 5% of innacuracy... :(
I like teachers having to read papers. It keeps them shorter. There is no reason a paper for some 3 credit class has to be 30 pages to get your point across. Although if a machine read it, you could probably bullshit more easily.
Are we this lazy as a country that we don't even want to go through the process of teaching? An essay that is gramatical error free and uses phrases like "in summary" and "because" doesn't mean anything if the student can not comunicate their feelings into a paper. High school is the time where students must develop the skill of relating their ideas and feelings on paper for a human to read. If they were to just write so a program would scan if for "errors" then why would they ever bother to take a risk and write something meaningful?
Humans have been putting too much responsibility in the hands of computers. But to make teaching, and especially writing for god sakes, an objective process shows nothing but our society's indifference for educating and improving ourselves.
For those who see objectivity as something positive because it levels the playing field, then they never had a teacher that would take a chance and go past their duty to help a student. This is something no program will ever be able to do.
Computers are objective, people are not. That is what makes us different and inherently better.
No one seems to be talking about another clear use for such software, helping students write better essays! If you give students a tool that will accurately estimate their grade, they will submit better essays, thereby indirectly making teachers lives easier. Teachers can tweak the grader software to their liking, then distribute those settings for the program on their class website (all English teachers have one of those right?). Students will get instant, and meaningful, feedback. Heck, you could require that students grade their essays beforehand, and submit that automated grade with the paper. This would give teachers a loose guideline when they really grade the papers.
Why are we feeling so threatened that a machine can do a human's job at this one, narrow task?
Haven't you learned anything from Deep Blue? How many problems must the AI field conquer before it's taken seriously?
Don't flatter yourself with delusions of superiority. If the computer agrees with human graders 99% of the time, then its response is definitely valid.
Besides, its not as if professors grade your work anyway. That's a job for TAs.
Class is more important than SATs too, if you come from the upper class (think George Bush) it helps you even more than being a female or minority would help you.
Despite what you might think, most minorities dont get free degrees from Yale, if you want a degree from Yale, you need a 1500 on your SATs just to get in, period, it does not matter if you are a minority, this is Yale.
Affirmative Action only helps people get into state universities, these are schools which any one of these minorities would get into if they went to community college for a while, so really Affirmative Action does nothing for a person with poor SAT skills but maybe help them get into University a year earlier than they should.
If you use Linux, please help development of Autopac
I'm thinking at least 6.
Software can, at best offer "educated(?)" guesses
on grammar and spelling.
In order to perform these tasks at a human level, software would need the capability of natural-language understanding; which is in the realm of science fiction.
A) This judges essays, not fiction, and no one is saying that prose would or should be evaluated this way.
B) If you're a professor (more likely a grad student) grading 50 essays in a single sitting, then you're already looking for little more than basic structure, grammar, and vocabulary, just like the software. If you, as a human, have just a few minutes to evaluate a 10-page essay, then your process will be very similar to that used by software.
every stain tells a story
Cat got your tongue? (something important seems to be missing from your comment ... like the body or the subject!)
No one will read this, but I bet that these automated essay graders are able to mimic human graders closely because the human graders themselves graded the 450 "config" essays under computer-esque conditions, i.e., a time constraint that forced them to skim, think as little as possible and generally act like a machine.
Automated is good because graders must read far too many papers. If your paper is on the top of the stack, you can write with eloquence, subtlety and wit and get a good grade. If your paper follows five or more papers out of the 50 or so that the grader must plow through, forget it. Write like you want to work for one of the New York tabloids. (Good student writing software, e.g. Inspiration or Study Star, will guide you to writing essays like a junior Mickey Spillane, slapping your reader back to conciousness with a brass-knuckles introduction and driving home your thesis with a haymaker conclusion.)
Ah so George Bush does not have the right to say "Bring em on!"?
Well, legally he does have the right (or does he? US Soldiers were killed and injured directly because of this comment, doesn't look legal to me)... but lets look at it in other terms... Lets look at the moral right. What is good for George Bush's people? What will kill his soldiers and what won't. What moral obligation does he have to keep his soldiers out of harms way? Should he also say on TV "hey fuckers in north korea, BRING THE NUKES ON BITCH!!!!"???? Does he have that right?
Any time you tell an enemy to "bring it on" and its not you who will be injured directly by such comments, but someone else. you absolutely do NOT have the right to make such statements. And this includes any human being, not just George Fucking Bush.
Two infinite things: your stupidity and mine. But I'm not sure about the latter. If my sig offends you, I'm sorry.
I believe that would be "from the sounds-better-THAN-my-12th-grade-teacher dept."
So all I have to do is figure out the eigenfunction (or eigenessay) to the system and my grades will soar. It would be a good thing if my math skills would help me gett better grades in litterature.
This is my sig, show me yours
Fact: Teachers would read only the introduction and conclusion paragraphs, and rely on the grading software to account for the quality of writing of the paper, and grade that way.
Brutal Truth: Teachers already read only the introduction and conclusion paragraphs, so use of this software actually would be an improvement.
If you can check for grammar, spelling, punctuation and other things, this would be a real help for e-mail. People who read a lot of e-mail could use this as a sort of chlorine in their mail pool. For example you could say someone needs at least a C+ to get read by me.
Mail written like "u r fukker I kil u" would never need to tarnish someone's consciousness again.
Very good point.
It is also likely that the 5% are split between the very high and very low ends of the normal distribution. Only exceptionally good teachers can deal with properly teaching & grading students at those extremes.
In particular there are many instances of "brilliant" students suffering at the hands of poor teachers.
Well it freaked out at a Deleuze essay... managed to swallow one on the Parmenides, and promptly gave me socket errors on everything else that was more than 2000 words.... bah.
What didn't impress me was that for two of the essays that were too long, I tried just pasting random segements of them in. Still got around 100 for everything...
i don't read slashdot anymore.
So I went to the demo site at http://testing.tc.iupui.edu/fipsedemo/ and c&p'd an essay generated by the Postmodernism Generator and guess what -- it got a nearly perfect score!
Overall: 100 3.2384
Content: 100 3.0584
Creativity: 99.973 2.9973
Style: 100 3.4543
Mechanics: 100 3.6429
Organization: 100 3.4075
Guikachu: Resource editor for PalmOS developers
I'd have read the damn article but I don't feel like sharing out my email address so that I can get even more crap.
Just the idea however I find very distasteful.
I don't like the fact that teachers (hey, they are human) weight scores on papers by likes and dislikes, style (not everyone like likes reading the same style of writing), organization techniques (ever had a teacher or prof _tell_ you how your were going to organize?), etc. I think that is bad enough. This grading software substitutes one person's (or commitee's) values. It will pick up a bias because of training. Sure, there are objective standards to be met but the software would miss the point because it can't really do analysis, just distill words.
Here on Slashdot, with the community (and still staff?) modding things up or down, the default (computer) is to subjectively award karma bonuses based on longevity and already high karmas. People, like, computers programmed by people, screw up or misunderstand. For example, my posts normally start off with a 2 unless karma bonus is shut off. I posted recently on something that ultimately was modded "off topic". It actually wasn't since I was pointing out a parallel between tactics used in the posted item and the tactics used by another group who most people I know find abhorant. Now this is fine that it was modded "off topic".
At least if you are someplace a paper is marked down where the professor graded it, you can defend it. Some teachers are really rigid and there are still alternatives. But with a computer, either there is no one to argue with or you get the easy answer "that's what the computer said so it's right" syndrome. We all seen that, it's been the subject of countless stories/movies, including Dick's short story "Minority Report". Okay, it's about more than that but it's a theme and it's what most of the characters in the story accept as gospel for much of it.
Perhaps the biggest danger in computerized grading is that objective analysis is a skill and this will weaken that skill in the people that otherwise would be doing the grading. Then what will happen when they aren't grading anymore? Additional work/class assignments that will tend to distance them from the projects and students they already have because they'll have more.
-Tez
Haskell, the static-typed, lazy, polymorphic, programming language.
First, an essay, or anything else written in a human language, is about communication. Has the student managed to communicate their idea? Personally, I feel this is more important than whether or not the student actually had a good idea to begin with. As a writer, you need to learn not only the formalities of the language, but how to play to your audience. There's no such thing as an objective reader and students shouldn't be taught to believe that there is an objectively perfect essay for which to strive. A computer, or a centrally controlled set of "grading standards" are simply other subjective readers. The value of a teacher of languages is in their unique subjective viewpoint, not just their ability to scan the page.
Second, language is a constant struggle between conformity and ingenuity. Spell-checkers and grammer checkers (when I use them) allow me to express my ideas without offending the eyes of typo-sensitive readers, but they are also "locking down" English and collapsing the variants (alas, poor "colour"). Automated checking and intolerance to mutation is causing us to lose the ability to evolve our language.
I fear that English is becoming the QWERTY keyboard of languages.
I would love to hear what Stephen King thinks about this too.
Yell & scream & rant & rave... it's no use... you need a shaaaave ~ Bugs Bunny
If there were any unix gurus around here, then they would know about AT&T's documentors workbench. Too bad that isn't open source. Its spell checker works much better than ispell ever will. It seems to know about errors such as nearby keys as well as other things. If you type "the" as "rgw" it will know that. If you perfer to type it as "qaz" it will learn that too and cope. It also seems to know about gramer and word tense. Too bad there isn't anything opensource that is even close and I'm not sure who even owns it now.
1. Suppose you could put together a bunch of stats that have nothing to do with the content of the paper at hand and use that to predict the grade that a human would have given the paper with 95% accuracy. E.g., you look at the writer's socio-economic level, which high school he went to, what his grades in high school were, how neatly he is dressed, how neat his handwriting is, etc. I am not saying you CAN achieve that kind of correlation (in fact surely you can't because there isn't that kind of correlation between papers by the same person, I suspect), but what if you could? Isn't it obvious this is not a good thing?
2. But here's a constructive use that would save us university faculty significant time. If the software really does grade spelling, grammar and syntax well, one could require of the student that before an essay gets handed in, it gets some high minimum score like 90% or even 98% (unless the student is dyslexic or something like that). Then we would not have to look at essays that had poor spelling, grammar and syntax, would have to do less red-inking, and would have more time to grade for content. (Which is all I grade for anyway in philosophy; though grammar, spelling, etc. get marked up but don't count unless they get in the way of my understanding.)
So soon the same people who ring up your burger by pressing the picture of a Big Mac will be teaching your kids. "Well, Sally, the machine says you got a D. I pressed the right button, see? The picture of the 'Book Report.'"
If research/rhetoric classes were only about getting all the grammar right and adequate use of a thesaurus, it'd be a killer app for the lazy and uninspired student. I honestly can't see myself or other writing teachers actually using it, except maybe as a research tool.
The interresting thing about this is that it should grade a given paper the same every time. Now, with that in mind, if a student were able to get a copy of the software and the grading rubric, it would be possible to test their paper before turning it in. This means that they will type a paper and just correct the errors the program finds, giving them a guarenteed A every time. ;)?
Where can I get a copy
According to the article, the essay grading software deducts it score from statistical analysis of the essay, this is the same a Bayesian spam-filter does to rate a message spam or ham.
The 'solution' spammers found to defeat Bayesian spam filters, is to fill the body and/or subject with common words, forming grammatically correct sentences, that are semantical rubbish.
Here is an example of a few sentences found in this kind of spam messages:
As you can see, this is clearly garbage, but to a computer grading this text using statistical analysis methods, this might well be an award-winning essay.
All you would need is a database of nouns, proper nouns, verbs and a few templates for sentences, questions etc. You could even do just with some templates for sentences and have your essay-writing program contact http://dictionary.com or some similar website to get the words to fill in your template.
Sven
-- Slackware linux... because wizards are for wussies
So, if I get a prof that just uses the tool as a shortcut, as long as I can create something that is well composed, has style and diction... I get an A!
My father told me of a classmate that wrote the first half of an essay answer as best he could, but ran out of ideas, and started writing about Mickey Mouse's great acheivements in science. The guy got an A, because the overworked grad student just looked at the first sentance to see if you were on track, and then gave you credit for that!
meh
Those two sentences are wonderful! I half suspect you of doing that on purpose, but I'll bite anyway...
The first sentence has a dangling preposition - it ends with 'in'. '...every program in which I've ever used it' is grammatical, and sounds more elegant IMO. In the second sentence, you use a possessive where you mean a subject and verb - it should be 'it's worthless' not 'its worthless'
What is the robbing of a bank, compared to the founding of a bank? -- Bertolt Brecht
I know you were just being a smartass with that paragraph, but I have to say, it was wonderfully amusing. "If I have not typed 500 words, this paragraph is not my penultimate, nor was my last. To assert otherwise is prudent, but lacking in elegance." You should try to write a "post-modern novel" and see if Salon will give it a good review.
God is real unless declared integer
Heh, while you can't do so RANDOMLY as grandparent claimed, for obvious reasons, you CAN do quite well.
;)
Here's how I managed A's in all the college religion courses here:
What is the Bible?
A) The inspired Word of God.
B) A collection of works holy to Christians.
C) Utter nonsense written by dead white males.
D) An alien mind control device.
(Yes, I'm exaggerating here.)
All you have to do is pick the "academic" choice that says nothing that anyone on either side of an issue should find controvertial (that's B, here, if you can't tell; the other answers are biased towards Christians, atheists and wierdos, even if that might be redundant
Also, if they wanted you to pick a name out of a list and you didn't know who it was, you would usually see names from *other* religions & such in the list, so by eliminating those silly choices.
So yes, you CAN do well on multiple choice tests without having to know it all. I got A's and did NONE of the assigned reading. Granted, I knew a fair bit to begin with, but the fact that I was able to divine the answers by checking the disposition of the questions gave me quite an edge.
The size of the school isn't the issue. It is the student/teacher ratio. I have no idea if anybody really understands how it happened. Huge loads are just reality in many of the biggest districts in the nation. I suggest you just ask an English teacher about his/her circumstances.
;-)
Everybody has a pet theory. The stats don't mean anything unless you know the population and reason for collection. For instance, lots of foreign countries have high math and science score averages. They also cull low performing students. (Bang!) I wouldn't be surprised to hear anything quoted.
Anyway, block scheduling helps a lot, but the math departments hate it like the plague and vote against adoption. My school had to hit rock bottom before it changed. Also, don't forget that the four period day has to fit a whole year of work into half a year, so students should write twice as much if the class is 90 min.
BTW, I met two foreign language teachers last week who had seven classes each. This happened because other elective teachers had been canned in fovor of more english and reading "units." So if things get better in one place, they suffer in another.
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
As someone else pointed out, if we're going to let computers grade the essays, we might as well let computers write them too. I grabbed a piece of gibberish from the Postmodernism Generator, and it got a whopping overall grade of 90.67.
Here is your grade inflation research straight from the university of texas.f lation/Inflation.html
http://www.utexas.edu/student/research/reports/In
If you use Linux, please help development of Autopac
I don't thing Hemmingway ever wrote a run-on sentence.
/. just means that people don't like what you said and has nothing whatsoever to do with fact unless it is about networking. :-))
Graders don't miss shitty grammar. They are forgiving if the student writes anything remotely interesting. That's all. They are bored by the drek that students try to pass of as thought and will go out of their way to reward something entertaining.
It is also inside current guidelines at my institution to overlook poor grammar in favor of original thought. Actually, I think it comes down from the district headquarters.
Moderation on
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
A progrm such as this one can only grade the construction of the essay, but will be blind to the concepts and ideas contained within that construction.
In all but the most basic writing courses, an essay is not given to test the students ability to construct an essay, but to measure the depth of understanding the student has gained of a subject by having the student demonstrate this understanding through the composing of an argument supporting a thesis that addresses a question or problem inherent to the subject of the course.
Proper grammar and stylistic constructs are necessary in order to convey the intended idea(s), but the ideas themselves are not contained in the construction itself, but in the relationships between the definitions and connotations of the chosen words that make up that construction, as well as in the order of presentation of the concepts (definitions and connotations), the implied relationships of separate constructs (similes, metaphors and ironies), and the properly drawn conclusions (by which I mean properly drawn from the preceding valid arguments and evidences. I do not mean "agreeing with the proffessor").
Unless a program can be written that is truly cognitive and therefore capable of understanding all possible relationships between the above concepts, as well as understanding all possible permutations of the legitimate classes of arguments (logos, ethos, and pathos), all possible combinations of those concepts (including the classes of argument), and be capable of deducing meanings from context, an essay grading program would only be able to provide perhaps half of the analysis for a proper grading of any essay.
In other words, a 100% from the grading program should only garauntee a 50% for an essay.
Read, L
Take your best A-papers and even some scholarly articles from real journals. I bet several of them score par for the course. Standardizing tests has gotten bad enough with unique skills and knowledge being required in different places; standardizing the criteria for measuring writing skills is just stupid, especially considering that there are dozens of ways to manipulate a sentence in the English language and have it still make perfect grammatical sense.
That, in my opinion, is the essence of this language. Nearly every rule has an exception, and that allows for a wide variety of styles, with regard both to diction and to usage.
....but I like to keep an open mind. Let's see how this thing works in action.
Automation in this instance seems like the lazy way out. I have always gained more personal insight by what the teachers have written on my papers than anything else. That is part of the learning process. I teach computer / graphic design and I am always writing down a student's feedback for them. If you don't want to do the work, don't become a teacher -- there are plenty of us who will.
--- Cassie
In response to HanzoSan: I would say, as a teacher, that part of a student's job is coming to class and participating. Part of this is: reading the materials and coming in to share their experience of that reading. I usually set up attendance as part of a class participation grade.
BTW, can you tell I teach English Lit? These are classes where discussion and participation (I feel) are important. On the other hand, if a student has indeed turned in kickass work without the standard hallmarks of student participation, I usually give them the benefit of the doubt.
I can see your attitude being very suitable for extremely large lecture classes, particularly since: discussion is impossible in class and is usually saved for discussion sections run by a GTA. Lots of the information can be obtained from the textbook used in class (though you get a sense of what the Professor in Charge wants by attending lectures or at least borrowing good notes from someone).
Over the years, I've learned to (in some ways) not worry about attendance. The students are adults. The simple fact is that good performance without consistent attendance is quite rare. So it's a problem that is self-correcting. I suppose my main problem among students is lack of empathy. Have *you* tried successfully teaching a course where 10 of 20 students show up? How can you hope to communicate the skills and information you're charged with communicating if not everybody shows up?
It's because of limitations on what we can detect. These will inevitably influence the grading. Style checkers will ding people for every use of the passive voice -- not because every use of the passive voice is bad, but because it's so easy to detect.
They will flag every dependent clause, punish people for using long words or unusual vocabulary. They will warn teachers when students use long sentences, complex language, long paragraphs. They'll give a Fog index, an understandibility rating, a grade-level requirement.
Teachers are busy, underpaid, and often bored (like when grading 120 essays). As someone who taught for seven years, I can guarantee that having teachers use this tool will influence the grading. "Yep, that's passive voice, all right. Ding!"
The ability to write something so that anyone with a third-grade reading level can understand it is important. The inability to write anything that a third grader can't read is a liability.
Submitting to the tyranny of these systems will punish students who express complex thoughts, students who use creative, counterintuitive language, and students who write in anything other than a series of simple declarative sentences.
Behold the riant ape! Beware, his crooked thumbs!
if you're so smart how come you jumped headlong into an argument with someone nicknamed "Gay Nigger"
how about you have the nuts to say that to me on K5
YHBT! I'm pwyshall!