Slashdot Mirror
Essay Grading Software For Teachers
asjk writes "Software to help teachers with grading has been around for sometime. This is true even with respect to grading essays. A new tool, called Criteria, will look at grammar, usage, and even style and organization. It works by being trained by at least 450 essays scored by two professionals. The difference this time? Here is a snip from the article: '"There's a lot of skepticism," Dr. Spatola said. "The people opposed see it dehumanizing the student's papers, putting them through some sort of mechanical, computerized system like the multiple choice tests. That's really not the case, because we're not talking about eliminating the human element. We're making the process more efficient."'"
that they've automated away a major part of a professors job, while we still need humans to pick spinach and deliver pizzas.
Don't drop the soap, Tommy!
I thought the point of an essay was to grade the ideas and how well they're expressed. I didn't realize they were spelling/grammar tests.
Maybe I'm just a bit jaded by this because of all the stupid grammar and spelling nitpicking that goes on here on Slashdot. Evidentally, it's much easier to criticize my spelling than it is to provided a rebuttal to my point.
"Derp de derp."
1 - the grammar check option in MS word is crap. this sounds awfully similar.
2 - your resume can suck, but with the proper buzz words, it'll come out looking like gold to those automated resume checkers.
1+2 = students who turn in good papers that aren't structured perfectly (and you have to admit, there is some fluidity to language) will get marked down, and those who know what bullet points to put in their papers will get good marks, even though the content is crap.
How long until you get kids selling manuals in the bathroom on what the machina are looking for?
--I don't want the world, I just want your half.
I bet that I can write a paper that satisfies this application's conditions for correctness of grammar, usage, style and organization, but is completely and utterly meaningless.
Then, let's feed this thing Ulysses and let's see how high it grades Joyce.
Anybody who can't see that this thing is useless for promoting any sort of creativity among students is off their rocker.
Then it is the students who are being cheated by a teacher using the software that doesn't double-check the material on his own. They will go through the class without having their mistakes caught. While the erosion of standards that a flawed proofing program might bring isn't likely to be enormous, it's kind of strange to think that the future of the English language would be in part determined by a development team piece of software.
Hope it works well, though, and gets used as a proper checking tool.
As long as this is merely an assistant and not the end-all be-all, as long as actual qualified instructors review the essay after this program does, I'm all for it.
The English language is so full of subtleties, nuances, combinations, and fantastic structural intracacies that make phenomenal writing in it possible (Faulkner, Bradbury, etc.). There's a reason English is a field of study for graduate degrees: it's absolutely worthy of them. There is no subsitute for the educated, refined judgment of someone who is exceedingly well-versed in the language.
The coolest voice ever.
There is no "humanity" in a modern constructed essay. There are certainly going to be "judgement calls" when standards are not as fully fleshed out for the computer as they should be, but as long as those are appealable, I have no problem having a computer assign me the other 95% of my essay points. The only instructors who will fear this are those who like to assign grades arbitrarily. And I don't feel too sympathetic toward those people.
"You're never ready, just less unprepared."
If the poem's score for perfection is plotted along the horizontal of a graph, and its importance is plotted on the vertical, then calculating the total area of the poem yields the measure of its greatness.
A sonnet by Byron may score high on the vertical, but only average on the horizontal. A Shakespearean sonnet, on the other hand, would score high both horizontally and vertically, yielding a massive total area, thereby revealing the poem to be truly great. As you proceed through the poetry in this book, practice this rating method. As your ability to evaluate poems in this matter grows, so will - so will your enjoyment and understanding of poetry.
(From the full script.
bash$
"The people opposed see it dehumanizing the student's papers, putting them through some sort of mechanical, computerized system like the multiple choice tests.
Actually it's about time! I don't see the essays themselves being dehumanized, but what I do look forward to is the day a middle school student doesn't receive a bad grade just because his book report was on the "Theory of Relativity" and the teacher couldn't comprehend the subject. (This is from experience) What it will do is take the human factor out of the grading process and grade all reports equally regardless of subject matter.
iRepairIT - iPhone, Mac, & PC Repair
is just one of many writers who would flunk using this system.
'Nuff said.
this software would be perfect for students majoring in comp sci or engineering who have to take a composition / writing class...
Course:
College of Liberal Arts / Sci: Rhetoric 105
- or -
College of Engineering: Pattern Analysis 202
Objective:
To teach the principles of essay-writing skills. Liberal Arts students will be encouraged to follow boiler-plate styles and formats, while Engineering students will be graded on their ability to analyze and defeat pattern recognition software.
- rabs
Sounds like everyone feels the same way too... We've got some automated testing software for MS Office at the local college and although it's getting better, it still makes really silly mistakes from time to time. Analyzing English composition has got to be many times more difficult than watching a bunch of clicks and key presses.
The only use I can see for this thing is as a "first pass" grading tool that quickly finds obvious mistakes (spelling, grammer, redundancy, etc) and flags them for the instructor. On the other hand, it's probably just as time consuming for the instructor to read over the flagged items as it is to just catch them on the first time reading through the paper.
So where does style come in? There are many, MANY forms of style, which make writters unique. For instance, I've found that when I write, even the shortest essays, I tend to break up my thoughts into multipart sentences... like this one. They tend to be very long and drawn out. I also use "granted" and "don't forget". I also seem to create a lot of sentences that are self contradicting: Though this, something else. It's part of my style.
My style isn't completely mine. I'm sure over-use would be bad. Granted this. Granted that. Where do those softer features of writing come in? Or are we all to be sterile and write with no tone or style.
--
"I'm not bright. Big words confuse me. But Wanda loves me and that should be enough for you." - Cosmo
As far as the achievements of ancient cultures go, it is all relative. We have harnessed fusion, mapped the genome, created antibiotics, peered deep into the hearts of galaxies a 100,000,000 light years away, forged fiber optics, designed the integrated circuit, et cetera. People three hundred years from now will look back upon us and wonder how a civilization that could barely put a man on the moon (a feat that will surely be trivial to them) was able to usher in the Information Age in only a decade worth of work.
;)
Some "dehumaniSing" could be a good thing, espcially when grading subjective material.
Objective material is factual, a simplification is "Most dogs have 2 eyes."
Subjective material is opinionated - "Australia should legalise heroin injecting rooms." Obviously this is controversial, and there are serveral positions on the matter.
Most teachers/lecturers/graders/tutors have their own (pre-existing) subjective opinions on certain topics. If you submit an essay that opposes their views, the chances are very high that you will get a lower grade, even if your essay is well formed/written/structured.
In high school, I always took this into account and wrote essays that agreed with the teacher's point of view, even if I didn't. Such software could lessen the need for writing what they 'want to read'.
You werent taught English. I'm not trying to insult you but thats one of the problems with our public schools, they dont do a good job teaching
When I went to high school 15 years ago, we didn't do any grammar in high school English class, it was all read-and-interpret (i.e. read-and-make-up-some-bullshit).
Yes and thats why when you got to college you couldnt write a good research paper.
We were supposed to learn the technical stuff in middle school (and we did to some degree).
You are supposed to learn English through highschool as well, if you want to get a 1500+ on your SATs. This is exactly why students get such low SAT scores in urban public schools, they dont get a focused education, when its time to take tests the test does not care how creative you are or even how intelligent you are, the only thing that matters to the SAT test is your technical knowledge.
Teach technical English and later on let a person learn creativity.
If you use Linux, please help development of Autopac
It's true. I've had teachers take their questions and quizzes directly off of websites (the curious may want to enter a few key words from their latest homework on google and see what turns up...). Now here's an ethical dilemma for you: if it's ok for them to get the questions of a website, is it alright for me to get the answers off that same website?
The good old fashioned teachers, on the other hand, would never do such a thing. No, they have been xeroxing the same handouts for the last two decades. You can tell by the fact that they have become half unreadable.
Homework is either trade it and grade it or credit/no credit. Major tests are all done on scantrons.
That's not to say I don't have good teachers.
But I've also had teachers who put in the bare minimum. At the end of the day, they're gone before the rest of us. They teach 5 periods (one's prep) for 180 days a year and gripe about having an average salary with mediocre benefits. On the other hand, the conservative holdouts who arrive at school before the janitors, who stay up all night meticuously grading essays, and who teach sheerly high schoolers simply because they want to (many have Ph.D's and could easily be working at the local University if they so desired) never gripe.
There has been a lot of proposals about reforming teacher pay. I don't know what can be done to attract better teachers. But sheerly for the sake of fairness I would like to see the good teachers taking a disproportionate share of the money. The trouble is, they would never ask for it.
If I have learned anything from my university career it is this: As class sizes get larger, testing becomes more frequent and more automated. Of course you say, if you have a class of one hundred or more people, it is simply not possible to mark that many essays. This usually means that essays don't need to be written at all! What do they do? Multiple choice! I heard a statistic once that if you chose answers randomly on a MC test that you could get a C by not knowing anything beyond how to circle a letter! ----- Discovering this, I made sure that I took all the obsure english classes that had no more than 30 people in them. An unexpected positive side effect to this system of choosing courses was that 90% of the other students in them were girls. Yea, life was good. ;-)
Apparently, the system uses statistical analysis as well as grammar checks to determine the score for the essay. Basically, they've built up a database of essays that have been graded by a bunch of humans, and then used these algorithms to figure out which bucket the essay belongs in. Sounds kinda like SpamAssassin, actually. I'd be willing to bet that with sufficient resources (in terms of essays and human grading time), this wouldn't be all that tough to duplicate. After all, what are spam filters but content analyzers? (Shameless plug for a system that requires human judges rather than computer judges)
--
Annotateit at Annotateit.com
Automated is good because theres less chance of error, and its almost always fair.
The only way to get fair grades in university is to be smart enough to pick the right teachers, and drop the ones who you dont get along with.
I heard a statistic once that if you chose answers randomly on a MC test that you could get a C by not knowing anything beyond how to circle a letter! ----- Discovering this, I made sure that I took all the obsure english classes that had no more than 30 people in them. An unexpected positive side effect to this system of choosing courses was that 90% of the other students in them were girls. Yea, life was good.
[ Reply to This ]
Who wants a C? Thats as good as an F in college, if you get a C you can just drop the class and take it again!
I dont really like small classes myself, there is no real benefit, what I notice from smaller classes is, teachers are more critical of you, you get greater punishment for poor attendence or for being late to class, you also get more focus from the teacher and this can be good or bad depending on if the teacher likes you or not.
If the teacher likes you, getting this extra focus is a very good thing because a personal connection with a teacher who likes you is to your benefit, however if the teacher dislikes you and decides to personally focus on you, this is bad.
If you use Linux, please help development of Autopac
You speak as if the use of computers to judge intellectual works will somehow make our society exempt from "rich upper class morons buying and pandering their way through school". Such an aristocratic model is something that exists beyond the scope of one's grades in school, and will not be eliminated, in any sense, by such a thing.
One of the primary purposes of essays are to learn how to write for a specific audience.
If you remove the human element, then you aren't writing for any audience, unless, of course, everyone starts writing for computers' entertainment and education.
At first glance, I thought that a tool used to analyze essays would be a nightmare -- it would kill creativity, homogenize style, etc. However, I then remembered my experience grading freshman *college* essays last year. This was in a small-classroom course, taught by an extremely good instructor that also offered office hours galore, all kinds of free tutoring on campus, free access to computers with MS Word, etc -- in other words, *plenty* of chances for the students to improve. I was given three distinct classes worth of essays to grade, and traded off with the instructor on which half of each class we'd each tackle. The papers were dreadful...while as an English & Creative Writing grad, I am extremely pro-creativity, these students weren't ready for that. Some of the sentences I ran across were so awful I would IM my friends and have *them* in hysterics at how terrible they were. For example -- and keep in mind, these are middle-class white kids, NOT English-as-a-second-language students. "There is not connection with what I know the same circles don't fit inside squares." "There is also numerous of shapes and designes which not to difined." "They get into the analization of the man..." The simple truth is, while creativity is great, there is a baseline level of grammatical ability that needs to be used in order for others to simply understand WTF the writer is saying. A lot of the time, it would take me 5 - 6 scans per sentence to figure out what my students were writing. The worst part of this is that because of department rules, I wasn't allowed to give any kid below a C if he/she *tried* to follow the essay guidelines. As long as the right number of pages, subject, etc. were touched on, the person would pass. I think the best use of technology in cases like that would be to sit the person down at the computer with the grammar-analysis program, and rather than have them ignore classroom lessons, interactively edit their own papers. Not the way that Word v.X does it, where the person just right-clicks to get a "correct" change, but the older method in which the program offers a series of alterations with explanations *why* the original is out of whack. Doing that in lecture isn't realistic, unfortunately. Despite the number of grammatical nightmares in the course, the students really were at varying levels of ability, each with a unique misunderstanding of the rules. Each needed individualized attention, though as far as I could tell none of them were trying to obtain it. Also, the campus requirements for Freshman English didn't leave time for stuff they should have learned in elementary school.
Now before you start up the flame throwers, this is not a message to deride high school students over their lack of creativity.
But when I was in high school, we were told that proper essay writing was an essential skill for the departmentals, and when they said "proper," they meant "Must conform to between five and seven paragraphs, with the first and last being this opening and conclusion with three to five paragraphs of body--each containing one topic of discussion."
Furthermore, it was made VERY clear that creative or unconventional ideas (let alone language!) would be strongly frowned upon. There was One True Way to write an essay, and One True Opinion on any given subject. Any deviations from that would cost you.
I hated it then, I hate it now, but I don't see any problem with having computers mark essays like this. After all, they were trying to turn us into computers to create them.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
"Rules were made to be broken", they say. And the same is true of the rules of English. Creative use of language necessarily breaks rules. Winston Churchill once remarked about bad grammar, "This is a situation up with which I will not put!"
What this will do is reward a class of students good at memorizing rules. Most of the great writers of English would fail were their work submitted to the machine. Let's see what the system does grading Shakespeare, or Donne, or Lincoln, or George Eliot, or Joyce, or Pynchon.
If they want to use this in high schools and middle schools then that assumes that all their students have computers with a word processor. What about all of the lower income kids at schools that don't allow after school activities (like using school computers) like public schools in NYC? It's a tool for the middle and upper classes that will give their teachers more time for their students and leave the teachers in poorer areas still overworked and struggling to keep up. It won't be feasible where it is needed the most.
Why are we feeling so threatened that a machine can do a human's job at this one, narrow task?
Haven't you learned anything from Deep Blue? How many problems must the AI field conquer before it's taken seriously?
Don't flatter yourself with delusions of superiority. If the computer agrees with human graders 99% of the time, then its response is definitely valid.
Besides, its not as if professors grade your work anyway. That's a job for TAs.
No one will read this, but I bet that these automated essay graders are able to mimic human graders closely because the human graders themselves graded the 450 "config" essays under computer-esque conditions, i.e., a time constraint that forced them to skim, think as little as possible and generally act like a machine.
Fact: Teachers would read only the introduction and conclusion paragraphs, and rely on the grading software to account for the quality of writing of the paper, and grade that way.
Brutal Truth: Teachers already read only the introduction and conclusion paragraphs, so use of this software actually would be an improvement.
1. Suppose you could put together a bunch of stats that have nothing to do with the content of the paper at hand and use that to predict the grade that a human would have given the paper with 95% accuracy. E.g., you look at the writer's socio-economic level, which high school he went to, what his grades in high school were, how neatly he is dressed, how neat his handwriting is, etc. I am not saying you CAN achieve that kind of correlation (in fact surely you can't because there isn't that kind of correlation between papers by the same person, I suspect), but what if you could? Isn't it obvious this is not a good thing?
2. But here's a constructive use that would save us university faculty significant time. If the software really does grade spelling, grammar and syntax well, one could require of the student that before an essay gets handed in, it gets some high minimum score like 90% or even 98% (unless the student is dyslexic or something like that). Then we would not have to look at essays that had poor spelling, grammar and syntax, would have to do less red-inking, and would have more time to grade for content. (Which is all I grade for anyway in philosophy; though grammar, spelling, etc. get marked up but don't count unless they get in the way of my understanding.)