Slashdot Mirror
Indiana First With Computerized Grading
Mz6 writes "Computerized grading has been talked about previously, however, the New York Times reports that Indiana has become the first state to grade high school English essays by computer. The computerized grading process, called 'e-rater', uses a 6-point rating scale and uses artificial intelligence to 'mimic the grading process of human readers'. The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. The big question is, will other states begin to emulate Indiana by tossing human grading?"
Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."
I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).
Kinetic stupidity has a new brand leader: Allen Zadr.
I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?
--
My username: hats off to George Carlin, and fuck the FCC. Freedom!
Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.
The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...
You are in a maze of twisted little posts, all alike.
Grading freely-written essays and structured computer programming code is two separate things *to a degree.* And India isn't the first to do the latter. Indiana is, however, according to the article.
I have taken plenty of essay exams where I felt I wasn't graded fairly by a human teacher/professor.
Some essays were graded out of a couple points. A paper out of 6 points carries less weight overall. If this is the only exam (ie AP tests) a 5/6 is looked at as a high score.
I don't see your point.
The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers
I think this says more about the training that the "trained readers" are receiving than it does about the software.
Good: The computer probably won't grade you down for writing an anti-Bush essay, and it probably won't get fired for it. Good: Computers won't play favorites, and you can't kiss up to a computer. Bad: The computer really can't grade you up for expressing original ideas. Bad: It's probably possible to fool the computer somehow.
-73, de n1ywb
www.n1ywb.com
At least the parent post proves one simple truth: human english teachers can be replaced by simple shell scripts.
Not only computers may not give fair grades, but there's a deeper problem with grading using computers: to me, students working to get good grades from a computer conjures up images of sheeps going in the wool-extraction machine. While this may be fine for sheeps, how do you think the students feel about it?
When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?
Not that there's anything in this post that serves as an example. I guess that's because I was graded by humans. Seriously, I don't recall getting any encouragement in writing back in the '70s in high school, and not much in college. I guess it wouldn't have been any worse if the Grade-O-Vac was inspecting my papers instead of my mostly-marginally-literate teachers. There were several exceptions, but they focused much more on reading than on writing. I suspect they had a lot greater effect that way--I know they had a great effect on me.
If I were a student, I'd want to get a copy of this software and use it to pre-grade my papers so that I could find out what's wrong and fix it before I turned it in.
When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?
You had a different school experience than I did apparently. I felt that the human's reading my papers were distant, uninteresting, and less than worthy of grading someone else's work.
Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.
On "bad papers" the comments were usually less than helpful. Don't just mark up the paper with "comma splice" or "vague". The teacher should have taught the class what a "comma splice" was or should have been following their own words of advice and kept themselves from being "vague" in their comments.
Sure, but that's the fault of the humans implementing the grade system, who don't understand the difference between Gaussian and uniform distributions. Don't blame the computers.
Not that computers are a great idea here - they can only grade at the shallowest level, and if they were grading like real teachers, then those "real" teachers weren't doing their jobs.
But this specific problem that you mention is entirely human based.
Imagine high school students re-writing their essays ...
Actually, anything that would encourage students to re-write their papers and improve their writing would be pretty amazing. Most students jot something down, run a spell checker and turn in their work. If they could pre-grade their work, they might be better motivated to put out more effort and improve their writing.
Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.
They're just converting "n/6" directly into a percentage grade?! That's ridiculous. So there's no such thing as a C (going straight from 83 to 67)?
My writing style is somewhat peculiar, though I can't exactly say how (or even approximately how). Partially as a result of this, my marks in English class over the years of high school ranged from C to A, depending not on me, but on who the teacher was. If the teacher happened to like my style, I got a good mark.
This is annoying, but at least each year there was a different teacher, who may like my style. If the marking is computerised, it will not change; if your writing doesn't fit what the computer likes, you're screwed; likewise, if it does like it, you might never learn to express yourself more creatively (ie you'll be punished for trying to write in a manner different from what you usually do).
There are possibilities in this technology, but I suspect that it will be a long while before the eccentric aren't labeled as poor writers.
A computer can check spelling and even grammar to a certain extent. However, it cannot evaluate factual accuracy, strength of argument. Even with spelling, the computer is not likely to catch improper use of homonyms. I can guarantee you that it will be possible to create a piece of writing that is utter crap that would get an A+ using this or any other possible computerized grading system. Unfortunately, there are probably many teachers out there who make poorer graders than this system does. The answer to the problem of poor-quality teaching is not replacing teachers with computers; the answer is a combination of better teacher pay and putting higher standards in place for our teachers via competency testing.
Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.
A computer can not replicate certain aspects of the grading process. Sure they can grade spelling and grammar and probably certain aspects of your writing style but there is plenty of important aspects of writing that they can not grade you on.
For instance, does your essay really grab the reader? Anyone here who reads technical documents knows what I'm talking about. There are some writers that, no matter how dull the subject, can make their work interesting and fun to read. A computer can not possibly grade one on that. I have a good friend who's a high school English teacher and occasionally I'll read some of the things written by his students. I've come across plenty of papers that are grammatically correct, have perfect spelling and are fairly well written from a syntactic and stylistic point of view, but are just plain boring to read. Then I'll move on to another paper, about the same subject, which is interesting and actually fun to read.
That's just one example of something a computer can not possible take into account when grading an essay. The bottom line is that a computer will never be able to grade you on certain subjective things, which although they are subjective and therefore open to a certain amount of interpretation depending on the person doing the grading, are nevertheless still very important aspects of good writing.
With spelling and grammar check, almost any average student can churn out a paper that is going to be mostly correct; however it still takes a good writer to produce something interesting. In my opinion, an interesting paper with a few minor spelling, grammar or syntactic errors is just as good as a boring paper with no spelling, grammar or syntactic errors.
The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.
Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?
They are not graded fairly, and they determine 10% of the final grade.
10% of the grade on the essay? Or in a particular class? 10% on the essay may actually be tolerable, because that means that at least a human actually read it to give the other 90% of the grade.
A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.
It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.
The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.
The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.
In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.
"How to Do Nothing," kids activities, back in print!
"That's because their HS teachers were too overworked to grade their writing, so they didn't assign much."
With all due respect, that is a failing of the teacher, or in other words, the difference between an excellent teacher and one that is average. My best high school teachers had no problem assigning an essay a week. Yes, they had a lot of grading, but they realized it was important and the best way for the students to learn. Any teacher unwilling to do the same is not worthy of being considered a good or excellent teacher.
Granted, a computer grading program could make it easier. In the hands of a good teacher, I don't have a problem. It certainly won't be any more biased than a person. But if the teacher wasn't willing to put forth the effort in the first place, how is this program going to make it better? Are they going to check all of the papers to make sure the program didn't screw up, use the extra time to emphasize advanced writing skills, etc.? Or are they going to treat it like an assembly line? I know which one I would place money on...
Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.
It is also the exact environment of the modern workplace, as designed. Results are irrelevant. Only the process matters.
Business isn't willing to pay for products, innovation and careers, so we get brands, mortgage commercials and layoffs.
This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.
As a college instructor, I tend to use a strict grading protocol -- and then "bump up" a few of the students. If someone comes in to my office every week and really struggles to understand the concepts, but the computer tells me that they earned a "C+" -- they're likely to find a "B-" on their transcript. But if someone who's smart enough to get an "A" blows an exam from being hung over, that person gets little or no sympathy.
Right. The software can grade things like spelling, vocabulary, grammar, and syntax, but it won't grade the quality of the ideas expressed or the depth of understanding displayed. Therefore those latter dimensions will cease to be considered important, since grading them is expensive compared to the more mechanical stuff.
The 5 paragraph essay is the cookie-cutter bottom line least common denominator for education these days.
I would estimate 90% of the essays given in the US educational system today are 5-paragraph essays.>br? Essentially, the 5 paragraph essay is a mold consisting of an introduction, 3 body paragraphs each focusing on one supporting fact, and a conclusion, and the teachers give students the ingredients in the form of a topic (definitely something that won't require too much thinking),and a style, The students generally mix these ingredients in their heads for about 30 seconds, vomit them into the 5 paragraph mold, and are graded on how aggressively the spell and grammar checked the paper.
Famous Last Words: "hmm...wikipedia says it's edible"
The postmodern essay generator
Unfortunately, the system described here doesn't return any such useful feedback. The Indiana system returns a grade from a six point scale. No comments, no criticism, no hint that the evaluation is meaningful.
Incidentally, what's this about "teaching instead of tedium"? Grading essays by evaluating construction, insight, and creativity should be part of the teaching process. Perhaps this is something that should be addressed earlier in the education of these students - if they're reaching college as "borderline illiterates" there is a problem - but grading in general is a part of teaching. If I were a student, I'd want to know that a human being - at some point - had bothered to look at the work that I did.
~Idarubicin
The thing is that a human can actually understand what you are writing. Clever arguments? A nice flow? Humor? Why would I concern myself with things that make reading interesting, if nobody is ever going to read it. Who cares if its dull, just get the punctuation right! Nevermind that it doesn't make much sense, just get a dictionary and show the computer your rich vocabulary...
Analogies don't equal equalities, they are merely somewhat analogous.