Slashdot Mirror

← Back to Stories (view on slashdot.org)

Indiana First With Computerized Grading

Posted by michael on from the replaced-by-a-small-shell-script dept.

Mz6 writes "Computerized grading has been talked about previously, however, the New York Times reports that Indiana has become the first state to grade high school English essays by computer. The computerized grading process, called 'e-rater', uses a 6-point rating scale and uses artificial intelligence to 'mimic the grading process of human readers'. The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. The big question is, will other states begin to emulate Indiana by tossing human grading?"

57 of 524 comments (clear)

  1. I smell lawsuits, how about you? by Allen+Zadr · · Score: 5, Insightful
    ... and produced results virtually identical to those of trained readers.

    Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."

    I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).

    --
    Kinetic stupidity has a new brand leader: Allen Zadr.
    1. Re:I smell lawsuits, how about you? by m0rph3us0 · · Score: 5, Funny

      I think you've taken one to many collage level english classes my friend.

    2. Re:I smell lawsuits, how about you? by m0rph3us0 · · Score: 5, Funny

      s/to/too/. I feel stupid now.

    3. Re:I smell lawsuits, how about you? by Shalda · · Score: 4, Interesting

      I disagree. I see lawsuits as no more likely. Furthermore, any process where you're subjectively evaluting something there has to be quality controls and an appeals process. My wife once held a part time job grading essay questions on a high school exit exam. Every few hours of grading exams, she would have to take and pass a "calibration battery" of 10 exams. Quality control is fundamental to the process.

      What I see as being problematic is kids learning to beat the system. Typically these systems are predicated on gramatical analysis (use of punctuation and sentence compeleteness) and evidence of citing the text the question is based off. I'd bet its a real easy system to beat.

      --
      bance.net
    4. Re:I smell lawsuits, how about you? by DrEldarion · · Score: 3, Insightful

      Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

      This is why essays SHOULD be subjectively graded instead of objectively graded. You need to take into account the writing abilities of the student and determine if it's a good or a bad paper based on what they're capable of in addition to technical aspects of the paper.

      Isn't this why there are remedial and accelerated english classes? To take into account the different levels of intelligence in students? If you took an 'A' paper from a remedial class, it's quite likely that it would be a 'D' in the accelerated class.

    5. Re:I smell lawsuits, how about you? by king-manic · · Score: 4, Insightful

      Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

      Why punish those with ability, why reward those who are not talented. Surely they won't be punished or rewarded in the same manner in real life. If the talented one was cruising and not expending any effort, thats his perogative. If he can be a productive member of society without effort thats fine, even if he has the potential to eb the next great mind, it's still his choice. The rewarding of those who work hard is important but if they work hard to only measure up to the minimium standard then they belong to that minimium standard regaurdless of how much effort they put in.

      --
      "There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
    6. Re:I smell lawsuits, how about you? by Anonymous+Custard · · Score: 3, Interesting

      The real answer is to adjust your teaching methods per student based on subjective analysis. A low objective mark on a paper or test would indicate that, ideally, the teacher needs to pay closer attention to the needs of that student, and teach him in a way he can learn. (VERY few teachers have the time and skill required for this, unfortunately).

      You can't grade subjectively because those grades will be compared objectively down the line. You can't say "this is pretty good for kevin, I'll give him an A", but then say "josh's paper is way better than kevin's paper, but josh is a bright kid, so I'm giving him a C". Kevin will think he's mastered the english language while Josh will go insane trying to achieve perfection.

      Grading, when used for anything other than helping the teacher learn about each students, just plains sucks, and is only used for competition.

      --
      $8.95/mo web hosting
    7. Re:I smell lawsuits, how about you? by clifyt · · Score: 5, Interesting

      Actually, I've been a part of writing software like this for their competitors and have worked with this software in the past as part of my duties as manager of development at the IUPUI Testing Center (thats Indiana University Purdue University Indianapolis). We've worked on this shit for about 10 years now.

      One of my tasks in the past was to push this type of software onto the local schools. We've used it for rating in class essays.

      The idea is that everyone knows that the only way you get better writing is to write more and get some feedback on it. It doesn't matter if you are an educator throwing the papers back at the student or a computer algorithm. It all forces the student to find the mistakes and not make them the next time.

      The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day. So this is where this type of software comes into play -- you assign 2x the work you can normally handle, and let the computers handle half of it. You don't tell the students which assignments will be computer rated. Thus the students grades got better. Not much better, but they were better than the students not using the system in the same types of classes.

      One of our smaller studies actually had us installing this software locally for instant feedback. It was a small percentage, but the students work was even better than before.

      Yeah, you 'steal a copy' if you can't seem to get one given to you, and run it through until it likes it. How is this cheating or anyway underhanded. One of the better and far more dedicated educators I know actually allows students to hand in papers and have them marked up as many times as they want until the paper is due. His students final works are generally light years ahead of other educators in his facility that don't have the dedication (and for $26k a year, do you really want to give up your nights and weekends???).

      Same thing here.

      Shit, even using the grammar checker under work will force you to learn to write better (up to a point). You learn what its looking for and you avoid it. I'm not a good speller and I know the spell checkers help me learn after I hit the same error over and over again.

      All these tools work for you in the learning process as long as you are willing to not just put this stuff on autopilot.

      As for the title of this thread -- Lawsuit? The only lawsuits will come from idiots. None of the high stakes testing does purely computer rating. They all put humans into the equation. You will most likely get better rating because instead of having three or four humans look at your paper for 30 seconds each before moving on, you will now have one that is able to devote some serious time to it. All these humans will still be working just as many hours as before, but studies have shown that the eyeballs on the paper are there longer with this type of software than without (sorry, but these studies belong to the bigger testing companies or I'd post links...I just get paid to crunch the data).

      Secondly, 5 years ago when I was working on this stuff full time, the software had a human agreeance of around 62% with a rater pool of 3 raters. Meaning that if you asked 3 people what they thought, took an average of this, and then asked a fourth, 62% of the time, the human agreed with the others. This was on a 12 point scale. The application, however, actually rated between 70 - 80% of the time depending on the model used.

      In both cases, the raters were all trained together with the same things to look for, and the models were designed around this rater pool -- in a sense trying to simply guess at what the others would pick. The computer:human agreeance was higher than the human:human agreeance.

      Back to the parent post, beating the system only means you beat learning to write.

      BTW -- My post is not indicative of my writing skills outside of a conversational and informal setting, sans spell checker and proof re

  2. OSS? by Karamchand · · Score: 3, Interesting

    Is this program available under an open source licence? It sounds really interesting!

    1. Re:OSS? by twistedcubic · · Score: 3, Insightful

      Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.

    2. Re:OSS? by Glonoinha · · Score: 4, Funny

      Slashdot First With Computerized Moderation.

      Now with Computerized Moderation the famous Slashdot message site can pre-emptively down-mod 'Redundant' posts long before they are actually 'Redundant.' The computerized modding process, called 'e-modder', uses a 6-point rating scale and uses artificial intelligence to 'mimic the modding process of human readers - including doing stupid shit like modding the first instance of a concept as Redundant'.

      --
      Glonoinha the MebiByte Slayer
    3. Re:OSS? by borkus · · Score: 4, Insightful

      Imagine high school students re-writing their essays ...
      Actually, anything that would encourage students to re-write their papers and improve their writing would be pretty amazing. Most students jot something down, run a spell checker and turn in their work. If they could pre-grade their work, they might be better motivated to put out more effort and improve their writing.

      Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.

    4. Re:OSS? by arose · · Score: 3, Insightful

      The thing is that a human can actually understand what you are writing. Clever arguments? A nice flow? Humor? Why would I concern myself with things that make reading interesting, if nobody is ever going to read it. Who cares if its dull, just get the punctuation right! Nevermind that it doesn't make much sense, just get a dictionary and show the computer your rich vocabulary...

      --
      Analogies don't equal equalities, they are merely somewhat analogous.
  3. I would have loved this is a kid by cheezus · · Score: 5, Interesting

    it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system.

    --
    /bin/fortune | slashdotsig.sh
    1. Re:I would have loved this is a kid by jeffy124 · · Score: 3, Interesting

      an opinion paper on why dyhydrogen monoxide must be banned would perhaps do the trick?

      --
      The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
    2. Re:I would have loved this is a kid by Kiriwas · · Score: 3, Informative

      In the florida school system this is a writting test called the Florida Writes. Its a standardized test, doesn't count for much, but we all had to take it. This is basically exactly what we had to do. Write the most horrible peice of trash you could think of, but make it adhere to a few preset guidelines and BAM instant grade. My first time taking it I wrote a rather good peice of work if I do say so myself. Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

    3. Re:I would have loved this is a kid by LnxAddct · · Score: 3, Informative

      Well I've read almost every post under this story and noone has mentioned how it actually works. From the e-rater site "E-rater learns to score essays on a particular topic by processing a significant number of essays on the topic, each of which has been scored by two or more faculty readers. While e-rater is a powerful scoring engine, it is not meant to replace a teacher whose judgment is essential to helping students improve their writing ability." This means that its essentialy a bayesian filter that instead of being fed spam and ham and told which is right from wrong, it is fed good papers and bad papers and told which is right from wrong. You could literally reproduce this with something like SpamBayes but instead of feeding it spam, feed it essays. Anyway, you can probably beat the system but the fact that it is based on preprocessed papers means that a) it will be much harder to game the system without knowing what papers were scanned, and b) all future grading is based solely on the fact that previous generations have written papers and the futre grades will be based on them. The good thing is that unless the teachers are willing to write 20 different papers all on the same subject and scan them in, then the computer won't expect anything better then what was previosuly written. This also means, assuming the teacher won't write essays but rather will use ones from students from prior years, that the essay topics can't be changed, otherwise the teacher would have nothing to scan from prior years and thus nothing to base the grades on. As a result of the essay topic not being able to be changed that simply means that you can find someone from a year above you and ask them to save their papers for you, or in a worst case scenario, everyone must be given the same topic so at least you can help each other out. Often times teachers would give everyone a different topic to avoid people "helping" each other, but with such a system that would be impossible. I sure wish I was back in highschool.
      Regards,
      Steve

  4. I already want a copy of this. by ShitPissFuckCuntTits · · Score: 5, Insightful

    I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?

    --

    --
    My username: hats off to George Carlin, and fuck the FCC. Freedom!
    1. Re:I already want a copy of this. by Kozar_The_Malignant · · Score: 4, Funny

      > I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's.

      I'll bet half the people here thought this as soon as they read the headline. The normal /. response is to spend 10 hours at the computer coding up something to do a task you can do in 1 hour with a pencil. :-) Of course the high school geek who does this will make piles of money selling the output to the football players, buy a nice ride, and get a prom date with a cheerleader. Isn't that the way it works?
      --
      Some mornings it's hardly worth chewing through the restraints to get out of bed.
    2. Re:I already want a copy of this. by Glonoinha · · Score: 3, Funny

      Bah, right now the program takes an essay as input and outputs a grade. Simply reverse the pipe streams, push in an A and have the programs spit out a essay worthy of that A. Magic!

      --
      Glonoinha the MebiByte Slayer
  5. Stupid by phreak0003 · · Score: 4, Interesting

    I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

    1. Re:Stupid by Anonymous Coward · · Score: 3, Funny

      Maybe you got docked for overuse of commas.

    2. Re:Stupid by servognome · · Score: 4, Funny

      Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.
      Your feedback was actual words!? In my day all we got was a scratch-n-sniff sticker and had to guess the meaning of getting a "watermelon" on our essay.

      --
      D6 63 0D 70 89 81 BB 8E 7B 7C 5F 5D 54 EA AB 73
    3. Re:Stupid by I_Love_Pocky! · · Score: 3, Insightful

      The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.

      Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?

      They are not graded fairly, and they determine 10% of the final grade.

      10% of the grade on the essay? Or in a particular class? 10% on the essay may actually be tolerable, because that means that at least a human actually read it to give the other 90% of the grade.

    4. Re:Stupid by rsadelle · · Score: 3, Interesting

      I certainly hope your essay was better written than this comment. If it wasn't, and you still got 5/6, that's pretty good evidence that the computer isn't doing accurate grading. IANAET (I am not an English Teacher), but I would grade this comment 3/6. Some comments:

      Do you have any evidence or thought behind your statement that "A computer can obviously not grade essays fairly, so it shouldn't be done"? Why is that obvious? Is it obvious only because the grading of your essay was, in your opinion, not fair?

      I suggest you review difference in usage between "good" and "well." Proper grammar/word choice is a large part of what makes a good essay.

      How is the conversion from the computer's 6 point scale to your teacher's/school's 100 point scale the fault of the computer? It appears that what you really mean is that the implementation of the computer grading system is not fair as it doesn't use an appropriate scale.

  6. comment moderation by Jotaigna · · Score: 4, Funny

    for the time being, i would trust more that program to moderate my comments.



    c'mon people i was only joking dont mod me down, not noooo!!

    --
    "The quality of life is inversely proportional to the number of keys on your keyring."
  7. What about tricking the software? by gtaluvit · · Score: 5, Interesting

    SPAM filters are tricked all the time depending on the text of an email. Google was f'd up not too long ago because of trackback linking in blogs screwing up their algorithms. Isn't this a similar situation? If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

    --
    - gtaluvit (prnc. GOT-tuh-LUV-it)
  8. So much for those essays by Zancarius · · Score: 5, Funny

    Perish the thought should students start writing about the dangers of artificial intelligence. They may very well fail!

    --
    He who has no .plan has small finger. ~ Confucius on UNIX
  9. Gaming the system by ePhil_One · · Score: 5, Insightful
    While it gives identical results for now, I can easily see the coming books: How to Write an A essay! Form essays to get you into Harvard.

    The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...

    --
    You are in a maze of twisted little posts, all alike.
  10. Not the First by dcocos · · Score: 4, Interesting

    My alma matter graded most of my computer programs with shell scripts and I graduated in 1997. So I don't think India is the first to do that.

  11. Those Indianans are ruining us! by L.+VeGas · · Score: 5, Funny

    Lets just outsource all our test grading to Indiana too.

    --
    Best Windows Freeware
  12. In fact, by ShitPissFuckCuntTits · · Score: 3, Interesting

    As a student the first thing I would hand in is twenty paragraphs from refreshing this. In political science class, of course.

    --

    --
    My username: hats off to George Carlin, and fuck the FCC. Freedom!
  13. Trained readers... by ari_j · · Score: 3, Insightful

    The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers

    I think this says more about the training that the "trained readers" are receiving than it does about the software.

  14. Perfect scores every time by Rorschach1 · · Score: 5, Funny

    Ok, since you know the grading software is going to make it into the hands of the students, here's my scheme for perfect essays:

    Step 1: Feed some encyclopedia articles, Wiki pages, and other random material on your subject into a Markoff chain generator.

    Step 2: Use a genetic algorithm to generate variations of the text. Fitness is determined by the grade calculated.

    Step 3: Repeat step 2 until desired grade is achieved. (And, of course, Profit!)

    The result is totally worthless, but at first glance would probably appear legitimate even to a human reader.

    Sort of like Slashdot posts.

  15. AI by Bugmaster · · Score: 5, Funny

    That's actually a pretty novel way to approach the problem of creating Strong AI. Making smarter machines is hard, so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test. Damn, this is so sad.

    --
    >|<*:=
  16. identical results to those of trained readers... by GGardner · · Score: 3, Interesting

    Maybe these says more about the readers than the computer program?

  17. This is both good and bad by n1ywb · · Score: 3, Insightful

    Good: The computer probably won't grade you down for writing an anti-Bush essay, and it probably won't get fired for it. Good: Computers won't play favorites, and you can't kiss up to a computer. Bad: The computer really can't grade you up for expressing original ideas. Bad: It's probably possible to fool the computer somehow.

    --
    -73, de n1ywb
    www.n1ywb.com
  18. I can't wait by proverbialcow · · Score: 4, Funny

    ...until some wiseass figures out a way spoof the grader, probably by sliding under the radar of whatever probabilistic models they've got that pass for spell- and grammar-checkers.

    For example:

    Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

    Grade: B+ Your use of double-negatives continues to haunt you, but I'm glad you've gotten over hanging participles.

    --
    The only surefire protection against Microsoft infections is abstinence. - The Onion
  19. At least the parent proves something... by Ieshan · · Score: 4, Insightful

    At least the parent post proves one simple truth: human english teachers can be replaced by simple shell scripts.

  20. In Other News by ThisIsFred · · Score: 5, Interesting

    Indiana parents are the first to buy (en masse) licenses for Essay Constructor Pro v2.0. The software produces essays that are indistinguishable from those written by real students, using the latest screen-scrape-from-Internet 'n' plagiarism-from-non-credible-sources techniques.

    Indiana Director of State Board of Ed comments: "Isn't it wonderful how technology is improving education?"

    --
    Fred

    "A fool and his freedom are soon parted"
    -RMS
  21. This says more about "trained readers"... by Phurd+Phlegm · · Score: 4, Insightful
    than it does about the software, methinks. I imagine it gives helpful hints like the ones I always turn off in Microsoft Turd. Any construction that deviates from the norm in a boring business document apparently triggers the "grammotron" or whatever they call it. A human reader has some appreciation for style and may actually accept something a little different for the sake of variety and sparkle.

    Not that there's anything in this post that serves as an example. I guess that's because I was graded by humans. Seriously, I don't recall getting any encouragement in writing back in the '70s in high school, and not much in college. I guess it wouldn't have been any worse if the Grade-O-Vac was inspecting my papers instead of my mostly-marginally-literate teachers. There were several exceptions, but they focused much more on reading than on writing. I suspect they had a lot greater effect that way--I know they had a great effect on me.

  22. Can students get this program? by Theovon · · Score: 4, Insightful

    If I were a student, I'd want to get a copy of this software and use it to pre-grade my papers so that I could find out what's wrong and fix it before I turned it in.

  23. India vs. Indiana by the+eric+conspiracy · · Score: 4, Funny

    Wouldn't it just be cheaper to grade the tests at call centers in India? What are those Indians doing when there are no incomming calls? Just slacking off??? They could be grading tests.

  24. Standard Grading by Tony · · Score: 3, Funny

    The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

    So it gives its favorite students 'A's without reading, least favorite students 'F's, and the rest arbitrary grades somewhere in between to mimic a bell curve?

    Excellent!

    "Artificial Intelligence is easy. It's artificial stupidity that impresses me." -- Arthur Oscar

    --
    Microsoft is to software what Budweiser is to beer.
  25. Antidisestablishmentarianism! by Bluesman · · Score: 4, Funny

    By creating a vernacular consisting of elongated words and sophisticated verbiage, obviously indifferent to definition but simultaneously observing grammar regulations while eschewing colloquialisms, perhaps students may increase individual chances of achieving substantial academic acclaim.

    If this works anything like the writing level indexes you find on word processors, it should be easy to fool.

    --
    If moderation could change anything, it would be illegal.
  26. Too Uniform by Roger_Wilco · · Score: 3, Insightful

    My writing style is somewhat peculiar, though I can't exactly say how (or even approximately how). Partially as a result of this, my marks in English class over the years of high school ranged from C to A, depending not on me, but on who the teacher was. If the teacher happened to like my style, I got a good mark.

    This is annoying, but at least each year there was a different teacher, who may like my style. If the marking is computerised, it will not change; if your writing doesn't fit what the computer likes, you're screwed; likewise, if it does like it, you might never learn to express yourself more creatively (ie you'll be punished for trying to write in a manner different from what you usually do).

    There are possibilities in this technology, but I suspect that it will be a long while before the eccentric aren't labeled as poor writers.

  27. Babelfish by Waffle+Iron · · Score: 3, Funny
    I will not trust computer grading until I see a computer-translated document that isn't laughable.

    To illustrate my point, I'll restate it. [English -> German -> English]:

    I do not trust the computer, which arranges, until I see a computer-translated document of this laughable isn't.

    That's about how well a computer "comprehends" language today.

  28. Whoever approved this should be fired by Retired+Replicant · · Score: 3, Insightful

    A computer can check spelling and even grammar to a certain extent. However, it cannot evaluate factual accuracy, strength of argument. Even with spelling, the computer is not likely to catch improper use of homonyms. I can guarantee you that it will be possible to create a piece of writing that is utter crap that would get an A+ using this or any other possible computerized grading system. Unfortunately, there are probably many teachers out there who make poorer graders than this system does. The answer to the problem of poor-quality teaching is not replacing teachers with computers; the answer is a combination of better teacher pay and putting higher standards in place for our teachers via competency testing.

  29. Essay grading is harder than science grading by sustik · · Score: 3, Interesting

    I believe that (English) essay grading is harder than grading science exams based on problem solving (no bubbles please), at least if essays are about content and not just grammatical correct sentences.

    I say this because there is an objective criteria for grading the solution to a physics or math problem: correctness. For essays I do not beleive that we (and the current state of AI) can come up with an exact criteria like that. You might determine whether an essay is too different from essays which were written by experts, but cannot a very different essay to be just as good?

    To my knowledge the AI programs can solve physics problems which are limited to some well defined domain (for example: http://www.cs.utexas.edu/users/novak/cgi/isaacdemo .cgi though that is from 1977...) I am not aware of programs grading physics problem solutions.

    I will accept an essay grading program after they grade solutions to math and physics problems.

    I conjecture that some writers would feel offended if their essay did well according to the program: they might think it means they are too conformist and conservative and not novel in their approach...

    Matyas

  30. From the web site. by Bluesman · · Score: 3, Informative

    >If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site.

    Submit "and" essay? I guess they haven't run the software on themselves.

    F.

    --
    If moderation could change anything, it would be illegal.
  31. the triumph of mediocrity by KMonk · · Score: 5, Insightful

    Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.

  32. As someone procrastinating grading right now... by Dr.+Spork · · Score: 4, Informative
    I don't think that automatized, high-school-level grading is an all-bad thing. We can call it unfair if we like, but as someone who grades a lot of stuff I can tell you that I'm nothing like fair. I don't always even know how to distinguish a B- from a C+, and I just go with my gut, which, as far as I can tell, is much like flipping an internal coin. If we looked at human grade assigners as an algorithm, we would find a whole lot of stuff wrong, even among those of us who try hard to be fair (cover author names, compare close grades for consistency, keep a constant mood, that sort of stuff).

    But I think that if a computer grading program which is no worse than humans could be devised, it would be a great learning tool. A lot of people make it to college as borderline illiterates. I'm not kidding. I read a lot of their crap. That's because their HS teachers were too overworked to grade their writing, so they didn't assign much. If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

    Sure, the first grading applications are going to make a few serious errors. This is the first stage of every application when a computer is asked to interpret rich data. Early voice recognition sucked. Now it sucks much less, and it will just keep getting better. Same with OCR, chess software, machine translation, etc. So the right debate to have is about when this will be good enough for school use, and not whether. I'm prepared to admit that the answer to the right question is "not yet" (I'm sure how deep the current problems go), but I fully support working on this system until it works right.

    1. Re:As someone procrastinating grading right now... by Idarubicin · · Score: 3, Insightful
      If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

      Unfortunately, the system described here doesn't return any such useful feedback. The Indiana system returns a grade from a six point scale. No comments, no criticism, no hint that the evaluation is meaningful.

      Incidentally, what's this about "teaching instead of tedium"? Grading essays by evaluating construction, insight, and creativity should be part of the teaching process. Perhaps this is something that should be addressed earlier in the education of these students - if they're reaching college as "borderline illiterates" there is a problem - but grading in general is a part of teaching. If I were a student, I'd want to know that a human being - at some point - had bothered to look at the work that I did.

      --
      ~Idarubicin
  33. I took this test by tundog · · Score: 5, Informative

    I live in Indiana (no, NOT India) and took this test. Being a techie, I figured I'd try to fake out the system. This test works out to be 10% of the final grade and since I had a 98 going into the test, I figured I could afford to gamble a little, figuring if it back-fired I could blame it on a computer error since every one would figure the kid with a 98 MUST be telling the truth.

    I almost wimped out. I wrote about 80 percent of the essay (about influence of pop-culture on society - and silly me I always thought society influences pop-culture but anyway). I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion. I reoreded the paragraphs, copied the one I felt was the best written and pasted it into the body 3 times.

    Guess what I got.....6/6 (six point grading scale which is pretty messed up because a 5/6 is an 83%). Hopefully they won't audit mine....

    --
    All your base are belong to us!
  34. How To Write An Essay by dpbsmith · · Score: 5, Insightful

    A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.

    It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.

    The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.

    The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.

    In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.

    --

    "How to Do Nothing," kids activities, back in print!

  35. Depends on the purpose of gtrading... by Dr.+Zowie · · Score: 4, Insightful
    If grading is intended as a motivator to encourage each student to perform his/her best, then more effort should yield a higher grade. Likewise, if grading is intended to reflect the student's ability to perform in a real-world situation, effort should probably yield a higher grade: folks who work hard tend to do better than folks who are marginally smarter but don't work hard, in real-life situations. But if grading is intended to reflect only the quality of the work that was submitted, then sure -- effort shouldn't count at all.


    This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.


    As a college instructor, I tend to use a strict grading protocol -- and then "bump up" a few of the students. If someone comes in to my office every week and really struggles to understand the concepts, but the computer tells me that they earned a "C+" -- they're likely to find a "B-" on their transcript. But if someone who's smart enough to get an "A" blows an exam from being hung over, that person gets little or no sympathy.

    1. Re:Depends on the purpose of gtrading... by hesiod · · Score: 4, Insightful

      > If grading is intended as a motivator

      A grade is used to show how well you know a subject. If I knew a subject completely before starting the class & wrote the same level of paper as someone who studied his ass off, we deserve the same grade. He should not be given a better grade than me just because he waited for this particular class to learn a subject. Why am I judged differently because I took the initiative to learn the information earlier than I absolutely had to?

      No, schools are not there to make you feel good about yourself (that's obvious), they are there to make sure you know the minimum information to pass a class. That'a a D. Then, if you know more, you get a better grade. The amount of work you put into it is irrelevant. In fact, if you put more work into it than you should have, it means you are not doing well, and once you get into the "real world," where you have strict deadlines, you don't have the option of getting paid more just because you worked harder for the same result. The exact OPPOSITE is true, in fact.

      School should help people prepare for life. If someone is given a grade they did not deserve, they are being improperly trained how to work.

      A problem, however, is with the PARENTS. Many students are C students, that's all there is to it. But they get all high & mighty towards the school if they see their child work very hard for a low grade. They figure their child isn't good enough if they don't have all As, but that they deserve them just for hard work.

      If life had a payrate based on how hard you worked, vs. your productivity, I would start working as an astrophysicist. I wouldn't get anything done, since I know nothing about the work, but I would sure as hell work my ass off. Do you think anyone wants an employee like that?
      (I mean the lack of knowledge -- everyone wants a hard worker, if they know enough).