Slashdot Mirror


Computer Program Makes Essay Grading Easier

phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"

39 of 666 comments (clear)

  1. Cheating by fembots · · Score: 4, Insightful

    How long until some students get hold of this program and tweak their essay until it's puifect? It's similar to spammers using spam filters to test their emails first.

    1. Re:Cheating by Anonymous Coward · · Score: 5, Insightful

      If you'd read the fucking article, you'd have known that the students get instact feedback on their scores, so it effectively is a filter for better papers.

    2. Re:Cheating by RazorX90 · · Score: 5, Insightful

      God, imagine what test prep courses like Princeton Review are going to do. Essay writing for the SAT will turn into a new branch of science. They'll teach you exactly what your ratio of compound sentences, to complex sentences, to clauses, to action verbs, etc. should be.

    3. Re:Cheating by subrosas · · Score: 5, Informative

      Qualrus doesn't operate using a set grading criteria, but trains based on the users' grading markups. Therefore, you'd need the teacher's copy (complete with its "learned" patterns) to fool the system. Actually Ed Brent encourages the students to use Qualrus to write rough drafts, as it gives instant feedback - arguably a better learning technique from a usability standpoint (faster feedback == more retention).

    4. Re:Cheating by LuYu · · Score: 4, Insightful

      The real question in this scenario is whether or not they will learn enough by cheating to have gained something valuable.

      If they have to write a program to beat the teacher's program, are the students not learning something very valuable (at least in the marketable business skills department)?

      Also, in order to write a program that creates essays that conform to the teachers program, will it not also be necessary to learn the grammar and logic rules the teacher considers to be important and even ponder those rules for extended periods of time?

      It seems to me that the cheater (or at least the first cheater) will do more work than the professor did and thereby become quite familiar with English grammar, organization of arguments, and computer programming. All of these are useful skills.

      --
      All data is speech. All speech is Free.
    5. Re:Cheating by mzieg · · Score: 4, Insightful
      It's no more cheating than spell-checkers and calculators.

      Is it cheating to run your HTML/XML through a validator?
      Is it cheating to test-compile your scripts with "perl -c"?
      Is it cheating to run a hardware diagnostic to check for faults?

      Humans are tool-users, technologists doubly so. This is how civilization advances: by developing processes to eliminate typical sources of error, allowing man to apply his thinking mind to higher-level problems.

    6. Re:Cheating by Lonewolf666 · · Score: 4, Interesting

      Heh. If so, it might be an interesting challenge to run a genetic algorithm against the program, trying to "evolve" the paper into a perfect match for Qualrus' preferences.

      If done successfully, you might get a good grade in sociology for something that would deserve a good grade in computer science ;-)

      --
      C - the footgun of programming languages
  2. Hmmm by Anonymous Coward · · Score: 5, Insightful

    I'm not so sure I like the idea of a computer grading my work.. I spent hours making it, but the guy doesn't even give it the time of day....
    Angst

    1. Re:Hmmm by CSMastermind · · Score: 4, Insightful

      What makes you think that your teachers read the papers to began with. At least smaller papers, I know that all they do is skim for keywords.

    2. Re:Hmmm by MoralHazard · · Score: 4, Insightful

      Hey, if it was an undergrad math class, like Calculus 1, how would you feel about automated grading? It's possible, trivial even, to write a program and have a way for students to solve problems (showing their work, of course!) in a machine-readable format. Then you can have a computer grade the homework and report a summary to the TA, who looks at what students are missing and deals with it. This isn't that different from the way things currently operate, because the TA manually grading homework is just using a mechanical process to check the students' work, anyway.

      What's the point of the class? In Calculus 1, the point is to learn concepts and methods that allow you to perform basic operations, as proven by your ability to work out problems on homework and tests. You're not asked to be creative or anything--that comes later, in 300 or 400 level classes or graduate work. First, you have to learn the basics.

      I imagine sociology isn't that much different--at least, it wasn't in Poli Sci when I was in college. First, you have to learn a bunch of basic facts and rules and concepts, and demonstrate that you have a know them. You should be able to talk about them, define them, and answer questions about them. Anybody who's being creative in a freshman sociology class is ahead of the game.

      And don't give me no shit about "I spent hours making it, you should spend hours reading it". That's like the .sig that says "I don't use comments--if it was hard to code, it should be hard to read." The fact that an 18-year old punk takes hours to craft an essay (that the professor could do in his sleep) doesn't make the effort more valuable.

      I mean, shit--it took me DAYS to write my first couple of C programs in CS 101. Does that mean that the professor is shorting my education if he takes 10 seconds to grade it?

  3. Structure by CHESTER+COPPERPOT · · Score: 5, Insightful

    "The program analyzes sentence and paragraph structure and can ascertain the flow of arguments and ideas."

    So it measures structure and argument.

    How's it going to measure creativity of thought? Are we going to just pump out logic machines from colleges?

    1. Re:Structure by JanneM · · Score: 4, Insightful

      If you believe teachers - even good, concientious ones - really delve into your work at more than an utterly superficial level, you are mistaken. No matter how dedicated you are, if you have fifty papers to grade, and five days to do so, you will be superficial about it. There is simply no time for anything else.

      Perversely, the worse a paper is, the more time it receives; it's more important, and more difficult, to motivate a failing grade than a good one. Also, to some extent all good papers (or assignments) are alike and can be spotted fairly easily; it's the bad ones that are (regrettably) unique and need individual attention.

      --
      Trust the Computer. The Computer is your friend.
    2. Re:Structure by Pxtl · · Score: 5, Insightful

      Given the abysmal failure of most people to demonstrate any mental flow of logic and ideas, but who have tons of very creative thoughts about reality, I think I'd appreciate more people like that. But maybe I follow too much politics.

    3. Re:Structure by El+Micko · · Score: 4, Insightful
      If his essay marking program is the real deal, then open it up for scrutiny. Allow "experts" to see what he's doing, under an NDA, of course... Lets put his reputation on the line as opposed to the education of the students he's neglecting.

      He's a sociology professor, and he's managed to write a natural language parser that can actually decipher meaning, and mark the relevance of the content of the essay against the question.
      <cough>BULLSHIT!</cough>
  4. Not the world's best plan by SparksMcGee · · Score: 5, Insightful

    The only way that a computer program can possibly analyze a paper fully as a paper is if it read it as a human. heuristic algorithms, however sophisticated, just aren't enough for things of this sort of importance--after all, the profs are paid to grade (well, them or the grad students). It doesn't seem like it would be too difficult to just throw in keywords and make sure that you use proper syntax in order to fool this thing, albeit the prof says keywords alone aren't enough. I find the claim that his program can "analyze argument flow" quite dubious. I'll stick to getting my several grands' worth out of my courses, thank you very much.

    1. Re:Not the world's best plan by MoralHazard · · Score: 4, Insightful

      This is a common misconception about algorithms. Effective methods don't have be consciously engineered in every step by a human designer, even when applied to extremely complex questions that traditionally require human judgement.

      It's possible to train computer programs to translate text between languages by feeding examples of good and bad translations to pattern-recognition algorithms, which start with simple rules. Most of these models are similar to neural-net machines, which is in turn based on the fundamental theory of how animal brains (including human brains) operate. You don't design and code an algorithm, you train the machine by example, with some human-assisted trail-and-error.

      This often works because that's how human judgement works: we learn just about everything by example and trial-and-error, and we're VERY good at it (look at what millions of years of evolution can accomplish!). This isn't to say that a trained neural net machine is "intelligent" or "conscious", just that solves problems by the same mechanism that a human brain does, albeit in a much more limited fashion.

      Of course, the effectiveness of a trained machine is limited by how big a computer you have, and how well you train it. Re-creating the complexity of the human brain in software with present-day techniques and equipment would be impossible (neural net software is VERY memory intensive when it gets complex). This may change in the future, but that's another debate that I won't get into.

      I'm not saying that this professor's software actually works or not--he could easily be full of shit. I'm also not saying that you can't game one of these machines the same way spammers game Bayesian anti-spam filters: use trial-and-error to figure out how to trick the machine consistently.

      In fact, I'm assuming that a canny student could steal the software and do exactly that. After all, the human brain is a much more powerful learning machine than the program, and could probably outsmart it in the same way that people can outsmart rats.

      But then again, this is a socialogy course, so his students probably won't think of it on their own.

  5. Intresting by Patrik_AKA_RedX · · Score: 4, Insightful

    Now if we could code a program to write those essays, the whole process would be automated. Press enter for another A. If you know how the program grades the essays, it should be possible to write a program that generates essays that comply perfectly to those rules, right?

    1. Re:Intresting by superpulpsicle · · Score: 4, Insightful

      Why don't they just make a program to print out a bachelors degree. Save me time, money and effort.

  6. And talent may remain unfound by cbreaker · · Score: 4, Insightful

    Sure, you could be great at grammar and sentence structure. You could be an ace at using proper english.

    But how would hidden talent and creativity be found? How will the teacher know if his students are actually trying hard to write their papers when all he does is check the thing with a computer program?

    It's a really terrible idea and I think it's really cheezy. Ohh, he saved some time. So does that mean he now gets paid less? Does this automation get the students a discount? Yea, right.

    If I'm going to put a lot of work into writing an interesting paper about something, I want someone to read it.

    --
    - It's not the Macs I hate. It's Digg users. -
  7. More importantly... by Chris+Pimlott · · Score: 5, Insightful

    What does this say about the field of Sociology? :P

  8. Cheating? Teaching! by ImaLamer · · Score: 5, Insightful

    Why not give the program to the English department and use it for teaching?

    Would be great for high-school students. Have students write an essay or paper and analyze it right in front of them. Then the program highlights their errors (or what the program perceives as an error). Even better, complaining students would help fix bugs in the software because they know their intent - they could send off a highlighted error-ridden version to the developers with an explanation of why they think they are right.

    Better yet, give it to everyone! It's not like you can cheat, you still have to rewrite and resubmit your papers. Shit, I say build it into text boxes on slashdot and wikipedia to start!

    please do not hold this post to the standard of the Qualrus (real page of the software)

    1. Re:Cheating? Teaching! by jonadab · · Score: 4, Insightful

      > Why not give the program to the English department and use it for teaching?

      That would be counterproductive. If the program actually works with even 70% reliability, I'll eat my hat. In other words, I guarantee it's worse than the average student. Natural language processing is AI-complete. Every six months somebody claims to have solved the problem, and it always turns out to be another Eliza ("Did you come to me because the fact that question that concerns you is the real reason?") or babelfish ("To celebrate the score and seven years, our suffered ancestors brought ahead on this continent a new nation, taken in freedom and devoted to the proposal which all gecreeerde people are equal") or, frequently, even worse.

      "I have a computer program that understands English sentences" is roughly the same as "I have some really great real estate a quarter-mile north of downtown Chicago that will fetch a fortune on the market, but because I'm in a hurry I'll let you have it for half price."

      --
      Cut that out, or I will ship you to Norilsk in a box.
  9. GMAT exam has been doing this for awhile now... by Golgafrinchan · · Score: 5, Informative
    The company who administers the GMAT (Graduate Management Admission Test) has been using a computer grader for the analytical writing portion of the exam for several years now. They call it the e-rater. Both a human and the e-rater grade every essay.

    According to ETS, the e-rater agrees with the human grader 98% of the time.

    --
    My userid is prime!
  10. The program DOES NOT grade the essays! by rainwalker · · Score: 5, Informative

    From TFA, which apparently no one has read yet:
    "The final papers, which he does read, are usually much better as a result of Qualrus, too."

    There you go! For the reading and comprehension impaired, here's a summary of what's actually happening, which even the reporter didn't get:
    1. Students write a draft of their essay, which they then upload via a Web form to this program
    2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.
    3. Students improve the pieces of their essay that the program suggests.
    4. Students submit the final draft to the professor, who reads and grades each one by hand. Due to steps 1-3, the quality of the final draft is much higher.

    This sounds like a great thing to me. Wish I had something similar for my students. I don't have the time to read through dozens of drafts for every student. Too bad I'm not in sociology.

    1. Re:The program DOES NOT grade the essays! by VeryProfessional · · Score: 5, Informative

      From TFA:

      The computer-generated scores count for about a third to a quarter of students' final grade for Brent's class.

      There you go! Make sure you RTFA very carefully before accusing others of being reading and comprehension impaired.

  11. Re:Wouldn't want to be in his class. by samtihen · · Score: 5, Interesting

    I go to the University of Missouri. I wouldn't want to be in his class either. Hmm, I don't have enough of the keywords in my paper? How about you actually read the paper you made me write?

    http://sociology.missouri.edu/Faculty_and_Staff/Fa culty/Edward_Brent.html

  12. Save $$$ on tuition now? by d474 · · Score: 5, Funny

    So since the professor's time is worth about $36.00/hour and he spends 200 less hours on reading papers...

    200 hours * $32.00 = $7200

    He teaches about 84 students...

    $7200 / 84 = $85.71 refund for each student. It's party time!

    --
    Authority questions you. Return the favor.
  13. Use it for Moderating SLASHDOT? by d474 · · Score: 4, Funny

    Okay Slashdot Editors, time to fork out the $$$ to get some auto-moderating going on in these threads! Wait, can this grading program test for humor? No? Fuck it then.

    --
    Authority questions you. Return the favor.
  14. Fire the professor... by John+Seminal · · Score: 4, Interesting
    This reminds me of something that was in the papers a few weeks ago. A professor and graduate students wanted to show that most journals will publish anything if it sounds "academic" enough. So they wrote a paper that was hog-wash, made no point, was just a bunch of academic sounding prose. And guess what? They got published.

    If a professor does not care enough to read my papers, then to hell with him. There is more that a professor does than just check grammer, or look for passages that deals with the question and used terms from the book. The best professors I had were the ones who wrote all over the margins, sharing their thoughts about my ideas. Those are the ones who I would meet in their office to chat with. They are the ones who I went to for advice.

    I had one teacher in english who graded the first paper, reading them all. She then never read another paper, only skimmed them. She pretty much gave out the same grade on all your papers you got on your first paper. I got an "A" on my paper, and another student got a "D". So I was working with the "D" student, and no matter what was done, the "D" grades went up to "C-" but stuck. So for the last paper, we switched our papers. Guess what? My paper was still an "A" even though it belonged to the other student, and the other paper was a "C". We went to the teacher to explain what we did, and rather than the professor owning up to what was done, we the students got blamed.

    This really pisses me off. Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union. Then they want to slack off. Fucking asshats. Something like this makes me want to vote to remove public funding from schools, to always vote no whenever there is a refferendum to increase property tax. With those kinds of professors, people might as well get their education at the public library.

    --

    Rosco: "If brains were gunpowder, Enos couldn't blow his nose."

    1. Re:Fire the professor... by mbrother · · Score: 4, Insightful

      I'd be pissed off, too, but don't let one bad professor sour you on all of us. My base pay is less than $60k a year, but with summer salary from grants I'm in your range. And I work way more than 20 hours a week. Way, way more. But what is this "union" you speak of? I know of no such union (although tenure can provide significant job security).

      I used to give short answer/essay questions to my astronomy students the first couple of semesters I taught the big non-major course. It took a tremendous amount of time to grade which was one reason I stopped, but not the primary reason. I'm a novelist, and I know how to write, and there was a consistently high fraction of exams written so badly it was very painful to read. Perhaps I should have kept at it, with the idea that it's good for the students. But a few essays in a science class won't dent the problem that starts in k-12 education.

      --
      Professor of Astronomy, Author of Spider Star & Star Dragon (Tor)
    2. Re:Fire the professor... by stephenbooth · · Score: 4, Insightful

      One of my teachers at school tried that. Unfortunately what happened is that in each group the rest of the members (the groups were arranged so each group was mixed ability) ganged up on the 'smart but weak/shy' one to do all the work then goofed off.

      Actually, now I come to think about it, that's exactly what happens in the workplace.

      Stephen

      --
      "Don't write down to your readers, the only people less intelligent than you can't read" - Sign on Newspaper Office Wall
  15. Calculus by John+Seminal · · Score: 4, Funny

    I can say this much, I have never had any other professor, outside of the Chem or Physics department, grade my papers like a math professor. Most of the humanities professors just skim over. But in my Calculus class, it was possible to turn in homework and get negative points. For example, you have a problem 1.0 + 1.00 = ?. You write 2. First, half a point off for not figuring in significant digits. Another half a point off for sloppy handwriting. And the full point off for not showing your work. Problem worth one point, your score is negative one point. In some cases, it was better to not turn in anything at all.

    --

    Rosco: "If brains were gunpowder, Enos couldn't blow his nose."

  16. So what? by raehl · · Score: 5, Insightful

    I have lots of programs that save me hours of "doing my fucking job". They're called scripts, and it's called efficiency.

    If I can write a program to automate a menial task so I don't have to do it, then by all means, I should do it. If grading undergrad papers is a menial task that can be automated, then it should be automated.

    I mean, just because a freshman writes a bad paper doesn't mean a professor has to actually read it.

  17. term papers... by Cryptnotic · · Score: 4, Insightful

    Unless it's an English class, they're just looking to see if you've learned the material for that course. Especially for the essay questions in the midterm and final, they will always just skim the papers for key words, phrases, and dates. They underline them and add up the points. That's just the way it works. The "B" student, who writes something interesting, but doesn't cover all of the relevant material, loses points.

    For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.

    These are things that I wish someone had told me when I was an undergrad.

    --
    My other first post is car post.
    1. Re:term papers... by John+Seminal · · Score: 5, Interesting
      Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus

      I really, really, really wish someone told me this. I lost a letter grade in a class because I had a differing point of view from the professor. I wrote a kick ass term paper, I spent countless hours in the library doing research, I had other people proof read my paper. It was one of the best papers I wrote. But it was the exact opposite of what the professor believed.

      We expect our teachers will grade us on our work. But every now and then we get a professor who probably spends too much time writing letters to the editorial section of the new york times.

      --

      Rosco: "If brains were gunpowder, Enos couldn't blow his nose."

    2. Re:term papers... by Stephan+Schulz · · Score: 4, Insightful
      Differing opinions out to be REWARDED--they show students who have formed their own opinions (generally).
      Well-reasoned, informed opinions should be rewarded. You will not get any credits for arguing 2+2=5 out of me (at least not in a Math/Science class). A different opinion is not inherent more valuable than a concurring one.
      --

      Stephan

  18. "A well thought out Slashshdot Post" by Headcase88 · · Score: 4, Funny
    Some profs at least claim to be a little more discerning when a page is longer than recommended. Everyone probably has at least one prof with the story (whether fact or fiction) of a student that handed in 3 pages of worthwhile material with multiple page data from a semi-related source sandwiched in, and how perople like that get a lower mark. Maybe I should make a long post and see what happens.

    First, moving around quickly, and with purpose, is a true sign of character. Secondarily, bustle(e.g. hustle) yields more product for the working types. "Hustle and bustle are like my right and left arms," said Li'l Spicy in his famous "Hustle and Bustle Are Like My Right and Left Arms" speech. Webster's defines bustle as "excited and often noisy activity; a stir." A stir, indeed. Finally, sometimes gross stuff can be funny.

    Here are some links:
    It is now my intention to play video games for several hours.

    Sources:
    The Brothers Chaps (2004).Homestar Runner. Retrieved April 8, 2005 from www.homestarrunner.com

    Random Source (2005). that you won't read because you were too lazy. Retrieved April 8, 2005 from www.toreadthisfar.com

    (I have four words for this post: "Too much half-asleep effort")
    --
    "When the atomic bomb goes off there's devastation...but when the atomic bong goes off there's celebraaaaation!"
  19. It's about following instructions. by jbarr · · Score: 4, Interesting
    For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.
    You make some interesting points.

    When I was in college in the late '80's, the "trick" to getting good grades really was to understand what the professors were looking for, and give it to them.

    For example, I had a professor who distributed pre-printed pieces of paper that had a line drawn around it indicating the margin (something like two inches at the top, a half inch on the right and bottom, and three inches on the left. The large margins on the left and top were for the prof's notes and comments to us.) We had to type (with a typewriter) our papers to fit within the bounds of the margins, and spelling and grammar counted (and this was a Psychology class, not an English class.) He would not accept more than the one page. Another professor required papers be written in a specific topical order. If you deviated from those models, you got marked down.

    The point was not that the profs were trying to be devious, but to make us take into account the instructions they gave us. If we followed the instructions, we got better grades. If we didn't, we would get marked down. And yes, content did count too.

    Yes, it was a hassle, but the result is that now, when my bossed give me instructions, I follow them. The times when I deviate are the times when I really hear about it. Lesson learned!
    --
    My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
  20. Re:Better teaching tool than grading tool by mopslik · · Score: 4, Interesting

    the red squigle under the mispelled word

    Back when I was grading papers, I used to recommend the exact opposite to the students -- turn off the "instant" spell-checker, then run the "full" spell-checker and re-read the paper. I found that, in many cases, students would correct anything that had a red squiggle underneath it, but would get a false sense of security that all of the errors had been detected by the word processor.

    Oddly enough, when they had a squiggle-free page before them, they were much more attentive to detail and caught the "spelled-correctly-but-used-inappropriately" words.