Slashdot Mirror
Computer Program Makes Essay Grading Easier
phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"
How long until some students get hold of this program and tweak their essay until it's puifect? It's similar to spammers using spam filters to test their emails first.
Rock that crushes, Paper & Scissors that don't matter.
I'm not so sure I like the idea of a computer grading my work.. I spent hours making it, but the guy doesn't even give it the time of day....
Angst
"The program analyzes sentence and paragraph structure and can ascertain the flow of arguments and ideas."
So it measures structure and argument.
How's it going to measure creativity of thought? Are we going to just pump out logic machines from colleges?
The only way that a computer program can possibly analyze a paper fully as a paper is if it read it as a human. heuristic algorithms, however sophisticated, just aren't enough for things of this sort of importance--after all, the profs are paid to grade (well, them or the grad students). It doesn't seem like it would be too difficult to just throw in keywords and make sure that you use proper syntax in order to fool this thing, albeit the prof says keywords alone aren't enough. I find the claim that his program can "analyze argument flow" quite dubious. I'll stick to getting my several grands' worth out of my courses, thank you very much.
Now if we could code a program to write those essays, the whole process would be automated. Press enter for another A. If you know how the program grades the essays, it should be possible to write a program that generates essays that comply perfectly to those rules, right?
Undergrad professors are usually not too excited about teaching these 18 year-old pizza-faced dorks. The problem is that the kids would rather be out drinking and screwing rather than debating the intricacies of pre vs post agrarian culture in the Southern States and the relationship between that and race relations as they exist today.
So more power to him. He is unlikely to be getting anything better or more insightful than a parroting of what he has already delivered in his monologues to his class. Same papers, year in and year out. No big deal to grade these kids with an automated program.
Lazy bastard.
XML is like violence. If it doesn't solve the problem, use more.
If a program is really good enough to mark an essay, then reversing the function should allow it to create an essay that is perceived by human assessors to be of good quality. And I suspect we are a long was away from that.
Has this guy even assessed the correlation between the marks the program gives and the marks he would give?
Sure, you could be great at grammar and sentence structure. You could be an ace at using proper english.
But how would hidden talent and creativity be found? How will the teacher know if his students are actually trying hard to write their papers when all he does is check the thing with a computer program?
It's a really terrible idea and I think it's really cheezy. Ohh, he saved some time. So does that mean he now gets paid less? Does this automation get the students a discount? Yea, right.
If I'm going to put a lot of work into writing an interesting paper about something, I want someone to read it.
- It's not the Macs I hate. It's Digg users. -
What does this say about the field of Sociology? :P
FWIW, this prof is at the University of Missouri (in Columbia, MO).
Gee - you'd think the submitter could RTFA...
... but costs $400 a copy. Software developed with NSF (taxpayer) money should have to be open source!
Why not give the program to the English department and use it for teaching?
Would be great for high-school students. Have students write an essay or paper and analyze it right in front of them. Then the program highlights their errors (or what the program perceives as an error). Even better, complaining students would help fix bugs in the software because they know their intent - they could send off a highlighted error-ridden version to the developers with an explanation of why they think they are right.
Better yet, give it to everyone! It's not like you can cheat, you still have to rewrite and resubmit your papers. Shit, I say build it into text boxes on slashdot and wikipedia to start!
please do not hold this post to the standard of the Qualrus (real page of the software)
Get your Unix fortune now!
It works by scanning text for keywords, phrases and language patterns.
Which is to say, this prof is asking students to regurgitate data. Given, a certain level of base knowledge is necessary in any class and topic, and regurgitation (aka parroting) is an easy way to check that base knowledge. If a paper is assigned on a particular topic that they've been studying, then this sort of program can easily check for base level ability to spit back key words and phrases.
But, I seriously doubt that the class is ONLY about that base knowledge -- or that the program can reasonably check for anything more. I've had classes where the prof or graders did basically the same thing that this program does (i.e. check only for key words, phrases, and patterns they want to see), and I have little respect for those profs.
If you don't want to put even a basic amount of effort into checking a paper, don't assign it -- find some better way to check students' progress.
This one is just nuts. Why on earth am I writing essays which are going to be marked automatically by a machine? It's bad enough that scantron cards have found their way into subjects where they're totally irrelevant (a multiple-choice test for a university level Shakespeare course?), this is just another reason why post-secondary education has become increasingly less complete.
If he's allowed to use a machine to save him the effort of reading an essay, I should be able to use a machine so I don't have to go through the effort of writing one. Trust me, as arduous as it is to read a 20 page essay on the relative merits of liquid rubber concrete compound fasteners, writing it takes a lot more effort, a lot more time, and it damn well deserves to be read by the professor who assigned it.
It seems like, instead of creating computer programs to grade essays, perhaps the schools should, you know, hire more teachers or pay the ones they have better! Couldn't this guy just hire a human assistant? There are plenty of people out there with education degrees who would enjoy having his job.
What if he grant application was reviewed by a program instead of a real person? I wonder how he would like it
If this professor's analysis can be "simulated" by a computer program, then he was obviously not doing a thorough enough analysis to begin with. I know plenty of professors that would laugh at the idea that a computer program would be able to "calculate" emotion, nuance, subtle sarcasm, humor, insightfulness, etc...
This professor should be fired.
Authority questions you. Return the favor.
I have lots of programs that save me hours of "doing my fucking job". They're called scripts, and it's called efficiency.
If I can write a program to automate a menial task so I don't have to do it, then by all means, I should do it. If grading undergrad papers is a menial task that can be automated, then it should be automated.
I mean, just because a freshman writes a bad paper doesn't mean a professor has to actually read it.
paintball
Unless it's an English class, they're just looking to see if you've learned the material for that course. Especially for the essay questions in the midterm and final, they will always just skim the papers for key words, phrases, and dates. They underline them and add up the points. That's just the way it works. The "B" student, who writes something interesting, but doesn't cover all of the relevant material, loses points.
For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.
These are things that I wish someone had told me when I was an undergrad.
My other first post is car post.
Announce to the class that they have two options. If they want their paper to be computer-graded, the maximum they can get on the paper is a B. Only hand-graded papers can earn an A - but they will be judged on original thought, well-chosen sources, and total structure. If they don't meet those criteria, they will be penalized (and as such, would score lower than the computer would give). That way, students who aren't confident in their capability for truly excellent writing and novel thought can choose the computer grader for a safe B, while the truly thoughtful and creative students can be justly rewarded.
Wait - what the fuck am I saying? ALL college students should be graded by the non-computer criteria I just listed, and those who can't do (or at least attempt) that kind of work shouldn't be in college.
If you mod me down, I shall become more powerful than you can possibly imagine.
I'd be pissed off, too, but don't let one bad professor sour you on all of us. My base pay is less than $60k a year, but with summer salary from grants I'm in your range. And I work way more than 20 hours a week. Way, way more. But what is this "union" you speak of? I know of no such union (although tenure can provide significant job security).
I used to give short answer/essay questions to my astronomy students the first couple of semesters I taught the big non-major course. It took a tremendous amount of time to grade which was one reason I stopped, but not the primary reason. I'm a novelist, and I know how to write, and there was a consistently high fraction of exams written so badly it was very painful to read. Perhaps I should have kept at it, with the idea that it's good for the students. But a few essays in a science class won't dent the problem that starts in k-12 education.
Professor of Astronomy, Author of Spider Star & Star Dragon (Tor)
From his Intro to Sociology Course Outline:
"There will be one paper based on a combination of Internet and library resources that must be written using a word processor worth 50 points."
If this is any indicator of his understanding of grammatical rules, I pity his students.
Question: How many points is your word processor worth?
If you're not going to mark the paper. DONT SET THE ASSIGNMENT...
I mean what's the point? if the paper doesn't really help to demonstrate your mastery of the subject, and it's not going to be marked properly anyway, why waste everyone's time.
Why not get the students to mark each others papers, for the papers that don't count anyway. And only mark a small sample, and then mark the final paper properly.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
That is an interesting idea. I had one unorthodox professor who did something like that. We had a term paper, but we had to work in groups of 4, and the group submitted the paper and everyone got the same grade. It cut down the 30 papers he would have had to under 8. And it forces the students to talk about what they wanted to include in the paper, how valid points of view were. In essance, we were teaching each other.
But I have never been one who liked the idea of having my grade tied into the work of other people. I asked the teacher about that, saying "it is unfair for you to give me a grade for what other people do, I want to be judged based on my work". His responce was "in the real world, the sucess of your business depends on how well your group works as a team, so consider this a heads up".
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
Dunno about you, but I'd expect someone's sociology grades to actually reflect some understanding of sociology. Same as I'd expect that their math grades reflect having learned some maths, or that the grades they get in their Java course reflect _some_ knowledge of Java.
Having sociology grades that reflect purely English grammar skills, is as sick a joke as grading someone's data structures course based purely on indentation. It misses the whole point and makes a mockery of the whole teaching process.
A polar bear is a cartesian bear after a coordinate transform.
I've always had a problem with the points off for handwriting thing too... until I ended up on the other side of the table as a teacher.
These days, there's no reason not to just type up your math if you can't write legibly or draw a decent curve on a graph. In some cases, it was actually fun asking a student "what is that letter"?
has gone right into the shitter, at least at the University of Missouri. This program was actually approved for school wide use? What the hell is wrong with the University of Missouri? I find the whole thing just reprehensible. I guess at least Ed Brent has finally defined exactly the qualities of a "good paper".
I'd be interested to put the Gettysburg address, MLKs "I Have A Dream" speech, major works of literature, etc through Mr. Brents meat grinder and see what grade they get. The whole thing reminds me of that scene in Dead Poets Society where they try to measure the "greatness" of a poem using trumped up terms like "importance" and "perfection". Sprinkle in a little computer wizardry, and suddenly you've got a mysterious, unbending, rule machine.
Frankly this kind of thing just disgusts me. I'm no romantic, but you can't analyse how good a paper is based on some algorithm. It's like the idiots who try to analyse a songs potential through computer analysis.
AccountKiller
One of my teachers at school tried that. Unfortunately what happened is that in each group the rest of the members (the groups were arranged so each group was mixed ability) ganged up on the 'smart but weak/shy' one to do all the work then goofed off.
Actually, now I come to think about it, that's exactly what happens in the workplace.
Stephen
"Don't write down to your readers, the only people less intelligent than you can't read" - Sign on Newspaper Office Wall
arguably a better learning technique from a usability standpoint
... but only for a content-free "discipline" like sociology or other pseudo-sciences.
Yes, it might well be a better learning technique from a usability standpoint
Cargo cults a la Feynman are all about form, and this tool can indeed detect the presence of form and even distinguish form that is considered "good" by some metric from form that is considered "bad".
But unless it actually understands what is being written through deep semantic analysis performed against a thorough database of relation-interlinked concepts, then there is no way the tool can detect hard scientific content (to the small extent that it occurs in sociology) from gibberish that just obeys the right forms.
As Brent himself says, "In sociology, we want them to learn the terms." And that pretty much sums it up.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
As a history teacher myself, my first thought was why on earth would any teacher want to abandon their students' writings to to a computer program? There is no possible way a programmer would have near enough time to fill that software with all the nuance of the written word. And far worse, what does the program do when it encounters an original thought? If someone approaches the problem with a solution unexpected by the program would it get a poor grade simply based on not being what the professor is looking for? I see answers that are different from what I expected on tests and it is refreshing and exciting to see students formulate their own ideas. That's the whole point! How can you possibly program something to react to what it is not expecting to see? Aside from the whole program grading pitfalls, what kind of teacher could possibly want to willingly take their fingers off the pulse of their students? Essay writing is one of the strongest academic bonds between teacher and student and to relegate that to computer work shows a desire to divorce oneself from the student. What meaningful comments and feedback can a teacher really give about a paper they have never read?? All in the name of saving 200 hours of reading... Ahhh yes... he has to publish! Publish 1st, students 47th.
All the "but if it was an English class" posts make a nice alternate history what-if scenario, and all. No doubt there. Also an a very insightful observation on the role of English classes and essays.
But if you actually RTFA you'll see that we're talking about a _sociology_ professor and _sociology_ papers.
"If it's an English class, that's the whole point."
Well, precisely. That's the whole point: it's not an English class.
He's grading a _science_ class based on form instead of content. _That's_ the problem.
Unlike the English class in your example (which you are right about) his job _is_ to read and judge the content.
A polar bear is a cartesian bear after a coordinate transform.
Same here. Base pay here is not much more than 50k but I'm not complaining. I work 40+ hours a week too. Most nontenured faculty do, as do most tenured who still want to get promoted or who actually want to stay active in the field. I know of a few faculty who just show up and teach but they are a year or two from retirement.
With regards to students' written work: My field is meteorology. I too used to give students in my survey-level meteorology class opportunities to "express themselves" via short answers (a paragraph or two) on exams. I stopped because it was so hard to grade many of them because they were written so poorly. In addition to that, it is very difficult to grade short answers in a consistent way. For many of the short-answer questions I would usually end up just writing a number down ("Hmm.. this feels like a 3-points-out-of-5 answer") which real doesn't feel right... but what do you do when the concepts are confused, spelling and grammar are terrible but they have expressed some knowledge of the material?
I have talked with professors who have been doing this stuff for a much longer time than I (some of whom are into the latest trends in teaching etc.) and many of them are gravitating towards all objective tests (multiple choice and true false) for their survey level classes (and some upper level). A well-written objective test should adequately test a student's knowledge of the material in a fair way, especially in the sciences where there truly are right and wrong answers. Still, I don't like giving these kinds of tests - it just doesn't feel right - but like grading the others even less.
In my upper level classes all of my testing is subjective, and I do assign papers such as case studies where a storm system is described and analyzed. Some of my seniors can write well, most of them are so-so and a few are truly terrible. I tell them up front that spelling, grammar, style etc. counts on these assignments, and I find that if you tell students that these things are part of their grade they will put in an effort to write well.
I suppose I could just "blame the high schools" but I think the problem is deeper than that. In the US grade inflation is a huge problem in many universities and at the college level, student evaluations of faculty are often very highly regarded (and if you are evaluated poorly it can keep you from getting tenured or promoted). So a logical response is for faculty to go easy on students, rightly assuming that this will return higher evaluations. I don't know if that is a part of the writing problem, but I know an A today isn't an A 20 years ago at many universities.
A squid eating dough in a polyethylene bag is fast and bulbous, got me?
If the profs at your school mark grades down because they have a different opinion you should change schools. No offense.
I had a professor that said he would give his opinion but he would grade based on how well the argument was crafted and backed up by facts/references.
ANother professor complained that too many papers where just repeating what he had said in class and he had marked those down for not having enough original thought.
Grammar and spellling are alway important..