Slashdot Mirror
Computer Program Makes Essay Grading Easier
phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"
How long until some students get hold of this program and tweak their essay until it's puifect? It's similar to spammers using spam filters to test their emails first.
Rock that crushes, Paper & Scissors that don't matter.
I'm not so sure I like the idea of a computer grading my work.. I spent hours making it, but the guy doesn't even give it the time of day....
Angst
"The program analyzes sentence and paragraph structure and can ascertain the flow of arguments and ideas."
So it measures structure and argument.
How's it going to measure creativity of thought? Are we going to just pump out logic machines from colleges?
The only way that a computer program can possibly analyze a paper fully as a paper is if it read it as a human. heuristic algorithms, however sophisticated, just aren't enough for things of this sort of importance--after all, the profs are paid to grade (well, them or the grad students). It doesn't seem like it would be too difficult to just throw in keywords and make sure that you use proper syntax in order to fool this thing, albeit the prof says keywords alone aren't enough. I find the claim that his program can "analyze argument flow" quite dubious. I'll stick to getting my several grands' worth out of my courses, thank you very much.
Now if we could code a program to write those essays, the whole process would be automated. Press enter for another A. If you know how the program grades the essays, it should be possible to write a program that generates essays that comply perfectly to those rules, right?
Undergrad professors are usually not too excited about teaching these 18 year-old pizza-faced dorks. The problem is that the kids would rather be out drinking and screwing rather than debating the intricacies of pre vs post agrarian culture in the Southern States and the relationship between that and race relations as they exist today.
So more power to him. He is unlikely to be getting anything better or more insightful than a parroting of what he has already delivered in his monologues to his class. Same papers, year in and year out. No big deal to grade these kids with an automated program.
Lazy bastard.
XML is like violence. If it doesn't solve the problem, use more.
Sure, you could be great at grammar and sentence structure. You could be an ace at using proper english.
But how would hidden talent and creativity be found? How will the teacher know if his students are actually trying hard to write their papers when all he does is check the thing with a computer program?
It's a really terrible idea and I think it's really cheezy. Ohh, he saved some time. So does that mean he now gets paid less? Does this automation get the students a discount? Yea, right.
If I'm going to put a lot of work into writing an interesting paper about something, I want someone to read it.
- It's not the Macs I hate. It's Digg users. -
Look Tom Landauer's work at University of Colorado.
It makes more sense than you'd think: it turns out that knowledgable essays in a particular domain cluster statistically in useful ways. Yes, it does mean that something like Molly Bloom's Soliloquy wouldn't necessarily score very well, but then if you didn't know it wsa a Nobel Prize winning classic, would you think it was well written?
What does this say about the field of Sociology? :P
FWIW, this prof is at the University of Missouri (in Columbia, MO).
Gee - you'd think the submitter could RTFA...
Why not give the program to the English department and use it for teaching?
Would be great for high-school students. Have students write an essay or paper and analyze it right in front of them. Then the program highlights their errors (or what the program perceives as an error). Even better, complaining students would help fix bugs in the software because they know their intent - they could send off a highlighted error-ridden version to the developers with an explanation of why they think they are right.
Better yet, give it to everyone! It's not like you can cheat, you still have to rewrite and resubmit your papers. Shit, I say build it into text boxes on slashdot and wikipedia to start!
please do not hold this post to the standard of the Qualrus (real page of the software)
Get your Unix fortune now!
It works by scanning text for keywords, phrases and language patterns.
Which is to say, this prof is asking students to regurgitate data. Given, a certain level of base knowledge is necessary in any class and topic, and regurgitation (aka parroting) is an easy way to check that base knowledge. If a paper is assigned on a particular topic that they've been studying, then this sort of program can easily check for base level ability to spit back key words and phrases.
But, I seriously doubt that the class is ONLY about that base knowledge -- or that the program can reasonably check for anything more. I've had classes where the prof or graders did basically the same thing that this program does (i.e. check only for key words, phrases, and patterns they want to see), and I have little respect for those profs.
If you don't want to put even a basic amount of effort into checking a paper, don't assign it -- find some better way to check students' progress.
This one is just nuts. Why on earth am I writing essays which are going to be marked automatically by a machine? It's bad enough that scantron cards have found their way into subjects where they're totally irrelevant (a multiple-choice test for a university level Shakespeare course?), this is just another reason why post-secondary education has become increasingly less complete.
If he's allowed to use a machine to save him the effort of reading an essay, I should be able to use a machine so I don't have to go through the effort of writing one. Trust me, as arduous as it is to read a 20 page essay on the relative merits of liquid rubber concrete compound fasteners, writing it takes a lot more effort, a lot more time, and it damn well deserves to be read by the professor who assigned it.
Just wanted to point out that the software was created originally for the purpose of qualitative coding. Grading essays is one of several other applications it has proved capable of addressing.
So the 1% of us who go to college because we enjoy learning about our field should just drop out and go drink and screw. I see.
I was required to take a lot of writing classes for my college (and still haven't finished them all) and I've observed the quality of my writing go up appreciably since I began school. However, the reason I've become a better writer is because my essay graders write copious comments about where I'm going wrong in my papers and what I should do to improve - and they read the next paper I write for the class with those things in mind, and tell me whether I've improved sicne the last one.
The article didn't say anything about what kind of feedback the program provides, but I can't imagine it's anywhere near as helpful as the paragraph-long evaluations of my logic, style, and structure, which I got back with every paper I ever turned in, and I'd be impressed but surprised if his program took each student's previous weaknesses into account in the course of the evaluation. In writing, practicing can only do so much - the real help is in constructive feedback, and I just can't imagine where these students are getting it if not from the human graders of their papers.
Introducing Microsoft Virtual Staircase XP Professor Edition 2000 Plus. Tired of moving up and down staircases to determine the flight distance and grades of your thrown paper essays? Enter the 21st century with MSVSXPPE2kP, Microsofts solution to choosing winners. Previously only available to in-house project managers for debugging code, it is now available for only $349.95!
According to ETS, the e-rater agrees with the human grader 98% of the time.
My userid is prime!
From TFA, which apparently no one has read yet:
"The final papers, which he does read, are usually much better as a result of Qualrus, too."
There you go! For the reading and comprehension impaired, here's a summary of what's actually happening, which even the reporter didn't get:
1. Students write a draft of their essay, which they then upload via a Web form to this program
2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.
3. Students improve the pieces of their essay that the program suggests.
4. Students submit the final draft to the professor, who reads and grades each one by hand. Due to steps 1-3, the quality of the final draft is much higher.
This sounds like a great thing to me. Wish I had something similar for my students. I don't have the time to read through dozens of drafts for every student. Too bad I'm not in sociology.
Actually Ed Brent encourages his students to use Qualrus on earlier drafts of the papers. This provides immediate, extensive feedback. And by "extensive" I mean more detailed and descriptive comments than those that a single teacher/TA could supply for each and every paper in a large lecture. The immediacy of this feeback is what is really important - immediacy is KEY to learning.
How long until a student specially crafts a paper which causes a buffer-overflow, followed by code to install spyware which makes all his papers recieve a perfect grade?
...Had this been an actual emergency, we would have fled in terror, and you would not have been informed.
What makes you think that the English teachers get paid as much as the math/science guys, especially at a college level?
My father was a high school math teacher. One day, a new assistant principal came by the math & science department lounge and told the chairman of the department (a physics teacher) that he didn't approve of the way he dressed (wearing jeans). The department chair replied "I was passing by the unemployment office the other day. I saw a whole line of assistant principals but not one tenured physics teacher." He wasn't bothered again.
William of Ockham had no beard. The most likely explanation is that it was chewed off by squirrels every morning.
I later found out why from someone who had been taught by him before. He would take 100% - the percentage of passive sentences found by the Word program. So I intstantly started handing in garbage essays with 0% passive sentences.
What if he grant application was reviewed by a program instead of a real person? I wonder how he would like it
Actually that's already been done. Quite effectivly too.
So since the professor's time is worth about $36.00/hour and he spends 200 less hours on reading papers...
200 hours * $32.00 = $7200
He teaches about 84 students...
$7200 / 84 = $85.71 refund for each student. It's party time!
Authority questions you. Return the favor.
...then a student should be able to write a program that develops an essay. That way the student isn't cheating too, because they will have created the essay, indirectly.
Saskboy's blog is good. 9 out of 10 dentists agree.
Okay Slashdot Editors, time to fork out the $$$ to get some auto-moderating going on in these threads! Wait, can this grading program test for humor? No? Fuck it then.
Authority questions you. Return the favor.
If this professor's analysis can be "simulated" by a computer program, then he was obviously not doing a thorough enough analysis to begin with. I know plenty of professors that would laugh at the idea that a computer program would be able to "calculate" emotion, nuance, subtle sarcasm, humor, insightfulness, etc...
This professor should be fired.
Authority questions you. Return the favor.
"But Professor, my original essay was really good! I just had to add a bunch of crap to get past the lameness filter ..."
Cheers,
IT
Power corrupts. PowerPoint corrupts absolutely.
"How long before he's replaced entirely by his own program to cut down on staff costs?"
I would say a long time. A program that tries to understand natural language requires some sort of "intelligence," a quality that humans definetly possess and computers, up to now, definetly do not.
AI still mostly consists of certain hacks to trick other people into thinking the programs are intelligent - basically attempting to fool the Turing Test. This can often produce great results and can be very useful, but almost never replaces a human in complex tasks (such as natural language processing).
The difficulty arises because humans cannot easily (or perhaps possibly) comprehend their own intelligence. It seems so natural to read a sentence and make sense of it, but when it comes time to program a computer to do it, most people try to emulate the behavior of their own comprehension. This may trick some people, but the simple nature of the programs cannot possibly be as powerful as an actual human.
The best solution, in my opinion, is a closer study of neuroscience and how it can be applied to silicon (or how new technologies need to arise to emulate the complex neural structure of the brain).
I know that people are starting to use computers to grade standardized essays, but there (currently) must always be a human checking the results because of the small number of unforseen cases that the hacked algorithms cannot do a good job. After all, the programs do not "understand" anything that is written. That is why I postulate it will be a long, long time before computers can truly emulate humans.
If a professor does not care enough to read my papers, then to hell with him. There is more that a professor does than just check grammer, or look for passages that deals with the question and used terms from the book. The best professors I had were the ones who wrote all over the margins, sharing their thoughts about my ideas. Those are the ones who I would meet in their office to chat with. They are the ones who I went to for advice.
I had one teacher in english who graded the first paper, reading them all. She then never read another paper, only skimmed them. She pretty much gave out the same grade on all your papers you got on your first paper. I got an "A" on my paper, and another student got a "D". So I was working with the "D" student, and no matter what was done, the "D" grades went up to "C-" but stuck. So for the last paper, we switched our papers. Guess what? My paper was still an "A" even though it belonged to the other student, and the other paper was a "C". We went to the teacher to explain what we did, and rather than the professor owning up to what was done, we the students got blamed.
This really pisses me off. Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union. Then they want to slack off. Fucking asshats. Something like this makes me want to vote to remove public funding from schools, to always vote no whenever there is a refferendum to increase property tax. With those kinds of professors, people might as well get their education at the public library.
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
I can say this much, I have never had any other professor, outside of the Chem or Physics department, grade my papers like a math professor. Most of the humanities professors just skim over. But in my Calculus class, it was possible to turn in homework and get negative points. For example, you have a problem 1.0 + 1.00 = ?. You write 2. First, half a point off for not figuring in significant digits. Another half a point off for sloppy handwriting. And the full point off for not showing your work. Problem worth one point, your score is negative one point. In some cases, it was better to not turn in anything at all.
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
That reminds me of a startegy some people had when I was an undergrad. Since they knew teachers only skimmed the writings, they used a shotgun approach. They threw everything in the essay including the kitchen sink. They figure somewhere in there, the terms the professor was skimming for would be included. Sadly, these were the "A" students. The "B" students, who tried to write a real paper and make a point, did not get any worthwhile feedback.
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
You remember peer editing in 4th grade? Did that have any value? Not really - but if you got instant feedback on papers, that makes it easier to just write better in the first place.
Especially if this technology is combined with this technology.
paintball
1. Students write a draft of their essay, which they then upload via a Web form to this program
2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.
This just in! University English professor discovers the POSIX toolchain. Novel misuse of cat, awk, and sed and friends expected. Film at 11!
I have lots of programs that save me hours of "doing my fucking job". They're called scripts, and it's called efficiency.
If I can write a program to automate a menial task so I don't have to do it, then by all means, I should do it. If grading undergrad papers is a menial task that can be automated, then it should be automated.
I mean, just because a freshman writes a bad paper doesn't mean a professor has to actually read it.
paintball
Unless it's an English class, they're just looking to see if you've learned the material for that course. Especially for the essay questions in the midterm and final, they will always just skim the papers for key words, phrases, and dates. They underline them and add up the points. That's just the way it works. The "B" student, who writes something interesting, but doesn't cover all of the relevant material, loses points.
For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.
These are things that I wish someone had told me when I was an undergrad.
My other first post is car post.
Announce to the class that they have two options. If they want their paper to be computer-graded, the maximum they can get on the paper is a B. Only hand-graded papers can earn an A - but they will be judged on original thought, well-chosen sources, and total structure. If they don't meet those criteria, they will be penalized (and as such, would score lower than the computer would give). That way, students who aren't confident in their capability for truly excellent writing and novel thought can choose the computer grader for a safe B, while the truly thoughtful and creative students can be justly rewarded.
Wait - what the fuck am I saying? ALL college students should be graded by the non-computer criteria I just listed, and those who can't do (or at least attempt) that kind of work shouldn't be in college.
If you mod me down, I shall become more powerful than you can possibly imagine.
Hmm...the NSF grant provisions I recall seeing the last time I submitted one discouraged, or even ruled out, supporting commerical activities. This does sound a bit funny to me, too.
Professor of Astronomy, Author of Spider Star & Star Dragon (Tor)
That's the first thing that came to mind as I was reading the summary: well, gee, so he can now grade purely on form, rather than content.
Sure, the program can analyze that the sentence flow and structure looks like it's analyzing/arguing/explaining/whatever a point. But is it even arguing the right point? Does that paper even _have_ a point at all, or is it just a babbledygook of random nouns/verbs/adjectives/etc that fits a structure?
I'm not even sure it has to end up "beautifully expressed babble", it can just be any collection of random words that fits the structure the program is looking for. I.e., I'm sure it can be _awful_ babble and still pass.
And indeed, a student then doesn't even have to understand how the program works or anything. A script will do just fine. Just download a paper that got good grades (hence, fits the idiot's program) and run it through a script that replaces each word with a random word that's in the same category. (Transient/intransient verb, noun, etc.)
E.g., take the following two sentences:
"The Electronic Frontier Foundation announced its prestigious Pioneer Awards today, and one of the three lucky winners for 2005 is Mitch Kapor."
"A Pink Smell Stew impaled its dormant Turkey Shaddow tomorrow, or five of the two spotted continuums towards 2005 equals Mitch Kapor."
It's just the random word substition I was talking about. It isn't even beautifully phrased babble, it's just awful even by dadaist criteria. Yet it has the exact same sentence flow and structure. Can the program even tell that the second is random blabber?
A polar bear is a cartesian bear after a coordinate transform.
Dunno about you, but I'd expect someone's sociology grades to actually reflect some understanding of sociology. Same as I'd expect that their math grades reflect having learned some maths, or that the grades they get in their Java course reflect _some_ knowledge of Java.
Having sociology grades that reflect purely English grammar skills, is as sick a joke as grading someone's data structures course based purely on indentation. It misses the whole point and makes a mockery of the whole teaching process.
A polar bear is a cartesian bear after a coordinate transform.
First, moving around quickly, and with purpose, is a true sign of character. Secondarily, bustle(e.g. hustle) yields more product for the working types. "Hustle and bustle are like my right and left arms," said Li'l Spicy in his famous "Hustle and Bustle Are Like My Right and Left Arms" speech. Webster's defines bustle as "excited and often noisy activity; a stir." A stir, indeed. Finally, sometimes gross stuff can be funny.
Here are some links:
- Turn It In
- Penny Arcade
It is now my intention to play video games for several hours.Sources:
The Brothers Chaps (2004).Homestar Runner. Retrieved April 8, 2005 from www.homestarrunner.com
Random Source (2005). that you won't read because you were too lazy. Retrieved April 8, 2005 from www.toreadthisfar.com
(I have four words for this post: "Too much half-asleep effort")
"When the atomic bomb goes off there's devastation...but when the atomic bong goes off there's celebraaaaation!"
Nah, you write as many sigfigs (significant figures) as is found in the least accurate information. Otherwise by the same logic 10 + 9 = 20. Like your logic though :)
I have a teacher this semester who doesn't teach sigfigs but expects you to get it right. Then he applies rules like "but with this info, you always show 3 sigfigs, with this info, 2 unless blah blah blah blah blah"... and he says this after marking everyone down in the test, mind you. Good thing I had a High School teacher who taught me this stuff.
His excuse? "You should have asked more questions in class". I swear, he must have expected us to actually ask "how do sigfigs work?" without knowing it would be an issue. Doesn't help that he makes you feel stupid when you ask a question in class.
He's not all bad, but still pretty bad. For more information: click.
"When the atomic bomb goes off there's devastation...but when the atomic bong goes off there's celebraaaaation!"
But can we teach it to spot dupes and use it on slashdot? (and does it have a "Rolland" filter?)
has gone right into the shitter, at least at the University of Missouri. This program was actually approved for school wide use? What the hell is wrong with the University of Missouri? I find the whole thing just reprehensible. I guess at least Ed Brent has finally defined exactly the qualities of a "good paper".
I'd be interested to put the Gettysburg address, MLKs "I Have A Dream" speech, major works of literature, etc through Mr. Brents meat grinder and see what grade they get. The whole thing reminds me of that scene in Dead Poets Society where they try to measure the "greatness" of a poem using trumped up terms like "importance" and "perfection". Sprinkle in a little computer wizardry, and suddenly you've got a mysterious, unbending, rule machine.
Frankly this kind of thing just disgusts me. I'm no romantic, but you can't analyse how good a paper is based on some algorithm. It's like the idiots who try to analyse a songs potential through computer analysis.
AccountKiller
arguably a better learning technique from a usability standpoint
... but only for a content-free "discipline" like sociology or other pseudo-sciences.
Yes, it might well be a better learning technique from a usability standpoint
Cargo cults a la Feynman are all about form, and this tool can indeed detect the presence of form and even distinguish form that is considered "good" by some metric from form that is considered "bad".
But unless it actually understands what is being written through deep semantic analysis performed against a thorough database of relation-interlinked concepts, then there is no way the tool can detect hard scientific content (to the small extent that it occurs in sociology) from gibberish that just obeys the right forms.
As Brent himself says, "In sociology, we want them to learn the terms." And that pretty much sums it up.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
WTF is a word processor? I write documents in LaTeX ;-)
What, his program can't read a DVI?
Tom
Someday, I'll have a real sig.
All the "but if it was an English class" posts make a nice alternate history what-if scenario, and all. No doubt there. Also an a very insightful observation on the role of English classes and essays.
But if you actually RTFA you'll see that we're talking about a _sociology_ professor and _sociology_ papers.
"If it's an English class, that's the whole point."
Well, precisely. That's the whole point: it's not an English class.
He's grading a _science_ class based on form instead of content. _That's_ the problem.
Unlike the English class in your example (which you are right about) his job _is_ to read and judge the content.
A polar bear is a cartesian bear after a coordinate transform.
When I was in college in the late '80's, the "trick" to getting good grades really was to understand what the professors were looking for, and give it to them.
For example, I had a professor who distributed pre-printed pieces of paper that had a line drawn around it indicating the margin (something like two inches at the top, a half inch on the right and bottom, and three inches on the left. The large margins on the left and top were for the prof's notes and comments to us.) We had to type (with a typewriter) our papers to fit within the bounds of the margins, and spelling and grammar counted (and this was a Psychology class, not an English class.) He would not accept more than the one page. Another professor required papers be written in a specific topical order. If you deviated from those models, you got marked down.
The point was not that the profs were trying to be devious, but to make us take into account the instructions they gave us. If we followed the instructions, we got better grades. If we didn't, we would get marked down. And yes, content did count too.
Yes, it was a hassle, but the result is that now, when my bossed give me instructions, I follow them. The times when I deviate are the times when I really hear about it. Lesson learned!
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
If the profs at your school mark grades down because they have a different opinion you should change schools. No offense.
I had a professor that said he would give his opinion but he would grade based on how well the argument was crafted and backed up by facts/references.
ANother professor complained that too many papers where just repeating what he had said in class and he had marked those down for not having enough original thought.
Grammar and spellling are alway important..
I've spent hundreds of hours over the last couple of years participating in essay-reading projects for national educational testing companies based in Iowa City, IA (and elsewhere), have developed a healthy regard and admiration for neural networks implemented in wetware (collaborating human test raters), and have to say I'm skeptical of claims that software can interpret anything important or consequential in the essays I've personally had before my eyes. In particular, I doubt the ability of software to recognize genius, especially genius buried under a thick layer of ESL ("English as a second language") errors or social disadvantage. I doubt the ability of software to recognize anything but a small core of mediocre constructs and pedestrian insights, and feel rather strongly that its use is a serious violation of civil rights.
``Tension, apprehension & dissension have begun!'' - Duffy Wyg&, in Alfred Bester's _The Demolished Man_
When was the last time a sociologist wrote a sophisticated program. I mean the way he is advertisng is it, he proclaims the program is capable of understanding the document. Thats insane. Do the guys at the AI lab at MIT know that some sociologist in Missouri is kicking their asses.