Slashdot Mirror
Bringing Auto-Graders To Student Essays
fishmike writes with this excerpt from a Reuters report:
"American high school students are terrible writers, and one education reform group thinks it has an answer: robots. Or, more accurately, robo-readers — computers programmed to scan student essays and spit out a grade. The theory is that teachers would assign more writing if they didn't have to read it. And the more writing students do, the better at it they'll become — even if the primary audience for their prose is a string of algorithms. ... Take, for instance, the Intelligent Essay Assessor, a web-based tool marketed by Pearson Education, Inc. Within seconds, it can analyze an essay for spelling, grammar, organization and other traits and prompt students to make revisions. The program scans for key words and analyzes semantic patterns, and Pearson boasts it 'can "understand" the meaning of text much the same as a human reader.' Jehn, the Harvard writing instructor, isn't so sure. He argues that the best way to teach good writing is to help students wrestle with ideas; misspellings and syntax errors in early drafts should be ignored in favor of talking through the thesis."
The best English professor I had in college would arrange to have every student come in to her office after papers had been turned in, reading each paper in the presence of the student who had written it and discussing it in depth while grading it.
Sphinx of black quartz, judge my vow.
Make the robots write the essays, then students can work on other subjects.
Table-ized A.I.
This is one area where automatic grading will cause massive skill decrease, as no auto-grader can actually assess contents.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I for one welcome all your hot grits during your first post that runs on Linux.
Table-ized A.I.
It's only a matter of time before someone writes a tool that generates top grade essays.
When asked why, the answer is almost always: "It's 2014".
...thus producing an entire country whose writing-skills were conditioned to game the auto-grader.
The best English teacher I had was my English instructor my first year of undergrad. Instead of concentrating on whether we were writing our papers to the curriculum and/or her own beliefs about the content, she was instead interested in developing our English skills.
I went from a C student in English to an A student. I never considered myself to have any ability to write, thankfully because someone took the time to actually think critically about my work instead of comparing it to their own preconceived notions I excelled and went on to complete a research and writing focused program. This degree later fed into my graduate degree which was also research and writing focused.
If this automated grading setup can provide students with clear expectations and explanations of the mechanics of their work while avoiding personal content expectations, I really do think it'll match the claims and help to foster a positive writing environment for many.
I think, therefore I spam
The best way to get students to write is give them something they enjoy writing about.
Allow me to introduce myself. I'm the founder of the Anti-My School Society. In this letter, I will tell you what made me form such an organization and how I plan to use it to strengthen our roots so we can weather the storms that threaten our foundation. Let me cut to the chase: Relative to just a few years ago, the worst sorts of flippant ogres I've ever seen are nearly ten times as likely to believe that the key to living a long and happy life is to provide contumacious conspiracies with the necessary asylum to take root and spread. This is neither a coincidence nor simply a sign of the times. Rather, it reflects a sophisticated, psychological warfare program designed by My School to work hand-in-glove with what I call intrusive vocabularians.
Even as I write those words I can feel My School cringe. That's okay. Cringe. I don't care because it appears to have found a new tool to use to help it make us the helpless puppets of our demographic labels. That tool is obstructionism, and if you watch it wield it you'll honestly see why it's good at one thing, and that's keeping its ulterior motives secret. Only a few initiates in the inner sanctum of My School's cabal know that it's planning to advocate fatalistic acceptance of a perfidious new world order. Even fewer of these initiates know that I don't need to tell you that we have fallen into My School's trap. That should be self-evident. What is less evident is that My School has two imperatives. The first is to judge people based solely on hearsay. The second imperative is to call for a return to that which wasn't particularly good in the first place.
If you were to tell My School that right is right and wrong is wrong, it'd just pull its security blanket a little tighter around itself and refuse to come out and deal with the real world. My School likes to talk about how cell-phone towers are in fact covert mind-control devices that use scalar waves to beam images into people's brains while they sleep. The words sound pretty until you read between the lines and see that My School is secretly saying that it intends to calumniate helpless rapscallions. I want to advance a clear, credible, and effective vision for dealing with our present dilemma and its most misinformed manifestations, but I can't do that alone. So do me a favor and point out that the emperor has no clothes on. That'll show My School that it's possible that it doesn't realize this because it has been ingrained with so much of Chekism's propaganda. If that's the case, I recommend that we enable adversaries to meet each other and establish direct personal bonds that contradict the stereotypes they rely upon to power their conceited ramblings. Not to put too fine a point on it, but My School's winged monkeys don't want us to disseminate as widely as possible all of the information we have regarding My School's cruel theatrics. That'd be too much of a threat to imperialism, simplism, and all of the other carnaptious things they worship. Clearly, they prefer seizing control of the power structure.
Efforts to create a factitious demand for My School's spleeny, uncouth analects are not vestiges of a former era. They are the beginnings of a phenomenon which, if permitted to expand unchecked, will push all of us to the brink of insanity. My School exists for one reason and for one reason only: to intensify or perpetuate hoodlumism. My goal is to challenge the present and enrich the future. I will not stint in my labor in this direction. When I have succeeded, the whole world will know that My School somehow manages to get away with spreading lies (big emotions come from big words), distortions (honor counts for nothing), and misplaced idealism (it has a "special" perspective on mandarinism that carries with it a "special" right to worsen an already unstable situation). However, when I try to respond in kind, I get censored faster than you can say "archaeopterygiformes".
While there's no dispute that My School is whiney and probably a little counterproductive, it's also cunning, implacabl
... one issue I take with my public education experience was the lack of mention by teachers that they would review or aide in writing for most papers. I recall only the final big paper for the class (whichever class that is) would have something akin to a draft-review event, and then a final draft.
I realize TFA suggests teachers would assign more work, and read less --- and maybe the robots would be useful in providing easy rapid review --- but I can't refrain from mentioning that, in my experience, teachers did not clearly express a willingness to aide in the writing process throughout the semester. (let me beat the critics by saying I was liked and respected by all of my teachers)
Some students are good/great writers and maybe they can be commissioned for honors credit or something in exchange for aiding peers. I know I was in my early 20's before I understood the power of the semicolon; and it is awesome!
My wife worked for Pearson as a "second tier" grader (or whatever they call them).
In her case, the tests went through the algorithm and were assigned a grade, then the grade and test were passed along to a human to read and check. Invariably, she would come home complaining about tests where the students had obviously studied specifically to answer the way the algorithm wanted: the algorithm would score the paper high, while the actual content of the test answer would leave a LOT to be desired. The answers would score high, but were more or less gibberish as read by a human.
This was about two years ago, so obviously the algorithms could have improved since then, but I have severe reservations about them becoming the sole arbiter of grading.
"lets see the grade on this one..........PC-load letter? The FUCK does that mean?"
Yeah, I know teaching is a drag and a lot of work, but that's what they're paying you for. I was up until 1:30 last night fixing/grading code and writing a final project assignment. It's not fun, and I could have easily just ran the programs and told them it didn't work so they got a zero. There's more value for the students when they get feedback from me telling them what specific errors they got, or a way they could have written code more efficiently, etc.
Taking guns away from the 99% gives the 1% 100% of the power.
If someone told me to write a paper, then told me it would be graded by some algorithm, I'd tell them to go fuck themselves. Who comes up with this crap?
The ditch-digging jobs will be filled by the robots that weren't allowed to read essays.
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." Feynman
The problem is not that the students don't get enough practice. The problem is that the students don't get feedback until they get their grade.
Having an auto-grader grade your work is a terrible idea because auto-graders can't handle complex English. I thought it might be a good idea to run a grammar checker across my novels before publishing them just to have an extra set of eyes, so to speak. So as an experiment, I fed some fragments of one novel (that I knew contained no grammatical errors) into about a dozen of these so-called grammar checkers, along with a list of deliberately broken sentences to see if they actually caught problems.
I just about died laughing at the ludicrous suggestions that the grammar checkers made, mostly stemming from them incorrectly guessing the parts of speech for words that could have more than one meaning. The best of these algorithms correctly reported about 80% of the correct sentences as correct, though many of those algorithms also failed to flag a lot of the incorrect sentences. The worst algorithms flagged more like 80% of the correct sentences as incorrect (and still failed to flag the actual errors in many of the incorrect sentences).
Based on that, I'd say that having someone's grade depend upon such poor algorithms is a really, really bad idea, I'm guessing it will be at least another 1-2 decades before I would trust a computer-based grader to actually perform grading that counts.
However, making those auto-graders available to students for online pre-screening of their writing before they hand in the final version would be a good thing, provided they can make them a lot better. Such software is great at catching simple errors, and anyone with poor writing skills can probably benefit from such software pointing those mistakes out, allowing them to correct their own mistakes before handing the assignments in. This allows the students to learn from the mistakes. A well-designed checker could even keep track of what mistakes a student makes regularly and point out the pattern so that the student can learn to watch for that type of mistake in the future. Unlike robo-grading, such software can actually teach students to improve their skills usefully.
Check out my sci-fi/humor trilogy at PatriotsBooks.
If I knew that a machine gets to grade my work I would feel like my time and efforts are worth so little that humans can't be bothered to read it. It defeats the purpose of even writing the thing.
When you write something you are trying to convey an idea. Knowing that the machine doesn't give a fsck proves my efforts are useless.
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
You mean, like proofreading and revising?
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." Feynman
Exactly. The way an idea is written is, for the purposes of a writing course, far more important than the idea itself, or even for that matter if the argument itself is well-made (although, obviously, that isn't completely incidental). I've seen many college-level students who simply cannot write well. Sure they may be able to spell decently, but their sentences tended to be organized poorly, and their paragraphs were even worse. An automated system could detect a lot of that. Besides basic spelling and grammar, there are stylistic things, like they reusing words unnecessarily, run-on sentences, even awkward syntax, that computers could be programmed to look for. It can't do everything, sure: humans will always be needed to provide feedback in important areas, but many of the basics of writing can be graded by computer.
Writing follows certain rules and patterns, and computers excel at determining that. More advanced stylistic issues can still be an issue, and of course logical validity needs a human to judge, but that is easier to do if the writer has all of the basic necessities of writing well down.
"None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
Computers just aren't up to understanding complex English well enough to decently grade it. The smartest students will very quickly grow apathetic and start gaming it, whilst forgetting the skills they do have. The less intelligent students will just learn to run it through MS Word's grammar and spelling check and add words that don't fit but are long.
If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
Lucky you. For me English class, fro 7th grade through undergrad was a constant string of "infer the hidden meaning behind this text" with nobody ever trying to teach us the process for inferring that hidden meaning. This lead to me being a C student in English for my entire academic career.
Despite all my efforts, in 8 years of English classes, I was never even able to get a single teacher or professor to explain to me how he knew there were hidden meanings behind the text that was assigned. Nor could I get anyone to tell me why they would put hidden meanings into text, when they could put the meaning the want in the literal text.
The funny thing is, my English is fine. IIRC I got a 760 on the English portion of the SAT. I always got As on papers in classes other than English, and complements on my writing were common. It seems to me that the way English classes are normally taught, they have nothing to do with English at all.
Give me Classic Slashdot or give me death!
One of the differences between good essays and poor essays that research has identified is although they both tend to have about the same amount of hedging ("it can be argued that...", "possibly...") the poor essays hedge the wrong things. The poor essays hedge well supported facts and fail to hedge personal opinions or unsupported facts. If the software can spot that, I'll be impressed.
Quidnam Latine loqui modo coepi?
It seems to me that the way English classes are normally taught, they have nothing to do with English at all.
You have found the hidden meaning behind English classes.
--
BMO
Writers have placed hidden meaning into their work to express opinions that are not socially accepted or possibly illegal for a very long time. In order to actually understand what any of it means you must first have foreknowledge of the writer, their culture, common issues of the time, and imagery. Even then unless the writer has later explained these things then you will probably never know if those hidden messages were really ever there. Grading someone on it means your teachers/professors were not very good at their profession.
said just about every English teacher I had in high school.
Also, anything beyond spell check is patently ridiculous. Even the best grammar checkers are still rubbish.
They grade C++ like that. If you did not write the code exactly as the grading system was looking for it was game over. I would have been fine with it if this was the standard in the 150 class. An automated system for grading essays will not detect the passion of the writer as it only can grade the mechanics acurately.
I'll be honest. I agree with this guy and I most certainly do not have aspergers. :)
High school level English consists of reading dull books and then writing character analyses and other crap about them. Unfortunately, at least in my experiences, being able to analyze characters and plots wasn't a skill that I gleaned by suffering through 1800's romance novels, it was a skill that I learned from writing fiction on my own. I started out very bad at writing, as most people do, because in all the time they spend forcing "classics" down your throat, they never teach you to write -- they just expect you to. (Sure, they teach you grammar and syntax and how to structure a paper, but they don't teach you a damn thing about how to write fiction, which is ironic since English classes focus on it so heavily.)
Interestingly, if I went back and took those classes now (with a mind for the teacher as my audience) I'd be an A student, but not because of anything I learned in English classes.
A Modest Proposal must have scared the shit out of you.
http://www.accountkiller.com/removal-requested
I believe they're called graduate students.
As someone who has taught writing, organization of ideas is the problem, not spelling, grammar and mechanics. That problem, in turn, comes from students who don't read enough. No automated grader is going to solve that problem.
If they had things that did that when I was in college, I probably would have spent most of my time trying to come up with syntactically correct nonsense.
A patriot must always be ready to defend his country against his government. -edward abbey
My middle school aged daughter recently wrote a paper that was autograded. I think it was an experiment by her teacher to see how well the autograder worked, since half the grade was to come from the autograder and the other half from the teacher actually reading it. At least she was allowed to run it through the autograder as much as she wanted to before handing it in.
Her first round before I read the paper, the autograder gave her a 92%. I read the paper, and it was hideous (sorry). The grammatical structure may have been technically correct, but the organization was awful, it was horribly confusing, and just didn't make much sense. I ran it through my own grader (a red pen), she fixed it, and it was clear, made sense, well organized, and still had correct grammar and spelling. The autograder gave it a 73%. Why? Because she didn't use advanced enough words. The words she used were perfectly appropriate for a middler-schooler or even a high schooler. So what does she naturally do? Pulls up a thesaurus, inserts a bunch of big words, and gets a 95%. I took a look and the words she used were not at all in the right context. I had to explain to her that, as she well knows, thesauruses don't provide *exact* synonyms, and the autograder is retarded.
So...I'm all for letting students run their paper through a set algorithms to give them hints about what *might* need changing, but relaying (that word passes spell checker and an autograder would have been happy with it) on an autograder to grade a papyrus is puerile (see, an autograder would have given me a 99% because I used those words from the thesaurus even though they're not in the right context). Also, as I think other commentors have pointed out, if you know a human is going to read your paper, even if you don't like your teacher and your teacher doesn't care, you're more likely to put more effort into getting your points across.
Unless they're the author, or have read the author saying so, they're creating the hidden meaning
That's a topic of some debate. Tolkien swore that the LotR books weren't the least bit allegorical, for instance, yet how can you read them without seeing Mordor as 1930s-era Germany?
It's pretty common for writers to insist that they weren't trying to plant the subtexts in their work that everyone else can see.
I've read them several times and that thought never crossed my mind.
"The space elevator will be built about 50 years after everyone stops laughing." - Arthur C. Clarke ~1980
Wow. That's great. How many sections was this professor teaching? How many students per section? How many writing courses? Currently I have 88 students per semester, with 5 five-page papers each. At my previous job, I had 135 per semester with 3 five-page papers each. I'm afraid to do the math to see what my life would be like if I gave each student a half-hour meeting for every paper (on top of class prep, my committee work, research work, stupid paperwork work, and basic bodily needs). Kudos to this teacher! (Or to hell with her for busting the curve. I know so many of us on uppers to get the work done. I hate them. Anyway, Gentle Reader, if you're about to do the math and tell me life would be great, be aware that I will perform the readerly equivalent of putting my fingers in my ear and yelling "la la la."