Automated System Developed To Grade Student Essays

RougeFemme points out this story at the Times about software that can be used to grade student essays and offer almost instant feedback. "Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the 'send' button when you are done and receiving a grade back instantly, your essay scored by a software program. And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade. EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."

Comments

This is horrid

[swm]

One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.
- go to the site
- get a problem
- solve the problem
- type in the numerical answer
- right answer? go on to the next problem
- wrong answer? try again
The web app allowed maybe 0.5% margin for rounding error, and you got 5 tries before it failed you on that problem.

It sounds reasonable in the abstract, but in practice it was utterly wretched.
All learning is, at some level, an interaction--a conversation--between student and teacher.
Even if it is nothing more than a red check mark or a red X on a homework paper,
you have communicated some thing to some person and gotten some response.
You don't realize how important this is until it is gone.

With nothing but a machine to talk to, it stops being about learning.
It is just about satisfying the machine by whatever means necessary.
In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google.
This would reliably return the general formula for solving that problem;
plugging in the numbers that the web app had generated for your instance of the problem would then yield the correct answer.
By the end of the school year, I was telling him that if he didn't want to deal with the web app, he should use google to get his grade,
and if he wanted to learn physics, I would teach it to him.

Automated essay grading is going to be even worse.
There is no point writing prose unless a human is going to read it.
When I want to talk to machines, I write code.

Writing songs, that voices never shared...
-- Paul Simon

Re:This is horrid

[Anonymous Coward]

I went through the same system and it taught me all sorts of useful things unrelated to my actual physics curriculum, like
1/2 != 2/4
0.5 != 1/2
x != x+1-1
x^2 != x*x

My TA had that 35 years ago

[gewalker]

Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.

Have a computer write your submission too

[hawguy]

Seems like it's a small step from this to having computer algorithms that automatically write your paper for you too - then you can let it go through thousands of submit-edit-submit cycles until the scoring computer gives you a perfect score.

Kind of like the guys that came up with software to generate nonsense scientific papers and actually had a few accepted at conferences and journals.

feedback...

[retchdog]

``Your grade is C. To improve your grade in the future, you need to do the following:

use 25-30 words per sentence; include more words from the wordnet entry for the topic of your essay; avoid simplistic or run-on sentences as measured by number of noun and verb phrases detected by our proprietary NLP tokenizer.

As a helpful reminder, our preparatory guides are available as a subscription service and include 100 practice submissions per week; only $29.95 per month."

Grading is about feedback

[FailedTheTuringTest]

Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student. Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student. Can a computer explain how to refine a research question or structure an argument? Sadly, many lecturers don't in fact give good feedback, but we should be looking for ways to enable lecturers to give better feedback, not accepting poor feedback as the norm.

Grades grammar not content. A.I. not ready yet.

[doug141]

"A director of writing at MIT Les Perelman says that because these robo-graders work according to an algorithm, it is not hard to find out what it values and thus beat the system. He found that if you write long essays with big words, even if they are nonsensical, you will score high. The algorithm does not like short sentences or paragraphs or sentences that begin with ‘and’ or ‘or’ nor is it enamored of sentence fragments. In other words, all the little rules that good writers will break to create a particular effect will cause your essay to be marked down.

Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically. The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.

For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants. “The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”

E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it. He could."

http://freethoughtblogs.com/singham/2012/05/03/how-to-fool-a-computer-grader/

Re:AI has not come far enough for this

[dgatwood]

Computers suck at even the most basic grammar checking. I once decided to try a bunch of online grammar checkers to see if they would be useful at providing a sanity check for my novels. I concluded that they report so many bogus mistakes that it simply wasn't practical to use their output at all. To test them, I fed them a block of content, some with intentional errors that the grammar checker should have caught, others with deliberately (or accidentally) tricky bits that should not have produced any errors.

• Upon seeing that, Joseph resolved to stop. Several grammar checkers thought "seeing that" was used idiomatically, and suggested replacing it with because. Upon because, Joseph resolved to stop. Yes. Much better.... Oh, and some others suggested that "Upon" is archaic.
• “Time to impact: seventy-six hours, fifteen minutes, twelve seconds,” the computer intoned. Oddly, several checkers suggested that "twelve seconds" was a fraction and should be hyphenated. Ugh.
• It's simple, really. There must be some mistake. Several spell checkers suggested "their". Others said that "must be" is passive voice. Uh, no, not every use of "to be" is passive construction.
• This isn’t your class anymore. Some checkers reported an agreement problem with "class". Huh?
• The room was dark, its plant-covered landscape shimmering green in the light of their headlamps. At least one checker suggested replacing "in the light of" with "considering". Eek!
• Joseph climbed up first. Several spell checkers suggested that "climbed up" is redundant. Apparently, their editors have never climbed down something.
• One checker even called "chided" archaic, but did not comment on the highly offensive swear word that I placed elsewhere in the sentence.

And so on. Heck, my phone doesn't even know the difference between "its" and "it's" and tries to auto-correct me into looking like I failed first grade English. And these folks expect me to believe that computers can feasibly help students learn to write better papers? Give me a break. Maybe in thirty to fifty years (*) we'll get there, but....

* Which many grammar checkers would probably suggest changing to "thirty-two fifty".