Slashdot Mirror
Automated System Developed To Grade Student Essays
RougeFemme points out this story at the Times about software that can be used to grade student essays and offer almost instant feedback. "Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the 'send' button when you are done and receiving a grade back instantly, your essay scored by a software program. And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade. EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."
One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.
- go to the site
- get a problem
- solve the problem
- type in the numerical answer
- right answer? go on to the next problem
- wrong answer? try again
The web app allowed maybe 0.5% margin for rounding error, and you got 5 tries before it failed you on that problem.
It sounds reasonable in the abstract, but in practice it was utterly wretched.
All learning is, at some level, an interaction--a conversation--between student and teacher.
Even if it is nothing more than a red check mark or a red X on a homework paper,
you have communicated some thing to some person and gotten some response.
You don't realize how important this is until it is gone.
With nothing but a machine to talk to, it stops being about learning.
It is just about satisfying the machine by whatever means necessary.
In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google.
This would reliably return the general formula for solving that problem;
plugging in the numbers that the web app had generated for your instance of the problem would then yield the correct answer.
By the end of the school year, I was telling him that if he didn't want to deal with the web app, he should use google to get his grade,
and if he wanted to learn physics, I would teach it to him.
Automated essay grading is going to be even worse.
There is no point writing prose unless a human is going to read it.
When I want to talk to machines, I write code.
Writing songs, that voices never shared...
-- Paul Simon
Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.
Seems like it's a small step from this to having computer algorithms that automatically write your paper for you too - then you can let it go through thousands of submit-edit-submit cycles until the scoring computer gives you a perfect score.
Kind of like the guys that came up with software to generate nonsense scientific papers and actually had a few accepted at conferences and journals.
Why I want to goto Harvard By P Q Student Up up down down left right left right B A
``Your grade is C. To improve your grade in the future, you need to do the following:
use 25-30 words per sentence; include more words from the wordnet entry for the topic of your essay; avoid simplistic or run-on sentences as measured by number of noun and verb phrases detected by our proprietary NLP tokenizer.
As a helpful reminder, our preparatory guides are available as a subscription service and include 100 practice submissions per week; only $29.95 per month."
"They were pure niggers." – Noam Chomsky
Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student. Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student. Can a computer explain how to refine a research question or structure an argument? Sadly, many lecturers don't in fact give good feedback, but we should be looking for ways to enable lecturers to give better feedback, not accepting poor feedback as the norm.
My friend wrote a story about his cat that was grammatically correct,and used big words, but made little to no sense. The auto-grader program told him he was approaching PHD level English. So he took his paper into school and showed it to the English teachers who reviled at it. He was like,"Show's what you know, the computer told me I'm university level."
God spoke to me
Every era has its snake-oil salesmen and their marks. Sadly, in this case it will not be the customers who suffer, but their hapless students.
Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety.
And you have been awarded 2500 extra HP.
Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically. The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.
For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants. “The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”
E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it. He could."
http://freethoughtblogs.com/singham/2012/05/03/how-to-fool-a-computer-grader/
The same as what they get hired and paid for now, research.
You notice the common statement is "publish or perish" not "educate or expire", right?
Computers suck at even the most basic grammar checking. I once decided to try a bunch of online grammar checkers to see if they would be useful at providing a sanity check for my novels. I concluded that they report so many bogus mistakes that it simply wasn't practical to use their output at all. To test them, I fed them a block of content, some with intentional errors that the grammar checker should have caught, others with deliberately (or accidentally) tricky bits that should not have produced any errors.
And so on. Heck, my phone doesn't even know the difference between "its" and "it's" and tries to auto-correct me into looking like I failed first grade English. And these folks expect me to believe that computers can feasibly help students learn to write better papers? Give me a break. Maybe in thirty to fifty years (*) we'll get there, but....
* Which many grammar checkers would probably suggest changing to "thirty-two fifty".
Check out my sci-fi/humor trilogy at PatriotsBooks.
It sounds reasonable in the abstract, but in practice it was utterly wretched.
No, the abstract does not sound reasonable: as with most things online you can always find bad ways to do it. I'm a physics prof working as part of a team to develop an open source, algebra capable question and content system. However even the current capabilities of something like Moodle (which is Open Source) is far in excess of what you describe. You can type in multiple "answers" to a problem and have the student get feedback and a partial grade if they get the problem wrong in a way that you managed to guess. Obviously if they find a new way to get it wrong then they will not get feedback though.
Commercial systems go even further with the student having the option to click on a help button which can break the question into steps for the student to complete in rder to guide them through to the right answer. This can be configured to give a grade penalty at the choice of the instructor - this is one of the features we want to add to an Open Source solution.
However even with current Moodle capabilities you can build a system that, I would argue, is better pedagogically for many physics problems (those with numerical or symbolic responses) than paper-graded assignments because, with an online system with some feedback and multiple attempts the student is encouraged to keep trying until they figure out how to get it right. This encourages them to think out the solution themselves whereas with a paper assignment they get one try and are then given the answer. To make this work though you need some means for students to come and talk to you and/or TAs to provide some help towards getting the right method. So you still need the student-teacher interaction but computers can provide a first line of contact and so let a teacher help more students.
That being said I find it exceedingly unlikely that this EdX system can work for written responses beyond checking that their english is good. For physics how can it possibly know that the statement "the Higgs boson has a mass of 140 GeV/c2" is wrong and "Dark Matter does not interact with photons" is correct? To be able to grade it will have to know a huge amount of information about a massive range of topics - and looking this stuff up on Google is not an option given all the crazy people and their wacky physics theories which they stick on a web page.
Any professors I've ever known or been taught by had their grad students doing the grading, anyhow.
Besides, what exactly are the professors being "freed up" for? Isn't their JOB to TEACH?
Oh, yeah, I'm thinking old school. Nowadays a professor's job is to find corporate grants...
I do not fail; I succeed at finding out what does not work.
I read the article and went to edX. At edX I signed up; but, I can not find out about this system. Quite Frankly, I am a teacher and I need my students to be writing more; however, I do not have the time to grade all of their papers so I have been assigning more objective homework that I would like.
A system like this may work as a first pass filter to do the bulk of the grading, allowing me to focus on identifying common problems and developing lessons based on common errors rather than tying myself down with a huge stack of papers. This would also benefit the students by providing them with more consistent grading and feedback.
This may not be what I go with; but, I would like to have a look at it. That takes me back to my question, can anyone point me to somewhere that I can get more information on this?
Face it, we're all going to get replaced by Expert Systems. They talked about this in the 80s and you didn't believe. 95% of us follow pretty simple patterns. There's damn little that most of us can do that a machine can't. Sure, there are exceptions. But most of us don't qualify, we just think we do.
You're being replaced. The real question is how are you going to deal with it? What do we do when 95% of us are completely unnecessary?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html
Archaisms are also not a grammatical problem. Even so, the checker chided him for chided, so why shouldn't he expect it for swearing?
Now assume you grow up somewhere where the teachers can't read and write proper English. We can call this place "public school in a non-educated community unless you're lucky." This kind of thing could be useful. The only problem is that people would not be smart enough to correct it, but it might still raise the bar.
All your GRE essays are evaluated by a machine and have been for years -- the e-rater. http://www.ets.org/research/topics/as_nlp/writing_quality/
The rating is also done by humans. It works well in practice and ensures that essays are graded fairly. If there is a significant discrepancy between the two ratings for a essay, that essay is examined further by another specialist. It prevents students from being victims of someone having a bad day at the office, and also does not encourage writing an essay to beat a machine.
The significance of the EDX news is not the concept of automated grading, it is that that such software is now free and opensource.
If I really care about what I am typing, I type it in word first. My students know I am the worlds worst typist when I type on the overhead; that is why I make an effort to use class material that I have written in advance. If I take the time to write well, which I am definitely not doing now, I can write, and have written, material suitable for publication
I will also add that I am a business teacher, I wrote well enough to make it through an MBA program (oddly enough, I was even called on to help edit for the ESL students). However, for all the amusement your jackass answer provided, it did not address the question. -- Grade F
According to Merriam Webster, yes, you can. That has been common English usage for at least as long as I've been alive.
Oxford English Dictionary agrees (#2), and uses the same example. So does Random House. And so on. So maybe you can't climb down something, but I hope your house never catches on fire, or else you're in serious trouble.
Check out my sci-fi/humor trilogy at PatriotsBooks.
The second part is a nominative absolute not a clause, so no, the comma is correct. That said, the construction is unusual enough that I've never been taught it in any English class, and I had to look it up just to figure out what it was called.
Perhaps the most famous example of this construction appears in the U.S. Bill of Rights (2nd amendment).
Check out my sci-fi/humor trilogy at PatriotsBooks.
It will be interesting to run some of the works of the great writers through this software. Joseph Conrad is well known for some quirky gramatical constructions. Proust is notorious for his long sentences and paragraphs. And would it make sense of Shakespeare's "This fell sergeant Death is strict in his arrest" ?
I expect all three of them would fail their exam. Oops, I started a sentence with "And".