Slashdot Mirror
Automated System Developed To Grade Student Essays
RougeFemme points out this story at the Times about software that can be used to grade student essays and offer almost instant feedback. "Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the 'send' button when you are done and receiving a grade back instantly, your essay scored by a software program. And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade. EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."
I wonder how these would do:
the postmodernism generator http://www.elsewhere.org/pomo/
the math paper generator http://thatsmathematics.com/mathgen/
FCKGW 09F9 42
Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically. The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.
For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants. “The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”
E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it. He could."
http://freethoughtblogs.com/singham/2012/05/03/how-to-fool-a-computer-grader/
I'm currently tutoring my daughter in statistics for the same reason. She's in college and while she's flipping through her homework appliication and her e-textbook, I'm flipping through my old statistics books, plus a couple of study guides I picked up. Also, sometimes the homework application is simply wrong. (Doesn't every tool/program have at least one bug?) My sister, a teacher, uses one - mandated by the community college where she teaches. Occasionally, she has to override the application so that she can mark correct problems that the application marked wrong. The students alert her, she checks and then overrides when the application is clearly wrong.
It sounds reasonable in the abstract, but in practice it was utterly wretched.
No, the abstract does not sound reasonable: as with most things online you can always find bad ways to do it. I'm a physics prof working as part of a team to develop an open source, algebra capable question and content system. However even the current capabilities of something like Moodle (which is Open Source) is far in excess of what you describe. You can type in multiple "answers" to a problem and have the student get feedback and a partial grade if they get the problem wrong in a way that you managed to guess. Obviously if they find a new way to get it wrong then they will not get feedback though.
Commercial systems go even further with the student having the option to click on a help button which can break the question into steps for the student to complete in rder to guide them through to the right answer. This can be configured to give a grade penalty at the choice of the instructor - this is one of the features we want to add to an Open Source solution.
However even with current Moodle capabilities you can build a system that, I would argue, is better pedagogically for many physics problems (those with numerical or symbolic responses) than paper-graded assignments because, with an online system with some feedback and multiple attempts the student is encouraged to keep trying until they figure out how to get it right. This encourages them to think out the solution themselves whereas with a paper assignment they get one try and are then given the answer. To make this work though you need some means for students to come and talk to you and/or TAs to provide some help towards getting the right method. So you still need the student-teacher interaction but computers can provide a first line of contact and so let a teacher help more students.
That being said I find it exceedingly unlikely that this EdX system can work for written responses beyond checking that their english is good. For physics how can it possibly know that the statement "the Higgs boson has a mass of 140 GeV/c2" is wrong and "Dark Matter does not interact with photons" is correct? To be able to grade it will have to know a huge amount of information about a massive range of topics - and looking this stuff up on Google is not an option given all the crazy people and their wacky physics theories which they stick on a web page.
http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html
I'm not sure what you meant, but
0.5 is most certainly not equal to 1/2. The latter is mathematically exact value, while 0.5 is not exact. E.g. 0.51 could become 0.5 when rounded.
regarding the difference between 1/2 and 2/4, I would accept only first one as an answer in a math test. While latter is also correct, it can be simplified. If you don't require the most simple form as an answer, then the answer to 5+4 could as well be 5+4.
"Isn't their JOB to TEACH?" Not completely, sometimes barely at all. At an R1, the typical humanities appointment is 25-40% teaching, 50% research, and the balance to service. Some faculty may only teach one class a semester, if they're administrating a department or subdivision of a department, or if they're running a onerous committee, like a hiring committee. At a teaching school, your "main" job is teaching, but you're still required to produce some token level of research and serve the university in other ways, such as by working on committees, being a public figure, and other stuff that you might not consider right away. So, at my job, at a teaching school, about 70% of my time goes into teaching. The rest goes into mandatory requirements to publish, present papers, do committee work, assist developing colleagues, and perform community service. (Note that in my annual performance review, I'm only allowed to indicate that teaching was a maximum of 60% of my effort, and this at a teaching school. This may be atypical, but I suspect it's not.) Now, in the sciences there are faculty with no teaching requirements. And in the humanities, at R1 schools, faculty get a year or a semester off periodically during which time they are expected to complete a research project, typically a book.