Slashdot Mirror
Tools for Automated Grading?
Dont tempt me asks: "As all teachers and students are well aware, it is back to school time. As a math/computer teacher, I am constantly looking for ways to automate repetitive tasks. The one that seems to take up most of my time is grading. As is typical for us nerds, I find myself looking at handwritten tests and thinking 'there's gotta be a better way...' Since I can't find any related open-source projects, I have been thinking about creating one. I have been toying with the idea of using OMR (Optical Mark Recognition) to make my own scannable multiple choice tests. Is anyone doing this? If not, where would be a good place to start? In addition to teachers, this could be a useful technology for questionnaires, or other processes that require manual data entry."
I taught astronomy at KU as a discussion section leader in 1991. We used scantron machines. These were #2 pencil IBM-card (~3 inches wide by ~8 inches tall) sized.
The machines could NOT have been expensive. Using them was dead simple. We (the section leaders) wrote several tests, and rearranged each test to have different orderings for the choices. Thus, on test version A-1, I had answers (a) Sun, (b) Moon, (c) Earth, then on A-2 I had (a) Moon, (b) Sun, (c) Earth, etc. Then, we looked at their version of the test, and put in the right key.
This kept cheating to a minimum; at the least they had to memorize the answers instead of the answer key. And, memorizing the answers was kind of okay in a sense since they at least paid attention to the subject material.
Unitarian Church: Freethinkers Congregate!
Autodata ExpertScan does this already. I've used it for surveys and forms, it should work for a test too, but admittedly, I've not tried it for that.
There's a legend about the time when OMR was a new technology: a student had marked every single checkbox. The software gave him 100% points, because it only checked whether the correct answers were marked. :-)
Use the low-tech solution. The answer key is created by using a hole punch on each correct blank. Then hold it up and mark in red any circle that isn't marked on the student's answer sheet.
Of course, you have to also look for students entering multiple answers (especially if they know how you're grading).
Once you scan the tests, you will have a graphic. You should be able to look at areas within that graphic and 'see' if it is dark. Take a look at imagemagick.org - an open source set of graphic manipulation tools that make this bone head easy in most of the popular languages. Make your life easy and place marks on the form so you can calibrate you fed the test in properly.
+++ UGUCAUCGUAUUUCU
Hi, I'm a CS student and always looking for ways to automate repetitive tasks. Is there any project out there that can do my work for me so that I don't have to?
Erik http://yakko.cs.wmich.edu/~rattles
> I have been toying with the idea of using
;)
> OMR (Optical Mark Recognition) to make my
> own scannable multiple choice tests.
Ooh! Ooh! Does this mean I can use a calculator when I take your test?
No? Hey, no fair!
assert(birth_date<time-86400)
Why not do automatic test creation as well. If you're going to do multiple choice, you can have a 'pool' of questions. Each question has a score, based on the percentage of students who get it wrong usually.
You could then automatically create tests with a certain percentage of 'A level' questions and so on. This would also let you more-or-less predict the curve... 10% will get an A, and so on.
Since the grading is automated as well, it would feed back into each question's score automatically
This may sound disturbing, but this is what the SATs do... those small sections at the end are just next years questions being tested
I also had a professor in college who did this, but it was through mental calibration over years. Yes, this does mean you can not give out the tests after for the students to review... but the test was surprisingly fair.
Is that most human beings are WAY too predictable in creating them- 75% of the answers will be B, unless it's a true and false question in which case 60% will be A.
My suggestion- get togehter with a programmer, and save class time by giving your tests on the web. You can code the tests in XML- the ASP or Java or python or PHP or whatever can randomize them for you so that no two students get the test answers in the same order. Students can log in from the computer lab to take the tests after school, or from home. Grading? All stored under their user name in your database, likely both as an already totalled value for the gradebook and a detailed view so that you can zoom in on what individual students are having a problem understanding.
I'm willing to bet there are not a few unemployed programmers out there willing to take on such a simple project as open source just for the resume fodder. I personally don't have the time, plus I'm not real good at web programming.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Depending on your testing needs/ability, you should easily be able to do optical testing of multiple choice or true/false tests (or have them submit tests via computer). A lot of more valuable tests require reading and thinking to grade. This still isn't fun.
But to make it less of a time sink, just multitask; grade during your commute or tv or laundry or workout or whatever.
- Take all the submitted assignments and collect them in a big pile.
- Throw the pile down a flight of stairs.
- Everything that makes it to the bottom gets an A. For each step above the bottom take off 5% of the grade.
- That's it.
A Multiplayer Strategy Game for Mac OS X, Windows, and Linux
This is something I've spent some time mulling over in my head, for no apparent reason, since I'm not even in the education industry at the moment. If you are relying on OMR, you are relying on a scanner. If you have access to a scanner with a bulk input, this might be ok, otherwise you're going to spend as much time scanning the pages as you would just grading them. If you have the resources in your classroom, setting up a randomized computer test for your students would eliminate paper usage (and save on your copying fees!) and provide automated correction, etc. but does not allow for the students to 'show work' (but neither do the multiple choice tests). One method I have had teachers use in the past (the long, distant past, come to think of it) is utilizing the students as graders. One of my teachers would have her next period's class grade the previous period's examinations. She had a ticket at the top of the tests which we ripped off when we turned them in that matched the test document to our name, which she kept. That cut down on people letting their friends slip by with wrong answers (or marking things wrong on foe's tests that weren't). She audited a few tests each round.
You say you got a real solution
Well, you know
We'd all love to see the plan
(The Beatles)
Assessment should be about the students knowing the material. Stuff like showing your work goes a long way. Math is the easiest to automate, but that would only show you that the student got the correct answer, not where the answer came from (like from a friend!).
To lower you work load, flip a coin on whether the students will hand in the work. If they aren't handing it in, trade with another student and grade it in class. Scantron only sends the message to your students that you are too lazy to look at their work, so why should they put any effort into it.
What, me worry?
When I was in high school, one of my teachers placed a small keypad/LCD on every desktop in the class (the seats were arranged as long booths, so wires weren't an issue). The LCD just prompted: "Answer to Question #X" and you typed in A, B, C, or D.
You could navigate back and forward, review your answers, and confirm completion. When you were done, the grade was instantly recorded and displayed to the student.
That was over 10 years ago! I see no reason why this shouldn't be possible (and inexpensive) in 2005. Furthermore, you could extend the idea to publish the grades for online viewing, e-mail the grade to students with a detailed answer key explaining why the selected answer was wrong, etc..
So many of the scantron-style tests had heavy-black marks on either or both sides of the exam. The machine use these marks to locate each question. (A big square would correspond to start/stop of the test and a line would correspond to an individual question or similar). Ths beats having to guesstimate where the questions were by feed-through rate & also led to high-speed machine-grading.
Well, a clever student decided to draw his own marks on the side of the exam. This managed to trick the idiotic machine into thinking all questions were one off & subsequently his test & all tests which were fed through after his had a low grade; the machine tested if question 1 had the answer expected of question 2 and so on.
Collect all of the tests, photocopy them, and allow students to grade themselves (walking through answers in class should take less than one class session & will help students understand what they missed). Cheaters can be caught by comparison to the photocopies.
If privacy isn't a concern, you can chop off the corner with the students' name & write a number on top & allow them to grade someone else's exam.
I've always had a strong dislike for multiple choice and true/false testing. Taking those tests is often more of an exercise in test taking than it is in the subject matter. A good test taker can eliminate a good portion of the answers right away and use fairly intuitive psychology to improve the odds of guessing correctly.
What ever happened to demonstrating competence in a field? Forget multiple choice and true/false. Ask your students to actually solve applied math problems or actually write some code (or pseudo code). Maybe you can't do as much testing that way and maybe you can't shorten the time it takes to grade the papers but at least you will be testing something worthwhile.
Sorry for the rant but after having survived more than a decade of "education" that consisted primarily of memorize foo and the regurgitate, I'm fairly traumatized by the horror that is the educational system. I learned orders of magnitude more useful information by simply reading everything and anything and trying to apply what I learned to my pet projects. I took one too many tests where I knew several multiple choice answers where justifiable and "right" depending upon unspecified information not contained in the question and having to guess what the test author thought the correct answer was. Multiple choice, true/false, and automated testing are big indicators of a "fast food" mentality and I'm firmly against that sort of foolishness. Grumble, grumble, etc.
Everyone I know uses scantron for multiple choice tests. If you're looking for some slightly more tech oriented solutions, here are some suggestions:
For multiple choice tests you could use off the shelf survey software like phpsurveyor or phpesp. Keep in mind these wouldn't necessarily be great at grading but it would let you easily analyze the test results question by question.
If you are grading programming assignments, you could develop your own testing suites using the *unit family of testing suites: nunit (.net), junit (java), phpunit (php), and I'm sure there are others. I think there's even some tools designed to evaluate test coverage like jcoverage (never used). Maybe you could have advanced students write test suites for the novice student assignments and evaluate/fix them with jcoverage... then use the test suites to automate testing of novice students.
Of course, there are only so many things that are easy to test in an automated fashion. You may have to give students exact specifications on interfaces and that may not always be desirable.
And make the students do the work...
;-)
Create three variations of the test with each question yielding a slightly different mathematical answer. You could have it as simple as adding all the answer so they match a certain sum or have an algebra equation at the end that they plug the values into.
All you have to do is check the variation number of the test and the result of the algorithm.
To shake it up and prevent cheating you can use variations on the test (either variables or questions), you can use variations on the algorithm at the bottom of the test.
A simple algorithm would be the sum or the product of primes. Each question's answer is mathematically related to a distinct prime. You could have a computer program break it back out and let you know which question they got wrong. Or maybe a derivative of the double entry booking system with questions matched in pairs to cancel each other out.
All in all, whatever method you use you are putting the weight on the student instead of individual grading the exams by hand. So any human error they make will still have to be compensated with a human's eye (and compassion) but it will get you 80-90% of the way there.
Overkill would be to write a program with a set of rules to generate the questions on the test, the primes, and the algorithm. Then you put in the unique identifier for the test and the one result and it spits out which answers they got right and wrong, but you'll have to work them out yourself if the student contests. How much do you trust your programming skills?
"Don't fear death... fear not living..." -me
In math, I think showing work is one of the most valuable aspects of the test. This shows a deep understanding of the problem at hand. Why not have final answers on a scantron like sheet and once marked wrong, review just the incorrect answers on the work sheet they turn in along with the scantron. Grading will still take time but you will instantly get to the incorrect problems and help the people who need it most.
Not sure if they have better or faster scantron machines today - but I would bet your school has something like it around somewhere.
Ask Slashdot - reinventing the wheel since the late 20th Century
I have mod points and I am not afraid to use them
What ever happened to demonstrating competence in a field? Forget multiple choice and true/false. Ask your students to actually solve applied math problems or actually write some code (or pseudo code). Maybe you can't do as much testing that way and maybe you can't shorten the time it takes to grade the papers but at least you will be testing something worthwhile.
Sorry for the rant but after having survived more than a decade of "education" that consisted primarily of memorize foo and the regurgitate, I'm fairly traumatized by the horror that is the educational system...
Look, obviously you're too insightful to be allowed to take part in this conversation.
And it's equally obvious that the original poster, "Dont tempt me" - who is too damned lazy to perform the most important aspect of his job, which, in this case, is to grade his students' math exams - has found for himself the perfect career path, that of public school educrat.
In twenty years, or thereabouts, he'll be raking in $250,000 per annum as some poor town's Superintendent of Schools.
So be happy for the guy.
And in the meantime, go take your Ritalin, like a good little plebe...
Not really a technological answer to what you were looking for, but I think it's worth mentioning. You say that you spend most of your time grading. I don't know how many students you teach, but I'm wondering if it's just a matter of you giving too much busywork to your students that you in turn have to grade. In my high school calculus class, my teacher assigned homework but he never graded it. At the beginning of every class, he make a quick pass around to see who had done it, and mark you off if you hadn't. He would then pass out an answer key for the assignment and go over any questions people had. His policy on homework was that it helps some people, and is just a pain in the butt for others. So the deal was that if your semester average was over 90%, there was no penealty for not doing homework. If your average was between 80 and 90, then you lost 0.5% for each homework not done, and so forth. As such, the only thing he graded for our class were the exams he gave every 3-4 weeks. So I'm just saying that perhaps the answer isn't to find faster ways to grade, but eliminating some of the pointless grunt work for you and for your students.
Maybe I've written some software that can help you. I'm a computer science teacher at the high school level, and I take my grading very seriously. But, I hate "busy work" and am equally serious about using technology to streamline my grading workflow.
Now, I assume you only have one PC in your classroom, whereas every student in my room has their own machine in front of them. And that's a huge advantage; if the work is in digital form to begin with, it grades much more quickly.
Anyway, I've done some things to improve my workflow:
It's not my stuff, but Jeff Burgard's Gateway to Mastery concepts are very, very good and encourage real accountability for learning in ways that keep the grading load manageable. (And he's got a lot for math teachers and doesn't assume you have access to more than one teacher PC.)
I don't have time to write more in this comment, but I do have more to say. Reply or email me if you're interested in discussing more.
Graham "Teach" Mitchell, computer science teacher, Leander HS
I'm working on an open-source Learning Management System called Sakai that has a tool for online test taking - you set up the test, tell the system what the correct answers are, then students sign in and take the test whenever.
There are obvious limitations to online test taking, but it *does* provide automated grading, and with the right institutional commitment it can have a positive impact on student learning.
-EvilMagnus
I'd simply suggest keeping away from automated grading, as it is a very bad way of grading.
I have taught math and CS for quite a few years, and I have never ever given an exercise in an examination for which the point was the final result. Most of the time, I do not even look at the final result. Of course, it is a lot more work to actually read what students write, how they get to the results they get, how they argue if they argue, &c, but that is what I am teaching them to do, so that is what I am grading.
I'm a highschool math / science teacher and for a while I've been playing around with moodle (http://moodle.org/). Though it may take a little to setup (PHP and MySQL are needed), it is a good system. Just make sure you have the power to use it.
All in all, it will allow you to make quizzes and lessons online that students can access. Questions can be auto sorted and even short answer questions with different possible answers. Its a beautiful system with the only flaw of facilitating a computer for each student to use. (I'm in an independant school so our kids have laptops at the ready, something we don't all have.)
The only other geek-oriented possiblity would be using scantrons or small LCD based devices, but from what I've seen nothing fits the bill. Possibly the best action might be changing how you grade and what work your students do (ie projects instead of tests and the similar). It works with a little imagination and there's alot less grading!
Works especially well if you don't happen to have a lesson plan that day.
This happened all the time in my high school. And despite that, I was still able to complete BC Calculus. Or whatever it's called now.
Suppose you have:
Submission 1:
print "hello world\n"
Submission 2:
print "hello "
print "w", "o", "r", "l", "d\n"
Suppose both submissions do the same thing when executed. An automated grader will note that, and score the submissions similarly. (Oops.) As a teacher, I've experimented with automated grading systems (including writing my own). For programming assignments, it seems to me that automation doesn't help either (a) me or (b) the students.
A lot of programs will transform the same inputs into the same outputs. Most of the possible programs are heinously bad, in terms of what I want the students to learn: good style. Getting the right answer (as output) is important, but how you get there is equally important.
What kind of teacher expects kids to do work that the teacher won't see and can't be bothered to deal with? If you want some fancy computer gizmo to do your homework grading for you, do your students get to have the same attitude and just copy and paste wikipedia articles?
Getting the right answer (as output) is important, but how you get there is equally important.
It sounds like correctness as potentially evaluated by a testing suite going off of a specification would be at least as important as code cleanliness/efficiency/style. Using regression testing suites would not solve all problems, but it seems like they would solve some. If you could work in regression testing into the curriculum somehow, there would be a value add for using the technology.
Perhaps the true value add is that you can see exactly which test cases fail while you are looking at a student's solution.
Another idea might be to let students run teacher supplied regression tests on their homework as they work on it. That way, they know when their solution is correct and when something is wrong. Maybe then you can focus more on evaluating style and maintainability and less time on correctness (because you will expect correct solutions).
I thought that the traditional way was to give them the test, collect them up, and throw all the papers down the stairs. Then the ones that fall furthest get the highest mark. Easy, and works for all non-practical subjects and some practical ones.
"...we should just trust our president in every decision that he makes and we should just support that." B.Spears 2003
On SourceForge, look up Webwork, Moodle, AIM (Assessment In Mathematics), and LON-CAPA, or look at STACK here. Webwork, AIM and STACK are primarily geared towards delivering math homework and I know that Webwork in particular can deliver fairly sophisticated problems. Webwork is entirely free and has several large free problem libraries, AIM is costly in that it uses Maple as one of its components, but STACK is similar and replaces Maple with Maxima or Octave (I think).
Moodle and LON-CAPA are more general. Moodle looks mangageable, but LON-CAPA looks huge and bulky to me and is probably very much a group project.
In the proprietary realm, I like the ALEKS online system for teaching basic mathematics, and their statistics looks good, too. No, I don't work for them or own their stock. I don't care at all for any of the other proprietary systems I've come into contact with, and I've seen many of them.
Warning: The intelligence of this post may be larger than it appears.
IMHO, your question is flawed. I would ask, "do multiple choice tests actually test students knowledge and problem-solving ability, or merely challenge their powers of deduction and 'gamesmanship'?" I used to ace standardized (SAT, ACT, ASVAB, and the like) and teacher-made MC tests all through my primary school career. Of course, once I got to University where I had to demonstrate actual learning, I got hammered - it took me years to learn 'how to learn' which is what is required to be truly educated, and also what is required to get the most out of a University education. Right now, there are probably few tools out there to help a public school teacher measure 'real learning', but I suggest you look for some or develop them - your students will be better served, IMHO. Good luck!
Just ask the kids this :
1.
You're in a desert, walking along in the sand when all of a sudden you look down and see a tortoise.
It's crawling toward you.
You reach down and flip the tortoise over on its back.
The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over.
But it can't.
Not without your help.
But you're not helping.
Why is that?
2.
Describe in single words, only the good things that come into your mind - about your mother.
Glonoinha the MebiByte Slayer
Memorization is stupid. Factual knowledge is a commodity. I can look anything up in seconds.
Anything worth knowing will be "naturally" memorized through use. Anything not worth knowing will be forgotten. Anything worth knowing that has accidently been forgotten can be looked up.
It's a beautiful system, and it's all natural!
Spoon not. Fork, or fork not. There is no spoon.
You need to learn to read. Or maybe reading comprehension (Though I'll admit that it would help if I was a better writer). You can look some stuff up in a book, but sometimes you should memorise that stuff before you need it.
When your clothes are burning off your back there is not time to find a book to look up "Stop Drop and Roll", so you have to memorize it in advance - in fact most people would never need to know what to do in that situation, but everyone should memorize what to do about it!
There is also value in memorization because it is brain exercise. You are unlikely to need to recite Poe's "the Raven", but that doesn't mean you shouldn't memorize a poem from time to time just to exercise your mind.
In fact your entire argument fails. You cannot look anything up in a book until you learn to read, and you cannot learn to read until you memorize your letters, phonics, and whatever else. True someday you will remember the rules of reading through use, but you need to memorize those rules before you can use them.