Tools for Automated Grading?

Dont tempt me asks: "As all teachers and students are well aware, it is back to school time. As a math/computer teacher, I am constantly looking for ways to automate repetitive tasks. The one that seems to take up most of my time is grading. As is typical for us nerds, I find myself looking at handwritten tests and thinking 'there's gotta be a better way...' Since I can't find any related open-source projects, I have been thinking about creating one. I have been toying with the idea of using OMR (Optical Mark Recognition) to make my own scannable multiple choice tests. Is anyone doing this? If not, where would be a good place to start? In addition to teachers, this could be a useful technology for questionnaires, or other processes that require manual data entry."

Comments

Scantron

[justanyone]

I taught astronomy at KU as a discussion section leader in 1991. We used scantron machines. These were #2 pencil IBM-card (~3 inches wide by ~8 inches tall) sized.

The machines could NOT have been expensive. Using them was dead simple. We (the section leaders) wrote several tests, and rearranged each test to have different orderings for the choices. Thus, on test version A-1, I had answers (a) Sun, (b) Moon, (c) Earth, then on A-2 I had (a) Moon, (b) Sun, (c) Earth, etc. Then, we looked at their version of the test, and put in the right key.

This kept cheating to a minimum; at the least they had to memorize the answers instead of the answer key. And, memorizing the answers was kind of okay in a sense since they at least paid attention to the subject material.

Re:Scantron

[Usquebaugh]

In my neighbourhood teachers and students are not allowed to have 'relations'.

Re:Scantron

[clifyt]

"If a person has correct answer but got it only by mistake then what good is it?"

Because by knowing how a test works, it tells a LOT about the student. There is a lot of science behind item response theory...for instance, a right answer on a specific item may tell you that the student has guessed. That's right...the reliability of the item may be that over 70% of the people that passed the test got it wrong, while 70% of the people the failed got it right. Quite a bit can be figured out about the student by what they got right or wrong.

In fact, the classic 3 parameter model guessing is a BIG part about the calibration:

http://www.rasch.org/rmt/rmt181b.htm

(difficulty and discrimination being the other two parameters)

Of course, this come more into play when you get into computer adaptive testing, but it still important on the standard multiple choice test (at least if its standardized).

As for math and computer science...I've never designed a CS test using multiple choice, but math is very useful. Again, adaptive testing with multiple choice is better, but its not like math is rocket science...as an instructor mentioned recently, math is not about memorizing formulas or otherwise, its about learning how to think creatively while breaking down a problem and coming up with a solution. If you can figure out the answers that don't belong and figure out the rest without having to do the problem, you've demonstrated knowledge about the ideas behind the problem in front of you...its just like the complaints we got about an essay rater we were working on several years back (it graded structure, not content)...people were submitting brilliantly written nonsense and then complaining that we were scoring it high -- the fact that someone was able to figure out ways around the writing rubric to game the system meant that they pretty much deserved the high scores they received because they were good writers -- regardless of the nonsense (unlike my own writing...this is not a good example of such writing).

All in all, this all depends on what you are after -- demonstration of mastery or for a diagnostic (or maybe to prove your students know as much as the students down the hall or the school across the state). Luckily, most educators are taught testing methods before they leave school and probably have a little more clue as to what they are looking for than students that think a test may just be multiple choice bullshit (though that could be considered negative face value :-)

clif (speaking for myself and not my office)

How about this?

[iseth]

Autodata ExpertScan does this already. I've used it for surveys and forms, it should work for a test too, but admittedly, I've not tried it for that.

Ask Slashdot: Automated Homework

[mr_rattles]

Hi, I'm a CS student and always looking for ways to automate repetitive tasks. Is there any project out there that can do my work for me so that I don't have to?

Not just grading..

[thefirelane]

Why not do automatic test creation as well. If you're going to do multiple choice, you can have a 'pool' of questions. Each question has a score, based on the percentage of students who get it wrong usually.

You could then automatically create tests with a certain percentage of 'A level' questions and so on. This would also let you more-or-less predict the curve... 10% will get an A, and so on.

Since the grading is automated as well, it would feed back into each question's score automatically

This may sound disturbing, but this is what the SATs do... those small sections at the end are just next years questions being tested

I also had a professor in college who did this, but it was through mental calibration over years. Yes, this does mean you can not give out the tests after for the students to review... but the test was surprisingly fair.

Re:Not just grading..

[big ben bullet]

This is actually an excellent idea.

But what really matters is the content. My wife teaches the courses in primary education (6 to 12y olds). She has a hard time making up questions for each course in each grade. If there was an openly licensed 'questions base' (maybe even a wiki) this would help her alot. She would be happy to contribute her existing material ;-)

You might want to start such an initiative along with the actual open source application.

Ohw... and *please* mind the localisation so there's a potential for every language speaking community in the world, and not only the english speaking community.

Re:It was a dark and stormy night...

[ragnarok]

I had a teacher in high school who graded OMR like sheets manually with an overlay that only had the holes cut for the correct answers.

Several times if I didn't know the answer I would mark more than one (you don't want to mark all of them, it stands out too much) and I always got credit.

The teacher seemed like a smart guy too, I wonder if he was doing it intentionally to see if people would figure it out.

Stairs method

[BortQ]

It works better for 'fluffy' subjects involving opinionated essay writing, but here's the method:

- Take all the submitted assignments and collect them in a big pile.
- Throw the pile down a flight of stairs.
- Everything that makes it to the bottom gets an A. For each step above the bottom take off 5% of the grade.
- That's it.

If you can automate, should you be grading?

[gozar]

My best math teacher assigned homework every night. She would flip a coin the next day on whether it would be for a grade or not. So 50% of the time she wouldn't have to grade anything.

Assessment should be about the students knowing the material. Stuff like showing your work goes a long way. Math is the easiest to automate, but that would only show you that the student got the correct answer, not where the answer came from (like from a friend!).

To lower you work load, flip a coin on whether the students will hand in the work. If they aren't handing it in, trade with another student and grade it in class. Scantron only sends the message to your students that you are too lazy to look at their work, so why should they put any effort into it.

Re:If you can automate, should you be grading?

[fm6]

Your teacher was a smart person. But she was lucky the bureaucracy let her get away with it. And that none of the students went whining to their parents about being made to do homework that wasn't graded. With school more bureaucratic and political than ever, I doubt if that would be allowed today.

Scantron Exploit

[Noksagt]

So many of the scantron-style tests had heavy-black marks on either or both sides of the exam. The machine use these marks to locate each question. (A big square would correspond to start/stop of the test and a line would correspond to an individual question or similar). Ths beats having to guesstimate where the questions were by feed-through rate & also led to high-speed machine-grading.

Well, a clever student decided to draw his own marks on the side of the exam. This managed to trick the idiotic machine into thinking all questions were one off & subsequently his test & all tests which were fed through after his had a low grade; the machine tested if question 1 had the answer expected of question 2 and so on.

Re:Scantron Exploit

[bluGill]

Back when I was in school, 15 years ago, my teachers were onto such tricks. It isn't hard to look at scores are you write them into your book. The teacher already knows 'Suzie' is smart, often getting a perfect score, so if he[1] misses most of the questions it is time to re-examine things by hand.

The most popular way to cheat was to mark the little box at the top that set this sheet to the master, which would re-program the machine to take your test as the correct answers.

None of these tricks were hard for a teacher to catch (if you knew about them it was easier, and the principal made sure they knew). Once you catch a student doing this you just write zero in for his score and re-run the tests.

[1]we miss you Johnny Cash

Multiple Guess

[99BottlesOfBeerInMyF]

I've always had a strong dislike for multiple choice and true/false testing. Taking those tests is often more of an exercise in test taking than it is in the subject matter. A good test taker can eliminate a good portion of the answers right away and use fairly intuitive psychology to improve the odds of guessing correctly.

What ever happened to demonstrating competence in a field? Forget multiple choice and true/false. Ask your students to actually solve applied math problems or actually write some code (or pseudo code). Maybe you can't do as much testing that way and maybe you can't shorten the time it takes to grade the papers but at least you will be testing something worthwhile.

Sorry for the rant but after having survived more than a decade of "education" that consisted primarily of memorize foo and the regurgitate, I'm fairly traumatized by the horror that is the educational system. I learned orders of magnitude more useful information by simply reading everything and anything and trying to apply what I learned to my pet projects. I took one too many tests where I knew several multiple choice answers where justifiable and "right" depending upon unspecified information not contained in the question and having to guess what the test author thought the correct answer was. Multiple choice, true/false, and automated testing are big indicators of a "fast food" mentality and I'm firmly against that sort of foolishness. Grumble, grumble, etc.

Re:Multiple Guess

[bluGill]

There are times for memorization, and there are times when you need to go farther. In math you always need to go farther and understand the concepts. In shop you must get 100% (no misses allowed!) on the safety memorization test before you are allowed to take the test on real tools. Of course knowing that the margin of safety on some saw is 10 inches doesn't mean you won't put your fingers closer to the blade, but if you don't know that number it means you will.

Memorization is important. Do not overlook the value of memorizing some things (even if you will never need to know that poem once you pass the class, it is still useful to do it). Though overall I agree that there is too much focus on memorizing (mostly on the wrong thing!) in school, that doesn't mean you can get rid of memorizing.

Re:Multiple Guess

[SlamMan]

At some level you have to think about what's logistically possible for a teacher. Say you teach 8th grade science. Your school has 8 mods (or periods, whatever they call em these days), 6 of which you teach students during, 1 for your lunch, and 1 for "planning", which is usually spent deal with school bureaucracy, possibly calling parents, or once in a while doing actual lesson plans for the next day.

Each of your 6 classes has 30-35 students.

Every time you give an assignment to you students, you get 180-210 papers back to grade. Thats 210 papers back about every day. How do you find the time to grade 210 papers every day?

Now imagine each of those papers was a free response, encouraging you to think, and show ability to use what you've learned.

Now add in how much time it would take to come up with those questions, for 180 school days.

suggestions

[Darth_Burrito]

Everyone I know uses scantron for multiple choice tests. If you're looking for some slightly more tech oriented solutions, here are some suggestions:

For multiple choice tests you could use off the shelf survey software like phpsurveyor or phpesp. Keep in mind these wouldn't necessarily be great at grading but it would let you easily analyze the test results question by question.

If you are grading programming assignments, you could develop your own testing suites using the *unit family of testing suites: nunit (.net), junit (java), phpunit (php), and I'm sure there are others. I think there's even some tools designed to evaluate test coverage like jcoverage (never used). Maybe you could have advanced students write test suites for the novice student assignments and evaluate/fix them with jcoverage... then use the test suites to automate testing of novice students.

Of course, there are only so many things that are easy to test in an automated fashion. You may have to give students exact specifications on interfaces and that may not always be desirable.

Do less grading!!

[MagicDude]

Not really a technological answer to what you were looking for, but I think it's worth mentioning. You say that you spend most of your time grading. I don't know how many students you teach, but I'm wondering if it's just a matter of you giving too much busywork to your students that you in turn have to grade. In my high school calculus class, my teacher assigned homework but he never graded it. At the beginning of every class, he make a quick pass around to see who had done it, and mark you off if you hadn't. He would then pass out an answer key for the assignment and go over any questions people had. His policy on homework was that it helps some people, and is just a pain in the butt for others. So the deal was that if your semester average was over 90%, there was no penealty for not doing homework. If your average was between 80 and 90, then you lost 0.5% for each homework not done, and so forth. As such, the only thing he graded for our class were the exams he gave every 3-4 weeks. So I'm just saying that perhaps the answer isn't to find faster ways to grade, but eliminating some of the pointless grunt work for you and for your students.

Try Moodle and such.

[Paladin2ez]

I'm a highschool math / science teacher and for a while I've been playing around with moodle (http://moodle.org/). Though it may take a little to setup (PHP and MySQL are needed), it is a good system. Just make sure you have the power to use it.

All in all, it will allow you to make quizzes and lessons online that students can access. Questions can be auto sorted and even short answer questions with different possible answers. Its a beautiful system with the only flaw of facilitating a computer for each student to use. (I'm in an independant school so our kids have laptops at the ready, something we don't all have.)

The only other geek-oriented possiblity would be using scantrons or small LCD based devices, but from what I've seen nothing fits the bill. Possibly the best action might be changing how you grade and what work your students do (ie projects instead of tests and the similar). It works with a little imagination and there's alot less grading!

Re:Make them grade each other...

[greenplato]

Besides concerns about cheating, that practice is likely a violation of the student's right to privacy. The Family Educational Rights and Privacy Act prohibits the release of any information from a student's education record. Because test scores make up a large part of any final grade, sharing these results with other students is probably a no-no.

This happened all the time in my high school.

Yeah, high school is a good place to stomp all over kid's rights. Somebody has to put them in their place...