Resources for Programming Course TA?

cndrr asks: "I'm a Teacher's Assistant for intro to Java at my university again this fall. The last time I taught, all the TAs had students turn in their assignments through email. I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. Has anyone else TAed and found a good solution that they would recommend?"

Comments

Don't do it.

[Andrew Sterian]

You'll waste as much time setting up and tweaking the system as you will doing it manually. Automatic submission and sorting into folders by course section is simple enough, but running the program and automatically grading the output??? That's madness.

Besides, trying to distance yourself from your students as much as possible by using technology is the exact opposite of what teaching is supposed to be about. If the students know that a real human will be reading the output and providing constructive feedback, they're much more likely to take it seriously.

Re:Don't do it.

[Anonymous Coward]

You'll waste as much time setting up and tweaking the system as you will doing it manually.

It really depends on the setup at his school. When I was a TA, we had one "head TA" who would take care of making automatic grading tools, and as a result the rest of us probabaly spent a lot less time doing tedious grading then we needed to.

Besides, trying to distance yourself from your students as much as possible by using technology is the exact opposite of what teaching is supposed to be about.

Agreed, but when grading programming assignments, part of the grade comes from being able to produce the correct output, and a good automated grading tool can make sure that students are graded consistently on that. Sure, you should still grade on style and give feedback and all that, but when strictly grading the output, who really cares if it was graded by a machine?

Re:Don't do it.

[kbielefe]

It's not madness. I actually preferred my classes with automated output checking, for the following reasons:

• It forces the teacher to make very clear program specifications.
• It forces the teacher to make a reference implementation of their own assignment, so they will see for themselves where potential problems are.
• Teachers provide some sample test cases and solutions, so that students can get instant feedback on the correctness of the program, even in the middle of the night. The final grading used those test cases, and some surprise ones to make sure the students thought about the entire algorithm and didn't just write to the test cases.
• The automated grading only ever counted for maybe 70% of the grade. I never had a class with automated grading where a human didn't check for style, implementation, simple mistakes, etc.
• The grader can still look at the output of failed test cases to provide constructive feedback, but they don't have to waste their time looking at it when the student got it right.
• I always knew exactly what a certain percentage of my grade would be, before I turned in the assignment.

As for implementation, all you need is a script that compiles the student's program, runs it against the test cases, then does a diff against the output from the reference implementation, and records how many test cases passed.

Re:Don't do it.

[feed_me_cereal]

If any student taking this class is good enough to circumvent the security for grading, they don't need to be taking the class and certainly don't need to worry about their grade.

If any student taking the class actually tried something like this, they would be severely busted as all the evidence would be right there in their code (so long as the system is like ours, where submissions are handled and logged externally from where they're tested).

Re:Don't do it.

[CliffEmAll]

As we try to teach our students, use the best tool for the job. In my opinion, that means only using automated testing when necessary. I am teaching a course (Programming in C & the UNIX Environment) the first time this summer. (I was fortunate enough to be granted a research assistantship when I was accepted to grad school, so I was never a grader.) I was offered the use of some automatic collection and grading scripts developed by someone in the department decades ago, but I decided to avoid them.

In my case, grading does not constitute a significant percentage of my teaching and prep work. I only have 5 students in the class. Although this system would have saved me some time, I remembered my own experiences with a professor when I was an undergraduate. I do not know whether or not this professor used automatic testing, but he was rather draconian about things being written exactly the way that he would have written them and that they produce exactly the output that he envisioned. Unfortunately, I have learned from trying to complete his assignments and from my experiences in industry that it is extremely difficult to completely and unambiguously write a specification document. Furthermore, I wanted my students to be able to use their creativity in their assignments -- I instruct them, for example, that their program should produce certain pieces of data as their output, and that these should be organized in some tabular form, but I do not require that their output look exactly like mine. Thus, automated testing would be of little use.

That said, you asked for resources, not to be talked out of it, and I assume your situation is significantly different from mine that the benefits of automation outweigh the drawbacks. The collection part is easy; either write a script they can call to copy their files to some specified destination, or write one that goes out at the submission deadline and copies from a standard location in their home directories. On UNIX, require the students to include a makefile with the all target. I am not very familiar with Java, but I presume they have a similar cross-platform approach. Assuming that all of the programs read from standard input and write to standard output, the testing part is also rather straightforward. Write your own implementation of the assignment. (An important step in any case!) Write some input files as test cases, capture your implementation's output for each of them, and then compare them to the output produced by the students' code with diff. For added security, do the entire compilation and testing steps in a chroot environment.

If you adopt this though, remember that you still have to look through everyone's code, even the ones that pass all tests. Hopefully you are grading for style in addition to correctness, and you need to verify that students aren't plagiarizing each other.

Good luck

Do the work.

[mukund]

I'd rather you manually grade it and provide valuable remarks to your students about their programs.

If universities were all about automated stuff, students can very well learn from course textbooks such as those prescribed by ocw.mit.edu by themselves. They go to university so that they can interact with their professors, get their amateur evaluated properly to shape their future work, and collaborate with their classmates.

Suggestion ...

[Monkelectric]

How about doing your job instead? Automated turn in is fine, automated grading is bullshit and a perversion of academic principles.

Re:Suggestion ...

[Dachannien]

If part of the assignment requirement is that, upon being run, your program should produce a specific output, I don't see where there's a difference between having a human run the program and determine whether the output is correct, and having a script do the same thing.

Sadly, automating any part of the grading process means you end up giving a lot less feedback to the students concerning their errors, but that's the prerogative of the instructor and assistants. In some classes, however, the student-to-TA ratio is so large that it would be impossible to grade every homework manually.

Also keep in mind that some TAs don't actually get paid for their efforts - and those who do usually make peanuts. At some schools, and in some academic programs, being a TA is a required part of your graduate degree. And since they're working on their own degree, the TAs actually have a lot of their own work to do in addition to grading hundreds of homeworks every week.

Do it, but don't do *only* it.

[feed_me_cereal]

For you and all the other people saying human eyes are better than a computer for grading:

They don't need to be a substitute, and they can help a great deal in doing a lot of the manual checking you would be doing anyway, as well as the organizational part. I'm a TA, and I use a script to check for just about every simple mistake I can think of, and then I go over every assignment by hand, with a printout of my test-script's results. The script doesn't so much grad ethe work, but point out any output that might not be exactly what I expect, over an assortment of tests. The students often comment that these scripts, which I also hand back with the assignments along with output from the scripts, help a lot to identify not only what their problems were, but what sorts of things they should have in mind when writing their programs. I then *carefully* go through their code by hand to insure they were using good style and didn't coincidentally happen to pass any test with code that isn't technically correct.

When you have over 60 students in a class and have to grade long programming assignments every week, these scripts are essential to getting my work done in a timely manner. My personal attention and comments are not replaced by my scripts, but are enhanced by my scripts. My time spent grading is made *more* effective.

Also (for cndrr): I might be able to provide the main part of the script we use, I'd just have to check with my instructor. The test scripts we use are fairly easy to write using it, if you don't mind doing it partially in scheme... (our department is in love with that language). Let me know if you're interested.

Re:Do it, but don't do *only* it.

[iMaple]

I was a TA for a programming course and we had managed to get a fairly automated system running very smoothly. It was a huge class with 7+ TA's and the submission script automatically alloted each TA his/her share (randomly to ensure fairness). The automated test script required TA intervention for student program. It complied and ran the program and compared it to the standard output and displaying the output with a fail/ pass result. If the test failed , it opened the souces files for the TA to review. And finally the TA would have large list of errors to chooses from (with optional comments), and predefined penalties. The final score was just written to a CSV text file (one for each TA).

This cut down the time required by almost 80% . The correct programs were easy to grade (automatically) and most programs with 'standard'(expected) errors did not take too long either. Once a while some one would have weird errors and the TA's would have small challenge finding those. It was the most fair system I could think of and took care of most of the drudgery without being unfair (the only important caveat was to make sure that test cases for the tester script were solid and the students couldnt cheat their way out, by pre defined responses).

Version control: your greatest bacon-saving device

[dsandler]

In COMP 314 (Rice University's sophomore-level programming & algorithms class, taught this past spring by Prof. Dan Wallach) we TAs solved this problem with Subversion.

An important part of real-world programming is teamwork; in 314 we embrace this and randomly sort students into groups of two for pair programming. The groups change for each project, and each project group gets a spot in a svn repository set up for the course. ACLs keep groups from peeking at one another's changes (for example, see this team list, which is actually just a slice of our Subversion access control file). Students were required to tag their submissions for each of the project milestones: specification, prototype, final; this was how students submitted code for these deadlines. From timestamps we could easily see which groups incurred "slip hours" for late turnins.

There were a number of reported incidences of lost work or conflicting changes which would have been disasters if not for svn, which saved their bacon. Groups that learned to check in early and often knew that accidental deletions or disk failures posed no threat to a successful project submission. A few enterprising teams even used tags and branches to help organize complex development efforts. In all, it was quite a successful adventure and we'll probably do something similar in the future.