Slashdot Mirror
Resources for Programming Course TA?
cndrr asks: "I'm a Teacher's Assistant for intro to Java at my university again this fall. The last time I taught, all the TAs had students turn in their assignments through email. I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. Has anyone else TAed and found a good solution that they would recommend?"
You'll waste as much time setting up and tweaking the system as you will doing it manually. Automatic submission and sorting into folders by course section is simple enough, but running the program and automatically grading the output??? That's madness.
Besides, trying to distance yourself from your students as much as possible by using technology is the exact opposite of what teaching is supposed to be about. If the students know that a real human will be reading the output and providing constructive feedback, they're much more likely to take it seriously.
I'd rather you manually grade it and provide valuable remarks to your students about their programs.
If universities were all about automated stuff, students can very well learn from course textbooks such as those prescribed by ocw.mit.edu by themselves. They go to university so that they can interact with their professors, get their amateur evaluated properly to shape their future work, and collaborate with their classmates.
Banu
Automatic grading apps are, uh, somewhat prone to breakage. I know at least one university that uses them found that certain students just wrote programs that did sneaky tricks rather than the assignment at hand (fork off a child process that sleeps for a bit then overwrites the grade file, then just output whatever). Perhaps more effort to do a one off program that way, but far less in the long term.
Said students got to keep the grades, too...
Are you sure it's a good idea to let students execute arbitrary code on your unattended machine?
I mean, I know *I* would get ideas...
Proud member of the Weirdo-American community.
"I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. "
Grade what? If they can automate it, then you're out of a job.
I had a class last fall in which the prof used an automatic grader. He had a couple page document describe all the nit-picky formatting requirements we had to follow so that his automatic grader could run it. And, if we missed even a signle req, we got docked for it (generally in multiples of 10% of the assignment). Many people scored below-average to failing on their first couple of assignments because of this system, even though their code was flawless. So, please, please don't use automatic graders. They're more trouble than their worth, and you students will appreciate and specific comments you give. Even if it runs perfectly, comments can still sometimes be useful to make it even better.
How about doing your job instead? Automated turn in is fine, automated grading is bullshit and a perversion of academic principles.
Religion is a gateway psychosis. -- Dave Foley
Automatic grading is unethical.
I've TA'd programming classes and I wouldn't suggest automatically grading anything. The classes I've graded for haven't had very difficult assignments and it's usually very easy to generate the correct output. I let them know that I *expect* them all to have correct output and I grade based on their code.
Setting up a site sounds like a good idea, though, and having it run and compare the output is a good idea that will save you a lot of time, but that should just be the first step.
We have an automated submission system that requires the program to compile for subnission. It also has the capacity to run the program and check its output (command line stuff only). It also ensures that the work is turned in on time and that it contains all of the files required (unless of course we are given an assignment giving us free reign on the class structure etc.) Though, the TA does go through all of the code levaing coments where appropriate and those comments have been a massive help. Overall I like my end of the system, and I can see how it could be likede on the TA's end. With those considerations, if you believe that you wouldn't waste more time setting a system up than you would save with it ,I would go for it.
Have any of the previous posters actually attended a large university in the computer science program?
Every class I've taken at UCSD does some sort of automatic grading on the programming assignments. It would be impossible to grade everything otherwise... Last quarter we wrote a compiler that ended up being a few hundred KB of source. There were over 200 test cases ran by the autograder.
All of the assignments are turned in from a unix prompt using a TURNIN command after prepping for the appropriate class.
The class I'm working on now (Operating Systems, using NACHOS) even has autograder() methods in the skeleton of the code that are used during grading...
I agree for a first class some feedback and hand grading might be necessary, but even with autograding you can add comments after looking at the code that causes similar test cases to fail.
Does your university have a subscription to one of those software scripts, such as WebCT? It's usually used to post lecture notes, assignments, and provides a bulletin board for questions. It also has a submission section (for WebCT) that allows students to submit their assignments by a certain time. Then you can download them all, each file separated into student accounts, and compile each one and grade.
Some amount of automation is helpful to avoid being swamped, but I don't think that a fully automated process can do the job. I require my students to generate an executable jar for submission, but also provide unit tests to them for the early assignments. Stage one of grading is to run a script which runs each of the jar files against the unit test. I then print the programs and mark them up for formatting and style, paying particular attention to those who ran afoul of the unit testing stage. Later in the course I start providing a spec and making them write their own unit tests, but I still use my own unit test modules for grading.
I agree. At U-M we pretty much autograde everything. As an added bonus, most classes require you to not only submit the source to your program that compiles and then passes the tests, but you're required to submit testcases that are then run against intentionally buggy programs written by the instructors.
This way you get a score not only based on how well you completed the task, but also your ability to write high-coverage test cases--and considering how many corner-cases get exploited these days, teaching kids to write their own test suites sounds like a great idea to me.
As for the malicious code factor, since all of the testing is done on a remote machine and you receive reports by e-mail, you'd have to either A) have inside knowledge of exploitable holes (in which case you're proabably a TA already) or B) write code that performs recon and reports back to you. Since I know our TA's at least scan through people source code submissions, writing recon code would probably get caught. Our school's Honor Code would not be very forgiving at that point etiher, you'd likely find yourself looking for another school to complete your degree at.
Bottom line: grading by hand is great until you've got 150 pipelined CPU simulators to go through by hand. Automated grading is an easy solution.
For you and all the other people saying human eyes are better than a computer for grading:
They don't need to be a substitute, and they can help a great deal in doing a lot of the manual checking you would be doing anyway, as well as the organizational part. I'm a TA, and I use a script to check for just about every simple mistake I can think of, and then I go over every assignment by hand, with a printout of my test-script's results. The script doesn't so much grad ethe work, but point out any output that might not be exactly what I expect, over an assortment of tests. The students often comment that these scripts, which I also hand back with the assignments along with output from the scripts, help a lot to identify not only what their problems were, but what sorts of things they should have in mind when writing their programs. I then *carefully* go through their code by hand to insure they were using good style and didn't coincidentally happen to pass any test with code that isn't technically correct.
When you have over 60 students in a class and have to grade long programming assignments every week, these scripts are essential to getting my work done in a timely manner. My personal attention and comments are not replaced by my scripts, but are enhanced by my scripts. My time spent grading is made *more* effective.
Also (for cndrr): I might be able to provide the main part of the script we use, I'd just have to check with my instructor. The test scripts we use are fairly easy to write using it, if you don't mind doing it partially in scheme... (our department is in love with that language). Let me know if you're interested.
"Question with boldness even the existence of a god." - Thomas Jefferson
Is very much automated-submission and grading -- or at least was from 00-04. Some professors do the grading by-hand (i.e. they run scripts by hand) but the majority of classes I took where you turned programs in, there were scripts that graded you that you could run part of yourself. (i.e. there were "public" and "private" scripts that used the same interface.)
In a large part, it depends on how big your class is, and if you plan on continuing teaching. One professor, (whom I idolize) has been doing this long enough to have set up an extensive and all-inclusive framework. Can't go (too badly) wrong when you roll your own, but YMMV.
-b
When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi. (Larry Wall)
I always made the grades up off the top of my head at the end of the semester, and found it much more fitting with academic tradition and principals.
hehe, jk.
www.GrenadeHop.com
If your students are all writing CGI-style programs, maybe reading and writing a file or two, why not set up a Linux or FreeBSD box for them? Open ftp and ssh for them and require them to build their programs there. Your requirements for each class will be that they compile and run their programs on your server, do a run under script(1), then hand in all source, output, data and log files via a script you wrote called handin assignment [file...], which copies [file...] into your directory $assignment/$studentid/[file...]. Since handin is suid, write it carefuly: It should contain no mkdir or similar commands.
This will be good for everyone. They will pick up the rudiments of Unix and using remote computers. You will pick up the rudiments of Unix administration, and all your class stuff will be in one convenient directory tree per assignment.
Just for fun, you could write yourself a little program called comment that lets you mark up their source code -- their stuff will be in mono and your comments in mono italics . Print 'em up and give them back at the beginning of class.
Make sure the students have access to a good variety of editors, including a couple of dumbed-down ones like pico.
The downside is that you'll spend a day or two going over the rudiments of Unix, and have to do some early handholding for them, but it will pay off in the end.
This is not my sandwich.
you are getting paid to mark the papers, to identify problem areas and to solve those problems for the students, you have to provide feedback to them. if you want to be lazy and skip on doing that please dont ever accept a teaching responsibility, i had to teach several friends how to program in java since our lecturer was lazy and incompetant. somewhere someone will have to take the responsibility. work out how much those students pay per class and per evaluation and you will quickly realise you are trying to screw them over. i am a student.
This is my sig.
In COMP 314 (Rice University's sophomore-level programming & algorithms class, taught this past spring by Prof. Dan Wallach) we TAs solved this problem with Subversion.
An important part of real-world programming is teamwork; in 314 we embrace this and randomly sort students into groups of two for pair programming. The groups change for each project, and each project group gets a spot in a svn repository set up for the course. ACLs keep groups from peeking at one another's changes (for example, see this team list, which is actually just a slice of our Subversion access control file). Students were required to tag their submissions for each of the project milestones: specification, prototype, final; this was how students submitted code for these deadlines. From timestamps we could easily see which groups incurred "slip hours" for late turnins.
There were a number of reported incidences of lost work or conflicting changes which would have been disasters if not for svn, which saved their bacon. Groups that learned to check in early and often knew that accidental deletions or disk failures posed no threat to a successful project submission. A few enterprising teams even used tags and branches to help organize complex development efforts. In all, it was quite a successful adventure and we'll probably do something similar in the future.
In one class I had a prof who would sit down and read the source and grade based on what he saw. He never ran a program but could (almost) always tell you exactly how the program would behave. The class was rather impressed by this. Most of my other classes did the auto submit and grade thing.
Except.... He gave a failing grade to one program I wrote - he said it wouldn't produce the desired output. We argued for a while and I finally agreed but insisted we run it to find out for sure. It did exactly what it was supposed to do but neither of us could figure out why. The program involved recursion.
-E
Considering it's an intro class, I am sure lot of students will unintentionally end up with similar programs. http://www.cs.berkeley.edu/~aiken/moss.html
I don't know about everyone else, but the instructors that I've had for programming will read through the code first before compiling and running it.
There's 3 reasons for this. 1. To make sure you followed coding standards. 2. To make sure you weren't doing something destructive. 3. To make sure you were actually using the programming constructs that the instructor wanted you to.
To expand on 3, for simple programs like you run into in an Introduction class, it's possible to do a recursion assignment using iteration. Or to use if/else if/else where you were asked to use switch/case. Reading the code will catch these things. Just checking the output won't.
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
Berkeley's been doing this sort of thing for years and years now. I'm sure someone there can help you identify software that can do what you're looking for. Unfortunately, I never TA'ed while attending, so I don't know what they use.
The Tufts Department of Computer Science use a homemade software called "provide" (Professor Alva Couch wrote it) It is a simple online grading utility. This program allows simple submission and online grading of electronic assignment submissions in Computer Science and Computer Engineering courses. It is specifically designed to be easy to use by graduate students working on their own, to leverage faculty resources and decrease faculty workload in managing grades. In production in EECS, not publicly distributed (1998-present).
Assignments can be graded manually or automatically. The Intro to CS and Data Structure instructor usually write an automatic grader for assignments. Programming style is graded manually. For algorithm, graphics, and theory assignments, students submit them via provide. The TA then grade the assignments, put in the grades, and students run a command "progress" to see their grade of their assignment. Many students say that the process work quite effectively.
One of the grad students at my school developed this system as his masters project...
Don't know if it's of any help but you might try contacting a Dr. Sun since he was the advisor.
It doesn't automatically grade the homeword, but it does let you set time limits and since it uploads directly to the server, you won't have to deal with students saying they got the wrong email address.
The system, currently, is only for homework and old code retrival but many other things are planned for it.
We had a T.A. automatically grade assignments at the school where I worked. He wrote an automatic scanner to count the number of words in the comments, and to automatically run the programs and compare the output. He then assigned grades appropriately, computing the grade only on the number of words in the comments.
He then quite proudly stood up in front of the class as explained to them what he had done. He had no idea what was about to happen. Essentially, everyone in the class simultaneously worked out that it didn't matter what they put in the comments. They just needed to boost the word count. The class turned red and almost rioted. The professor's jaw dropped.
The professor recalled the assignments and had them regraded by a different T.A. After that day, the original T.A. was never allowed to mark another assignment.
How about not being a lazy bastard and doing what you're paid for?
why? forty-two.
I'm a student at the University of Technology in Vienna, Austria. Since we're lots of students it wouldn't be feasible grading the assignments of a few hundred people by hand.
Some courses use an online grading facility where students just log in, submit their assignments over a web interface and receive an email how they scored. Some other courses give you a shell account on a UNIX box. Your programs are expected to compile in this environment and you can submit subdirectories of your home directory using tools they have written themselves. This concept is used from simple java classes in Inruduction to Programming to simulated CORBA stock trading clients and servers in Distributed Systems.
As far as I have seen everything behind the scenes is fully automated. Some systems just run your class in a sandbox with a java security manager, feed the application several test inputs, compare the output and error handling to the specifications of the assignment and assign scores based on that. Others invoke your classes via Reflection and run about 20 - 50 JUnit test cases against some components as well as the entire application. You get an email with simple descriptions of the test cases, whether the specific test case went through OK or failed and if available exceptions that were triggered.
As for cheating: they used different test cases all the time, some were to some extent randomized so that static output wouldn't work. They also use similarity checking tools to find suspiciously similar submissions and asked students whose submissions got flagged to come and explain the code to them (afair there was a discussion about such similarity checking tools on ask slashdot not too long ago). The tests of such courses contain simple questions about the environment and tools they should have used to see whether they have written the code themselves.
I'd recommend you to write such an automated grading tools yourself. It shouldn't take more than some simple JUnit tests and a bit of perl code to set up a clean environment, run the tests, evaluate the results and write them to a database. Just don't forget to set up a java security manager so that nothing can compromise the test environment and restore any resources like files and databases the submissions have to access to their default state in case someone screwed up. Or you could do it the academic way and get grad students to do all the work.
:/- spoon(_).
At RIT we used a program called "try", developed by one of the professors there. You can download it from his page: http://www.cs.rit.edu/~kar/software.html
Search first, ask questions later.
I am also a TA for an introductory course to JAVA. I am not sure how many others here are but I have found auto-marking to be essential. I am a single TA and I have 180 students. Each assignment has 3-4 substantial programs. Trying to mark them all by hand would be impossible given time limitations. I do have to agree with others though in that feedback is always nice. Therefore what I have done is I mark half of each assignment and the other half is automarked. I have automarking compare output given input and allocate this to a mark called 'Correctness'. I then mark certain functions (usually the harder ones assuming that if they could do these ones they could do the others). This way the students still get feedback. I also encourage them to come in and see me if they have questions so we could go over it together in more depth. The test cases are also put online (after all assignments are in).
The submission system I use however is unix based. So the first or second tutorial/lab we run is all about how to submit your assignment through unix. We have had a website submission system in the past but we found this negativly affected some students who in third year had yet to touch unix at all.
Just as a warning though, I have found with introductory courses, no matter how many times I tell them to output something in a specific way, the students will not do it. I usually have my automarking scripts do a 'diff' with multiple possible answer or ways of formating an answer. They will still fail all the test cases forcing me to hand mark anyways. I have even given them Strings in provided code to use and they will either change the output, ignore it, or have extra output. Sometimes I just give them 0 in the correctness area. The main reason for this is the students don't care. Some are just in the course because there program requires them to have one CS course.
Therefore, if you plan on doing this then you might want to consider having final results stored in a variable, and have them make a public get method. That way you can simply redirect standard out to something like /dev/null when you run their code to get the final result, and then set it back to the screen and use the get method to output just the result. This only works in the case of computing numbers though.
Anyways, good luck with whatever you decide to do
If you're TA'ing, you should have the Java-fu to do it. It's not hard. We used one for our classes, and we worked out a really nice compromise.
When the program was submitted, it was compiled and trivially rejected if it failed to compile. This is worth the price of admission right here, since people whose code "compiles on their machine" get multiple chances to recognize the error of their ways. (Some people think students should get points taken off for this; that's a philisophical discussion I won't go into.)
We ran it in a separate TA account so that if one student does "rm -rf *" he doesn't hurt anything before he's escorted out of the department.
There are a suite of tests that are run on the program, and a score is given, generally 10 points per test. That's the "base score." Now, if your students are good and your assignments are fair, with multiple tries and different outputs at least half the students will get 100. The point of the "try until you get it right" method is to guide the students to the right answer.
Then the TA hand-grades the non-100s in the traditional way, but he doesn't try TOO awfully hard, because the students get the opportunity for a "regrade." The TA steps through the grading with the student in person, explaining what went wrong.
This is a great solution for courses that use homework to get students to explore the language and possible solutions; it may not be the best solution for classes where the grades for homework are significant.
In my introductory engineering/programming classes at Purdue University, the professors/TAs use a program called MOSS to check for code plagiarism. Apparently it's free to use; just email them to get an account.
At Drexel University, our CS department has students submit assignments by checking code into an SVN repository, too. The advantages of this are many.
I don't know what your qualifications are to be a TA for this course. I was a TA for an intro to object oriented programming course using Java during my junior year. I also don't know what the difficulty level of your course is, but that should not change my opinion. 1. The students in this course will most likely put alot of time and effort into these assignments. 2. I would hope that YOU could implement a system like the one you described with an equivalent amount of effort. 3. I would hope that your STUDENTS, after taking this course, given adequate guidance and framework, could implement a system like the one you described. If you are suggesting this system as anything other than a progress indicator for students to test early drafts of their work, I would seriously reconsider your role as a mentor/teacher/guide. At this point in a student's education, it is most important to reinforce good habits. As far as some of the other comments posted here are concerned, I am worried about the level of education others are receiving. Get your acts together folks! Students will live up to, or down to your expectations of them.
why not just have each student submit a branch to subversion, then have a bunch of test cases that pass or fail?
You could then use ANT to pull each version in series, compile it, grade it, etc.
Stanford's CS 106a class is famous not only for being a popular class (1/3rd of all Stanford students take it) but for producing amazing TAs as well. When I took the coursea long time ago not only did the TA had grade a print out of your code, but they then scheduled a 15 one on one session with you to talk about what you did right and how you could improve. This sort of mentoring in the intro to programming course preped students for writing more complex code for other courses and meant that TA resources in later classes could concentrate on the topics at hand and not have to worry about spaghetti.
The TAing program became so famous that Silicon Valley companies would recruit people that had been TAs and pay big bucks for someone that had been head TA.
Lasers Controlled Games!
1) At least one student will either be lazy or make repeated mistakes trying to get their submission into the correct format. Because you are the TA and have no power, students will complain to you and demand that you make a special exception for their code. You'll find yourself making multiple variations to handle all the ways people screwed up your basic turn in instructions.
2) Studying code from other people is yes time consuming, but very valuable for you as a student. You will see patterns of solutions emerge and you'll see first hand where people had problems, and where they are inefficient. This gives you excellent data to share with students during recitation. It's valuable for everyone to discuss the different ways the problem was attacked and you being in the position of seeing everyone's code can make thoughtful assessments of what were ultimately good and bad choices.
3) Spending time automating can turn into a black hole; better to just spend time automating a turn in procedure that sorts people's submissions into folders.
4) Why not put testing the code onto the student's shoulders? If you have the need for seeing output why not instead post a series of expected inputs and outputs and ask students to turn in results that document this. Yes, it is more work for them. But becoming careful about work and careful to check it before turning in is or should be a key goal in a programming course.
5) Too much automating can set you up for grief from the professor if several students privately go complain to him or her about your choice. If they perceive your methods are unfairly depriving them of points, even if it is not true, you'll get complaints and depending on the character of the prof, he or she may not defend your actions and choose to make you out as a bad TA.
6) Take care not to go against the prof, it would be advisable to directly ask what their opinion is - it is after all their class.
7) Studying the code, rather than pushing it through an automated checker, is a good way to see who might be cheating and unfairly copying code. It's also quite hysterical to see the lengths that some desperate people will go to hide this from you like changing variable names and being very fancy with comments, but logic is logic and after looking at code you'll see cheaters really easily. Unless it's a group assignment where this was allowed, cheating shouldn't be tolerated. Some TAs have automated searching for cheating in coding assignments - perhaps this might be a better use of your time.
Good luck and have fun with those late night emails begging for help 8 hours before the assignment is due...
You'd automatically grade based on output? Are you serious? So if you asked students to reverse an array and print it out, they could simply print the array out backwards and that would be adaquate for you? The correct results should be part of the grade (obviously), but much like a Physics or Math problem, how they get the answer is just as (if not more) important than the answer. As a TA, it's your job to look at your student's code.
Please quit asap or let us know what university/college/devry you're at so we can avoid hiring anyone from there. TIA.
Automarking is used extensively by most of the computing courses at my uni, the course I tutor does things quite well: Assignments are submitted through the command line, where they are run through some scripts to make sure they compile and pass a few simple tests (if they don't, you can still submit them, you are just warned). There is a reference implementation of the assignment available for execution (web based and commandline versions). When it comes to marking, one guy is responsible for writing the automatic test cases, though he usually asks tutors for things which should be tested (also we have done diabolical things like making students submit test cases, and just using students test cases to do the marking... and more diabolical things like separating the assignments into modules, and pairing the modules of different students together (*)).
Tutors (and students) are responsible for making sure the auto-marking was correct. If the tutor sees a mistake, they can notify the automarking guy or over-ride the mark. If the student makes a small mistake (e.g. leaving out a space in the output) which makes them lose heaps of marks, then the tutor can modify their code and re-run all the tests over it, then manually penalise this mistake. The modified code is given back to the student, with comments by the tutor. This also ensures that a mistake is only penalised once.
The tutor also reads through the code and marks it for style. A strict style guide is given to the students which describes the little things (e.g. bracing styles, naming conventions etc), some of these things are tested for by the auto-marker, but this is just for the tutor's benefit (it doesn't change marks, just notifies the tutor). They are also marking for big-picture style - a clear algorithm, well broken down, no repetition etc. The tutor puts comments in the code in specific locations to point out the positives and negatives.
The students can then collect their assignments (commandline again), the collected assignment contatins:
* Code that was marked, including comments (and any changes made by the tutor)
* Diff of output and expected output for all test cases that were failed.
* Summary of automarking tests.
* Mark breakdown
* Final comments by tutor
Students are free to challenge marks, and seek further explaination of stuff. The auto-marking test cases are made available for downloading so students can reproduce the tests.
All in all, this system works pretty well. I don't think any students feel ripped off by being automarked, and by having lots of tests applied, the marks can be quite granular (more granular than I expected, given that typically a program either works or doesn't), and the tutors can spend their time looking at the important stuff, rather than running tests by hand. I obviously took this system for granted though, I assumed that all universities would be doing something similar (at least, all computing departments).
So yeah, this system is available to all the computing courses at my uni, though some subjects don't include as much of the time consuming stuff. (In later years, you often get assignments which are purely automarked).
(*) Mixing and matching of student modules is a fairly big task I think, but pretty cool. The system is made of modules A and B, so each student write A, B, testA and testB. The purpose of the test* modules is to test another student's modules - if student1's testB module passes student2's B module and student2's testA module passes student1's A module, then you can match those modules together, and any marks lost or gained go to both students. If a student's test* module fails another student's module, then they have to come to agreement about wh
I see multiple ideas here. First, automating of submission handling. This is a good thing. All it does is reduce errors and speed up grading.
Second is automated testing. This is good for when you can test for the program working in general and also for specific errors you anticipate. However, this automated testing should NOT just be plugged in as a grade. Instead, a thorough review of the code should be performed. Not only does this help provide better feedback, but it helps you to spot cheating. Finally, there are those programs that just won't run properly for one reason or another. These are the fun ones. When I was a TA I first downgraded them due to the failure to submit the program in fully working order, then did what I could to get it to run, and based an additional deduction on what I did (or not, in the case of ambiguities in specifications). If I still could not get it to work, I made a severe deduction and let the students schedule a time to meet with me so that they could demonstrate what their program did.
Purdue's CS department uses some sort of automated submission for their CS assignments. You submit your code, it actually runs algorithms on it to see if it is *too* similar to other students' assignments to help find cheating. Simple things like merely changing variable names will NOT make it through this system. I know a CS roommate of mine had to submit his homework with all source files and a makefile. The entire project had to compile and run in a certain environment to which the students had access. The automated submission process actually put the program through compilation and execution and provided input and checked the output.
I will say to the naysayers out there: Get Over It! No one is arguing that the students need feedback. No one is arguing that automated grading is not without it's own problems. However, automated grading is very useful on some levels and it's here to stay. There is nothing you can do about it. We still need instructors/TAs to look over code, for sure. But, with hundreds to thousands of students submitting code and no budget to hire the manpower to manually grade the submissions (some exceeding 10,000 lines) we're left without too many options. Let the TAs cover style and other forms of manual checking on the small code the students submit early in their studies, as in 100 and 200 level courses. Even then, make sure it is only on the small projects. Asking a prof or TA to look over 1000+ line submissions for hundreds of students each week is asking the impossible. And, I think there is nothing wrong with asking students on the 300 and 400 level to be content with just getting the program running correctly. In their jobs, they will be given tasks which must be completed by the deadline. When they whine and complain to their employers that they should still be given some praise for their uncompleted task because they *mostly* got it done, they will be fired. Automated grading could help them with this. The program works as advertised or it doesn't.
And, as an aside: write it yourself. I wouldn't want to take programming from a school that didn't have people capable of writing an automated grader.
Use CVS or other version-control system. Set it up so that each student gets their own module, and each student has access _only_ to their module [if individual assignments -- group assignments should naturally be set up for group access]. Then you can obtain and compile the code by performing a checkout rather than the student running a turnin program.
Teach 'em how to use Make or Ant. Specify the directory structure and the name of the build-control file. This lets you automate the "checkout, compile, run" process. Give them the script that does this, so that they can test this part of their process.
Then go through the assignment, and make a list of everything you told 'em to do and not to do. This includes stuff like "every file will have your name, the class, the assignment name, and the date in a comment at the top of the file' -- the finer-grained the list, the easier it is for you to grade. Go through this list and assign weights to the items on this list. Adjust the weights so that you come up with something reasonable -- you don't want to have an assignment that is worth 191.125 points.
As you're teaching a Java class, take advantage of the Java sandbox. Take a couple of days to learn how to use the policytool to create a policy file. If it's pure computation, stdin/stdout are good enough; otherwise, it's easy enough to set up filesystem access.
The start of your script should cat the documentation -- GRADER.NOTES file, README, whatever. Then it should cat all of the source files. Then it should compile the program. Then run the program w/ the appropriate policy file, with the inputs as desired. (Don't give the students _all_ of the possible inputs, but just a couple.) Capture all of this (script is a good tool), and print it out.
Inspect and markup the printout. Do NOT get hung up on correcting their crappy code. Do not let their crappy code slide by, or the next assignment will be just as hard to inspect. (And they'll get in the habit of writing really ugly code, which is a habit that's really, really, really hard to break, and they'll say, with all seriousness, "if it's hard to write it should be hard to read", and I'll just have to stake them through the heart when I encounter them.) Use the checklist.
At least 1/3 but no more than 2/3 of the grade should come from passing the tests. You're teaching 'em HOW to program, which involves following instructions, abiding by constraints and specifications, and writing readable code.
My problem was always that I spent too much time showing 'em how to do something right, rather than marking it as a problem and letting 'em come to me in office hours.
Finally, speaking of office hours, determine your appeals policy. Do not, ever, discuss the assignment with an individual student in class. They should come to your office during office hours to discuss, and _there_ you can point out why "if ( expression ) return TRUE; else return FALSE;" got them dinged, or why "i++; // incr i" does not count as commenting in your book. (The caveat being... if a sizable fraction of the class did the same sort of stupid error, that's a sign you need to cover that topic.)
P.S. The version-control history is a useful tool for tracking down cheaters. When three people with very nearly the exact same code commit the entire project the day it's due, that's supporting evidence.
Pick One: http://www-rohan.sdsu.edu/~stremler/sigs/sigs.html (Note - disable Javascript first!)
Automarkers are the WORST thing in the world.
,'s are in the right place is the worst way to get students interested in computing.
First semester the automarker wouldn't accept unless the output was character perfect!
Took off a mark each attempt too!
Spending more time making sure . _ and
When I was a TA, one of the things that seemed to make the most difference for students was help them find the problems in their code on their own. Few of them came to talk to me about any comments I ever wrote on their submissions.
If you expose your automated grading system so that anyone can run it (at least with some subset of the standard test data and expected results) you let every student check their grade as they go. You also make it more like the real world.... run your program against several data sets, and try to characterize the behavior that amkes it fail in certain situations. Once you know what makes an aspect of your program fail, you know where to go dig for solutions.
If it concerns you that they won't work any harder than they have to to meet the passing grade, make sure you save out some dedicated grading data sets that expose the more tricky boundary conditions, and that your students know that.
If you want to be proactive about helping students that need help, track the number of submissions and the associated scores. If you see someone entering a lot of submissions without seeing a steady improvement in their score, they've likely hit a roadblock and may need some questions answered. If someone is seeing steady progress to a bug free solution, give them a pat on the back.
It seems like this could be more than just a tool for automated grading, but also a tool for giving students immediate feedback (think standardized unit tests as a grading method), as well as helping keep track of students' progress and which may need more individual attention (but may not be asking for it).