Slashdot Mirror
Resources for Programming Course TA?
cndrr asks: "I'm a Teacher's Assistant for intro to Java at my university again this fall. The last time I taught, all the TAs had students turn in their assignments through email. I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. Has anyone else TAed and found a good solution that they would recommend?"
You'll waste as much time setting up and tweaking the system as you will doing it manually. Automatic submission and sorting into folders by course section is simple enough, but running the program and automatically grading the output??? That's madness.
Besides, trying to distance yourself from your students as much as possible by using technology is the exact opposite of what teaching is supposed to be about. If the students know that a real human will be reading the output and providing constructive feedback, they're much more likely to take it seriously.
I'd rather you manually grade it and provide valuable remarks to your students about their programs.
If universities were all about automated stuff, students can very well learn from course textbooks such as those prescribed by ocw.mit.edu by themselves. They go to university so that they can interact with their professors, get their amateur evaluated properly to shape their future work, and collaborate with their classmates.
Banu
Are you sure it's a good idea to let students execute arbitrary code on your unattended machine?
I mean, I know *I* would get ideas...
Proud member of the Weirdo-American community.
I had a class last fall in which the prof used an automatic grader. He had a couple page document describe all the nit-picky formatting requirements we had to follow so that his automatic grader could run it. And, if we missed even a signle req, we got docked for it (generally in multiples of 10% of the assignment). Many people scored below-average to failing on their first couple of assignments because of this system, even though their code was flawless. So, please, please don't use automatic graders. They're more trouble than their worth, and you students will appreciate and specific comments you give. Even if it runs perfectly, comments can still sometimes be useful to make it even better.
How about doing your job instead? Automated turn in is fine, automated grading is bullshit and a perversion of academic principles.
Religion is a gateway psychosis. -- Dave Foley
We have an automated submission system that requires the program to compile for subnission. It also has the capacity to run the program and check its output (command line stuff only). It also ensures that the work is turned in on time and that it contains all of the files required (unless of course we are given an assignment giving us free reign on the class structure etc.) Though, the TA does go through all of the code levaing coments where appropriate and those comments have been a massive help. Overall I like my end of the system, and I can see how it could be likede on the TA's end. With those considerations, if you believe that you wouldn't waste more time setting a system up than you would save with it ,I would go for it.
Have any of the previous posters actually attended a large university in the computer science program?
Every class I've taken at UCSD does some sort of automatic grading on the programming assignments. It would be impossible to grade everything otherwise... Last quarter we wrote a compiler that ended up being a few hundred KB of source. There were over 200 test cases ran by the autograder.
All of the assignments are turned in from a unix prompt using a TURNIN command after prepping for the appropriate class.
The class I'm working on now (Operating Systems, using NACHOS) even has autograder() methods in the skeleton of the code that are used during grading...
I agree for a first class some feedback and hand grading might be necessary, but even with autograding you can add comments after looking at the code that causes similar test cases to fail.
For you and all the other people saying human eyes are better than a computer for grading:
They don't need to be a substitute, and they can help a great deal in doing a lot of the manual checking you would be doing anyway, as well as the organizational part. I'm a TA, and I use a script to check for just about every simple mistake I can think of, and then I go over every assignment by hand, with a printout of my test-script's results. The script doesn't so much grad ethe work, but point out any output that might not be exactly what I expect, over an assortment of tests. The students often comment that these scripts, which I also hand back with the assignments along with output from the scripts, help a lot to identify not only what their problems were, but what sorts of things they should have in mind when writing their programs. I then *carefully* go through their code by hand to insure they were using good style and didn't coincidentally happen to pass any test with code that isn't technically correct.
When you have over 60 students in a class and have to grade long programming assignments every week, these scripts are essential to getting my work done in a timely manner. My personal attention and comments are not replaced by my scripts, but are enhanced by my scripts. My time spent grading is made *more* effective.
Also (for cndrr): I might be able to provide the main part of the script we use, I'd just have to check with my instructor. The test scripts we use are fairly easy to write using it, if you don't mind doing it partially in scheme... (our department is in love with that language). Let me know if you're interested.
"Question with boldness even the existence of a god." - Thomas Jefferson
Is very much automated-submission and grading -- or at least was from 00-04. Some professors do the grading by-hand (i.e. they run scripts by hand) but the majority of classes I took where you turned programs in, there were scripts that graded you that you could run part of yourself. (i.e. there were "public" and "private" scripts that used the same interface.)
In a large part, it depends on how big your class is, and if you plan on continuing teaching. One professor, (whom I idolize) has been doing this long enough to have set up an extensive and all-inclusive framework. Can't go (too badly) wrong when you roll your own, but YMMV.
-b
When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi. (Larry Wall)
In COMP 314 (Rice University's sophomore-level programming & algorithms class, taught this past spring by Prof. Dan Wallach) we TAs solved this problem with Subversion.
An important part of real-world programming is teamwork; in 314 we embrace this and randomly sort students into groups of two for pair programming. The groups change for each project, and each project group gets a spot in a svn repository set up for the course. ACLs keep groups from peeking at one another's changes (for example, see this team list, which is actually just a slice of our Subversion access control file). Students were required to tag their submissions for each of the project milestones: specification, prototype, final; this was how students submitted code for these deadlines. From timestamps we could easily see which groups incurred "slip hours" for late turnins.
There were a number of reported incidences of lost work or conflicting changes which would have been disasters if not for svn, which saved their bacon. Groups that learned to check in early and often knew that accidental deletions or disk failures posed no threat to a successful project submission. A few enterprising teams even used tags and branches to help organize complex development efforts. In all, it was quite a successful adventure and we'll probably do something similar in the future.
Berkeley's been doing this sort of thing for years and years now. I'm sure someone there can help you identify software that can do what you're looking for. Unfortunately, I never TA'ed while attending, so I don't know what they use.
One of the grad students at my school developed this system as his masters project...
Don't know if it's of any help but you might try contacting a Dr. Sun since he was the advisor.
It doesn't automatically grade the homeword, but it does let you set time limits and since it uploads directly to the server, you won't have to deal with students saying they got the wrong email address.
The system, currently, is only for homework and old code retrival but many other things are planned for it.
We had a T.A. automatically grade assignments at the school where I worked. He wrote an automatic scanner to count the number of words in the comments, and to automatically run the programs and compare the output. He then assigned grades appropriately, computing the grade only on the number of words in the comments.
He then quite proudly stood up in front of the class as explained to them what he had done. He had no idea what was about to happen. Essentially, everyone in the class simultaneously worked out that it didn't matter what they put in the comments. They just needed to boost the word count. The class turned red and almost rioted. The professor's jaw dropped.
The professor recalled the assignments and had them regraded by a different T.A. After that day, the original T.A. was never allowed to mark another assignment.
At RIT we used a program called "try", developed by one of the professors there. You can download it from his page: http://www.cs.rit.edu/~kar/software.html
Search first, ask questions later.
I am also a TA for an introductory course to JAVA. I am not sure how many others here are but I have found auto-marking to be essential. I am a single TA and I have 180 students. Each assignment has 3-4 substantial programs. Trying to mark them all by hand would be impossible given time limitations. I do have to agree with others though in that feedback is always nice. Therefore what I have done is I mark half of each assignment and the other half is automarked. I have automarking compare output given input and allocate this to a mark called 'Correctness'. I then mark certain functions (usually the harder ones assuming that if they could do these ones they could do the others). This way the students still get feedback. I also encourage them to come in and see me if they have questions so we could go over it together in more depth. The test cases are also put online (after all assignments are in).
The submission system I use however is unix based. So the first or second tutorial/lab we run is all about how to submit your assignment through unix. We have had a website submission system in the past but we found this negativly affected some students who in third year had yet to touch unix at all.
Just as a warning though, I have found with introductory courses, no matter how many times I tell them to output something in a specific way, the students will not do it. I usually have my automarking scripts do a 'diff' with multiple possible answer or ways of formating an answer. They will still fail all the test cases forcing me to hand mark anyways. I have even given them Strings in provided code to use and they will either change the output, ignore it, or have extra output. Sometimes I just give them 0 in the correctness area. The main reason for this is the students don't care. Some are just in the course because there program requires them to have one CS course.
Therefore, if you plan on doing this then you might want to consider having final results stored in a variable, and have them make a public get method. That way you can simply redirect standard out to something like /dev/null when you run their code to get the final result, and then set it back to the screen and use the get method to output just the result. This only works in the case of computing numbers though.
Anyways, good luck with whatever you decide to do
Stanford's CS 106a class is famous not only for being a popular class (1/3rd of all Stanford students take it) but for producing amazing TAs as well. When I took the coursea long time ago not only did the TA had grade a print out of your code, but they then scheduled a 15 one on one session with you to talk about what you did right and how you could improve. This sort of mentoring in the intro to programming course preped students for writing more complex code for other courses and meant that TA resources in later classes could concentrate on the topics at hand and not have to worry about spaghetti.
The TAing program became so famous that Silicon Valley companies would recruit people that had been TAs and pay big bucks for someone that had been head TA.
Lasers Controlled Games!
1) At least one student will either be lazy or make repeated mistakes trying to get their submission into the correct format. Because you are the TA and have no power, students will complain to you and demand that you make a special exception for their code. You'll find yourself making multiple variations to handle all the ways people screwed up your basic turn in instructions.
2) Studying code from other people is yes time consuming, but very valuable for you as a student. You will see patterns of solutions emerge and you'll see first hand where people had problems, and where they are inefficient. This gives you excellent data to share with students during recitation. It's valuable for everyone to discuss the different ways the problem was attacked and you being in the position of seeing everyone's code can make thoughtful assessments of what were ultimately good and bad choices.
3) Spending time automating can turn into a black hole; better to just spend time automating a turn in procedure that sorts people's submissions into folders.
4) Why not put testing the code onto the student's shoulders? If you have the need for seeing output why not instead post a series of expected inputs and outputs and ask students to turn in results that document this. Yes, it is more work for them. But becoming careful about work and careful to check it before turning in is or should be a key goal in a programming course.
5) Too much automating can set you up for grief from the professor if several students privately go complain to him or her about your choice. If they perceive your methods are unfairly depriving them of points, even if it is not true, you'll get complaints and depending on the character of the prof, he or she may not defend your actions and choose to make you out as a bad TA.
6) Take care not to go against the prof, it would be advisable to directly ask what their opinion is - it is after all their class.
7) Studying the code, rather than pushing it through an automated checker, is a good way to see who might be cheating and unfairly copying code. It's also quite hysterical to see the lengths that some desperate people will go to hide this from you like changing variable names and being very fancy with comments, but logic is logic and after looking at code you'll see cheaters really easily. Unless it's a group assignment where this was allowed, cheating shouldn't be tolerated. Some TAs have automated searching for cheating in coding assignments - perhaps this might be a better use of your time.
Good luck and have fun with those late night emails begging for help 8 hours before the assignment is due...