Slashdot Mirror
Resources for Programming Course TA?
cndrr asks: "I'm a Teacher's Assistant for intro to Java at my university again this fall. The last time I taught, all the TAs had students turn in their assignments through email. I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. Has anyone else TAed and found a good solution that they would recommend?"
You'll waste as much time setting up and tweaking the system as you will doing it manually. Automatic submission and sorting into folders by course section is simple enough, but running the program and automatically grading the output??? That's madness.
Besides, trying to distance yourself from your students as much as possible by using technology is the exact opposite of what teaching is supposed to be about. If the students know that a real human will be reading the output and providing constructive feedback, they're much more likely to take it seriously.
I'd rather you manually grade it and provide valuable remarks to your students about their programs.
If universities were all about automated stuff, students can very well learn from course textbooks such as those prescribed by ocw.mit.edu by themselves. They go to university so that they can interact with their professors, get their amateur evaluated properly to shape their future work, and collaborate with their classmates.
Banu
Automatic grading apps are, uh, somewhat prone to breakage. I know at least one university that uses them found that certain students just wrote programs that did sneaky tricks rather than the assignment at hand (fork off a child process that sleeps for a bit then overwrites the grade file, then just output whatever). Perhaps more effort to do a one off program that way, but far less in the long term.
Said students got to keep the grades, too...
Are you sure it's a good idea to let students execute arbitrary code on your unattended machine?
I mean, I know *I* would get ideas...
Proud member of the Weirdo-American community.
"I'm thinking about scripting a site that will let students turn in their programs automatically and in some cases, run the program and (based on the output) automatically grade it. "
Grade what? If they can automate it, then you're out of a job.
I had a class last fall in which the prof used an automatic grader. He had a couple page document describe all the nit-picky formatting requirements we had to follow so that his automatic grader could run it. And, if we missed even a signle req, we got docked for it (generally in multiples of 10% of the assignment). Many people scored below-average to failing on their first couple of assignments because of this system, even though their code was flawless. So, please, please don't use automatic graders. They're more trouble than their worth, and you students will appreciate and specific comments you give. Even if it runs perfectly, comments can still sometimes be useful to make it even better.
How about doing your job instead? Automated turn in is fine, automated grading is bullshit and a perversion of academic principles.
Religion is a gateway psychosis. -- Dave Foley
Automatic grading is unethical.
We have an automated submission system that requires the program to compile for subnission. It also has the capacity to run the program and check its output (command line stuff only). It also ensures that the work is turned in on time and that it contains all of the files required (unless of course we are given an assignment giving us free reign on the class structure etc.) Though, the TA does go through all of the code levaing coments where appropriate and those comments have been a massive help. Overall I like my end of the system, and I can see how it could be likede on the TA's end. With those considerations, if you believe that you wouldn't waste more time setting a system up than you would save with it ,I would go for it.
Have any of the previous posters actually attended a large university in the computer science program?
Every class I've taken at UCSD does some sort of automatic grading on the programming assignments. It would be impossible to grade everything otherwise... Last quarter we wrote a compiler that ended up being a few hundred KB of source. There were over 200 test cases ran by the autograder.
All of the assignments are turned in from a unix prompt using a TURNIN command after prepping for the appropriate class.
The class I'm working on now (Operating Systems, using NACHOS) even has autograder() methods in the skeleton of the code that are used during grading...
I agree for a first class some feedback and hand grading might be necessary, but even with autograding you can add comments after looking at the code that causes similar test cases to fail.
Does your university have a subscription to one of those software scripts, such as WebCT? It's usually used to post lecture notes, assignments, and provides a bulletin board for questions. It also has a submission section (for WebCT) that allows students to submit their assignments by a certain time. Then you can download them all, each file separated into student accounts, and compile each one and grade.
Some amount of automation is helpful to avoid being swamped, but I don't think that a fully automated process can do the job. I require my students to generate an executable jar for submission, but also provide unit tests to them for the early assignments. Stage one of grading is to run a script which runs each of the jar files against the unit test. I then print the programs and mark them up for formatting and style, paying particular attention to those who ran afoul of the unit testing stage. Later in the course I start providing a spec and making them write their own unit tests, but I still use my own unit test modules for grading.
For you and all the other people saying human eyes are better than a computer for grading:
They don't need to be a substitute, and they can help a great deal in doing a lot of the manual checking you would be doing anyway, as well as the organizational part. I'm a TA, and I use a script to check for just about every simple mistake I can think of, and then I go over every assignment by hand, with a printout of my test-script's results. The script doesn't so much grad ethe work, but point out any output that might not be exactly what I expect, over an assortment of tests. The students often comment that these scripts, which I also hand back with the assignments along with output from the scripts, help a lot to identify not only what their problems were, but what sorts of things they should have in mind when writing their programs. I then *carefully* go through their code by hand to insure they were using good style and didn't coincidentally happen to pass any test with code that isn't technically correct.
When you have over 60 students in a class and have to grade long programming assignments every week, these scripts are essential to getting my work done in a timely manner. My personal attention and comments are not replaced by my scripts, but are enhanced by my scripts. My time spent grading is made *more* effective.
Also (for cndrr): I might be able to provide the main part of the script we use, I'd just have to check with my instructor. The test scripts we use are fairly easy to write using it, if you don't mind doing it partially in scheme... (our department is in love with that language). Let me know if you're interested.
"Question with boldness even the existence of a god." - Thomas Jefferson
Is very much automated-submission and grading -- or at least was from 00-04. Some professors do the grading by-hand (i.e. they run scripts by hand) but the majority of classes I took where you turned programs in, there were scripts that graded you that you could run part of yourself. (i.e. there were "public" and "private" scripts that used the same interface.)
In a large part, it depends on how big your class is, and if you plan on continuing teaching. One professor, (whom I idolize) has been doing this long enough to have set up an extensive and all-inclusive framework. Can't go (too badly) wrong when you roll your own, but YMMV.
-b
When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi. (Larry Wall)
I always made the grades up off the top of my head at the end of the semester, and found it much more fitting with academic tradition and principals.
hehe, jk.
www.GrenadeHop.com
If your students are all writing CGI-style programs, maybe reading and writing a file or two, why not set up a Linux or FreeBSD box for them? Open ftp and ssh for them and require them to build their programs there. Your requirements for each class will be that they compile and run their programs on your server, do a run under script(1), then hand in all source, output, data and log files via a script you wrote called handin assignment [file...], which copies [file...] into your directory $assignment/$studentid/[file...]. Since handin is suid, write it carefuly: It should contain no mkdir or similar commands.
This will be good for everyone. They will pick up the rudiments of Unix and using remote computers. You will pick up the rudiments of Unix administration, and all your class stuff will be in one convenient directory tree per assignment.
Just for fun, you could write yourself a little program called comment that lets you mark up their source code -- their stuff will be in mono and your comments in mono italics . Print 'em up and give them back at the beginning of class.
Make sure the students have access to a good variety of editors, including a couple of dumbed-down ones like pico.
The downside is that you'll spend a day or two going over the rudiments of Unix, and have to do some early handholding for them, but it will pay off in the end.
This is not my sandwich.
you are getting paid to mark the papers, to identify problem areas and to solve those problems for the students, you have to provide feedback to them. if you want to be lazy and skip on doing that please dont ever accept a teaching responsibility, i had to teach several friends how to program in java since our lecturer was lazy and incompetant. somewhere someone will have to take the responsibility. work out how much those students pay per class and per evaluation and you will quickly realise you are trying to screw them over. i am a student.
This is my sig.
In COMP 314 (Rice University's sophomore-level programming & algorithms class, taught this past spring by Prof. Dan Wallach) we TAs solved this problem with Subversion.
An important part of real-world programming is teamwork; in 314 we embrace this and randomly sort students into groups of two for pair programming. The groups change for each project, and each project group gets a spot in a svn repository set up for the course. ACLs keep groups from peeking at one another's changes (for example, see this team list, which is actually just a slice of our Subversion access control file). Students were required to tag their submissions for each of the project milestones: specification, prototype, final; this was how students submitted code for these deadlines. From timestamps we could easily see which groups incurred "slip hours" for late turnins.
There were a number of reported incidences of lost work or conflicting changes which would have been disasters if not for svn, which saved their bacon. Groups that learned to check in early and often knew that accidental deletions or disk failures posed no threat to a successful project submission. A few enterprising teams even used tags and branches to help organize complex development efforts. In all, it was quite a successful adventure and we'll probably do something similar in the future.
Considering it's an intro class, I am sure lot of students will unintentionally end up with similar programs. http://www.cs.berkeley.edu/~aiken/moss.html
I don't know about everyone else, but the instructors that I've had for programming will read through the code first before compiling and running it.
There's 3 reasons for this. 1. To make sure you followed coding standards. 2. To make sure you weren't doing something destructive. 3. To make sure you were actually using the programming constructs that the instructor wanted you to.
To expand on 3, for simple programs like you run into in an Introduction class, it's possible to do a recursion assignment using iteration. Or to use if/else if/else where you were asked to use switch/case. Reading the code will catch these things. Just checking the output won't.
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
Berkeley's been doing this sort of thing for years and years now. I'm sure someone there can help you identify software that can do what you're looking for. Unfortunately, I never TA'ed while attending, so I don't know what they use.
The Tufts Department of Computer Science use a homemade software called "provide" (Professor Alva Couch wrote it) It is a simple online grading utility. This program allows simple submission and online grading of electronic assignment submissions in Computer Science and Computer Engineering courses. It is specifically designed to be easy to use by graduate students working on their own, to leverage faculty resources and decrease faculty workload in managing grades. In production in EECS, not publicly distributed (1998-present).
Assignments can be graded manually or automatically. The Intro to CS and Data Structure instructor usually write an automatic grader for assignments. Programming style is graded manually. For algorithm, graphics, and theory assignments, students submit them via provide. The TA then grade the assignments, put in the grades, and students run a command "progress" to see their grade of their assignment. Many students say that the process work quite effectively.
One of the grad students at my school developed this system as his masters project...
Don't know if it's of any help but you might try contacting a Dr. Sun since he was the advisor.
It doesn't automatically grade the homeword, but it does let you set time limits and since it uploads directly to the server, you won't have to deal with students saying they got the wrong email address.
The system, currently, is only for homework and old code retrival but many other things are planned for it.
We had a T.A. automatically grade assignments at the school where I worked. He wrote an automatic scanner to count the number of words in the comments, and to automatically run the programs and compare the output. He then assigned grades appropriately, computing the grade only on the number of words in the comments.
He then quite proudly stood up in front of the class as explained to them what he had done. He had no idea what was about to happen. Essentially, everyone in the class simultaneously worked out that it didn't matter what they put in the comments. They just needed to boost the word count. The class turned red and almost rioted. The professor's jaw dropped.
The professor recalled the assignments and had them regraded by a different T.A. After that day, the original T.A. was never allowed to mark another assignment.
How about not being a lazy bastard and doing what you're paid for?
why? forty-two.
At RIT we used a program called "try", developed by one of the professors there. You can download it from his page: http://www.cs.rit.edu/~kar/software.html
Search first, ask questions later.
I am also a TA for an introductory course to JAVA. I am not sure how many others here are but I have found auto-marking to be essential. I am a single TA and I have 180 students. Each assignment has 3-4 substantial programs. Trying to mark them all by hand would be impossible given time limitations. I do have to agree with others though in that feedback is always nice. Therefore what I have done is I mark half of each assignment and the other half is automarked. I have automarking compare output given input and allocate this to a mark called 'Correctness'. I then mark certain functions (usually the harder ones assuming that if they could do these ones they could do the others). This way the students still get feedback. I also encourage them to come in and see me if they have questions so we could go over it together in more depth. The test cases are also put online (after all assignments are in).
The submission system I use however is unix based. So the first or second tutorial/lab we run is all about how to submit your assignment through unix. We have had a website submission system in the past but we found this negativly affected some students who in third year had yet to touch unix at all.
Just as a warning though, I have found with introductory courses, no matter how many times I tell them to output something in a specific way, the students will not do it. I usually have my automarking scripts do a 'diff' with multiple possible answer or ways of formating an answer. They will still fail all the test cases forcing me to hand mark anyways. I have even given them Strings in provided code to use and they will either change the output, ignore it, or have extra output. Sometimes I just give them 0 in the correctness area. The main reason for this is the students don't care. Some are just in the course because there program requires them to have one CS course.
Therefore, if you plan on doing this then you might want to consider having final results stored in a variable, and have them make a public get method. That way you can simply redirect standard out to something like /dev/null when you run their code to get the final result, and then set it back to the screen and use the get method to output just the result. This only works in the case of computing numbers though.
Anyways, good luck with whatever you decide to do
Did he amend your grade then? I mean, if you don't understand how your own program worked, you don't deserve a passing grade, even if it did work correctly.
If you're TA'ing, you should have the Java-fu to do it. It's not hard. We used one for our classes, and we worked out a really nice compromise.
When the program was submitted, it was compiled and trivially rejected if it failed to compile. This is worth the price of admission right here, since people whose code "compiles on their machine" get multiple chances to recognize the error of their ways. (Some people think students should get points taken off for this; that's a philisophical discussion I won't go into.)
We ran it in a separate TA account so that if one student does "rm -rf *" he doesn't hurt anything before he's escorted out of the department.
There are a suite of tests that are run on the program, and a score is given, generally 10 points per test. That's the "base score." Now, if your students are good and your assignments are fair, with multiple tries and different outputs at least half the students will get 100. The point of the "try until you get it right" method is to guide the students to the right answer.
Then the TA hand-grades the non-100s in the traditional way, but he doesn't try TOO awfully hard, because the students get the opportunity for a "regrade." The TA steps through the grading with the student in person, explaining what went wrong.
This is a great solution for courses that use homework to get students to explore the language and possible solutions; it may not be the best solution for classes where the grades for homework are significant.
At Drexel University, our CS department has students submit assignments by checking code into an SVN repository, too. The advantages of this are many.
I don't know what your qualifications are to be a TA for this course. I was a TA for an intro to object oriented programming course using Java during my junior year. I also don't know what the difficulty level of your course is, but that should not change my opinion. 1. The students in this course will most likely put alot of time and effort into these assignments. 2. I would hope that YOU could implement a system like the one you described with an equivalent amount of effort. 3. I would hope that your STUDENTS, after taking this course, given adequate guidance and framework, could implement a system like the one you described. If you are suggesting this system as anything other than a progress indicator for students to test early drafts of their work, I would seriously reconsider your role as a mentor/teacher/guide. At this point in a student's education, it is most important to reinforce good habits. As far as some of the other comments posted here are concerned, I am worried about the level of education others are receiving. Get your acts together folks! Students will live up to, or down to your expectations of them.
Stanford's CS 106a class is famous not only for being a popular class (1/3rd of all Stanford students take it) but for producing amazing TAs as well. When I took the coursea long time ago not only did the TA had grade a print out of your code, but they then scheduled a 15 one on one session with you to talk about what you did right and how you could improve. This sort of mentoring in the intro to programming course preped students for writing more complex code for other courses and meant that TA resources in later classes could concentrate on the topics at hand and not have to worry about spaghetti.
The TAing program became so famous that Silicon Valley companies would recruit people that had been TAs and pay big bucks for someone that had been head TA.
Lasers Controlled Games!
1) At least one student will either be lazy or make repeated mistakes trying to get their submission into the correct format. Because you are the TA and have no power, students will complain to you and demand that you make a special exception for their code. You'll find yourself making multiple variations to handle all the ways people screwed up your basic turn in instructions.
2) Studying code from other people is yes time consuming, but very valuable for you as a student. You will see patterns of solutions emerge and you'll see first hand where people had problems, and where they are inefficient. This gives you excellent data to share with students during recitation. It's valuable for everyone to discuss the different ways the problem was attacked and you being in the position of seeing everyone's code can make thoughtful assessments of what were ultimately good and bad choices.
3) Spending time automating can turn into a black hole; better to just spend time automating a turn in procedure that sorts people's submissions into folders.
4) Why not put testing the code onto the student's shoulders? If you have the need for seeing output why not instead post a series of expected inputs and outputs and ask students to turn in results that document this. Yes, it is more work for them. But becoming careful about work and careful to check it before turning in is or should be a key goal in a programming course.
5) Too much automating can set you up for grief from the professor if several students privately go complain to him or her about your choice. If they perceive your methods are unfairly depriving them of points, even if it is not true, you'll get complaints and depending on the character of the prof, he or she may not defend your actions and choose to make you out as a bad TA.
6) Take care not to go against the prof, it would be advisable to directly ask what their opinion is - it is after all their class.
7) Studying the code, rather than pushing it through an automated checker, is a good way to see who might be cheating and unfairly copying code. It's also quite hysterical to see the lengths that some desperate people will go to hide this from you like changing variable names and being very fancy with comments, but logic is logic and after looking at code you'll see cheaters really easily. Unless it's a group assignment where this was allowed, cheating shouldn't be tolerated. Some TAs have automated searching for cheating in coding assignments - perhaps this might be a better use of your time.
Good luck and have fun with those late night emails begging for help 8 hours before the assignment is due...
I see multiple ideas here. First, automating of submission handling. This is a good thing. All it does is reduce errors and speed up grading.
Second is automated testing. This is good for when you can test for the program working in general and also for specific errors you anticipate. However, this automated testing should NOT just be plugged in as a grade. Instead, a thorough review of the code should be performed. Not only does this help provide better feedback, but it helps you to spot cheating. Finally, there are those programs that just won't run properly for one reason or another. These are the fun ones. When I was a TA I first downgraded them due to the failure to submit the program in fully working order, then did what I could to get it to run, and based an additional deduction on what I did (or not, in the case of ambiguities in specifications). If I still could not get it to work, I made a severe deduction and let the students schedule a time to meet with me so that they could demonstrate what their program did.
Purdue's CS department uses some sort of automated submission for their CS assignments. You submit your code, it actually runs algorithms on it to see if it is *too* similar to other students' assignments to help find cheating. Simple things like merely changing variable names will NOT make it through this system. I know a CS roommate of mine had to submit his homework with all source files and a makefile. The entire project had to compile and run in a certain environment to which the students had access. The automated submission process actually put the program through compilation and execution and provided input and checked the output.
I will say to the naysayers out there: Get Over It! No one is arguing that the students need feedback. No one is arguing that automated grading is not without it's own problems. However, automated grading is very useful on some levels and it's here to stay. There is nothing you can do about it. We still need instructors/TAs to look over code, for sure. But, with hundreds to thousands of students submitting code and no budget to hire the manpower to manually grade the submissions (some exceeding 10,000 lines) we're left without too many options. Let the TAs cover style and other forms of manual checking on the small code the students submit early in their studies, as in 100 and 200 level courses. Even then, make sure it is only on the small projects. Asking a prof or TA to look over 1000+ line submissions for hundreds of students each week is asking the impossible. And, I think there is nothing wrong with asking students on the 300 and 400 level to be content with just getting the program running correctly. In their jobs, they will be given tasks which must be completed by the deadline. When they whine and complain to their employers that they should still be given some praise for their uncompleted task because they *mostly* got it done, they will be fired. Automated grading could help them with this. The program works as advertised or it doesn't.
And, as an aside: write it yourself. I wouldn't want to take programming from a school that didn't have people capable of writing an automated grader.
Use CVS or other version-control system. Set it up so that each student gets their own module, and each student has access _only_ to their module [if individual assignments -- group assignments should naturally be set up for group access]. Then you can obtain and compile the code by performing a checkout rather than the student running a turnin program.
Teach 'em how to use Make or Ant. Specify the directory structure and the name of the build-control file. This lets you automate the "checkout, compile, run" process. Give them the script that does this, so that they can test this part of their process.
Then go through the assignment, and make a list of everything you told 'em to do and not to do. This includes stuff like "every file will have your name, the class, the assignment name, and the date in a comment at the top of the file' -- the finer-grained the list, the easier it is for you to grade. Go through this list and assign weights to the items on this list. Adjust the weights so that you come up with something reasonable -- you don't want to have an assignment that is worth 191.125 points.
As you're teaching a Java class, take advantage of the Java sandbox. Take a couple of days to learn how to use the policytool to create a policy file. If it's pure computation, stdin/stdout are good enough; otherwise, it's easy enough to set up filesystem access.
The start of your script should cat the documentation -- GRADER.NOTES file, README, whatever. Then it should cat all of the source files. Then it should compile the program. Then run the program w/ the appropriate policy file, with the inputs as desired. (Don't give the students _all_ of the possible inputs, but just a couple.) Capture all of this (script is a good tool), and print it out.
Inspect and markup the printout. Do NOT get hung up on correcting their crappy code. Do not let their crappy code slide by, or the next assignment will be just as hard to inspect. (And they'll get in the habit of writing really ugly code, which is a habit that's really, really, really hard to break, and they'll say, with all seriousness, "if it's hard to write it should be hard to read", and I'll just have to stake them through the heart when I encounter them.) Use the checklist.
At least 1/3 but no more than 2/3 of the grade should come from passing the tests. You're teaching 'em HOW to program, which involves following instructions, abiding by constraints and specifications, and writing readable code.
My problem was always that I spent too much time showing 'em how to do something right, rather than marking it as a problem and letting 'em come to me in office hours.
Finally, speaking of office hours, determine your appeals policy. Do not, ever, discuss the assignment with an individual student in class. They should come to your office during office hours to discuss, and _there_ you can point out why "if ( expression ) return TRUE; else return FALSE;" got them dinged, or why "i++; // incr i" does not count as commenting in your book. (The caveat being... if a sizable fraction of the class did the same sort of stupid error, that's a sign you need to cover that topic.)
P.S. The version-control history is a useful tool for tracking down cheaters. When three people with very nearly the exact same code commit the entire project the day it's due, that's supporting evidence.
Pick One: http://www-rohan.sdsu.edu/~stremler/sigs/sigs.html (Note - disable Javascript first!)
Automarkers are the WORST thing in the world.
,'s are in the right place is the worst way to get students interested in computing.
First semester the automarker wouldn't accept unless the output was character perfect!
Took off a mark each attempt too!
Spending more time making sure . _ and
When I was a TA, one of the things that seemed to make the most difference for students was help them find the problems in their code on their own. Few of them came to talk to me about any comments I ever wrote on their submissions.
If you expose your automated grading system so that anyone can run it (at least with some subset of the standard test data and expected results) you let every student check their grade as they go. You also make it more like the real world.... run your program against several data sets, and try to characterize the behavior that amkes it fail in certain situations. Once you know what makes an aspect of your program fail, you know where to go dig for solutions.
If it concerns you that they won't work any harder than they have to to meet the passing grade, make sure you save out some dedicated grading data sets that expose the more tricky boundary conditions, and that your students know that.
If you want to be proactive about helping students that need help, track the number of submissions and the associated scores. If you see someone entering a lot of submissions without seeing a steady improvement in their score, they've likely hit a roadblock and may need some questions answered. If someone is seeing steady progress to a bug free solution, give them a pat on the back.
It seems like this could be more than just a tool for automated grading, but also a tool for giving students immediate feedback (think standardized unit tests as a grading method), as well as helping keep track of students' progress and which may need more individual attention (but may not be asking for it).