Slashdot Mirror
Cheating Detector from Georgia Tech
brightboy writes "According to this Yahoo! News article, Georgia Tech has developed and implemented a "cheating detector"; that is, a program which compares students' coding assignments to each other and detects exact matches. This was used for two undergraduate classes: "Introduction to Computing" (required for any student in the College of Computing) and "Object Oriented Programming" (required for Computer Science majors)." Cuz
remember programmers: in the real world you are fired if you consult
with a co-worker ;)
This is new? They used something like this when I was at the University of Maryland a few years ago. And it did more than just check for exact matches, it compared parse trees and so on to check for similar program structure (any matches were, of course, double-checked by a human before ringing the cheating bell). It caught quite a few people I knew.
It better check for exact duplicates only, down to the variable names. Many undergraduate CS assignments are programs so basic that there are really only a few ways to implement them. It would suck to be a student who from scratch used the same algorithm as another student, and have them both flagged as cheaters.
A few years ago, when I was a 2nd or 3rd year at Virginia Tech, some professor implemented a cheating detector into the automated grader for a class called Intro to C++.
Prior to that year, VT had an average of 75 cheating violations for the WHOLE university (25000+ students). For that one class, on one assignment, 150 students were found cheating by the cheating detector... out of the 500 or so students in the class.
Funny as hell
Nosce te Ipsum
but there is a lot of cheating in undergraduate courses.
I was one of the better students in my comp-sci classes and so other students looked for me for help etc. I would routinely point them to my own finished assignments as example of how to do something or provide listings in which we would discuess the assignment and how to do things.
This worked well until I got called before the teacher in regards to two students having taken my listings and typed them in ( with practically no modification whatso-ever ). I explained the truth - that I provided it for purposes of instruction not stealing and managed to escape. The other students were forced to retake the course.
After this incident I kept my eyes wider open and noticed more students "copying"...
It happens. Whether this program is really needed or not I think is more an indication of how well the teacher stresses the students on final exams and such.
There's a gorilla from Manilla whose a fella that stinks of vanilla and has salmonella.
Good point. I know a 4th year student who doesn't know C, but watch out his Counter-Strike skills are amazing.
Sigh... when will schools implement the other kind of cheating detector?
Any sufficiently well-organized community is indistinguishable from Government.
CmdrTaco says: ;)
Cuz remember programmers: in the real world you are fired if you consult with a co-worker
As someone who TAed classes at GA Tech, I take a lot of offense at this comment. There is a difference between working as a team on project based classes (of which GA Tech has a good number off including classes where we got to hack the Linux kernel and another where we got to deliver a product to a customer) once you've shown you understand the basics of programming and wholesale copying of other people's work in entry level classes where you are supposed to be learning to program on your own.
Beginning programmers need to learn how to program, find information from MAN pages & API docs, and come up with solutions on their own before being introduced into team based environments. If not they never learn how to be self sufficient or even if they are cut out for programming at all.
It is true that in the real world no man is an island but on the flip side, how many people have worked with co-workers who completely clueless about how to perform their jobs but held degrees or certifications that implied they shoould be knowledgeable about programming? These are the kind of people who hid behind the work of others in team based projects and submitted others work on individual projects.
For all of you who posted : "gee, they invented diff again", it's a little more involved than just "diff". I'm sure other schools have similar cheat-detecing programs as well. Also, why Yahoo decided to pick up on this now and pass it off as news is beyond my comprehension. Maybe they had nothing else better to pass off as news. In my entire 4 years at GA Tech, I only heard about this program once and it's not a big deal. "There's nothing here to see people. Please move on with your lives."
Biodiesel : domestic, renewable, clean, and in the fuel tank of my bone stock 2002 New Beetle TDI
One of my CS profs created a program to do something similar for himself. It would take two programs and compare them and give a similarity score between 0.0 and 1.0. Seeing anything up to 0.6 in intro courses was considered normal, since the assignments were easier, but much above that and things got suspicious fast. Of course, any red flags were hand checked. Seeing as this is the prof that taught the compiler courses, I don't think there were many false positives. :-)
:-) Having a URL to an identical file from an algorithm archive helped too.
It caught a few guys that I know. When confronted they tried to say that they didn't cheat. So the prof does the only sensible thing that a CS prof should do when dealing with cheating intro students: Single out a common line of code in their programs and ask them what it did. Hint: How many of you knew the ternary operator in your first forays into C?
Pax, Ardax
The responses here, at least the ones along the lines of "But collaboration is allowed in the real world" sicken me. I would (and HAVE) fired programmers who couldn't program simple stuff on thier own. The collaboration in industry is not anywhere near the level of syntax and elementry algorithm design.
A University degree is supposed to signify that you demonstrated knowledge in certain areas.
Cheating is not demonstrating knowledge.
Undergraduate level programming assignments do not require even consultation with other students, IMHO. They are too simple. If you can't code an undergraduate programming project without extensive "consulting", then you can't program. Period.
I am sickened by the number of people with CS degrees only because of "teamwork" and "consulting". I would guess, from my experience, 95% of people with CS degrees can't write a sort routine. Widespread use of these kinds of programs might fix some of this. As would harsher grading. In the real world, you don't get partial credit for a program that only dumps core or doesn't meet any of the design objectives. (in my opinion, any program which doesn't properly run a set of tests, provided to the students in the project instructions, should receive an "F" grade)
No wonder the software industry is such a mess. I've seem CS *GRADUATE* students who couldn't use malloc(). Note that I did not say "who use malloc() wrong - no, these students could not even figure out how to call malloc() nor explain what it did. There's something strange happening (I call it cheating) when someone can graduate with a CS degree yet never use dynamic memory allocation knowingly...
First, the standard disclaimers: my comments are my own and should not be taken as necessarily representative of the GA Tech administration.
There's a much better and more accurate article on the topic at the AJC. Take the AP version with a grain of salt.
The fact that GA Tech uses software to detect possible cheating should not come as a surprise to anyone. Such systems have been in use at many schools across the country for many different disciplines besides CS. Nor should anyone be disturbed by the use of such systems: their purpose is to detect possible cheating, which according the AJC article was clearly verboten to the students in the class.
In the real world, a completely different set of rules may exist, but the fact remains that if your boss tells you he wants you to do something on your own, then you'd damn well better do it on your own. When a teacher instructs a student to perform a task on his own, he so instructs not to make life more difficult for the student, but to ensure that the student is capable of independently executing the skills necessary for the completion of the assignment. When that student eventually enters the real world, he has demonstrated the ability to perform the skills to be expected of him in the real world, so when he then has the ability to collaborate with his peers, he can actually contribure to the group's performance. A student who has always relied on others to get by will offer minimal assistance to a group and will typically act as a hinderance.
So sure, in the real world you won't be fired for collaborating with your peers, but you will be if you can't get anything done without collaborating with your peers.
Here's some more info, from the perspective of a former TA (once for one of the classes in question). First, everyone at GaTech is required to take the first CS class, not just CS majors (== people in the CoC). Second, GaTech doesn't restrict collaboration in all classes. The first tier of classes are strictly individual so everyone has to be in front of the computer. In the second tier, CS2130 - Languages and Translation explicitly allows colloboration as long as people turn in their own code. Going further, later classes involve heavy amounts of group work.
With regards to the cheater-detecter program (called 'cheatfinder'), it's significantly more complicated than diff(1). It involves checking the structure of the code (ignoring variable names , indentation, and whatnot). Admittedly, I've never seen the source for it (very few people have), but it's been around since at least 1997. The output of the program is a single number indicating the probability that two people colloborated on an assignment. The threshold is typically set fairly high (0.90+), so false-positives are less likely. 187 students, the number caught this time around, is definitely the highest I've heard of, but it's definitely not the first time we've hit a large number -- just the first time it made the cover of the local newspaper.
Interestingly, many students (including myself before becoming a TA) think (well, thought now) cheatfinder is just something the profs made up to scare students.
One assignment: modify this open source program to fix an assigned bug.
Another assignment: Modify this (Same) open source program and add a feature that they've been wanting for a while.
It's not like the solutions aren't readily available and well documented, it's just like in the real world: it hasn't been done here yet.
As long as intro courses use textbook problems with textbook solutions, students should be penalized for not doing their own work. The point in these classes is to provide an educational foundation. As soon as the foundations are laid, the students should be given work that isn't straight out of a textbook, and should be allowed to use any legal method of getting a solution. It's not like they'll find an exact solution, so they will have to do some tweaking and patching to get it to work anyway.
This was the philosophy my teaching cohort and I presented to the college Provost and the head of the CS department. They bought into it and gave us the class.
Network Security: It always comes down to a big guy with a gun.
Here at Stanford, where CS classes account for more than half of all cheating incidents, we've been using a system like this for a couple of years. Apparently, rather than comparing source code, it actually compares object code so that it can detect people who change variable names and so on. The theory is that no two people should actually write code which compiles to the exact same object code, no matter how similar their algorithms, unless they really are cheating.
- Copy the original code.
- Change every variable name (even if to a less sensible name - HalfCircleWidth instead of Radius).
- Rephrase most comments, but in the most transparent manner (e.g. "incerment the counter": becomes "the counter is incremented").
- Grab one or two lines of code near the top and rewrite them in the most awkward manner possible. Presumably, this is to prove to themselves that they're more clever than the teacher and that they could've actually done the assignment if they'd bothered.
Inevitably, it was the trivial stuff (indentation, comment structure) that set off my alarms. Then, I'd give them a moment of truth and sit them down to try to explain how "their" code works. If they didn't, I'd kick their tails out. If I was teaching a seminar at someone's workplace, I might or might not inform their management. Since all these penalties were spelled out in my syllabus, I never lost any sleep (in fact, putting them in my syllabus tends to ensure no one tries it).As to the the differenece between "consulting" with another and "cheating", I've found that the "explain your own code" is a pretty good yardstick. If I spend 2-3 hours preparing to teach a lecture, I have no sympathy with someone who doesn't spend enough time to do the assigned work but instead cheats.
"Prepare for the worst - hope for the best."
>Cheating is not demonstrating knowledge.
I agree, so why not use our ability with networking to solve the problem. The goal is to see that students can code their way out of a paper bag.
Why not have a class of networked boxes and then 'test' the students by having them come in and write their code while the network is shut down, preventing the students from getting access to any help during the test. Take the floppys and CDROMs out and they can't bring in outside help. There could be bonus marks for speed.
Experiment!
I took Intro to Computing in the Spring of 1996. It was cake for me because I was a Computer Science major and I dig this stuff. But a lot of non-CS people dreaded that class above all others, especially Management, International Affairs, and Architecture majors, but also some engineering people, such as Aerospace and Industrial Engineering.
(And can you really blame them? How many civil engineers really need to know how to sort numbers in O(N log N) time? Or insert into a linked list for that matter? They write hacked-up FORTRAN if they write anything at all.)
Kurt Eiselt came to the first lecture and gave us a scare speech about Cheatfinder. Knowing that it looks for similarities between two students' works, I was worried constantly about my homework answers. A typical problem was to write an inorder binary search tree traversal routine in pseudocode. Honestly, how many different ways are there to do this? And there are 500 people in all sections of the class?
Fortunately, I was never flagged, but I have heard a few stories (which may not be true, you know how that goes) of people who were flagged, and were only vindicated after losing student jobs and failing classes.
I don't think an automated cheat detection system is applicable to small problem sets like binary search, stacks, and Mergesort. For the later classes, say Sophomore level, I have no problem with it though.
Besides, many Greek orders and clubs on campus have extensive "word" banks--archives of previous homeworks and tests, with solutions, from previous class offerings. Are they going to check against all previous students' work too?
LAMP hosting on Debian, SSH, no bandwidth cap, PayPal accepted - http://secondbrainhosting.com/
Well we got every assignment in the last 5 years on file. We run hacker (which produces functionally equivalent programs = renames functions and vars) and the like (rotate ORCAD schemes) and got a cgi script that does most of this automagically.
Those dumb enough to carbon copy stuff deserve to be caught.
Here's 7 lines of C to reverse an array. The assembly would be more or less identical. I don't feel like dredging up my memories of 8086 assembler... it would probably end up screwing up my Perl for the next hour or so :-)
int list[] = {0,1,2,3,4,5};
int i,j,len=sizeof(list)/sizeof(int);
for (i=0; i < len/2; i++) {
j = list[i];
list[i] = list[len - i - 1];
list[len - i - 1] = j;
}
Reversing a linked list would be marginally longer, but a doubly linked list would be just as short or shorter than this. Only a real novice would take 100 lines of C to do it. BTW, how could you possibly learn assembly before learning what a stack is?
* And remember, it's spelled N-e-t-s-c-a-p-e, but it's pronounced "Mozilla."
CMU 15251 Course Document and Cheating Policy
His policy encourages collaboration and specifically forbids cheating. It itemizes various types of cheating, for example copying from another student, letting another student copy you, and looking at someone else's files online (even if they forgot to set their file permissions).
Furthermore, he requires all of the students in his class to sign a statement saying that they have read and understand the cheating policy. Not only does that discourage some students from cheating, but it also makes it much easier for him to get students into serious trouble with the school when they are caught.
In addition to the course document, here's more or less what he had to say on the first day of class: (I apologize for paraphrasing; this is how I remember it) "Nobody plans to cheat. You all must be very smart, or you wouldn't be here. You think you're going to try hard and do well in this class. But later in the semester you'll get busy with other classes and activities, and all of a sudden an assignment will be due in one day and you haven't started. Or you'll be taking a test and realize that you forgot to study an important equation. Or you'll work hard on an assignment and almost completely get it working, but get stuck on one subroutine. Even though you never planned on cheating, all of a sudden you'll find yourself in a circumstance like that and it will seem tempting."
(BTW, I shouldn't have to say this, but Prof. Rudich's cheating policy is copyrighted. If you're a teacher or T.A., don't copy his cheating policy without his permission. That would be just as dishonest as cheating!!! If you want to use it, contact him and I'm sure he'd be delighted to let you use it, as long as you give him credit.)
As you may have read, bestselling historian Stephen Ambrose was recently caught having lifted sentences and even passages from other sources, and passing them off as his own writing in his books. (While he mentioned the source books in footnotes/endnotes, he did not put the cribbed text in quotes.) At least four different Ambrose books have now been shown to have the same pattern of lifted, unattributed passages.
These instances only came to light because an author of a lifted passage noticed it while reading Ambrose's book. Subsequent episodes came about because other authors started looking, and now some people are checking out new likely sources; this works because Ambrose only lifted passages from books that he admired and heavily footnoted (at least, so far as we know!).
Perhaps Ambrose was really just lazy, as he was fairly open about crediting others for the ideas (he "just" failed to credit them for the words, too). There are many cases of sneakier plagiarism than that, both in academia and in journalism.
So, class, the programming problem for today is, given the text of two books, spit out the most likely candidates for lifted passages, based on length and similarity of words. You get a B if you can do this for exact, verbatim matches, an A if you can do it with individual word substitution, and an A+ if you can recognize re-ordered clauses. The end users for this tool would be 1) authors everywhere who want to protect their own writing, and 2) journalists looking for juicy plagiarism scandals.
Dear professor Andrew S. Tanenbaum of the free university of amsterdam and his colleagues already used this kind of program to check the hand-in programs we students wrote in modula-2 back 8 years ago or so, and actually it was kinda a big thing in dutch news back then. Guess these guess just reinvented the wheel.
so
I used to work for two brothers that were coding (seperately) modules for a project. Their code looked almost identical; variable names the same, data structures and algorithms almost identical...
These two both had the same training, and guess what...their code looked really similar.
Besides, how many different ways can you code "hello world" without failing for using inefficient/bad programming style?
-ted
This really isn't that new at all. I did some development on an autograding system at UCSD and incorporated a cheat detector using an age old system developed at Berkeley called MOSS. You simply tar up all your files and send them via mail to moss and they send email back to you with a link to an html page that gives percentages and all. Great system, uses program structure and logic to determine matches not commments and variable names. Give Berkeley the credit. DWP