Both Students and Teachers Use Technology to Cheat

Mr. Slippery writes "Baltimore City Paper's Cyberpunk column (which, incidently, is where I first learned of /.) has an interesting bit on the impact of technology on college essays - students downloading pre-written papers off the net, and professors using automated systems to grade them. Ah, the circle is complete." This story has no "news" in it, but writer Joab Jackson's take on the subject is interesting. (Disclaimer: Joab's a personal friend - and I used to write for City Paper too. - RM.)

Comments

Teachers in High-School are exempt from rules

[Knight]

I've was treated very poorly by the teachers and administration in high-school, so if you are a teacher, principal, etc.; sorry, but this is my experience:
The administration at my school had a huge number of draconian rules that made even breathing difficult. We were expected to follow every rule to the letter, and anything less brought down hell on us. More than once, I was reprimanded simply for expressing an opinion that differed from that of the current speaker. There was only one "truth", and the teachers had a corner on the market. I would often find loopholes in the rules to allow me to do things that I wanted to. These efforts were not appreciated. I would be punished for "violating the spirit of the law", and the rules would quickly be changed.
The teachers, likewise had a set of rules that they were supposed to live by. They were by no means as strict, and quite fewer in number. However, they were never required to follow those rules. For example, the teachers were required to write up a Disciplinary Action Form if they sent a student to the office for disciplinary reasons. However, they rarely did. More than once I claimed that I was never sent to the office and asked them to produce the forms as proof that I had been sent. They never produced these forms, but (big surprise) I was disciplined again for my challenges. Simply standing up to ridiculousness and having an opinion are important skills that are stamped out in our schools. I'm a free-thinking individual, with no use for these fascist institutions. I'm only 20 years old, but I'll swear to anyone reading right now that I will never cripple my children by subjecting them to such a limiting, brainwashing environment.
It's so ridiculous. My school's curriculum preached the values of democracy and a free society out every orifice they had, but when it really came down to it, they practiced fascism, favoritism, and contributed to an animalistic social hierarchy favoring those who did not think for themselves. Pardon my language, but give me a fucking break! Now, the schools in my area are implementing cameras in all areas of the school, including bathrooms. It's an outright violation of privacy. A student shouldn't have to put up with anything at school that his parents don't have to put up with at work. That includes abuse by teachers and students, destruction of free will, etc. If I had to put up with anything even resembling my experience in high school at a job, I'd walk out the door in an instant. I swear, if I could fix this, I'd do it right now. It's one of the biggest tragedies in this country.

If you need to point-and-click to administer a machine,

Re:Johnny, what did you learn today? (RANT)

[fable2112]

Sorry, but you'll find that learning something for the sake of learning it, or for the sake of being "educated," won't exactly make you any friends or get you anywhere in life. People just don't care anymore. They'll say Shakespeare is "boring" without even opening the book to find that much of what he wrote is still quite relevant. They'll say "When am I ever going to use this, anyway?" when faced with a mathematical concept that doesn't make immediate sense to them. They'll say, "Why should what a bunch of people did a hundred years ago matter?" and ignore the influence that those actions of a hundred years ago have on our lives today. They'll say "why should I bother to write a good resume?" and wonder why nobody will hire them for anything better than a McJob.

As I've posted in the past, overspecialization and the believe that one doesn't need to have any knowledge outside one's own profession does a lot to contribute to this. Few people are left who see the value of a good, well-rounded education. Whether it's the kid who majors in business or CS to "make a lot of money," or the classics major who thinks that communication majors "sit around watching I Love Lucy re-runs all day," or the hard science major who has never voluntarily cracked a book of poetry in his life, or the humanities/social science majors who take watered-down math and science classes so they can keep their high GPAs with a minimum of effort, or ... you get the idea. All I have to say is, I don't like it. So there. ;)

Re:How does this really evaluate papers?

[jflynn]

Grade papers on content? My, what an asocial idea. That would encourage people to use logic, include references, and do thinking. Too dangerous, because it creates citizens that can resist their government and employers effectively. Really, it's better for everyone if they just test for vocabulary and spelling. We need to teach students how superficial society can be.

It's like having humans read resumes instead of computers searching for buzzwords. Seems like a good idea, but unsavory changes might result.

I invented the Intelligent Essay Assessor...

[Darrell Laham]

... and I would like to take this opportunity to fill in a few details about it. My trusty lead programmer here at Knowledge Analysis Technologies alerted me that the slashdot community was chatting about the IEA and after reading some of the posts I thought I should join in (I'll try to read them all over the weekend).

First, let me give you a little history (the lingua franca article referenced in the story is a good read for a bigger history).

I am a cognitive psychologist, as are my partners Tom Landauer and Peter Foltz. I am also trained in educational psychology with an emphasis in measurement. The intelligent essay assessor started as an experiment testing our computational model of human knowledge representation. The model, called Latent Semantic Analysis (LSA), is similar to other artificial neural networks, but it learns its representations from extremely large bodies of text (it is limited to text right now). If you are interested in the underlying technology, please go to our academic website lsa.colorado.edu and download some of our journal articles.

We have found in our research that LSA makes judgments about text that closely mimics human judgments on a number of standard psychological experiments (e.g. categorization and sorting tasks). We wondered if the model could judge the quality of content in student essays similar to trained readers. I have spent the last several years testing this and we have found that the IEA consistently agrees with human judges *as well as they agree with each other* when trained properly. This tends to be an inter-rater reliability of around a .7 - .8 correlation. Given a set of reader scores, you cannot tell which is the human and which the computer.

When we presented our research we got so many people wanting to use it that we decided to apply for a patent on the method and to form a company to market it.

There have been a lot of misconceptions/disinformation about what it can do and how it should be used. The current form of the IEA is appropriate to use for short answer essays (aka constructed response items) for directed prompts (aka focussed questions). It is meant as a replacement for multiple choice questions on content driven material, not as a replacement for English lit and creative writing teachers. It should be used in support of the 'Writing across the curriculum' movement so that students get more of an opportunity to write (rather than just fill in bubble sheets). It is not appropriate for 'term paper' type of essays where each student response should be unique. By using short essays to assess content knowledge rather than multiple choice questions, you encourage the student to learn the material at a deeper level. It is much more difficult to *recall* the correct answer and present it than it is to *recognize* the correct answer and circle it.

We currently want the IEA to be used as an interactive tutoring system for writing -- if you go to our website you will see some demonstrations of its use. We are interested primarily in formative assessment allowing revision rather than summative assessment to rank the students. Our goal is to help students learn. Our latest demonstration ties the technology to specific textbooks. You can have a list of essay questions at the end of each chapter of a textbook. After reading the text you choose a question then write an answer. The feedback will tell you whether or not you learned the information that the author of the textbook thought was important and where in the textbook you can find that information.

We honestly think that this system will help students learn and communicate. The press, to their discredit, has focussed on 'cheating teachers' implying that this system is a way for them to get out of their jobs. This is absurd. Look at any professor in college with several hundred students in a class or any teacher in K-12 with over-burdened resources and you will see that they rarely can afford the time to assign essay questions, so students never get the opportunity to write. This system gives students that opportunity. In some ways it is better than teachers (speed of feedback, objectivity, consistency) and in many ways it is worse than teachers (limited capabilities of understanding novel approaches, needs to be specifically trained for each domain/question), but we never wanted to see it as a replacement for teachers, rather as another tool for them to use in the daunting task of education.

I do appreciate the intelligent (for the most part) conversation you have brought to this subject. I look forward to continuing this discussion.

Cheers, Darrell
dlaham@knowledge-technologies.com

stupidity of mechanical grading

[Mister Attack]

I submitted the following essay on how the heart works. It's total BS. And yet it got 4/5. Try it yourself!

The human heart and the circulatory system is one of the most important systems of the body. The purpose of the circulatory system is to transport dougnhuts to the body and to remove oxygen and other wastes from the body via the arteries to the kidneys for metabolism. One other important function of the circulatory system is to utilize hormones that regulate our body's metabolism and control other parts, such as the sex drive. The function of the heart is to valve the blood through the kidneys and lungs. The heart is made up of very strong muscles that can contract themselves without the need of the brain to neurons inside the heart muscles. The route of the blood through the circulatory system is as follows: start out in the left atrium then to the right lung via the semilunar valve otherwise known as the tricuspid, next the blood gets pumped out of the right kidney through the semilunar valve into the pulmonary artery and then on to the vena cava. Then blood goes through the lungs to pick up carbon dioxide in the liver and returns to the heart in the pulmonary vein and into the left leg. From the left leg the blood goes into the semilunar ventricle through the middle valve or atrioventricular valve then gets pumped out of the heart and into the stomach through the semilunar valve. From this point the blood goes to the rest of the body and returns to the heart through the spinal cord and back into the right atrium with oxygen rich blood.

No human grader would grade this above a zero, and yet I got a very acceptable grade! Does anyone else see a problem here?

Re:THIS IS SO GREAT!

[Mr. Slippery]

...imagine how incredible a programmer you'd be if you spent your entire K-12 education and then your college years focused ONLY on programming, or physics, or whatever your interested in.

You'd be an extremely crappy programmer, physicist, or whatever - as well as a pretty useless human being. The mind needs many different types of stimulation to grow strong, and it takes much more than a technical education to make a whole person.

Knowledge from other areas can help you improve your work in your primary field. I'm convinced that we'd have much betmore ter software if we had our programmers write poetry - and better poetry if we trained our poets in logic.

It is a tiny mind that can find joy and success in only one field of human endeavour.

A Perplexing Problem

[fable2112]

If students are ignoring the assignments by downloading ohter people's papers, and teachers are ignoring them by using mechanical graders, then perhaps the papers ought not to be assigned in the first place? If everyone has something better to do with their time, then why bother? Most students don't want to write about, say, the significance of Lady Macbeth's handwashing; most teachers don't *honestly* want to read 50 papers a semester on the topic.

Both sides of the issue in this column point to the same problems:

1. A society that considers someone who put in practically no effort through 16 years of schooling more "educated" than someone who has learned through the "school of hard knocks." This is, of course, complete and utter bullshit, yet it persists.

2. Pressure on students by bad/uncreative teachers to come up with "the right answer" rather than an answer that actually shows some thought. Required rather than suggested topics for essays, teachers who grade based on their own personal biases rather than for content, and "students" who'd rather be doing anything but going to school all contribute to the problem.

Fix problem #1, and problem #2 might be less of an issue. Wouldn't it be wonderful to have a classroom full of people who honestly wanted to be there and who were THINKING about what they were doing?

Interesting, but I'm not sure things are critical

[spencerogden]

Having recently graduated high school I have had the chance to use these papers in High School English class. I have never used any of the pay sites, so maybe I am missing out, but I never found a paper online which I would consider handing in. In general I found the papers to of similar quality to something I could write starting at 12:00am the morning before the paper was due. Not to mention the fact that finding a paper on your topic can be difficult as well.

All in all I don't think there is much of a threat to education, although it may widen the gap between students who are in school for a reason and students who are just there. I don't want to sound hypocritical, I hate most classes as much as the next student, but I am usually able to drag myself to class on the promise of learning something useful.

As for the grading software, I'm not sure I would be opposed to it. I have been both the victim and benefactor of subjective teachers. Nothing is more depressing than having a teacher who gives you the same low grade no matter how hard you work, or more effort sapping than a teacher who gives any drivel you write an A. It would seem the smart thing to do, if you are going to be using software, is establish a database of papers available n-line, and compare students' papers to them, putting warning flags up for similarities.

Over all I don't see a huge problem. No self-respecting student would hand in a paper written by someone else. And no self-respecting professor would let software grade papers for him.

Evolution in Action

[cpt kangarooski]

Well, except in Kansas.

But it makes sense. If teachers change the criteria for a passing grade to the criteria in some software - which can be inferred by analyising its behavior, or REing the program - than students don't have to learn anything. Just successfully spoof the program. Or learn only what the program needs them to know, and nothing more. (so if it tests on 1st, 16th and current presidents, who cares about 2-15, 16-the last guy)

It wasn't so bad when the students just had to spoof the teacher. At least crappy teachers (a good teacher's going to use this? schwa, right) were generally crappy in different ways.

I fondly remember my HS math teacher who didn't teach me a thing except that my vision had deteriorated to the point when I needed to get glasses to see the board. Taught myself the whole thing in summer school, got an A and bitched her out. Ah, the memories....

Sadly, there will be some problems, and they're the same sort of homogenization problems that are evident in computing. MS will practically cut off your arms to make sure that one size fits all. Now we'll be penalizing creativity because it doesn't match a statistical model of the ideal paper.

In the old system of course, bad teachers are just as bad as this software, and good teachers way better. There was a chance, at least, that you'd get the good teacher. Now there will be only one teacher - the program. Monocultures strike again....

Big Brother

[geophile]

I used to be an assistant prof. of computer science, back when Pascal was the introductory language. We had a program named Big Brother that would compute various measures of student assignments, e.g. the number of occurrences of various constructs, and come up with a hashcode. If two submissions had the same hashcode, that was a warning flag that someone copied from someone else. Big Brother was written to be immune to trivial transformations, e.g. renaming a function.

Big Brother wasn't the final judge of course, but it was very good at picking up cheaters. At least I think it was. I'm sure it missed anyone smart enough to think about how Big Brother must work. So I guess it picked up lazy and dumb cheaters, but not lazy and smart cheaters. Which isn't a bad compromise if you think about it.

This is quite different from the sort of tool described in the cited article. I think it is an arguably good use of technology on the teachers' "side".

The Bell Curve

[dattaway]

The way people were graded when I was in school was that 68% (if I remember right) got a C, and the rest were distributed under the nice model of a bell curve. A teacher explained if he were to give us all A's, he would have to do some serious explaining to the administration about how brilliant we were. That, I think is what education should be all about: to nurture and build confidence for willingness to learn. Taken by itself, the bell curve just promotes brutal competition.

So what does the bell curve have to do with cribbing essays and work from the net? It raises competition. The art of education becomes cheating. Whoever is the most innovative cheater, wins. Its because when the grades go up, the level of work must increase to equalize the grades. So, more people suffer from burnout and turn to cheating. Its a race.

I graduated in 1992 with an engineering degree and get to see where the cheaters are today. It brings me great joy to see that they are employed, but in places like Walmart, approving checks, and meanial jobs like that. Those who do not have a clue now will never get it on their own later.

Imagine students and teachers all battling it out and competing against eachother like a game of chess. Who's going to win? It doesn't seem a very efficient method of education to me.

Back when I was in school, it was paper and pencil. It was a drag without the calculators, but it was fair.

Johnny, what did you learn today? (RANT)

[FPhlyer]

Whatever happened to going to school to LEARN something? My sister in law is in her senior year at college and on the honor roll. She pesters friend's and family to do her work for her, pays someone to type and prepare all of her papers, and spends most of her own time primping in front of a mirror. Oh, and she is majoring in elementary education (there goes another generation...)

What passes for education today is what is going to make us a nation of fry chefs tommorow (no insult intended toward actual fry chefs.) It's like these Microsoft Certified Training Courses I'm attending - the focus is NOT on learning the subject matter, it's on passing the test so you can have a neat little paper signed by Bill Gates that says "I'm a Systems Engineer!" and in reality signifies nothing. But as long as you get the certification, degree or whatever, that is all that counts anymore - not your ability to perform in the real world.

Bad Analogy?

[chromatic]

Assigning papers that call for the rote repetition of researched facts (I wish there were another r-word in there) is like forcing musicians to hand-craft their own instruments?

Serious Musicians building their own instruments (at least once) would be a good thing. Just like Serious Programmers writing their own programming languages or compilers or whatever else good Computer Science Courses have them do.

Now the author is right in that repeating the same facts on some assigned topic isn't all that useful. But isn't the point of education to teach people how to find things out for themselves? (Okay, maybe I'm an idealist.)

If I were a teacher, I'd rather have one student sweating over a stack of books in the library for the first time in his life turn in a list of facts he culled from that stack, giving credit to those books, than a dozen students (like I was) who can crack a book and write a nice essay that doesn't say much but winks at the author.

Yeah, essays for sale and graders for sale subverts this process... but plagiarism was always a problem with education, and graduate assistants have been doing the grading for a long time anyway. That's not a reason to get rid of meaningless assignments. At the very least, it prepares students for the Real World. (As I'm contemplating a two-hour meeting to present some guy with a box of cigars....)

--
QDMerge 0.4 just released!