Indiana First With Computerized Grading

I smell lawsuits, how about you? by Allen+Zadr · 2004-05-20 05:00 · Score: 5, Insightful

... and produced results virtually identical to those of trained readers.

Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."

I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).

--
Kinetic stupidity has a new brand leader: Allen Zadr.

Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 05:03 · Score: 0

That's nothing which wouldn't also apply to human grading though.
Re:I smell lawsuits, how about you? by m0rph3us0 · 2004-05-20 05:04 · Score: 5, Funny

I think you've taken one to many collage level english classes my friend.
Re:I smell lawsuits, how about you? by DrEldarion · 2004-05-20 05:05 · Score: 1

I'm sure they wouldn't make it so there was no dispute process. They'd have to be complete morons to make it so the grade the computer gave you was final. (not that that's unheard of from high-school personnel...)

I'm sure the computer can't tell sarcasm, or tell when a paper is a parody, or give extra points to a particularly good paper from a stupid student or take away points for a particularly bad paper from a brilliant student.
Re:I smell lawsuits, how about you? by m0rph3us0 · 2004-05-20 05:07 · Score: 5, Funny

s/to/too/. I feel stupid now.
Re:I smell lawsuits, how about you? by magefile · 2004-05-20 05:10 · Score: 2, Insightful

extra points to a particularly good paper from a stupid student or take away points for a particularly bad paper from a brilliant student.

A "stupid" student doesn't deserve more points just for improving than a non-"stupid" student, and vice versa.
Re:I smell lawsuits, how about you? by Shalda · 2004-05-20 05:10 · Score: 4, Interesting

I disagree. I see lawsuits as no more likely. Furthermore, any process where you're subjectively evaluting something there has to be quality controls and an appeals process. My wife once held a part time job grading essay questions on a high school exit exam. Every few hours of grading exams, she would have to take and pass a "calibration battery" of 10 exams. Quality control is fundamental to the process.

What I see as being problematic is kids learning to beat the system. Typically these systems are predicated on gramatical analysis (use of punctuation and sentence compeleteness) and evidence of citing the text the question is based off. I'd bet its a real easy system to beat.

--
bance.net
Re:I smell lawsuits, how about you? by ari_j · 2004-05-20 05:11 · Score: 1

I don't think that bad grading is what's hurting your ability to get a writing-based scholarship, bud. Your comment is barely comprehensible, has almost no examples of correct English grammar, and has some spelling issues and capitalization deficiencies, as well.
Re:I smell lawsuits, how about you? by dtfinch · 2004-05-20 05:11 · Score: 0, Redundant

You to.
Re:I smell lawsuits, how about you? by Troed · 2004-05-20 05:16 · Score: 1

Exactly. How would the computer know that I tried to make a mockery out of 1337 sPeAkErS in the middle of a s3nt3nc3?

"It looks like you're writing a letter ... "

--
it's in my head
Re:I smell lawsuits, how about you? by Allen+Zadr · 2004-05-20 05:17 · Score: 1

I disagree. The computer spits out a number, and you get to live with the results. I imagine a paper that's both on topic, and well organized... but uses different verbage in the initial description and the conclusion. Perhaps using words that were introduced throughout the body.
A computer will probably not be able to recognize this. A Human would.

--
Kinetic stupidity has a new brand leader: Allen Zadr.
Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 05:18 · Score: 1

I read this article on DrudgeReport.com HOURS ago. Way to be on top slashdot.

As for lawsuits... why? They've done tests and found that the computer grading gave pretty much the same grades/choices that real life professionals gave on the same tests. In addition, a computer doesn't have the bias that teachers have. Ever been a kid that a teacher just loathed and despised - for no discernable reason whatsoever and had to make over a HUGE curve just to get a "fair" result? Or the opposite - a kid that the teacher favors and so you get a grandious positive curve in your grade thanks to that preferential treatment?
Re:I smell lawsuits, how about you? by DrEldarion · 2004-05-20 05:19 · Score: 3, Insightful

Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

This is why essays SHOULD be subjectively graded instead of objectively graded. You need to take into account the writing abilities of the student and determine if it's a good or a bad paper based on what they're capable of in addition to technical aspects of the paper.

Isn't this why there are remedial and accelerated english classes? To take into account the different levels of intelligence in students? If you took an 'A' paper from a remedial class, it's quite likely that it would be a 'D' in the accelerated class.
Re:I smell lawsuits, how about you? by micromoog · 2004-05-20 05:19 · Score: 1

Then what's the point of education, if not improving one's own abilities?
Re:I smell lawsuits, how about you? by Glonoinha · 2004-05-20 05:19 · Score: 1

-I'd bet its a real easy system to beat.

Ya think? Yea, it's called stealing a copy of the program, running your essay through it to see what it doesn't like, and continue to tweak it until the program gives the essay an A. Submit essay for grading, rinse, repeat.

--
Glonoinha the MebiByte Slayer
Re:I smell lawsuits, how about you? by ibpooks · 2004-05-20 05:21 · Score: 1

Typically these systems are predicated on gramatical analysis (use of punctuation and sentence compeleteness) and evidence of citing the text the question is based off. I'd bet its a real easy system to beat.

Hey, if kids are learning grammar, spelling and sentence-writing just to beat the grading program, then it was worth it. Granted there's much more to good writing than syntax, but we have to start somewhere.
Re:I smell lawsuits, how about you? by Moofie · 2004-05-20 05:22 · Score: 1

So, sort of like submitting drafts to the teacher for him/her to review?

Yeah, I'd certainly call that cheating.

--
Why yes, I AM a rocket scientist!
Re:I smell lawsuits, how about you? by Allen+Zadr · 2004-05-20 05:25 · Score: 1

I agree with this to a point, yet, there are considerations that a human will take into account. For instance bad grammar produced by quoting or paraphrasing a source.
For instance, an essay on Tom Sawyer would naturally be full of intentionally bad grammar, and words that don't exist (archaic slang). This would be easily recognized by a human, but would be difficult for a computer to grade fairly.

--
Kinetic stupidity has a new brand leader: Allen Zadr.
Re:I smell lawsuits, how about you? by Bombcar · 2004-05-20 05:26 · Score: 2, Funny

I wonder if it has bounds checking on input..... perhaps you could submit /proc/kcore and cause a buffer overflow and get r00t on the gr4d3b00k!

--
Fellowship 9/11
Re:I smell lawsuits, how about you? by nine-times · 2004-05-20 05:26 · Score: 2, Insightful

The way I read it was "Makes sense. With some of the stupid teachers I had in public school, a chimpanzee with a box of crayons could produce results that are virtually identical."
Re:I smell lawsuits, how about you? by king-manic · 2004-05-20 05:29 · Score: 4, Insightful

Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

Why punish those with ability, why reward those who are not talented. Surely they won't be punished or rewarded in the same manner in real life. If the talented one was cruising and not expending any effort, thats his perogative. If he can be a productive member of society without effort thats fine, even if he has the potential to eb the next great mind, it's still his choice. The rewarding of those who work hard is important but if they work hard to only measure up to the minimium standard then they belong to that minimium standard regaurdless of how much effort they put in.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:I smell lawsuits, how about you? by Anonymous+Custard · 2004-05-20 05:30 · Score: 3, Interesting

The real answer is to adjust your teaching methods per student based on subjective analysis. A low objective mark on a paper or test would indicate that, ideally, the teacher needs to pay closer attention to the needs of that student, and teach him in a way he can learn. (VERY few teachers have the time and skill required for this, unfortunately).

You can't grade subjectively because those grades will be compared objectively down the line. You can't say "this is pretty good for kevin, I'll give him an A", but then say "josh's paper is way better than kevin's paper, but josh is a bright kid, so I'm giving him a C". Kevin will think he's mastered the english language while Josh will go insane trying to achieve perfection.

Grading, when used for anything other than helping the teacher learn about each students, just plains sucks, and is only used for competition.

--
$8.95/mo web hosting
Re:I smell lawsuits, how about you? by Ryne · 2004-05-20 05:36 · Score: 1

I definitely think that the same paper paper produced by students with differing talent would get the same grade.
If you try to weigh in how much effort a student have put into a paper, you'd get a very uncertain process.

You would also get a very weird outcome. Namely that a student that works his ass off, but still can't produce papers of the same level as a student who cruises through school would get a higher grade. So, in essence the grade would be based on effort, not on quality, and that I think is absurd.

I'm not very familiar with the US school system, but we have something like accelerated classes (not as clear though, I believe) and I think what you're saying is a problem with them. A student that produces papers that are consistently of a higher quality than the papers from students in the remedial class may get a lower grade just because the other students in his class also are better.

Talk about taking obective judgment away.

IMO grades should show the quality of the work a student has produced, nothing else, and the standard against which the grades should be measured should be a global one, not a local one.
(Then there's the question if grades are a good measurement, but that's another story.)
Re:I smell lawsuits, how about you? by king-manic · 2004-05-20 05:36 · Score: 1

In my area of the world (Alberta, Canada), our standardized tests are administered and taken in one sitting. No chance to revise the essay or polish it. If I have learn proper gramar and writing skills to cheat the system, then I've learned all the system shoudl have imparted any way. My gramar and spelling is sub par simply because I'm an individual who cheated the other way. I never learned proper gramar or spellign because our system switched to a more subject and idea oriented english. This english is geared towards girls. The stats show ever since they switched the ciriculum girls have done much better and boys have slid down. I cheated the system because I'm a good speech writer. Even though my gramar sucks and my spelling is horrible I can portray my side very well and convince a grader that I knwo the subject, even though I'm suppose to be graded on the structure of my essay not just my arguments. Thus I got amazing marks in english without trying hard but I always feel my gramar and spelling is horrible and I don't knwo how to write a well structured essay.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:I smell lawsuits, how about you? by realdpk · 2004-05-20 05:36 · Score: 1

There's no need for life to be harder for the smart students. It isn't fair to ask them to work extra hard to get the same credit, and at the end of the day, the very same piece of paper that a less-smart student receives.

You do need to understand the writer's abilities so you can encourage them to do better, if necessary. But there's no reason to withold giving a student an A, simply because they've had A's the whole time and haven't shown any improvement.

In the case of a student doing way better than their peers, it's more appropriate to challenge them further. Move them in to the accelerated classes. Let them take on the advanced cirriculum, and pass or fail it there.

At least then they can say they took AP classes, which will help them stand out from the rubber-stamp high school diplomas if they decide to go to college.
Re:I smell lawsuits, how about you? by jeffmeden · 2004-05-20 05:39 · Score: 1

Yeah its really easy to beat. Get this, the student simply takes words that fit grammatically into sentences, then arranges some of these sentences together into contiguous-thought paragraphs! oh, wait.
Re:I smell lawsuits, how about you? by RoderickMcDougall · 2004-05-20 05:39 · Score: 1

I smell an uprising from the teachers when they realise they are being replaced by robots :) Man I hope they don't code software to emulate unemployed government support sapping wastes of space or I'll be out of a job.
Re:I smell lawsuits, how about you? by cloudmaster · 2004-05-20 05:42 · Score: 2, Interesting

Darn straight. I'm all for making dumb people feel not so bad about things they can't control - like being dumb - but not at the expense of making the non-dumb feel dumb. There's a lot to be said for working smarter, not harder, and a lot to be said for those who have figured that out on their own. English provides several correct ways to do something and some incorrect ways of communicating. An "A" paper is an "A paper, regardless of the source and effort put in by said source.
Re:I smell lawsuits, how about you? by kiatoa · 2004-05-20 05:46 · Score: 1

Ya think? Yea, it's called stealing a copy of the program, running your essay through it to see what it doesn't like, and continue to tweak it until the program gives the essay an A. Submit essay for grading, rinse, repeat.

Write a genetic algorythm script to generate papers using the test evaluation program as a fitness test. Would be facinating me thinks.

--
90% of the wealth is in 2% of the pockets. Bummer to be in the majority.
Re:I smell lawsuits, how about you? by Ateryx · 2004-05-20 05:49 · Score: 1

dropping out of collage[sic] level english classes

Slowly but surely everything is going automated. Now I am probably a bigger advocate of computerized systems taking care of everything making life easier for John and Jane Civilian, but essay grading is among the very short list of no's. Think about the best teachers you had in high school/college... did they just toss your crappy paper back or try to put some serious thought into the review? I know my best teacher said she calculated once that comparing what she gets paid to her salary, she is making as much as her brothers uneducated roofers... roughly $20 an hour... with 2 masters degrees.

--
"The truth suffers from too much analysis"
Re:I smell lawsuits, how about you? by CaptainVic · 2004-05-20 05:50 · Score: 1

I agree that subjective grading in an English class is appropriate. It is a fallacy to believe that an employer will value average work from a clearly above-average worker the same as average work from an average or below-average worker. Employers perceive abilities when they hire you, and working below their perceptions is always a risk. That is why enhancing a resume to get a job is a bad idea. I have seen workers fired for not meeting the hiring interviewers perceptions of what he could do. --vicki
Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 06:08 · Score: 0

So, sort of like submitting drafts to the teacher for him/her to review?

Yeah, I'd certainly call that cheating.

No, actually that's called an education, which isn't happening either.
Re:I smell lawsuits, how about you? by Dr.+Evil · 2004-05-20 06:08 · Score: 1

It's the glue.
Re:I smell lawsuits, how about you? by coyote_oww · 2004-05-20 06:14 · Score: 1

What I see as being problematic is kids learning to beat the system.
The whole point is to get the kids to "beat the system". It's otherwise known as learning. Produce some writing that meets certain criteria...
If your point is that the criteria are wrong, then we can change that. If the point is that "writing is something that can't be judged" then why have teachers grade it either? Just expose the kids and call it good - base the grade strictly upon attendance (since that is your measure of exposure). The only alternative to this is to have some performance metric - do ____ to demonstrate that you can write. Using a computer to measure ____ seems like an improvement, as a computer won't mark you down for race, religion, opinion, etc - things that tend to drive humans to have lower opinions of each other (and consequently can bias grading).
Re:I smell lawsuits, how about you? by TyrranzzX · 2004-05-20 06:16 · Score: 1

Nah, I smell a time when us hackers finally get straight A's in the classroom :D. These pieces of software always have bugs, weither intentional or unintentional. Kids'll cheat, as they always do, cept they'll have to be real smart about it. Remember a typing program where you could hold down the space bar and it'd give you 0 errors, and tons of words per minute.

In any case, why the hell is a computer teaching a human language? I can see it helping to teach math, helping to teach science, and helping to teach social studies, but english can't be taught by a machine. English is learned by reading books and writing inspired writing, a machine can help you check grammar and spelling errors, it can't give you advice on a story through. This is why those of us who rot away on UBB's and do a lot of arguing have excellent spelling and grammar and writing in general (although everyone makes mistakes and has a different style, grammar tends to restrict the style imo).

--

Candy-Coated Knowledge
Re:I smell lawsuits, how about you? by corngrower · 2004-05-20 06:18 · Score: 1

Grading should always be as objective as possible. The grade received on a paper should never depend on who the student is. If you are an instructor, you can certainly comment on the fact that you feel that the work wasn't up to the student's usual standards, or that the work exceeded the usual standards, but the actual grade should not depend on the student.
If the student's are that much different in abilities, they shouldn't be in the same class. One class, one standard for grading. If you were a teacher in my school district, I'ld see that you'ld be fired I ever found out you were doing this shit!
Re:I smell lawsuits, how about you? by slam+smith · 2004-05-20 06:21 · Score: 1

Or take points away from you when they disagree with the premise of the paper.

Or take points away when you arent' one of thier pet students,

etc.

--
My Weblog
Re:I smell lawsuits, how about you? by wcrowe · 2004-05-20 06:23 · Score: 1

Surely they won't be punished or rewarded in the same manner in real life.

True. The less-talented one will be promoted to management.

--
Proverbs 21:19
Re:I smell lawsuits, how about you? by clifyt · 2004-05-20 06:32 · Score: 5, Interesting

Actually, I've been a part of writing software like this for their competitors and have worked with this software in the past as part of my duties as manager of development at the IUPUI Testing Center (thats Indiana University Purdue University Indianapolis). We've worked on this shit for about 10 years now.

One of my tasks in the past was to push this type of software onto the local schools. We've used it for rating in class essays.

The idea is that everyone knows that the only way you get better writing is to write more and get some feedback on it. It doesn't matter if you are an educator throwing the papers back at the student or a computer algorithm. It all forces the student to find the mistakes and not make them the next time.

The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day. So this is where this type of software comes into play -- you assign 2x the work you can normally handle, and let the computers handle half of it. You don't tell the students which assignments will be computer rated. Thus the students grades got better. Not much better, but they were better than the students not using the system in the same types of classes.

One of our smaller studies actually had us installing this software locally for instant feedback. It was a small percentage, but the students work was even better than before.

Yeah, you 'steal a copy' if you can't seem to get one given to you, and run it through until it likes it. How is this cheating or anyway underhanded. One of the better and far more dedicated educators I know actually allows students to hand in papers and have them marked up as many times as they want until the paper is due. His students final works are generally light years ahead of other educators in his facility that don't have the dedication (and for $26k a year, do you really want to give up your nights and weekends???).

Same thing here.

Shit, even using the grammar checker under work will force you to learn to write better (up to a point). You learn what its looking for and you avoid it. I'm not a good speller and I know the spell checkers help me learn after I hit the same error over and over again.

All these tools work for you in the learning process as long as you are willing to not just put this stuff on autopilot.

As for the title of this thread -- Lawsuit? The only lawsuits will come from idiots. None of the high stakes testing does purely computer rating. They all put humans into the equation. You will most likely get better rating because instead of having three or four humans look at your paper for 30 seconds each before moving on, you will now have one that is able to devote some serious time to it. All these humans will still be working just as many hours as before, but studies have shown that the eyeballs on the paper are there longer with this type of software than without (sorry, but these studies belong to the bigger testing companies or I'd post links...I just get paid to crunch the data).

Secondly, 5 years ago when I was working on this stuff full time, the software had a human agreeance of around 62% with a rater pool of 3 raters. Meaning that if you asked 3 people what they thought, took an average of this, and then asked a fourth, 62% of the time, the human agreed with the others. This was on a 12 point scale. The application, however, actually rated between 70 - 80% of the time depending on the model used.

In both cases, the raters were all trained together with the same things to look for, and the models were designed around this rater pool -- in a sense trying to simply guess at what the others would pick. The computer:human agreeance was higher than the human:human agreeance.

Back to the parent post, beating the system only means you beat learning to write.

BTW -- My post is not indicative of my writing skills outside of a conversational and informal setting, sans spell checker and proof re
Re:I smell lawsuits, how about you? by Short+Circuit · 2004-05-20 06:39 · Score: 1

Why punish those with ability,

It's a penalty for being lazy.

why reward those who are not talented.

It's a reward for those who work hard.

What's at issue is how well students are utilizing the abilities they have. An ability is worthless if the person who has it doesn't use it and extend it.

--
tasks(723) drafts(105) languages(484) examples(29106)
Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 06:40 · Score: 0

ETS (creator of e-rater) claims that the grades match over 98% of human grading. This number is deceptive. For example, lets say that the system is good at judging essays with "average" english, but cannot understand / fairly judge essays with nuances, uncommon (but correct) grammar, etc. Since 98% of students may be "average" or worse, grades for their essays may match close to 100%, and the "above-average" students may get the shaft.
Re:I smell lawsuits, how about you? by WalksOnDirt · 2004-05-20 06:42 · Score: 1

For now, this automated grading is probably meant for standardized tests, where adjusting grades for student ability does seem completely inappropriate.

In a classroom setting, however, a teacher's main goal is not evaluating students, but educating them. If marking down a student for not not trying hard enough motivates the student to improve, that's good teaching. If this technique is used, I do feel strongly that it must not be allowed to hurt their permanent record, i.e. report cards.

--
a,e,i,o,u and sometimes w and y (at be if of up cwm by)
Re:I smell lawsuits, how about you? by king-manic · 2004-05-20 06:45 · Score: 1

An ability is worthless if the person who has it doesn't use it

Using it and working hard aren't the same. If I have an ability that makes me able to spew good essays at will, then I may spend 20 min to write one but I used it. I didn't work hard but I used my abilities. Work and effort are much less important then ability and talent. If I finish a web page in 30 min, it looks good I get rewarded. If I work hard and make one in 30 hours thats just as good I also get rewarded. However if the two compete the talented one will win. If I work hard and have talent then I will be rewarded more. Thus the papers shoudl be marked and thus will we be treated in the real world.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:I smell lawsuits, how about you? by Short+Circuit · 2004-05-20 06:49 · Score: 1

Ah, but to extend it, you have to work hard at it. It's like exercise. If you don't push yourself, you don't really improve.

--
tasks(723) drafts(105) languages(484) examples(29106)
Re:I smell lawsuits, how about you? by king-manic · 2004-05-20 06:54 · Score: 1

Ah, but to extend it, you have to work hard at it. It's like exercise. If you don't push yourself, you don't really improve.

That may be. But you don't have to extend it for it to be useful. The objective to life is to live, and to be happy if you can manage it. Not to die the most talented or have the most money or to measure up to someone else criteria. If you can be happy cruising then who are you to critisize, if you can be happy workign your ass off to be average then that too is something I cannot assail.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:I smell lawsuits, how about you? by Eccles · 2004-05-20 06:58 · Score: 1

Bell's commandment: Judge not syntax, lest ye be judged.

--
Ooh, a sarcasm detector. Oh, that's a real useful invention.
Re:I smell lawsuits, how about you? by hesiod · 2004-05-20 07:02 · Score: 1

> An ability is worthless if the person who has it doesn't use it and extend it.

Let's say I am a stone cutter. I've apprenticed for years, and read just about everything there is to read. I know my shit, but I am lucky enough to be able to do it without much thought, just because of my innate artistic skill. I make a decent living because of my abilities. I don't feel like extending it, but is it still worthless? Not at all.

> It's a penalty for being lazy.

Hey, here's a message from the real world: please come back.

Two people can do the same work, but one takes half the time to do it. Which one is going to be hired? By your logic, the "lazy" one.

The problem is that you mistake efficiency for laziness. Punishing those with ability is punishing those who are more efficient. Efficiency should be the goal, not something to be punished for.

I am lucky enough to be able to slack a bit and read /. because I am pretty efficient at what I do. I don't expect to be fired and replaced by someone dumber just so that they spread the same 6 hours of work into 8 hours. That means they are inept, not deserving of a payraise.
Re:I smell lawsuits, how about you? by galego · 2004-05-20 07:03 · Score: 1

And you haven't even gotten to the lost jobs and unions factor, yet.

--
Que Deus te de em dobro o que me desejas
[May God give you double that which you wish for me]
Re:I smell lawsuits, how about you? by Flexagon · 2004-05-20 07:04 · Score: 1

This is why essays SHOULD be subjectively graded instead of objectively graded. You need to take into account the writing abilities of the student and determine if it's a good or a bad paper based on what they're capable of in addition to technical aspects of the paper.

And would you pay more for a contractor who clearly worked harder and longer but didn't complete what you asked for, versus one who came in and did meet the specification in 1/2 the time? Include brain surgeon or oncologist among the the class of contractors. I certainly wouldn't pay more. So I'd expect their grades to be directly proportional to their results, not their efforts, however sincere.
My father told me about an elementary school teacher he had. He got a B+ on a paper, but wanted an A. So he spent a lot more effort on the next paper, but got a B-. He asked the teacher why, on a paper she agreed was better then the previous one, and she said, "Well, I expect more of you now.". How supremely motivating.
Or, a TA I had in a programming class taught by a famous professor. The weekly assignment included writing a review of another team's previous week's project. Our team did a full QA effort on another team's work, including running test cases through it in addition to a code review. We got a B. I looked over the reviews that got A's, and found that they were all superficial. Most didn't bother to run any tests at all, and many were heavy on style (as in tabs versus spaces) versus substance. So the next week we wrote a superficial review in a fraction of the time and got an A. Again, how supremely motivating.
The two best classes I took, by different professors, had the same grading algorithm: after getting a project approved for reasonable scope at the beginning of the term, you got an A if you finished coding and it worked, a B if you finished coding and were close to working but not done, and a C if you were done or close to done with coding but not working at all. A far more rational and realistic grading scheme.

Isn't this why there are remedial and accelerated english classes? To take into account the different levels of intelligence in students? If you took an 'A' paper from a remedial class, it's quite likely that it would be a 'D' in the accelerated class.

This sounds like social promotion to me.
Re:I smell lawsuits, how about you? by david_reese · 2004-05-20 07:08 · Score: 1

The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day. So this is where this type of software comes into play -- you assign 2x the work you can normally handle, and let the computers handle half of it. You don't tell the students which assignments will be computer rated. Thus the students grades got better. Not much better, but they were better than the students not using the system in the same types of classes.
I think the general problem is in implementation. If it works as you describe, that'd be great (and impressive). The problem is where the teacher assigns 2x the workload, then has the software grade ALL of it.... consistently.
Life has shown me that people are lazy. What kind of safeguards are in place? Is there process (hard-coded) that will eliminate this pathological use case I presented? If not, then I do think that even providing teachers/schools this service is going to cause more harm in the long term than good.
That being said, I do appreciate and admire the amount of work to get an AI (expert system, perhaps?) that achieves this work... but I fear the misuse.
Re:I smell lawsuits, how about you? by hesiod · 2004-05-20 07:09 · Score: 1

> Employers perceive abilities when they hire you, and working below their perceptions is always a risk

I disagree, in a way. The employer has a job that needs to be done. Anyone who can get it done efficiently deserves the job. HOWEVER, someone who does it MORE efficiently should have more opportunity for advancement and payraise.

The problem you point out is not a problem of employer's perception, but a problem of some doofus lieing on his resume. If I said I can write multiprocessor compilers, then can't when asked to, I obviously lied, which is generally a violation of the employment contract and I deserve to be fired. If I said I could, I get there & can, I do not deserve to be fired. If I had the ability to do it twice as fast, but choose not to, the employer has no right to fire you, as long as you aren't pokey (excessively slow) about it.
Re:I smell lawsuits, how about you? by hesiod · 2004-05-20 07:13 · Score: 1

> Then what's the point of education, if not improving one's own abilities?

That IS the point of education. The point of grading, however is to show your level of competency, which a subjective grade does not show.
Re:I smell lawsuits, how about you? by Hatta · 2004-05-20 07:13 · Score: 1

The problem isn't getting students to write more, its getting educators to grade more. There isn't enough hours in the day.

I think what you mean is, "There aren't enough teachers per student."

--
Give me Classic Slashdot or give me death!
Re:I smell lawsuits, how about you? by Eccles · 2004-05-20 07:14 · Score: 1

It is a fallacy to believe that an employer will value average work from a clearly above-average worker the same as average work from an average or below-average worker.

An employer will pay the above-average worker more to begin with.

--
Ooh, a sarcasm detector. Oh, that's a real useful invention.
Re:I smell lawsuits, how about you? by Eccles · 2004-05-20 07:21 · Score: 1

In that case, you've done a hell of a lot of work on that paper, a good bit of which is educational if the program is "competent."

--
Ooh, a sarcasm detector. Oh, that's a real useful invention.
Re:I smell lawsuits, how about you? by clifyt · 2004-05-20 07:25 · Score: 1

No, it can work either way.

With the right tools at ones fingertips, an efficient teacher can do the work of two or three and still have time to have a life out of the class.

Having more teachers is one option -- if you are willing to pay more in taxes. As one that doesn't have children and knowing the current president of the US gives tax breaks to those that do have children -- to the point where those of us childrenless have to carry a larger load in the tax system, I say there are enough teachers and not enough tools for them.

I've seen what a dedicated educator can do with an 'overloaded' class. Having more not so dedicated educators is not always the answer.
Re:I smell lawsuits, how about you? by Short+Circuit · 2004-05-20 07:26 · Score: 1

Two people can do the same work, but one takes half the time to do it. Which one is going to be hired? By your logic, the "lazy" one.

Academic environments are usually separate from business environments. And for the employee with talent, there's a choice. Do they do an OK job on twice as much work, or do they do a superb job on the work that's due?

In a business environment, quality vs quantity comes into play, as well as timeliness and set deadlines.

In an academic environment, a student is normally given a due date for the assignment, and he's expected to do the best damn job you can at it.

The primary purpose of attending an academic environment is to learn and improve your skills.

On the other hand, it's also pretty hard to stay employed until retirement without learning and improving. I'm not saying it doesn't happen, but it's also not something a student should be taught to expect.

--
tasks(723) drafts(105) languages(484) examples(29106)
Re:I smell lawsuits, how about you? by slam+smith · 2004-05-20 07:28 · Score: 2

It does seem unfair. After all at least the roofers are doing something useful.

--
My Weblog
Re:I smell lawsuits, how about you? by servognome · 2004-05-20 07:35 · Score: 1

Ugh, I always hated the "living up to your potential" teacher. Guess what there isn't a level playing field, students should learn that early on so they don't feel a sense of entitlement. If I play basketball should every basket I make count as 10 points because I'm under 6' tall?
In the end if you grade on effort you only hurt the students. How do you think that stupid kid who got straight A's all through highschool will feel when he's flunked out of college in the first year. Did you really help him?
The student who gets a "C" because he just didn't turn in a papers you thought were fitting for his intelligence level; when colleges see his transcript they don't see "wrote a wonderful essays in class, they just weren't as brilliant and polished as I expected" they just see a "C" = average student.

--
D6 63 0D 70 89 81 BB 8E 7B 7C 5F 5D 54 EA AB 73
Re:I smell lawsuits, how about you? by clifyt · 2004-05-20 07:40 · Score: 1

The same way you make certain any educator is doing their job. Do you think there are no peer reviews going on or that the school board doesn't ever ask for educators to be evaluated?

Seriously -- you make it sound like an educator goes in and has no guidance about anything.

The fact is, the software does not do everything. It does enough to give you pointers and thats about it. It takes a human to interpret this for you. Even if ALL the papers went through the system, you'd need a human to guide the students -- and that wouldn't be a bad thing either. Have the computer do the manual work, and let the human explain this to them.

One of the great things about the rating system I initially designed -- and this idea has been used in most designs I've seen -- is that the truely good and the truely bad are those that the computer cannot rate with any efficiency. These get flagged for human review regardless.

Its not hard for folks that know what they are doing to max out certain rating models on the upper or lower end. If above or below, a human *WILL* read it. For instance, one of the early models had a student quoting several lines of Haiku. That gave all but the creativity conglomerate nightmares and scored lower to the point a human had to look at it (don't even ask about the Poetry / Prose identifier that one of our collegues ended up writting in regards to these situations).

There are safe guards in place for educators of all ilk in their grading methods. Like anything, some will ignore these and get away with it. Students will cheat on tests, educators will skim papers and the world will be as it is. You can't provide a system that will catch everything, nor should you try.

By trying to build a system that tries to catch gaming of the system, you only encourage it to happen and you put the idea out that one might be able to out think the computer.

Like a hammer, you can put nails into a wall or you can bash someones brains out. Its a tool. Tools can be used inappropriately. Get over it. I don't see anyone outlawing hammers -- those that use them wrong get thrown in jail -- or at least given the threat of jail. Educators that do nothing to help their students out will be asked to leave their employment.

Do you stay awake at night fearing the misuse of hammers?
Re:I smell lawsuits, how about you? by APDent · 2004-05-20 07:51 · Score: 1

It all forces the student to find the mistakes and not make them the next time.
[...]
the software had a human agreeance of around 62%
[...]
My post is not indicative of my writing skills outside of a conversational and informal setting, sans spell checker and proof reading,[...]

You make several good points (which means I'm a dick for saying this), but unless agreeance is a technical term in this context, the preferred choice would be agreement.
Re:I smell lawsuits, how about you? by clifyt · 2004-05-20 08:09 · Score: 1

Actually, its a perfectly good word and used within some statical jargons.

The OED would claim that Agreeance is a correct word.

Preferred choice would have only to do with ones vernacular and nothing else. On the street, I would look like an overeducated dick for using this word. Well, not me -- but anyone else using the word :-)

The word has been in place since early 1700s as noted by the OED. Feel free to look it up if you choose to do so. Just remember, because of my research, I have language dictionaries in a dozen tongues as well 300 years of English (mainly so we could scan them in and build a computer generated euclidean dictionary with a z plane for timelines -- never worked as well as we thought it would).

I'm not saying anything else is going to be spelled right (or anywhere near gramatically correct), but I am in fact sure of this word.

So, doesn't that make you an even bigger dick for correcting someone when you are in fact wrong and the one you corrected had stated he might be wrong alerting one to the fact you shouldn't be verifying the text purely for anything but informational means :-)
Re:I smell lawsuits, how about you? by AndroidonPPC · 2004-05-20 08:22 · Score: 1

yes, it could lead to lawsuits indeed.
starting rant here....

hmmm.... The problem is, can a computer recognize content as valid? Will it reward for creativity, for originality (in the case of fictional writing), on-topic content, plagerism checks intelligent enough to distiguish a thought being ripped off? anyone can change word order, make use of synonymns, and say exactly the same thing as a quote from a peice of writing, but essentially, can it recognize 'credit where credit is due' type situations?

Sure, this may sound somewhat technophobic, but essentially it will rob writing of a certain dimension, because all one would have to do to get a proper rating is to appeal to the algorithm.

2004 continues to become like 1984

And I do feel like this all could lead to lawsuits. Writing is more then good grammer, diverse wording, and correct spelling. Certain clever writing devices (as in methods of skilled writing) are very hard to quantify and measure and judge. Is the computer going to say that using alliteration x number of times puts writing at y skill level, for example?

Use of this computer system could lead to teachers teaching to the test. It happens already in subjects such as math, chem, physics, and so on (my highschool ap chem spent a certain amount of time teaching to the test. however, my ap calc class spent alot of time teaching calc.)

Could this be the start of the end of creative writing for these schools?
Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 08:50 · Score: 0

Why punish those with ability, why reward those who are not talented.
Because those who have ability but don't use it are punks that deserve to be punished for squandering their gifts.
Re:I smell lawsuits, how about you? by neurojab · 2004-05-20 08:58 · Score: 2

>Using a computer to measure ____ seems like an improvement, as a computer won't mark you down for race, religion, opinion, etc - things that tend to drive humans to have lower opinions of each other (and consequently can bias grading).

Yes, a computer has no bias. It also cannot comprehend what is written AT ALL. All it can do is process a few heuristics. This means that a child that's incapable of forming an intelligent thought will get rewarded for simply putting random words in a sentence structure.

Sorry, but that's not an improvement. We already have spelling and grammar checkers that students can use at home. What we need are intelligent people grading essays to give feedback on things like coherency, strength, voice, etc.

The point of writing is to communicate. How can you test communication when the target of that communication cannot understand what is written? To give you an example: Suppose I want to test your ability to speak french. I hire a Russian who can't speak a word of French, and ask him to count the words you speak and determine if your accent sounds French. You could be speaking Swahili with a French accent and he would never know.

Using a computer to test essays is even more ridiculous because not only does it not know the language, but it's incapable of even intelligently guessing at the meaning of an expression.
Re:I smell lawsuits, how about you? by ari_j · 2004-05-20 09:11 · Score: 1

98% of students may be "average" or worse

There is no way for more than 50% of a group to be below average. So, unless 48% of the group is exactly at the average (meaning that 2% of the group somehow is so far ahead of the pack that they offset the 98% who are at or below the average), your statement is a complete fallacy.
Re:I smell lawsuits, how about you? by binain · 2004-05-20 09:11 · Score: 1

And what makes you think the teachers are gonna get the sarcasm?
You decide if I'm smiling as I type this...
Re:I smell lawsuits, how about you? by APDent · 2004-05-20 09:23 · Score: 1

So, doesn't that make you an even bigger dick for correcting someone when you are in fact wrong and the one you corrected had stated he might be wrong alerting one to the fact you shouldn't be verifying the text purely for anything but informational means :-)

Indeed it does! And I also learned something new, today, which is always nice. Seriously: thank you!
Re:I smell lawsuits, how about you? by david_reese · 2004-05-20 09:35 · Score: 1

Its not hard for folks that know what they are doing to max out certain rating models on the upper or lower end. If above or below, a human *WILL* read it. For instance, one of the early models had a student quoting several lines of Haiku. That gave all but the creativity conglomerate nightmares and scored lower to the point a human had to look at it (don't even ask about the Poetry / Prose identifier that one of our collegues ended up writting in regards to these situations).
There are safe guards in place for educators of all ilk in their grading methods. Like anything, some will ignore these and get away with it. Students will cheat on tests, educators will skim papers and the world will be as it is. You can't provide a system that will catch everything, nor should you try.
Well, I'll agree that you probably know much more about teacher accountability than I, and your response is adequate. I guess I was mistaken in thinking (from the /. article as well as the news article) that this would FULLY replace human grading (gee, overstatements in /. story sumbissions? I'm shocked!).
By trying to build a system that tries to catch gaming of the system, you only encourage it to happen and you put the idea out that one might be able to out think the computer.
I will disagree with you on this; I think that you want to prevent gaming to a large enough degree that you can deal with the "fallout" (ie, no security method is perfect, but some are good enough). Having safeguards in place doesn't neccessarily make people complacent, it's the reliance on those safeguards that create complaceny. Whether your product is game-proof enough is a question that will only be answered after the paint dries, so to speak.
Do you stay awake at night fearing the misuse of hammers?
I'll say this: Hammers have been around for a long time; software based grading has not. I have concerns about anything that is new and complex enough that most people don't understand or are aware of it.
In the end, your description of the software/process has me more interested than scared in seeing it's implementation, but you'll have to excuse my scepticism for now.
Re:I smell lawsuits, how about you? by Jardine · 2004-05-20 09:44 · Score: 1

If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper?

Of course not. If two students handed in the same paper, they both get an F because one copied the other.
Re:I smell lawsuits, how about you? by jimbolaya · 2004-05-20 12:24 · Score: 1

Sounds like what is needed is a moderation (and meta-moderation) system for essays...

--
There ain't no rules here; we're trying to accomplish something.
Re:I smell lawsuits, how about you? by Dirtside · 2004-05-20 13:53 · Score: 1

I think you've taken one to many collage level english classes my friend.
To be fair, this could be interpreted as meaning, "I think you've taken between one and many classes" :)

--
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
Re:I smell lawsuits, how about you? by autopr0n · 2004-05-20 14:04 · Score: 1

Why punish those with ability, why reward those who are not talented.

Because the purpose school is to improve. Letting a student coast on their innate skill isn't going to do them any favors either. If they can already write at that level, why even bother taking the class.

surely they won't be punished or rewarded in the same manner in real life.

No, but the smart student is going to be a lot better off if be a lot better off in the long run if they learn to bust their ass rather then coasting on their intelegence.

Also, the less intelegent student is going to be better off learning to bust their ass to produce something, rather then beliving that they can't do anything and not trying.

Treating everyone the same will ultimately let down both the smart and the stupid, and only help the average.

--
autopr0n is like, down and stuff.
Re:I smell lawsuits, how about you? by TheOnlyCoolTim · 2004-05-20 14:37 · Score: 1

If this think becomes at all widely used it is probably going to be all over the various illicit distribution channels, and everyone will just practice writing the essay that it loves.

Tim

--
Omnia vestra castrorum habetur nobis.
Re:I smell lawsuits, how about you? by Anonymous Coward · 2004-05-20 14:57 · Score: 0

Well if you are going to be nice about it I will just admit that I made it up and when you called me on it I thought to myself 'there is no way I'm gonna let some dick on Slashdot catch me making words up ...' so I hammered Google mercilessly until (luckily) I found something in agreeance with me.

But since you are cool about it I can come clean.
Re:I smell lawsuits, how about you? by stanmann · 2004-05-26 23:51 · Score: 1

How about we eliminate the bloat.

I was reading on my headlines the other day that there were more employees than students in most districts, And many of these employees have no function other than as a layer between teachers and administration

[ob officespace quote]I'M A PEOPLE PERSON DAMMIT[/ob officespace quote]

--
Food not Bombs is a nice platitude but it breaks down when you notice that the Bombees are usually well fed
Re:I smell lawsuits, how about you? by CaptainVic · 2004-06-01 04:53 · Score: 1

Whether or not he deserves it, an employee who appears to work below his potential will often be the first let go when business slows down. Maybe that's not fair, but I have been in the workforce for 22 years and have seen it. --Captain Vic
Re:I smell lawsuits, how about you? by CaptainVic · 2004-06-01 04:56 · Score: 1

Assessment at hiring time does not always show that a worker deserves to be paid an appreciable amount more, but a worker in the workforce for a while has likely established himself as deserving of more money. But straight out of the college starting gate, this might not be the case. --vicki

OSS? by Karamchand · 2004-05-20 05:01 · Score: 3, Interesting

Is this program available under an open source licence? It sounds really interesting!

Re:OSS? by twistedcubic · 2004-05-20 05:02 · Score: 3, Insightful

Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.
Re:OSS? by Anonymous Coward · 2004-05-20 05:05 · Score: 0

how the fuck is the parent redundant?!
Re:OSS? by Glonoinha · 2004-05-20 05:17 · Score: 4, Funny

Slashdot First With Computerized Moderation.

Now with Computerized Moderation the famous Slashdot message site can pre-emptively down-mod 'Redundant' posts long before they are actually 'Redundant.' The computerized modding process, called 'e-modder', uses a 6-point rating scale and uses artificial intelligence to 'mimic the modding process of human readers - including doing stupid shit like modding the first instance of a concept as Redundant'.

--
Glonoinha the MebiByte Slayer
Re:OSS? by Anonymous Coward · 2004-05-20 05:19 · Score: 0

Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.

Huh? You only need the binary to do that, whether it's open or closed source doesn't come into it.
Re:OSS? by borkus · 2004-05-20 05:20 · Score: 4, Insightful

Imagine high school students re-writing their essays ...
Actually, anything that would encourage students to re-write their papers and improve their writing would be pretty amazing. Most students jot something down, run a spell checker and turn in their work. If they could pre-grade their work, they might be better motivated to put out more effort and improve their writing.

Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.
Re:OSS? by dtfinch · 2004-05-20 05:37 · Score: 1

I'd just write a program using genetic algorithms to brute force a good paper using words from the program's training set. And release it under the GPL.
Re:OSS? by rsadelle · 2004-05-20 06:30 · Score: 1

In my senior year AP/IB English class in high school, we had that option with a human grader.

Here's how it worked: Turn in a paper, even if it's just an introduction, on the date it's due. Get a grade. We could then rewrite/resubmit the paper as many times as we wanted until we got the grade we wanted. The limit was some period of time before the end of the semester that was sufficient for the teacher to have a chance to grade everything before she had to turn in grades.

Given that we were all smart kids who tended to procrastinate, my friends and I often turned in nothing more than an introduction on the due date. Considering the other things we ruined for future classes via our gaming of the system, I wouldn't be surprised if the teacher later changed her policy.
Re:OSS? by Mad+Marlin · 2004-05-20 08:39 · Score: 2, Interesting

My guess is that they are talking only about things like this. I used to use a similar program back when I was taking English classes, in order to bring my papers down to an 8th-grade reading level.

These are encredibly easy to mess around with. For example, the fog index is:
Fog Index = 0.4*(words/sentences+100*((words >= 3 syllables)/words))
Which is roughly equal to the school grade reading level required for the essay. If I remember correctly, Associated Press articles are written to a 4th-grade reading level, which is why all of the paragraphs are only one sentence long.

--
Best Slashdot comment ever
Re:OSS? by Mad+Marlin · 2004-05-20 08:42 · Score: 2, Interesting

For example, my post above (with the formula removed): readability grades: Kincaid: 6.4 ARI: 6.6 Coleman-Liau: 9.2 Flesch Index: 77.8 Fog Index: 8.5 Lix: 35.8 = school year 5 SMOG-Grading: 8.0 sentence info: 408 characters 96 words, average length 4.25 characters = 1.33 syllables 6 sentences, average length 16.0 words 50% (3) short sentences (at most 11 words) 33% (2) long sentences (at least 26 words) 3 paragraphs, average length 2.0 sentences 0% (0) questions 100% (6) passive sentences longest sent 28 wds at sent 2; shortest sent 6 wds at sent 4 word usage: verb types: to be (9) auxiliary (0) types as % of total: conjunctions 1(1) pronouns 9(9) prepositions 14(13) nominalizations 1(1) sentence beginnings: pronoun (3) interrogative pronoun (0) article (0) subordinating conjunction (1) conjunction (0) preposition (0)

--
Best Slashdot comment ever
Re:OSS? by arose · 2004-05-20 08:43 · Score: 3, Insightful

The thing is that a human can actually understand what you are writing. Clever arguments? A nice flow? Humor? Why would I concern myself with things that make reading interesting, if nobody is ever going to read it. Who cares if its dull, just get the punctuation right! Nevermind that it doesn't make much sense, just get a dictionary and show the computer your rich vocabulary...

--
Analogies don't equal equalities, they are merely somewhat analogous.
Re:OSS? by psoriac · 2004-05-20 15:42 · Score: 1

Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.

You must not work for the government.

--
I browse Slashdot at +3, Funny

I would have loved this is a kid by cheezus · 2004-05-20 05:01 · Score: 5, Interesting

it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system.

--
/bin/fortune | slashdotsig.sh

Re:I would have loved this is a kid by jeffy124 · 2004-05-20 05:04 · Score: 3, Interesting

an opinion paper on why dyhydrogen monoxide must be banned would perhaps do the trick?

--
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
Re:I would have loved this is a kid by jbrader · 2004-05-20 05:16 · Score: 0, Redundant

Well of course it should be banned. Hundreds of people drown in huge pools of it every year.

--
You are so boring that when I see you my feet go to sleep.
Re:I would have loved this is a kid by Kiriwas · 2004-05-20 05:34 · Score: 3, Informative

In the florida school system this is a writting test called the Florida Writes. Its a standardized test, doesn't count for much, but we all had to take it. This is basically exactly what we had to do. Write the most horrible peice of trash you could think of, but make it adhere to a few preset guidelines and BAM instant grade. My first time taking it I wrote a rather good peice of work if I do say so myself. Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.
Re:I would have loved this is a kid by LnxAddct · 2004-05-20 05:46 · Score: 3, Informative

Well I've read almost every post under this story and noone has mentioned how it actually works. From the e-rater site "E-rater learns to score essays on a particular topic by processing a significant number of essays on the topic, each of which has been scored by two or more faculty readers. While e-rater is a powerful scoring engine, it is not meant to replace a teacher whose judgment is essential to helping students improve their writing ability." This means that its essentialy a bayesian filter that instead of being fed spam and ham and told which is right from wrong, it is fed good papers and bad papers and told which is right from wrong. You could literally reproduce this with something like SpamBayes but instead of feeding it spam, feed it essays. Anyway, you can probably beat the system but the fact that it is based on preprocessed papers means that a) it will be much harder to game the system without knowing what papers were scanned, and b) all future grading is based solely on the fact that previous generations have written papers and the futre grades will be based on them. The good thing is that unless the teachers are willing to write 20 different papers all on the same subject and scan them in, then the computer won't expect anything better then what was previosuly written. This also means, assuming the teacher won't write essays but rather will use ones from students from prior years, that the essay topics can't be changed, otherwise the teacher would have nothing to scan from prior years and thus nothing to base the grades on. As a result of the essay topic not being able to be changed that simply means that you can find someone from a year above you and ask them to save their papers for you, or in a worst case scenario, everyone must be given the same topic so at least you can help each other out. Often times teachers would give everyone a different topic to avoid people "helping" each other, but with such a system that would be impossible. I sure wish I was back in highschool.
Regards,
Steve
Re:I would have loved this is a kid by cubicledrone · 2004-05-20 06:11 · Score: 2, Insightful

Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

It is also the exact environment of the modern workplace, as designed. Results are irrelevant. Only the process matters.

--
Business isn't willing to pay for products, innovation and careers, so we get brands, mortgage commercials and layoffs.
Re:I would have loved this is a kid by glwtta · 2004-05-20 06:15 · Score: 1

it would have been my goal to make the most wrong essay I could
Sounds like you wouldn't have too much trouble.

--
sic transit gloria mundi
Re:I would have loved this is a kid by Kiriwas · 2004-05-20 06:16 · Score: 1

Thankfully I do not currently work in an environment like that.
Re:I would have loved this is a kid by coyote_oww · 2004-05-20 06:17 · Score: 1

it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system. Which would be a success for the system, since to do this you would need exacting knowledge of English spelling and grammer. i.e. you would have learned quite a bit.
Re:I would have loved this is a kid by Anonymous Coward · 2004-05-20 06:43 · Score: 0

In the florida school system this is a writting test called the Florida Writes.

No wonder Floridians cant differentiate between Al Gore and Pat Buchannan
Re:I would have loved this is a kid by pilkul · 2004-05-20 06:48 · Score: 1

I don't see this as being necessarily a bad thing. Of course the system won't teach you to be a great, publishable writer, but most people don't have what it takes to do that anyway. What it will teach you is how to follow guidelines without going off in your own crazy direction when it's not called for, and how to write an essay that has a minimal amount of structure. It seems to me that these are worthwhile things to learn.
Re:I would have loved this is a kid by mrtroy · 2004-05-20 06:55 · Score: 1

definately
I would likely write perfect gramatical essays, yet with horrible topics that are blatently wrong.
Like why dyhydrogen monoxide must be banned, or proof that my teacher was having inappropriate relations with classmates of mine. A+!

--
[I can picture a world without war, without hate. I can picture us attacking that world, because they'd never expect it]
Re:I would have loved this is a kid by Krach42 · 2004-05-20 06:57 · Score: 2, Informative

I'll let you in on how it works, as I ported some software that does exactly what this essay grader does, as work for a professor who worked on this stuff.

It's not a baysian filter, it's Latent Semantic Analysis. LSA works by taking large amounts of text, and comparing the usage and application of the words within paragraphs. It learns very quickly what words mean, and the interesting thing is, that once it's trained far enough, it starts gaining more meaning to its words by where they're not, than by where they are.

LSA has been put through a variety of tests. And has taken tests even. LSA has been shown to produce "average" results on a synonym test. ("A Doctor is: A) Nurse, B) Practitioner, C) Politician, D) Numerologist") Producing incorrect results mainly only when one word given is more associated (strongly linked) to the word than another more suitable word. Such as in my example, it would pick "Nurse" not "Practitioner" because it's seen Nurse used more often with Doctor.

LSA has been seriously tested by the designers to see if they could write a bad essay that gets a good grade. The answer? Yes, it's possible. But you have to REALLY know the subject well, (as you'd have to produce garbage that relates words accurately between each other) and a lot of time.

The recommended the best way to cheat the system, was to do your research, know your topic, and... write a good essay. Any other way requires too much effort, and a vastly superior knowledge of the subject.

Interesting is that this system can identify plagarism, give it two papers, and it looks how closely the papers match. This gets not just exact copies, but also paraphrased plagarism. The system doesn't really care what the words are, as it looks at their similarity. It could tell that "The doctor studied the patient." is just paraphrased "The practitioner examined his customer."

If train it right, it will even do this between two languages. It's also useful as a spam detector, as it will get "Enlarge your member" from just one marked "Make your dick bigger."

(So, I was told from the professor, Apple's Mail.app is supposed to use LSA)

For any interested. The professor at New Mexico State University was Peter Foltz, and some college up in Colorado was doing a lot of work on it also.

--

I am unamerican, and proud of it!
Re:I would have loved this is a kid by Krach42 · 2004-05-20 07:43 · Score: 1

I'd not say that with much authority without knowing the underlying technology.

Most automated essay grading is now done with Latent Semantic Analysis (if you read about it, the same stuff that Mail.app uses to catch Spam)

So the problem of writing a bad essay that recieves a good grade, becomes a similar problem to writing a piece of spam that won't get caught by a well trained Mail.app filter.

--

I am unamerican, and proud of it!
Re:I would have loved this is a kid by Krach42 · 2004-05-20 07:47 · Score: 1

Pretty close. You don't need so much the grammar and spelling part. You need word usage. Which is much more important in any context than spelling and grammar.

We place so much pointed interest on spelling and grammar in school, because native English speakers have few problems with word usage, but rather with the formalized rules of spelling and formal grammar. (No one has real problems producing correct grammatical text/speech. It just usually doesn't agree with the formal rules that have been arbitrarily established for English)

Now, considering that your spelling and grammar need no be perfect, there's one point to conceed. You can write a complete piece of garbage that gets a good grade. But the people, who designed it, and tested it, have found that the only way to do this, is to have an extremely good grasp of the subject, since you have to synthesize garbage around the correct word usages.

--

I am unamerican, and proud of it!
Re:I would have loved this is a kid by Anonymous Coward · 2004-05-20 08:21 · Score: 0

Just skimming your post, I see 3 misspellings:
definately -> definitely,
gramatical -> grammatical,
blatently -> blatantly.

I see countless grammatical mistakes:
"definately" not a sentence,
", yet" preceding a subordinate clause,
"Like why..." not a sentence,
", or" preceding a subordinate clause,
etc.

I doubt that you "would likely write perfect grammatical essays." C-!
Re:I would have loved this is a kid by Anonymous Coward · 2004-05-20 08:27 · Score: 0

C-? You're being generous. The article is about grading high school papers, not middle school papers.
Re:I would have loved this is a kid by Anonymous Coward · 2004-05-20 08:51 · Score: 0

More likely it will teach you how to follow guidelines without going off in your own crazy direction....ever.

"Sit down. Shut up. Do your job."

That crap is why I deeply despise the corporate world. A little inefficiency is a small price to pay to allow for creativity. Especially since a little creativity is likely to lead to more efficiency.
Re:I would have loved this is a kid by Anonymous Coward · 2004-05-20 09:13 · Score: 0

You can write a complete piece of garbage that gets a good grade. But the people, who designed it, and tested it, have found that the only way to do this, is to have an extremely good grasp of the subject, since you have to synthesize garbage around the correct word usages.

I'd be more worried about people writing good essays that were marked badly, because the student was more original in his/her use of vocabulary than the sample on which the system was trained. Or maybe they did deeper research and so found different sources. Or whatever.

Bad students aren't going to succeed once they reach a human marker, but the problem with e-marking is the potential for a good student to fail because they never do reach a human marker.

Imagine Shakespeare failing his English class because a machine didn't like his habit of coining new words, while a hack poet got top grades because his entry, entitled "Roses are red", matched so well with the existing corpus?
Re:I would have loved this is a kid by Hoch · 2004-05-20 12:18 · Score: 1

I took the pilot program 2 years ago as a junior. One of the questions was something to the effect of : Choose one invention throughout history and write about how it has contributed to society or hurt society. So my friend wrote about the wheel and stuck only to how it has hurt society. He made sure to use unnessecarily big words and stick only to how it made people lazy. It was the most absurd essay ever, but he still got one of the highest grades in the class. It was damn funny.

--
2*31*37*263
Re:I would have loved this is a kid by Niet3sche · 2004-05-20 21:40 · Score: 1

My first time taking it I wrote a rather good peice of work if I do say so myself. Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks.

Christ, are they teaching the 5-paragraph essay? Ugh. That's pretty sad in itself. :( HUMAN instructors should not be pulling out that old trashpile, much less requiring that for a schema/template/rubric from which the program grades.
Re:I would have loved this is a kid by Krach42 · 2004-05-21 14:19 · Score: 1

Hm... very true, and good point.

Although, from what I read and had been informed from (again from one of the guys who developed the technology) it's not meant for analysing poetic, and even specific prosaic pieces.

It can't grade creativity, and there you're dead on. But it's not being asked to grade creativity, it's being asked to grade essays, which deal specifically with predefined, and researched topics.

A failure to not use words correctly in an essay is a fundamental failure at the assignment. It's not wrong, it's just a failure at the assignment given.

Of course, this probably sounds like the tech response: "It's a problem."

--

I am unamerican, and proud of it!

Google Bombing by m0rph3us0 · 2004-05-20 05:02 · Score: 1, Interesting

I can't wait til someone figures out how to google bomb the grading system.
I wonder if it will be as simple as repeating a high ranking sentence?

Re:Google Bombing by superpulpsicle · 2004-05-20 05:07 · Score: 2, Funny

Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.
Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.

Essay Result = A+

I already want a copy of this. by ShitPissFuckCuntTits · 2004-05-20 05:02 · Score: 5, Insightful

I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?

--

--
My username: hats off to George Carlin, and fuck the FCC. Freedom!

Re:I already want a copy of this. by glenrm · 2004-05-20 05:15 · Score: 1

If I was the teacher I would give the a student the student what the deserve an A.

--
Onward to the Aether Sphere!
Re:I already want a copy of this. by Kozar_The_Malignant · 2004-05-20 05:16 · Score: 4, Funny

> I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's.
I'll bet half the people here thought this as soon as they read the headline. The normal /. response is to spend 10 hours at the computer coding up something to do a task you can do in 1 hour with a pencil. :-) Of course the high school geek who does this will make piles of money selling the output to the football players, buy a nice ride, and get a prom date with a cheerleader. Isn't that the way it works?

--
Some mornings it's hardly worth chewing through the restraints to get out of bed.
Re:I already want a copy of this. by Glonoinha · 2004-05-20 05:22 · Score: 3, Funny

Bah, right now the program takes an essay as input and outputs a grade. Simply reverse the pipe streams, push in an A and have the programs spit out a essay worthy of that A. Magic!

--
Glonoinha the MebiByte Slayer
Re:I already want a copy of this. by legoleg · 2004-05-20 05:44 · Score: 1

I'll just have my spam do my homework for me.
Re:I already want a copy of this. by ViolentGreen · 2004-05-20 05:46 · Score: 1

And why would an essay of gibberish deserve an A? Just because something is cool doesn't change the fact that it is not what was assigned.

--
Not everything is analogous to cars. Car analogies rarely work.
Re:I already want a copy of this. by 4of12 · 2004-05-20 07:22 · Score: 1

a program to create nonsensical gibberish that always gets A's.

Well, in my world there are too many writers producing nonsense already.

I'd like to do my part not to contribute to that specific problem.

If this grading program were open source and freely available perhaps I could use it to self grade what I've written.

Even better would be if the program were to suggest ways to improve my writing so that I could improve my score.

Perhaps the Slashdot submissions would not only reject postings that lack a specified subject, were too frequent, too lengthy, but also bounce writing that was really bad, really ugly.

All that notwithstanding, I still have a hard time believing that an automated program can adequately gauge how well organized any piece of writing happens to be. The concepts embodied in any sentence or paragraph are several levels of abstraction above any combination of vocabulary words and grammatical constructs.

--
"Provided by the management for your protection."
Re:I already want a copy of this. by commodoresloat · 2004-05-20 08:48 · Score: 1

Isn't that the way it works?
All except for the prom date. The cheerleader still doesn't want to date a geek.

Stupid by phreak0003 · 2004-05-20 05:02 · Score: 4, Interesting

I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

Re:Stupid by RoderickMcDougall · 2004-05-20 05:06 · Score: 2, Interesting

I agree. Did you receive any comments from the computer or was it just an unjustified percentage? I would be most interested to know how they have managed to justify this questionable practice. Almost as someone is rushing to boast about being at the forefront of technology without the goods to back it up. Oh the children! Won't someone please think about the children!
Re:Stupid by garcia · 2004-05-20 05:07 · Score: 2, Insightful

I have taken plenty of essay exams where I felt I wasn't graded fairly by a human teacher/professor.

Some essays were graded out of a couple points. A paper out of 6 points carries less weight overall. If this is the only exam (ie AP tests) a 5/6 is looked at as a high score.

I don't see your point.
Re:Stupid by Anonymous Coward · 2004-05-20 05:09 · Score: 3, Funny

Maybe you got docked for overuse of commas.
Re:Stupid by Anonymous Coward · 2004-05-20 05:11 · Score: 0

Oh give the kid a break. His score of 5/6 was extemely well.
Re:Stupid by Roland+Piquepaille · 2004-05-20 05:13 · Score: 2, Insightful

Not only computers may not give fair grades, but there's a deeper problem with grading using computers: to me, students working to get good grades from a computer conjures up images of sheeps going in the wool-extraction machine. While this may be fine for sheeps, how do you think the students feel about it?

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?
Re:Stupid by Anonymous Coward · 2004-05-20 05:16 · Score: 0

was extremely well

Perhaps that is where the 5/6 comes from.
Re:Stupid by garcia · 2004-05-20 05:17 · Score: 2, Insightful

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?

You had a different school experience than I did apparently. I felt that the human's reading my papers were distant, uninteresting, and less than worthy of grading someone else's work.

Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.

On "bad papers" the comments were usually less than helpful. Don't just mark up the paper with "comma splice" or "vague". The teacher should have taught the class what a "comma splice" was or should have been following their own words of advice and kept themselves from being "vague" in their comments.
Re:Stupid by micromoog · 2004-05-20 05:22 · Score: 2, Insightful

They're just converting "n/6" directly into a percentage grade?! That's ridiculous. So there's no such thing as a C (going straight from 83 to 67)?
Re:Stupid by servognome · 2004-05-20 05:23 · Score: 4, Funny

Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.
Your feedback was actual words!? In my day all we got was a scratch-n-sniff sticker and had to guess the meaning of getting a "watermelon" on our essay.

--
D6 63 0D 70 89 81 BB 8E 7B 7C 5F 5D 54 EA AB 73
Re:Stupid by Moofie · 2004-05-20 05:29 · Score: 1

But if the grading system is such that a really good (but not perfect) paper results in an 83%, which (depending on your grading scheme) is a B-, or "average to good", then the grading scheme is skewed.

Taking a six point scale and rendering it onto a 100 point scale is really unfair.

--
Why yes, I AM a rocket scientist!
Re:Stupid by ibpooks · 2004-05-20 05:32 · Score: 2, Interesting

A computer can obviously not grade essays fairly, so it shouldn't be done.

The article states that comparisons between the computer grading and human grading revealed nearly identical scoring. By this data, I don't think it's obvious that a computer can't be as fair as a human.

I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

What does the computer have to do with it? You would have been in the same situation if a human had given you the 5/6. Perhaps what you mean to assert is that the entire test isn't a fair assessment of performance; but that sounds like sour grapes from someone who didn't get a 6. :-)
Re:Stupid by Anonymous Coward · 2004-05-20 05:32 · Score: 0

Not only computers may not give fair grades
students working to get good grades from a computer conjures up images
sheeps
I guess humanizing the system had little practical effect....
Re:Stupid by kunudo · 2004-05-20 05:33 · Score: 1

The schooling system already lacks humanity, why de-humanize it even more?

Because the humans that work there are lacking in the skills department?
Re:Stupid by Anonymous Coward · 2004-05-20 05:33 · Score: 0

It meant you were fat.
Re:Stupid by sTalking_Goat · 2004-05-20 05:41 · Score: 1

humans can't even grade essays fairly. Case in point. I took SAT and got 720 on verbal (my math score sucked though) In my first year college though I had to take an English placement exam which was to write an essay. Whoever graded my essay for some reason thought it was bad enough to stick me in remedial English.
I actually went to the remedial English that being my fiurst year in college, I assumed that they must know what they're doing. Why else would they be working at a University. After my first essay the professor told me I didn't belong there. She made a call to the Dean and had him dig up and read my placement exam Essay. A week later I was in an English Honors class.
So if Humans can be so divergent in their grading how do you program software to do it right?

--
My days of not taking you seriously are certainly coming to a middle...
Re:Stupid by king-manic · 2004-05-20 05:44 · Score: 1

In general high school isn't exactly a fair enviroment. It's aimed at those who learn by seeing or hearing. It's ciriculum has been hijacked numerous times by numerous groups. Most ciriculum aroudn the western world now favors girls abilities over boys. Western thought and ideals are stressed over others. Science have become political issues instead of a academic one. And soem individuals will always game the system. I know I did. Each english teacher I had failed me on my first essay. From grade 6-12. Each succesive essay I would find out more about the teacher and write essays with the same structure(horrible) but with ideas more in line with the teacher. At the ened of every year I'd have 80+ on my last essay, in my final year I had a 90. I gamed the syetem and won. This is a reflection of a important ability that will help me succeed so I don't figure it as a bad thing.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Stupid by I_Love_Pocky! · 2004-05-20 05:47 · Score: 3, Insightful

The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.

Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?

They are not graded fairly, and they determine 10% of the final grade.

10% of the grade on the essay? Or in a particular class? 10% on the essay may actually be tolerable, because that means that at least a human actually read it to give the other 90% of the grade.
Re:Stupid by X_Bones · 2004-05-20 05:52 · Score: 1

Whine. This piece of software, never before used in high schools, didn't give me the grade I thought I deserved (which is only worth 10% of the final score anyway). This "obviously" means computers are completely and forever unfit to grade student essays, even though they by nature don't get tired of grading papers, don't display favoritism, don't deviate from established criteria at all, and have the ability to be instantly and continuously refined.

Ever heard the phrase about not throwing out the baby with the bathwater? So you were ETS' guinea pigs, and you think you got shafted. That's why it was only worth a fraction of your grade. Get over it.

--

the coolest club on /.
Re:Stupid by Jorkapp · 2004-05-20 05:54 · Score: 1

Allow me to consult my "Guide to scratch and sniff stickers - Teachers Edition"

Consulting...

Lets see... Water Buffalo, Waterfall, ah! Here we are! Watermelon.

Watermelon:
This student is good. Not great, but good. Possibly a geek, nerd, or other such social class striving for the top. Will likely go on to college - possibly university, but will struggle financially after losing their McDonalds job for trying to install IRC on teleprompters.

Next time - hope for a smile face.

--
Frink: Nice try floyd, but you were designed for scrubbing, and scrubbing is what you shall do.
Re:Stupid by rsadelle · 2004-05-20 05:56 · Score: 3, Interesting

I certainly hope your essay was better written than this comment. If it wasn't, and you still got 5/6, that's pretty good evidence that the computer isn't doing accurate grading. IANAET (I am not an English Teacher), but I would grade this comment 3/6. Some comments:

Do you have any evidence or thought behind your statement that "A computer can obviously not grade essays fairly, so it shouldn't be done"? Why is that obvious? Is it obvious only because the grading of your essay was, in your opinion, not fair?

I suggest you review difference in usage between "good" and "well." Proper grammar/word choice is a large part of what makes a good essay.

How is the conversion from the computer's 6 point scale to your teacher's/school's 100 point scale the fault of the computer? It appears that what you really mean is that the implementation of the computer grading system is not fair as it doesn't use an appropriate scale.
Re:Stupid by NaugaHunter · 2004-05-20 05:57 · Score: 1

Your feedback was actual words!? In my day all we got was a scratch-n-sniff sticker and had to guess the meaning of getting a "watermelon" on our essay.

You had scratch-n-sniff stickers? All we had were:

- Gold Stars or Unhappy Faces
- Check, Check Plus, or Check Minus
- Letter grades

--
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
Re:Stupid by Surt · 2004-05-20 06:05 · Score: 1

Yes, our (corporate america's) goal is to churn out as many identical producer/consumer drones as possible.

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
Re:Stupid by cardshark2001 · 2004-05-20 06:05 · Score: 1

I got a 5/6, which, according to the computer, was extremely well.
Normally, I just let stuff like this pass, but this being a discussion about english essays, I was forced to respond. (plus you were grouching about getting a bad grade)
The proper sentence would read:
I got a 5/6, which, according to the computer, was extremely good. OR
I got a 5/6, which, according to the computer, meant I did extremely well.
I'll ignore your comma fetish for the time being.

--
WWJD? JWRTFA!
Re:Stupid by skifreak87 · 2004-05-20 06:05 · Score: 1

Nothing to do w/ the computer, has to do w/ possible point values assigned. I always fought w/ teachers who'd claim i only took 1 point of for your little careless mistake since you clearly know the material but it's be 1 point out of 10 or something like that, which would be a 90 which would lower my gpa if factored in.

That's a naive teacher problem. My band teacher never understood that because I took almost entirely honors/AP classes that an A+ in band lowered my gpa (since I, like 7% of the class had over a 4.5 which was an A+ in a normal class). Same w/ the 9-pt essay grading in history (althoug h they curved that nicely w/ a "fair" mapping of score to letter grade).
Re:Stupid by CoolToddHunter · 2004-05-20 06:10 · Score: 1

The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.

If you start grading on content (read: opinion of the author), then only the students that parrot the teacher will get good grades. Much better to remove content from the equation, so the student can effectively learn how to present ideas in a useful and persuasive manner regardless of what the idea may be.

Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?

I don't think this argument works well at the high-school level. Students are still learning about grammar and spelling, so all mistakes will be unintentional except by the most gifted students (about three standard deviations away). This is the same priciple by which artists are trained: master classical techniques so you can utilize them (through use or intentional mis-use) in your modern art.
Re:Stupid by Unknown+Lamer · 2004-05-20 06:12 · Score: 1

The verb, to conjure, is referring to the concept of "students working." The students are being modified by the gerund[ive? I can't remember which one was the adjectival and which one was the noun form] "working" and the phrase "to get good grades from a computer." It is this concept that does the conjuring, hence the original sentence was correct and you are dumb.

--

HAL 7000, fewer features than the HAL 9000, but just as homicidal!
Re:Stupid by Anonymous Coward · 2004-05-20 06:21 · Score: 0

It sounds like six points isn't providing a fine enough granularity.
Re:Stupid by I_Love_Pocky! · 2004-05-20 06:27 · Score: 1

All I know is that when I was in high school we were expected to have already mastered the basics of English. The only thing that was emphasized was content.

By the way, you can grade content without grading opinions. The grade is based on how well formed an argument is, or how well an idea is fleshed out. In other words, you are grading the an author based on what they intend their words to mean, rather than just their words. These are things that a computer just can't do. ou don't get an A because of what you think, but how you convey what you think.
Re:Stupid by argStyopa · 2004-05-20 06:29 · Score: 1

I got a 5/6, which, according to the computer, was extremely well

You mean "was extremely good" or "was extremely well-written"?

--
-Styopa
Re:Stupid by Just+Some+Guy · 2004-05-20 06:58 · Score: 1

Your classes directly assign a letter grade to each assignment? It seems that "n/6 percent" is meaningless in the context of one single portion of the overall grade.
In other words, in my math classes, I didn't panic if I got 3 of 5 points on a random pop quiz. My final grade wasn't computed from "median letter grade received", but from "total points earned out of total points possible". If you're used to getting letter grades on all assignments, I would hate to have been at your school.

--
Dewey, what part of this looks like authorities should be involved?
Re:Stupid by Anonymous Coward · 2004-05-20 07:25 · Score: 0

I hear that. The first month of class is always the hardest for me since I haven't figured the teacher out yet. Once I figure him or her out, her political affiliations, what kinds of things impresses them (often it's technology; I've written complete garbage but filled with false technology buzzwords and gotten good grades), and so I 'beat the system' by conforming.

This of course can and probably is done with this software too, figuring out what it likes, how to impress it and so forth.

Pretty much like the real world.

Unfortunatly, I don't see how this software can check for actual fact-based information. Since anything that isn't fact is fiction, and fiction is much more subjective than fact, I really can't see how this works very well. Other than running it through a grammar and spelling check.
Re:Stupid by Dhalka226 · 2004-05-20 07:51 · Score: 1

Does anybody else find this to be too highly exaggerated at best (complete BS at worst)?
How can an 83% on something worth 10% of the final grade bring down his grade "significantly?" If he had a 100%, he would end up 98%. Hardly significant. If he had a 91%, he'd end up at 90%. Also not significant. In fact the only (integer) number that would make a difference, at least according to my high school grading scale, is if he was sitting right on 90% to begin with, in which case he'd slide down to 89.3%, a little too low for most teachers to round back up. He shifted down 2% at most. Even if that DID move him down a grade, I can hardly consider a drop of no more than 2% to be significant. It means he was on the border to begin with and this toppled him over.
Re:Stupid by arkanes · 2004-05-20 07:57 · Score: 1

In fairness, I scored 740 on verbal and I can't write my way out of a paper bag. I read a tremendous amount and have a feel for proper English but little formal understanding of the rules, very little patience for those rules, and none of the creative spark required to write engaging essays. SAT verbal doesn't really test any of that stuff.
Re:Stupid by Anonymous Coward · 2004-05-20 08:01 · Score: 0

Um, his use of commas was perfect. His only error was the improper use of an adverb ('well') when an adjective was called for.
Re:Stupid by Anonymous Coward · 2004-05-20 08:02 · Score: 0

If the computer's grade counts for 10%, and the computer gave you an 83%, then the most the computer could have brought down your grade would be 100-(90+8.3)=1.7%!

If you meant it the other way around, then the most it could bring your grade down would be 100-((83*.9)+10)=100-(74.7+10)=100-84.7=15.3%, in which case, I'd agree that assigning 90% of your grade by way of a process with a 17% granularity is a bad idea, given a typical grading scale with a grade granularity of 10% or so (A=100-91, B=90-81, C=80-71, D=70-61, F=60-0).
Re:Stupid by egomaniac · 2004-05-20 08:47 · Score: 1

Much better to remove content from the equation, so the student can effectively learn how to present ideas in a useful and persuasive manner regardless of what the idea may be.

Are you suggesting that there exists a machine which can judge if an idea is presented in a "useful and persuasive manner"?

Funny, the machine on my desk has enough trouble just telling whether a comma is in the right place or not. I can't wait to see something that can judge how useful and persuasive English prose is.

--
ZFS: because love is never having to say fsck
Re:Stupid by phreak0003 · 2004-05-20 09:53 · Score: 1

A few things I'd like to say to defend my case. I'm only a freshman, which means I'm bound to make mistakes, and often (especially over the internet when I'm not paying attention).
This program, after I cracked into it, discovering its algorithm, found that the things that were missed. The computer counted the word "reiterate" misspelled since it wasn't in its dictionary. It also counted off for writing 10 sentences in a paragraph instead of 11.
It has created a common mediocrity, which means that if you don't have 11 sentences in each paragraph, you don't get proper points, and if you "misspell" a word because it's not in the program's dictionary, it's counted wrong. You also get counted off quite frequently for lack of transitions, which are not necessary when the paragraph is running smoothly. Indiana's grading scale is a 5-point which means that a 94% is a B and an 87% is a C. This knocked my grade down to a 94.4, since I had near to a 96%.
Being counted off for programming errors isn't fair at all.
Re:Stupid by Jardine · 2004-05-20 09:56 · Score: 1

You had scratch-n-sniff stickers? All we had were:

- Gold Stars or Unhappy Faces
- Check, Check Plus, or Check Minus
- Letter grades

Man, you were lucky. We didn't get any of those things. If our essay (which was written in the dirt during a rainstorm) wasn't perfect, we were stabbed by the teacher with a knife. The more times you were stabbed per essay, the worse it was. It was on a scale of 0 to 100 stabs. So if you were stabbed 12 times, that was the equivalent of 88%
Re:Stupid by Anonymous Coward · 2004-05-20 10:31 · Score: 0

YOU
ARE
A
FUCKHEAD

this is more of a reply than your post deserved.
Re:Stupid by airhed13 · 2004-05-20 11:55 · Score: 1

Based on what I read into the article, as well as posts given by others who apparently have done work on the topic, it's not very surprising that the computer graded similarly to the humans.

Why should we be surprised, after all, when the humans we're comparing the machines to have first been trained to grade like they themselves were machines?

I don't think that the real concern is at all whether or not the humans grade similarly to machines based on the same criteria. I think that the real concern is how those criteria are defined and applied to individual papers. Such definition is what makes the difference between a good and bad essay grader, in my experience- and that is something which can't be done by computer.

Of course, if the graders were to define their own criteria, that would preclude any sense of fairness in grading when it comes to large-scale (such as state-wide or nation-wide) essays. We're stuck in a catch-22, and the only way that I see to get out of it is to redefine what these large-scale essays are assessing.

Standardized writing exams do not (and I would argue can not) provide absolute feedback as to a particular writer's ability to write. All that they can consistently do on a fair basis is rate the writer's skill in the mechanical (IE, non-subjective) aspects of the English language. The very fact the other aspects of the essay are subjective implies that it's not possible to grade them fairly except in the most extremely bad or extremely good cases anyway.

A computer is superb at evaluating mechanical aspects of language. Better at it than people. If we train our people to look for specific sentence or essay structure, we've simply extended the definition of the mechanical aspects of the language to include such structures. We've stated, "Active voice is good, Passive voice is bad." Such generalizations are hogwash in interesting literature. There's even a time and place for comma-splices and sentence fragments in good writing.

The whole issue could be put to bed if we simply admit that these tests aren't measuring the quality of an essay, they're evaluating a complex set of mechanics which define some set of things to be good and some set of things to be bad.
- Airhed (Who's not a huge fan of state-wide writing proficiency tests. Smack down lawsuits on teachers who flunk inept students, instead.)
Re:Stupid by Afromelonhead · 2004-05-21 15:46 · Score: 1

I am a senior at an Indiana high school, and I printed out a copy of the article to show to my English teacher. After looking over it, she said a couple of things to me:
They have decided at my school to grade all the exams (I'm not sure which grades) by hand because they have had several issues with the system. She claims that the computer doesn't really recognize sophisticated diction or syntax patterns such as inverted sentences; in fact, the computer apparently grades these lower than students who choose to use simple diction and sentence structures.
Now for an example: she was in the library administrating one of her freshman English classes. She had three students in front of her, with the middle one being an excellent writer and the other two being mediocre at best. The middle writer was having some problems with the system, as it claimed that her essay was too short for the computer to grade, except that my teacher looked at all three essays and discovered that the person in the middle had written the most (100-200 words more).
Meanwhile, the student on the left had gotten a score of 5, and the person on the right said that he had also gotten a 5 and was happy with it. My teacher started reading the right person's essay, and discovered a misuse of both "its"/"it's" and "they're"/"their"/"there" in the first sentence. The second sentence wasn't even that; it was a fragment.
She didn't tell me how the rest of the essay went, but I would imagine that it followed the same general pattern. Yet, somehow, it received a 5. Something's screwy here...

--
Procrastination sucks.

I guess this rules out... by Loco3KGT · 2004-05-20 05:03 · Score: 2

schmoozing with the teacher to get higher grades.

In unrelated news, Delicious Red Apples have suffered a terrible sales slump.

--
Blessed be he who reads this post, Cursed be he who tells my boss.

Re:I guess this rules out... by underworld · 2004-05-20 05:11 · Score: 1

I guess now you have to give the teacher a Macintosh instead of a Red Delicious.... ;-)

comment moderation by Jotaigna · 2004-05-20 05:03 · Score: 4, Funny

for the time being, i would trust more that program to moderate my comments.

c'mon people i was only joking dont mod me down, not noooo!!

--
"The quality of life is inversely proportional to the number of keys on your keyring."

Re:comment moderation by BillyBlaze · 2004-05-20 05:39 · Score: 1

That would actually be a good idea: a Bayesian filter, trained on previous +5 comments, to moderate. linux riaa vorbis grits micro$oft profit.

What about tricking the software? by gtaluvit · 2004-05-20 05:03 · Score: 5, Interesting

SPAM filters are tricked all the time depending on the text of an email. Google was f'd up not too long ago because of trackback linking in blogs screwing up their algorithms. Isn't this a similar situation? If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

--
- gtaluvit (prnc. GOT-tuh-LUV-it)

Re:What about tricking the software? by Anonymous Coward · 2004-05-20 05:05 · Score: 0

Since when is SPAM an abbreviation and therefor written ALL UPPERCASE?
Re:What about tricking the software? by Marxist+Hacker+42 · 2004-05-20 05:12 · Score: 1

If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

Which, if Harris Miller of the ITAA is actually right, is exactly what we need kids to learn so that we can compete with India.

--
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Re:What about tricking the software? by wrp103 · 2004-05-20 05:29 · Score: 1

Good point.

I would think that even non-techies should realize this. Anyone who has looked at the "suggested corrections" in MS-Word should realize that sometimes that program has no idea what you are trying to say, and its "corrections" actually make it worse, not better.
Re:What about tricking the software? by NaugaHunter · 2004-05-20 06:03 · Score: 1

SPAM filters are tricked all the time depending on the text of an email.

I'm pretty sure misspelling words or using letters for numbers in an essay, or even tacking a hundred unrelated words to the end of the essay, would all properly lower your grade.

While it might be easy to beat the system grammar-wise by constructing well formed sentences, if it has been close to person grading for two years it problem has a method of ensuring the statements relate to each other. Ironically, anyone spending a lot of time to beat this will probably learn how to write better essays anyway.

--
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
Re:What about tricking the software? by aardwolf204 · 2004-05-20 06:11 · Score: 1

Something. Posing. As. Mail.

--
Im dreaming ofa big bndwdth, That can resist the /.crowd.May ur days b merry & bright & may al
Re:What about tricking the software? by drooling-dog · 2004-05-20 07:16 · Score: 2, Insightful

we'll have students learning to write to beat software, not form a well written essay.
Right. The software can grade things like spelling, vocabulary, grammar, and syntax, but it won't grade the quality of the ideas expressed or the depth of understanding displayed. Therefore those latter dimensions will cease to be considered important, since grading them is expensive compared to the more mechanical stuff.
Re:What about tricking the software? by PalmerEldritch42 · 2004-05-20 09:12 · Score: 1

I haven't tested out this specific software, but if we can take it at face value, it looks as if the way to "beat the software" is to write a grammatically correct paper. If it can at the very least check spelling, sentence structure, etc. then perhaps all the time that people spend trying to beat it will, in fact, get them to write a more correct paper. I personally doubt that it can understand the content of the paper in any real way, but I can't even count how many papers I've written where I know the teacher couldn't follow what I was writing about, so I was being graded more on correctness of Works Cited and parenthetical documentation.

--
Ceci n'est pas une sig.
:wq!
Re:What about tricking the software? by ClausCCC · 2004-05-20 09:21 · Score: 1

".....The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. " I assumed the system was trained with regular essays. The students who wrote those worked under the premise that their essay would be graded by a real person. If I were a student now in Indiana I would no longer try to write good essays but try to trick the system. Since the automatic grading system lacks true understanding it should be easy to find some basic rules (like: use fancy words, make long sentences, obey grammar, ...) which lead to good grades. The next step would be to write a program which produces top graded essays and needs only a list of key-words as input.
Re:What about tricking the software? by complete+loony · 2004-05-20 11:57 · Score: 1

Gee you guys really don't know what a computer can do in this area. But I guess this is slashdot so I shouldn't be surprised.
10 years ago while I was at Uni, there was a guy writing one of these semantic analysis programs that another poster already mentioned. As a test he was running alice in wonderland through it. This book has sentences that go for half a page, sections of french and all sorts of crap you shouldn't be getting in an emglish paper. Yet it still did a reasonable job of understanding the text and could describe what each section was talking about.
And that was 10 years ago.

--
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
Re:What about tricking the software? by jimbolaya · 2004-05-20 13:08 · Score: 1

But to get to that point, where you have figured out how to "trick" the program, you would have had to feed it hundreds if not thousands of essays. Aside from the issue of writing or acquiring all those essays, there's the problem of access to the grading program. And if a copy "leaked", there's no guarantee that its the same version as that which will actually be used to grade your paper. If you were a student, would you be willing to gamble with your GPA in hopes of "tricking" the grader?

--
There ain't no rules here; we're trying to accomplish something.

So much for those essays by Zancarius · 2004-05-20 05:03 · Score: 5, Funny

Perish the thought should students start writing about the dangers of artificial intelligence. They may very well fail!

--
He who has no .plan has small finger. ~ Confucius on UNIX

Re:So much for those essays by identity0 · 2004-05-20 06:16 · Score: 1

I shudder to think how that grading session would go...

"Mr. Anderson, it seems you have been leading two lives... in one, you are a respected student, all A's, boy scout... in the other, you are a hacker named Zancarius, spreading rumors and dangerous allegations against the benevolent all-knowing AI. One of these lives - has a future. Which will it be, Mr. Anderson?"

"How about, I give you the finger, and you get me to college."

"College? But how can you go to college - when you fail this test?"

"Mrrrrph!! Mmmmph!! EEEEEE!!!"
(insert horrific scene of test machine probing your bellybutton)

Gaming the system by ePhil_One · 2004-05-20 05:03 · Score: 5, Insightful

While it gives identical results for now, I can easily see the coming books: How to Write an A essay! Form essays to get you into Harvard.

The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...

--
You are in a maze of twisted little posts, all alike.

Re:Gaming the system by CrazyTalk · 2004-05-20 05:29 · Score: 1

More than that, the GMAT essays are already graded by computer. I believe they have a human "spot-checker" so if you manage to write gibberish that somehow the computer figures out to be a brilliant essay because you included all the key buzzwords, you will get caught. But for the most part, the computer does all the reading and grading.

Ridiculous by RoderickMcDougall · 2004-05-20 05:03 · Score: 1

If Dr Sbaitso can't cure what ails me how the hell is a computer supposed to miraculously rise to the level of intelligence required to grade creative literacy. Perhaps its all a ruse and Dr Sbaitso really is behind the grading program. Hint, look for: How does it make you feel?

Indiana? I thought it said India by Anonymous Coward · 2004-05-20 05:03 · Score: 0

And I was going to make a joke about India outsourcing teaching jobs to computers... alas...

Not the First by dcocos · 2004-05-20 05:04 · Score: 4, Interesting

My alma matter graded most of my computer programs with shell scripts and I graduated in 1997. So I don't think India is the first to do that.

Re:Not the First by gregarican · 2004-05-20 05:07 · Score: 2, Insightful

Grading freely-written essays and structured computer programming code is two separate things *to a degree.* And India isn't the first to do the latter. Indiana is, however, according to the article.
Re:Not the First by gregarican · 2004-05-20 05:12 · Score: 1

Actually I meant the former, not the latter. Everyone makes mistakes I guess :-)
Re:Not the First by Anonymous Coward · 2004-05-20 05:15 · Score: 0

The difference here is that these are essays, not programs. The grading of essays is (or at least I hope) a lot more involved than making sure that a program has the correct output, etc.

It makes me wonder just how accurate this program can be. Is there really a formula for good essays or writing styles? What happens if its a well written essay, but all the facts are wrong? I really wish the article had gone into greater detail.
Re:Not the First by Sporkinum · 2004-05-20 05:16 · Score: 2, Funny

So are all IT jobs being outsourced to Indiana?

--
"He's lost in a 'floyd hole"
Re:Not the First by Anonymous Coward · 2004-05-20 05:16 · Score: 0

My alma matter graded most of my computer programs with shell scripts and I graduated in 1997.

Yes, but the professor can test the correctness of your program with known inputs that should give known outputs. This is easy to automate.

English essays have far more complexity.
Re:Not the First by king-manic · 2004-05-20 05:46 · Score: 1

Indeed, india hasn't done it all. It's indiana.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Not the First by Anonymous Coward · 2004-05-20 06:30 · Score: 1

Of course, a computer program has a functional aspect--it either works, or it does not. My own college had one class where the instructer gave us very detailed descriptions of the expected output, and did the grading by feeding both our program and a reference program the same set of test cases, then running a diff on them. Not rocket science.

On the other hand, I'm sure that the search engine optimizer types should know how to spam the hell out of any algorithm they use for grading. Hell, I bet you can probably just pass it the same way you can get a grade-level of '12' in Word (turn on all the useless spell checking/word count nonsense and get the huge window to pop up). All you have to do is put up with some of the inane pronouncements of the Word grammar checker (it often rejects perfectly good sentences for things like being "too long" even if they're not), use big words to up the Flesch-Kinkaid readability thingy, make sure there are no misspelled words (might be difficult for those systems which do not recognize your last name...), and (possibly) make it somewhat relevant to the question you're answering.

In short, the only thing stopping SEO types from reversing an algorithm like this would probably be their poor grammar :) But who knows? Maybe you can defeat this like the Bayesian spam, by quoting random disconnected bits of Hamlet?

I hope they keep a human in the loop to adjudicate any mechanized stupidity (ooh! fun new term! *saves it for later*)
Re:Not the First by Bull999999 · 2004-05-20 06:34 · Score: 1

I guess you'll only be getting a score of 5/6 for you post then.

--
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Re:Not the First by PrestoChango · 2004-05-20 10:56 · Score: 1

Mine had been grading their Lisp submissions via script when I took a particular class in 2000. One of the assignments was to implement the Davis-Putnam algorithm for testing satisfiability of CNF expressions. The year I took it, they added a style portion to the grade so TAs had to actually look at the submissions. Someone the previous year had defined his davis-putnam function to always return true. Since 87 percent of the test cases were supposed to be satisfiable, he got a B on the assignment.
Re:Not the First by Anonymous Coward · 2004-05-20 11:25 · Score: 0

Indiana? Is that near New Mexico?

Yeah... by Paulrothrock · 2004-05-20 05:04 · Score: 1

Sure, this won't be hacked... Some script kiddie is gonna get the algorithm and get straight A's for his whole life on gibberish and devote more time to a new virus.

--
I'm in the hole of the broadband donut.

Re:Yeah... by Hatta · 2004-05-20 05:41 · Score: 1

Sure, this won't be hacked... Some script kiddie is gonna get the algorithm and get straight A's for his whole life on gibberish and devote more time to a new virus.

And this is different from normal grading how? In HS and most of college writing essays is about how well you can bullshit, not how well you think and write.

--
Give me Classic Slashdot or give me death!
Re:Yeah... by Paulrothrock · 2004-05-20 05:55 · Score: 1

This is different because this can be automated and distributed via the Internet instead of learned through trial and error. Bullshitting (and recognizing bullshit) is very necessary behavior in life, particularly for dealing with management. ("This is BS, so I can stop listening and think about something important.")
If you can download Get_an_A_in_IN.exe and have it spew gibberish so you can get a good grade you're missing out on a valuable experience. At least with manual bullshitting people had to learn how to write something.

--
I'm in the hole of the broadband donut.
Re:Yeah... by PalmerEldritch42 · 2004-05-20 09:18 · Score: 1

I totally agree with you, but I would expect that there is some sort of check or balance in place where an actual human would at least glance at the A paper and perhaps lower the grade if it looked like:
A11 0f U R Ownz0red by My 1337 sK1lls @t t3rm p@P0rz. Th1s skr1pt ru1ez. U w1ll g3t an A e@zy.

--
Ceci n'est pas une sig.
:wq!

Oh, my. by eddy · 2004-05-20 05:04 · Score: 1

But does it do any semantic analysis? I assume a real person at least reads the material once, or this is going to pass some very interesting material.

--
Belief is the currency of delusion.

First mistake? by macshune · 2004-05-20 05:05 · Score: 1

My Great Grandfather's Most Famous Quote
By H. Jones III

"We named the cat Indiana!"

Re:First mistake? by Anonymous Coward · 2004-05-20 05:18 · Score: 0

Henry: What did you find, Junior? Indy: Junior?! Dad... Sallah: Please... what does it always mean, this... this "Junior?" Henry: That's his name. Henry Jones, Junior. Indy: I like Indiana. Henry: We named the dog Indiana. Brody: May we go home now, please? Sallah: (To Indy) The dog!? (Laughs) You are named after the dog... Indy: (Embarrassed) I've got a lot of fond memories of that dog.
Re:First mistake? by mcb · 2004-05-20 05:29 · Score: 1

eh, "we named the dog indiana" you mean.

Those Indianans are ruining us! by L.+VeGas · 2004-05-20 05:05 · Score: 5, Funny

Lets just outsource all our test grading to Indiana too.

--
Best Windows Freeware

Re:Those Indianans are ruining us! by Anonymous Coward · 2004-05-20 06:00 · Score: 0

dude, we're Hoosiers, not Indianans

It's About Time by USAPatriot · 2004-05-20 05:05 · Score: 1, Interesting

Many geeks like me did not like English class for the simple reason that grading was entirely subjective to the teacher's tastes.

If he or she didn't like what you wrote, or took a point of view opposite to theirs, you would get a lower grade. Frequently, the "special" students would get the benefit of the doubt, and easy grading just for exceeding their own limitations. An 'A' paper in one English class could be a B- in another, etc, etc.

With this computer grading, these students now know that they will be treated equally, and not bitch about potential human biases. Then, everyone will have a fair shot.

--

Slashdot Moderation: From positive to terrible in 2 "insightful" posts.

Re:It's About Time by Anonymous Coward · 2004-05-20 05:17 · Score: 0

...subjective to the teacher's tastes...
That's gibberish. Please learn to communicate.
Then, everyone will have a fair shot.
Wrong again. Read the article: "The system ...produced results virtually identical to those of trained readers."
In other words, the system is doing exactly what the teachers always did. What's that, you ask? Okay, I'll tell you: There are several components to a "grade" awarded by the system. Much of it comes from a database of personal information about the students. Female? +5%. African-American? +10%. Actual African from Africa? No extra points for that one, sorry. White male? -10%. Bible-believing Christian? -20%.
Under the provisions of the USA Patriot Act, students who have "displayed an abnormal/pathological interest in technology" (as defined by DSM IV) or who practice Islam are penalized and the content of the essay is scanned for "key words"; the results of the latter analysis are forwarded automatically to the US Dept. of Homeland Security for further scrutiny, without the student's knowledge or consent.
Finally, the content of all exams will be scanned for key "consumer words", and appropriate advertisements will be printed in the margins.
Re:It's About Time by Anonymous Coward · 2004-05-20 05:51 · Score: 0

But surely you can tweak the program to give better grades to essays that express the student's patriotic feelings towards their homeland? This would mean that instead of mindless government-bashing and rioting, the youth could actually give some thought to what it really means to be an American.
PS. Slashdot Math: -1 + 2*Interesting = (Score: 3, Interesting) ('Reply' to the parent post to see this)

In fact, by ShitPissFuckCuntTits · 2004-05-20 05:06 · Score: 3, Interesting

As a student the first thing I would hand in is twenty paragraphs from refreshing this. In political science class, of course.

--

--
My username: hats off to George Carlin, and fuck the FCC. Freedom!

Even more efficient? by yndrd · 2004-05-20 05:06 · Score: 1

Let's just use a random generator to give out grades. We'll get the same bell curve distribution, won't we?

Re:Even more efficient? by Surt · 2004-05-20 05:49 · Score: 1

Better specify a gaussian random generator. Otherwise you'll get a uniform distribution (unless you use java).

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
Re:Even more efficient? by egomaniac · 2004-05-20 08:51 · Score: 1

Otherwise you'll get a uniform distribution (unless you use java).

Huh? The 'simple' way of generating random numbers in Java, java.lang.Math.random(), is uniform. You have to use java.util.Random.nextGaussian() if you want a Gaussian distribution.

--
ZFS: because love is never having to say fsck
Re:Even more efficient? by Surt · 2004-05-20 10:55 · Score: 1

java.lang.Math.random() is painfully ununiform, at least in 1.4 it is biased all sorts of interesting ways.

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking

hmm by Anonymous Coward · 2004-05-20 05:06 · Score: 0

Does that mean you could have a program that would tell you what grade you'd get on your essay before you turned it it? And then fiddle with it until it's an A? Would that be cheating?

Trained readers... by ari_j · 2004-05-20 05:07 · Score: 3, Insightful

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers

I think this says more about the training that the "trained readers" are receiving than it does about the software.

Re:Trained readers... by Anonymous Coward · 2004-05-20 05:37 · Score: 0
This program came from ETS, which should know a thing or two about statistics. Notice what we aren't being told:
- What sort of sample data was used?
- Did they construct any test cases for gaming the system? The obvious one would be a well-written essay that is off-topic but mentions the right references.
- What were the characteristics of the 2% that differed from the results of human readers? False positives? False negatives? Mistakes by the human readers?
- How well does it cope with misspellings and typos?
- How easy is it to adjust the program for changes in the grading policy?
- Can the program provide an explanation for the grades it produces?
Re:Trained readers... by ari_j · 2004-05-20 07:08 · Score: 1

Don't forget: Can the program be bribed? :)

This would be wonderful, if it works by magefile · 2004-05-20 05:08 · Score: 1

As it is now, my current teacher doesn't like my essays, but isn't clear about what I can do to improve - a program might help with that. And it would be more consistent.

But somehow, I doubt it's that good. Plus, kids like me are just waiting for it to be released so we can figure out how to scam it. :) Too bad I don't live in Indiana.

Re:This would be wonderful, if it works by Steeltoe · 2004-05-20 06:05 · Score: 1

Take it up with the headmaster. You can accept the situation internally while doing all you can externally to fix it. Some teacher on your school should be able to tell you what you can improve on.

--
http://www.debunkingskeptics.com/

Perfect scores every time by Rorschach1 · 2004-05-20 05:08 · Score: 5, Funny

Ok, since you know the grading software is going to make it into the hands of the students, here's my scheme for perfect essays:

Step 1: Feed some encyclopedia articles, Wiki pages, and other random material on your subject into a Markoff chain generator.

Step 2: Use a genetic algorithm to generate variations of the text. Fitness is determined by the grade calculated.

Step 3: Repeat step 2 until desired grade is achieved. (And, of course, Profit!)

The result is totally worthless, but at first glance would probably appear legitimate even to a human reader.

Sort of like Slashdot posts.

Re:Perfect scores every time by protohiro1 · 2004-05-20 05:43 · Score: 1

Oh man that sounds so fun. I would love to see what this output. Maybe it WOULD fool a human grader.

--
Sig removed because it was obnoxious
Re:Perfect scores every time by king-manic · 2004-05-20 05:49 · Score: 1

30s in a program or 20 min from a human grader. the difference is slight wouldn't you say? It's all a fairly aribitrary way of judging ones worth. Structure is important in both and ideas ussually don't get analyzed too much. The human grader just doesn't have the time while the Program problbly isn't diverse enough to accomadate all themes.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Perfect scores every time by Precipitous · 2004-05-20 06:16 · Score: 1

You'd probably learn more, writing this program, then you would writing the essay. You'd probably also spend much more time. A well deserved A!

--
My motto: "A cat is no trade for integrity."
Re:Perfect scores every time by Anonymous Coward · 2004-05-20 06:43 · Score: 0

You mean a Markov chain generator.

I actually have one written up in perl that you would be free to use.

Sadly, the lameness filter won't let me post it :(

I did a lot to make it useful/pretty, but all you really have to do is first make a hash, then move a sliding window along the text. Suppose the window size is 10 chars (which will basically have you repeat bits of the source material randomly, but that's pretty good for our purposes), then if you have the sentence:

This is an example sentence.

You wind up making hash entries like so:
'This is an' -> ' '
'his is an ' -> 'e'

And so on. Mind you, you always *append* that next letter to the hash entry. You do this over all the source material and wind up with a 'chain' (actually, more like a big, fugly tangle... it can be hard to "tie up the ends" of it, too, as the last lines of text in your source can go nowhere, which gives the generator trouble if it runs down them).

Then, to generate something, figure out about how much you want, start with any old line of source that has a capital letter (e.g. the beginning of a sentence, we hope), and do this process in reverse, stopping whenever you find some punctuation (end of a sentence) and are somewhere inside the range you specified. Note that I was fancy in mine, I had it make its own paragraphs every 4-9 sentences or so, and I turned my source material into nothing but long single paragraphs so it would work better on them.

E.G. we randomly choose the hash entry 'This is an', and we pick at random any single letter in that entry. Since we only did one line of text in the example, the only char in the hash is the ' ', so we append this to our output string, getting 'This is an ', and then we look in the hash under 'his is an ', to find the e, we append that as well, and so on until we reproduce 'This is an example sentence.' as we had above.

At least, this was the most convenient way to do Markov chains that I found, and mine works remarkably well when you don't care so much about repeating yourself a lot and just want to munge some texts :)

I should try it on Darl-speak sometime. It might eventually turn out something more lucid than Darl's actual writings...
Re:Perfect scores every time by Anonymous Coward · 2004-05-20 08:58 · Score: 0

If Step 1 requires you to research your topic, then you're going to be in a really good position to write a paper about the subject you just researched, and skip the whole GA CS BS.

What you need, is to write a program to do your research for you. Then, you can sip drinks in Barbados while getting A's on gibberish.

We have computers to help grade assignments... by Anubis333 · 2004-05-20 05:09 · Score: 1

Now if we could only get computers to help teach our children!

But.. by abscondment · 2004-05-20 05:09 · Score: 0, Offtopic

Some idiot will figure out how to trick the program and we'll have gobs of script kiddies getting A's in English, when in reality they write so poorly that they should be beaten.

Haxored!

Content? by sokk · 2004-05-20 05:09 · Score: 1

Well, what about the content of the essays? Anyways, any way to get a hold on the software :)? Would've been nice with a "clippy" that said what grade the current document is.

AI by Bugmaster · 2004-05-20 05:09 · Score: 5, Funny

That's actually a pretty novel way to approach the problem of creating Strong AI. Making smarter machines is hard, so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test. Damn, this is so sad.

--
>|<*:=

Re:AI by Anonymous Coward · 2004-05-20 07:11 · Score: 0

so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test

When I get up in the morning, I know my coffee pot could beat me in the Turing test :)

e-Rater result? by ackthpt · 2004-05-20 05:09 · Score: 2, Funny

Input: All your base are belong to us!

Output: A+

--

A feeling of having made the same mistake before: Deja Foobar

identical results to those of trained readers... by GGardner · 2004-05-20 05:10 · Score: 3, Interesting

Maybe these says more about the readers than the computer program?

And the era of virtual teacher begins! by gnuman99 · 2004-05-20 05:10 · Score: 1

We dont ned a techer to marc mi worc anyway. The komputer said I writee good!

Seriously, how can a program replace a human when the program cannot comprehend structure like language? Computers cannot and should not replace English teachers or math teachers (well, beyond grade shool at least!). How can a computer program mark an English paper? How can a computer program check that a mathematical proof is correct? How can a computer program say that a particular train of thought is interesting, or pointless.

But above all, does the equal result of a computer software mean that the software is so good or does it mean that the markers grade papers like computers?!? (ie. without any insign)

Re:And the era of virtual teacher begins! by Anonymous Coward · 2004-05-20 05:35 · Score: 0

actually, proofs can be checked by computer, so long
as they are written in a formal enough language. It is a
easy (but tedious) task to translate just about any
proof you can find into say the language of ZFC, at
which point a computer can check it.

Re:Neat by James+Lewis · 2004-05-20 05:10 · Score: 1

Well, as most people around here already know, if you past the link into google, and then click on the link google gives, NYTimes will let you in without registering.

This is both good and bad by n1ywb · 2004-05-20 05:11 · Score: 3, Insightful

Good: The computer probably won't grade you down for writing an anti-Bush essay, and it probably won't get fired for it. Good: Computers won't play favorites, and you can't kiss up to a computer. Bad: The computer really can't grade you up for expressing original ideas. Bad: It's probably possible to fool the computer somehow.

--
-73, de n1ywb
www.n1ywb.com

Re:This is both good and bad by Surt · 2004-05-20 05:43 · Score: 1

I don't know about the original ideas issue. Just input the set of all previously expressed ideas, and compare against the list.

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking

copyright and plagiarism by Anonymous Coward · 2004-05-20 05:11 · Score: 0

Possibly off-topic, but, IIRC, there was a computerized method of detecting plagiarism by students submitting college level essays/theses (I forget the URL) by comparing them to a library of previously submitted work. This seems to me to be a simple thing to link to such a system. When will I have to start copyrighting my assignments, giving my teachers a limited license to grade my work but not store in an information retrieval system?

I can't wait by proverbialcow · 2004-05-20 05:12 · Score: 4, Funny

...until some wiseass figures out a way spoof the grader, probably by sliding under the radar of whatever probabilistic models they've got that pass for spell- and grammar-checkers.

For example:

Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

Grade: B+ Your use of double-negatives continues to haunt you, but I'm glad you've gotten over hanging participles.

--
The only surefire protection against Microsoft infections is abstinence. - The Onion

Re:I can't wait by Anonymous Coward · 2004-05-20 05:16 · Score: 0

I would often get a grade A based on my handwriting and illustrations. My text was peppered with intermittent obscenities. So I's say this sort of system is probably pretty accurate.
Re:I can't wait by proverbialcow · 2004-05-20 05:22 · Score: 1

A friend of mine wrote a short story for 12th-grade English which featured a pact with the devil that made a guy gut himself with a sword every night and bathe in his own blood. He got an A on the story, and a lot of concerned "Are you feeling okay today?"'s from the teacher for the following month.

Luckily for him this was pre-WAVEamerica.

--
The only surefire protection against Microsoft infections is abstinence. - The Onion
Re:I can't wait by DeVilla · 2004-05-20 05:58 · Score: 1

For some reason, this brings to mind Winter Games on the Commadore 64. In the figure skating, I think you only had to get the first move right and then spend the rest of the run falling as much as possible. You'd get a perfect score and the Gold every time.
Re:I can't wait by NaugaHunter · 2004-05-20 06:11 · Score: 1

Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

F-. The only word in that sentence recognized by my spell checker was 'dar', which is apparently an acronym.

--
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
Re:I can't wait by Anonymous Coward · 2004-05-20 08:43 · Score: 0

Your spellchecker doesn't have "jut" (to protrude)?
Re:I can't wait by tepples · 2004-05-20 11:01 · Score: 1

Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

Abconnae ishk fen conkin Geberquen a'Slashdot. Well da malan?

Copying by rf0 · 2004-05-20 05:13 · Score: 1

What about if one person copies another will it pick that up and flag it?

--
Cheap UK and US VPS

Re:Copying by Anonymous Coward · 2004-05-20 05:15 · Score: 0

Probably not... but lets patent the idea of doing both grading and plagiarism checking at the same time!

At least the parent proves something... by Ieshan · 2004-05-20 05:13 · Score: 4, Insightful

At least the parent post proves one simple truth: human english teachers can be replaced by simple shell scripts.

Re:At least the parent proves something... by sheetsda · 2004-05-20 05:43 · Score: 1

Erm... Insightful? Funny I'll grant you (and I think that's how the post was intended), but the script provided has no sense of grammar/spelling and would replace "too" with "tooo" and "stop" with "stoop" (OK, easily fixed with s/\bto\b/too/, so not really an issue) but "to" is a word and has proper uses, the script would arbitrarily replace the first occurence it with "too". If you can write me a script to find an improper use of "to" versus "too" (versus "two") it would seem you're well on your way to solving the Natural Language Problem. "To how many people did you send a dollar?" "I sent too many." (yum. abiguity, although arguably using "too" here wouldn't answer the question.) "to" or "too" here?
Re:At least the parent proves something... by eclectus · 2004-05-20 06:01 · Score: 0

and the truly ironic thing is that there was a second mistake in his pattern replacement that you neglected to catch; the fact that he was supposed to replace the 'To' with 'Two'.

--
This signature is a waste of 42 characters
Re:At least the parent proves something... by Anonymous Coward · 2004-05-20 06:19 · Score: 0

Ah, this is a troll, right?

You really know 'too' is correct, and you add capitalization to get more complaints!
Re:At least the parent proves something... by Short+Circuit · 2004-05-20 06:30 · Score: 1

Teachers? No. Graders? Maybe.

( Though I wouldn't call it a simple script. )

--
tasks(723) drafts(105) languages(484) examples(29106)
Re:At least the parent proves something... by Anonymous Coward · 2004-05-20 06:46 · Score: 0

So now all we need is a "simple shell script" that will generate essays that bias the software towards a better grade.
Re:At least the parent proves something... by Anonymous Coward · 2004-05-20 07:05 · Score: 0

I had my post corrected by a shell script tooo, and I believe that there are some bugs too be worked out.
Re:At least the parent proves something... by jav1231 · 2004-05-20 09:04 · Score: 1

This is kinda funny esp. considering the report that came out last year, I think, saying that teachers as a group represented those who did poorly-to-average in college. Now we see this. Are we making their jobs easier because they can't handle them? Just food for thought, as I think teachers as a group have a monumental job and are grossly underpaid.
Re:At least the parent proves something... by phrogeeb · 2004-05-20 09:12 · Score: 1

No.

Nu-uh.

Absolutely not.
I am a high school senior. My scholastic abilities are primary literary. I'm nothing special in math, but in English, especially essay-writing and creative writing, I really shine. (I'm modest, too.) I've written entire papers without punctuation that received A's. I've written papers constructed entirely of fragments that received A's as well. In fact, almost every paper I write has either a run-on sentence, a dangling participle, or a sentence fragment. And I've only gotten one or two B's in English ever.
Real writers - the kind that we want to encourage - play with grammar, punctuation, even spelling. Their constructs are wierd. Their arguments don't always appear logically linked and aren't always guided by transition statements. That's the way it is.
Teaching conformity is for Soviet Russia. Never should we encourage people to treat writing as a mindless exercise that follows a formula or a set of rules. We may as well have robotic art critics.
Screw the machines. I hope every student exposed to them submits a bastardized wikipedia article that fools the formula and takes them 5 minutes to write, and then goes and spends a few hours working on their own, more worthwhile projects, because that's all this system encourages.

--
------
"Will the highways on the Internet become more few?" --George W. Bush, in Jan. 2000
Re:At least the parent proves something... by Ieshan · 2004-05-20 09:18 · Score: 1

Totally depends for which discipline you're writing. Though, I agree. The g-parent was a joke. =)
Re:At least the parent proves something... by Typhon100 · 2004-05-20 09:39 · Score: 1

Exactly. Now, I don't think we should be writing critical essays about Shakespeare without using the letter e, but you do have a point.
How does the software feel about starting sentances with conjuctions? Software (that I've seen) is notoriously bad at picking out clauses - how many times has a grammar checker tried to pluralize the wrong adjective?
My other concern would be word usage...would the following sentance recieve a high grade?
My canine uncle, who was a bio-chemical engineer in the Russo-Martian War, bakes automobiles in his pocket. ??
You can't encode style.

In Other News by ThisIsFred · 2004-05-20 05:13 · Score: 5, Interesting

Indiana parents are the first to buy (en masse) licenses for Essay Constructor Pro v2.0. The software produces essays that are indistinguishable from those written by real students, using the latest screen-scrape-from-Internet 'n' plagiarism-from-non-credible-sources techniques.

Indiana Director of State Board of Ed comments: "Isn't it wonderful how technology is improving education?"

--
Fred

"A fool and his freedom are soon parted"
-RMS

Re:Indiana? I thought it said India by Anonymous Coward · 2004-05-20 05:14 · Score: 0

Yeah well, racist Americans are a dime a dozen. No need to stand up and call yourself a jackass in every /. article.

I have an algorithm they can use... by Anonymous Coward · 2004-05-20 05:14 · Score: 0

... and save a bunch of money.

Randomization!

Statistically speaking, 75-80% of the people won't question their grades. For the rest of them, let a lengthy, expensive, and convoluted appeals process sort 'em out. Either that, or threaten to grade them again on the chance that their score is lowered.

I wanna try, I wanna try! by lpangelrob2 · 2004-05-20 05:14 · Score: 1

So if I did this, what would happen?

The word emission generally means sending something out. Because of this argument, Hauser is a city located in Kootenai County, Idaho. After Idaho, the Liberals formed the government in Alberta for the first 15 years of the province's existence.

Yeah, those were random snippets from the Wikipedia. Who knows? Maybe this technology got around?

Grade: A+++

--
-Rob
Marriage doesn't have to suck!

Re:I wanna try, I wanna try! by CrimsonAvenger · 2004-05-20 05:57 · Score: 1

"Because of this argument" is poor grammar. You'd probably be graded down for that. Beyond that, looks good to me.
I wonder to what extent your essay can be meaningless gibberish with proper form and still get a good grade?
On the other hand, if kids are forced to learn grammar and spelling, it can't be all bad, even if they CAN turn in meaningless gibberish.

--

"I do not agree with what you say, but I will defend to the death your right to say it"

Other applications by jdbarillari · 2004-05-20 05:14 · Score: 0, Redundant

A 6-point computerized grading system. Hmmm. Moderators, watch out -- your jobs are about to become obsolete!

Re:Other applications by Gannoc · 2004-05-20 05:22 · Score: 1

A 6-point computerized grading system. Hmmm. Moderators, watch out -- your jobs are about to become obsolete!

Don't worry! Our moderaters rate on a _FIVE_ point system, and are in no danger of being outsourced. Bush '04!
Re:Other applications by Anonymous Coward · 2004-05-20 05:43 · Score: 0

I wish I could see the look on your face when you realize this post will be modded -1, Troll.
Re:Other applications by king-manic · 2004-05-20 05:52 · Score: 1

5 points positive, but also 0 and -1. So we have a 7 point system here.
-1 0 1 2 3 4 5

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."

This says more about "trained readers"... by Phurd+Phlegm · 2004-05-20 05:14 · Score: 4, Insightful

than it does about the software, methinks. I imagine it gives helpful hints like the ones I always turn off in Microsoft Turd. Any construction that deviates from the norm in a boring business document apparently triggers the "grammotron" or whatever they call it. A human reader has some appreciation for style and may actually accept something a little different for the sake of variety and sparkle.

Not that there's anything in this post that serves as an example. I guess that's because I was graded by humans. Seriously, I don't recall getting any encouragement in writing back in the '70s in high school, and not much in college. I guess it wouldn't have been any worse if the Grade-O-Vac was inspecting my papers instead of my mostly-marginally-literate teachers. There were several exceptions, but they focused much more on reading than on writing. I suspect they had a lot greater effect that way--I know they had a great effect on me.

Re:This says more about "trained readers"... by booyaar · 2004-05-21 00:05 · Score: 1

The same grammotron as the Grammaton Cleric? Would certainly make sure you got your grammar correct.. :-)
Re:This says more about "trained readers"... by Anonymous Coward · 2004-05-22 06:28 · Score: 0

My father is retired from academia (I won't say from where exactly to protect the guilty). He told me that essay exams for 100-level courses would be graded by T.A.'s by giving the T.A.'s a checklist of concepts to look for in each essay. The more of the concepts were in the essay, the higher the grade--grammar & spelling didn't count & organization counted for little. One day, he had a T.A. come to him for guidance on how to grade a particular essay: the essay wasn't even written in complete sentences, it was just a series of bullet point terms and phrases relating to the subject matter of the question. It wasn't even as organized as an outline. The T.A. didn't know how to grade it because all the concepts on the checklist were in the "essay" (which would make it an "A" paper) but it wasn't really an essay-- the T.A.'s instinct was to give it a much lower grade. If I recall the story rightly, it was finally decided to give the essay an "A" because to do otherwise would be to unfairly change the grading criteria. They changed the criteria for grading essay exams after that.

My point is that I can easily imagine a human reader using a checklist and a computer using a checklist awarding similar grades on essays.

No way this is sound by Hays · 2004-05-20 05:14 · Score: 1

I cannot believe they are doing this. It is undoubtedly a deep AI problem to properly grade an essay. You have to be able to understand if statements connect logically to form an argument, you have to understand analogy, you have to understand sarcasm, etc... this is so far beyond our current capability.

As implemented, I'm sure there are easy ways to scam this system by writing gibberish. Who sold them this idea? I want to know the person responsible for this abuse of computing.

Re:No way this is sound by IWannaBeAnAC · 2004-05-20 05:44 · Score: 2, Interesting

Edsger Dijkstra would no doubt have something profoundly and humerously offensive to say to the writers of this software ;-) There doesn't seem to be anyone to take over his mantle, which is presumably why the software industry is going pot at an ever increasing rate. sigh.
Re:No way this is sound by corngrower · 2004-05-20 06:03 · Score: 1

I'll agree with you. It should be fairly easy to scam the system since I doubt that the software analyzes the content of the essays all that well. I'm sure the program can look for good grammer, sentence structure, and spelling. It may also be able to do some analyitical reasoning on the text. But I doubt it would be able to really do a good job of determining things like paragraph cohesion or analyzing reasoning that depends on intelligence commonly shared by the readers.

I see as a potential problem the fact the the software was probably 'trained' using text written by students/participants expecting a human grader. When the students know that there essays are graded by computers, they likely will write essays that they know will give them a good grade, and not necessarily write what most human readers would call a good essay.

Students by Gettinglucky · 2004-05-20 05:14 · Score: 2, Funny

Now all the students need is e-writer so that they can just type in the subject and the score they want to achieve and then e-reader will grade it accordingly!!!

Can students get this program? by Theovon · 2004-05-20 05:14 · Score: 4, Insightful

If I were a student, I'd want to get a copy of this software and use it to pre-grade my papers so that I could find out what's wrong and fix it before I turned it in.

Re:Can students get this program? by nine-times · 2004-05-20 05:36 · Score: 1

Now, that's actually a pretty decent idea for how this software could be used legitimately. Would that be really different than spell-check and grammer-check, merely because it's more advanced?
Re:Can students get this program? by RoderickMcDougall · 2004-05-20 05:36 · Score: 1

I suggest you ask the closest teacher to you. I reckon they will just give it to you because you asked.

What you are suggesting isn't all that unreasonable. It would be far better if students were given this to encourage them with feedback to refine their essays and consequently become better students with good grades.

Personally I've always felt it is best to work back from the answers (otherwise known as cheating mostly) but it really is the most efficient way of learning.
Re:Can students get this program? by Psychotext · 2004-05-20 06:43 · Score: 1

Personally I've always felt it is best to work back from the answers (otherwise known as cheating mostly) but it really is the most efficient way of learning.

University taught me to agree with that statement completely. I thought I knew pretty much everything on the subjects I was taking, so I didn't really bother looking at the past papers and solutions. Big mistake. It turns out that they weren't interested in well constructed conclusions backed by evidence - They were more interested in the answers as they (The lecturers) saw them. Just as a note, these were management / business questions so there weren't really any exact answers - Only opinion.

The next few years I didn't bother doing any work at all. I just made sure I knew the previous years questions and answers verbatim. Strangely I never failed another exam. I can only imagine that this electronic system is even worse than that due to its reliance on scripted logic.

--
People that believe in their opinions don't post AC.
Re:Can students get this program? by Man+of+E · 2004-05-20 07:21 · Score: 1

And if it's not immediately available to students, rest assured that it'll show up on various P2P systems shortly. Somebody is bound to leak it.

--
Ceci n'est pas une sig

That's ok by paranode · 2004-05-20 05:15 · Score: 2, Funny

schmoozing with the teacher to get higher grades

This works better for the Slashdot crowd. They are much better at romancing computers than people to get what they want.

dangerous dangerous dangerous by tplayford · 2004-05-20 05:16 · Score: 1

Especially for a subject like English. Over the years teachers will begin to understand how the computer makes and they'll tell their students what style will get the beast marks.

It becomes a study of the marking algorithms, not a study of English.

I suppose they could make alteratoins every year... but then it could become unfair.

Re:dangerous dangerous dangerous by denis-The-menace · 2004-05-20 05:44 · Score: 1

I see your point.
It would be the high school version of Google.
People would come out with a bunch of tricks in order to get get a better grade or ranking.

This e-grade-ing will work for about 3-4 years tops. After that, only hackers will get good grades.

Jocks and bimbos won't be able to use their "talents" to help their grade. PCs don't care for sports or Blow-jobs.

--
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
Re:dangerous dangerous dangerous by The+UberDork · 2004-05-20 06:15 · Score: 0

Why not? Don't hacker's like blow jobs? Don't hackers NOT want to get their ass kicked?

India vs. Indiana by the+eric+conspiracy · 2004-05-20 05:17 · Score: 4, Funny

Wouldn't it just be cheaper to grade the tests at call centers in India? What are those Indians doing when there are no incomming calls? Just slacking off??? They could be grading tests.

Re:India vs. Indiana by Surt · 2004-05-20 05:46 · Score: 1

It's not actually multiple Indians. It's just one guy. He answers all the phone calls. That's why the wait times are sometimes really long.

Trust me, he's working hard for his $0.90/hour.

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking

Is this a joke? by Anonymous Coward · 2004-05-20 05:17 · Score: 0

e-rater will generate an advisory if it has difficulty scoring or identifying some or, all, of the writing sample.

I wonder if e-rater flagged that incorrectly placed comma.

Seriously, does anyone remember a program for Apple II circa 1985 called "Babble"? You'd give it a couple of keywords and a number of paragraphs, and it would churn out pages of grammatically valid, completely readable nonsense about any topic.

From the FAQ:

I don't see how a program can be much more than a super-sophisticated grammar-checker. As the FAQ says, e-rater doesn't actually "read" anything. I wonder how it would score the output of Babble.

How would e-rater would judge essays written by, say, Hemingway, or another writer who has a sophisticated and unique style. How does a machine judge sarcasm or humor in general? How does it judge accuracy or depth of thought? How would a well-written but totally racist essay be graded?

From the FAQ:

e-rater compares the new essay to samples of essays previously scored by faculty readers-looking for similarities in sentence structure, organization, and vocabulary.

Great. The more you conform with your peers, the higher your score.

Last point-- writing an essay to be read by a human and one to be processed by a machine is, I suspect, a different psychological exercise. To write for a human is called "communication". To write for a machine is called "programming" or "data entry".

Re:Is this a joke? by stanmann · 2004-05-27 00:26 · Score: 1

How would a well-written but totally racist essay be graded?
So on a paper where the student must select and write 3-5 pages on a historical figure of his choice, the student who selects hitler or stalin should be penalized vs the student who selects MLK or George washington, or what about the student who selects Jesus vs the student who selects Plato or Socrates??

Should they be evaluated on factors other than historical accuracy and writing proficiency?

--
Food not Bombs is a nice platitude but it breaks down when you notice that the Bombees are usually well fed

Makes life easier by Dracolytch · 2004-05-20 05:17 · Score: 1, Redundant

Since I know a real person won't be reading the paper, all I need to do is come up with an AI to write the paper.

2: Sell it to my classmates
3: Profit!

~D

--
This sig has been enciphered with a one-time pad. It could say almost anything.

Re:Makes life easier by king-manic · 2004-05-20 05:55 · Score: 1

If they found out they'd find statistical markers of your program (unless yoru friggin brilliant it won't have the same word mix/ structure as a normal essay) and regrade all the essays (it's automated so it's nto that time consuming), and flunk all who tested positive.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Makes life easier by Dracolytch · 2004-05-20 06:27 · Score: 1

Depends on how you write the software. If you used a pure statstical technique to write software, yes, that could happen. Of course, you need a pretty large number of essays to detect these stastical markers with any accuracy (Of course if they got a copy of the program, they'd have it). On the other hand, you could have the program use different writing styles, or adjust the weights on the words used. You could even have each computer learn to train differently, so the output structure would be based on that instance of the software, not the codebase as a whole.

Want to profit even more? Sell the anti-cheating software to the schools under a different brand name.

--
This sig has been enciphered with a one-time pad. It could say almost anything.

Completely offtopic papers by fulldecent · 2004-05-20 05:18 · Score: 2, Interesting

...and how would one ever devise software capable detecting papers that are

-1 Offtopic

while completely gramatically correct?

--

-- I was raised on the command line, bitch

For grading grammar, a computer would be unbiased by tmundar · 2004-05-20 05:19 · Score: 1

I wrote many essays when I was in school, and I remember being graded down because the teachers didn't like my ideas. The quality of my writing was acceptible, but how I wrote the topic was not. I didn't write anything offensive, but I occasionally took creative liberties with the point of view (i.e. I changed the gender of the person who wrote the article, and then interpreted it from that point of view). Maybe I write an essay about atheism, but the grader is intensly religious and has a bias against my point of view.

I just want to point out that having a computer grade a paper using rules would be much more 'fair', because the computer won't have any personal bias about the content. I am sure that the computer would grade an essay against computerized grading identically to an essay promoting computerized grading.

This will create more techies... by MoeMoe · 2004-05-20 05:19 · Score: 1

As many have said regarding this topic so far, the kids will try to find out what makes this system tick and find an exploit... Having said this, wouldn't that end up making atleast a few of these kids smarter through trying to "crack the code"?

It's just my opinion, but I say if you can figure out how to use the tricks of the trade, whether on some grading system or a complex business scenario, you deserve the grade you set for yourself... That's what learning (and to some extent grading) is about, right?

--
Business \Busi"ness\, n.;
A scam in which all people involved perceive as beneficial...

Necessary != Sufficient by uncadonna · 2004-05-20 05:20 · Score: 1

Amazing.

To dispel skepticism over computer scoring, student essays were simultaneously graded by a computer and trained readers during a two-year pilot program. Using artificial intelligence to mimic the grading process of human readers, the computer's automated scoring engine, known as e-rater, generated grades on a six-point scale that were virtually identical to those of the readers.

Clearly this is an inadequate test. The fact that the system performs reliably on essays that were going to be graded by humans does not mean it will perform reliably on essays that no human (other than the author) will see. In practice, students will reverse engineer the probably very shallow AI pretty quickly and a contest to get high grades on particularly stupid submissions will ensue.

This will amount to an education of sorts, I guess...

--
mt

Don't blame the computer by Scott+Richter · 2004-05-20 05:20 · Score: 2, Insightful

I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

Sure, but that's the fault of the humans implementing the grade system, who don't understand the difference between Gaussian and uniform distributions. Don't blame the computers.

Not that computers are a great idea here - they can only grade at the shallowest level, and if they were grading like real teachers, then those "real" teachers weren't doing their jobs.

But this specific problem that you mention is entirely human based.

Brought to you by... by Anonymous Coward · 2004-05-20 05:21 · Score: 0

Indiana, The people who declared 3 equal to pi.

Standard Grading by Tony · 2004-05-20 05:21 · Score: 3, Funny

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

So it gives its favorite students 'A's without reading, least favorite students 'F's, and the rest arbitrary grades somewhere in between to mimic a bell curve?

Excellent!

"Artificial Intelligence is easy. It's artificial stupidity that impresses me." -- Arthur Oscar

--
Microsoft is to software what Budweiser is to beer.

Re:Standard Grading by MooseByte · 2004-05-20 05:54 · Score: 1

"So it gives its favorite students 'A's without reading, least favorite students 'F's, and the rest arbitrary grades somewhere in between to mimic a bell curve?"
The AI system replaces the venerably robotic system that would toss the essays down the stairs with its robotic arm and then visually scan for location, A's in the first 3 steps, B's in the next 3, etc.

Missing feedback by Hamster+Of+Death · 2004-05-20 05:21 · Score: 2, Interesting

How are these system supposed to scribble in the margins and tell you your ideas don't fit together?

How do they judge the content? What if you submit an excellent paper on middle ages history but the assignment was on socialism?

Human feedback is required in order to learn how to write well, you can't just expect a machine to tell you how to improve your writing. Grammar perhaps, but not ideas and how to let them flow coherently.

In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?

Seems like they are trying to solve the wrong problem with this system, or a problem that dosen't exist. (Are there really so many papers to mark you need a machine to do it?)

Re:Missing feedback by RzUpAnmsCwrds · 2004-05-20 06:44 · Score: 2, Interesting

"How are these system supposed to scribble in the margins and tell you your ideas don't fit together?"

They can't. But on a standardized test, you don't get any feedback anyway.

"In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?"

Because it takes too long.

"Are there really so many papers to mark you need a machine to do it?"

Yes. Human graders for standardized tests get about 1-3 minutes per paper. Human graders don't have time to really read your essay, so they grade you on the same kinds of criteria as this software does (grammar, spelling, a clear layout, etc.). Thus, it's not hard to create a computerized system which performs the same tasks as well as human graders.

"How do they judge the content? What if you submit an excellent paper on middle ages history but the assignment was on socialism?"

As I said, human graders don't have time to evaluate this anyway. The computer systems actually tend to be better at this.

Look, this system can and should not be used to relpace English teachers grading papers. A good English teacher will spend at least 30-60 minutes on a paper, and will write lots of comments.

What this system *is* good for is standardized tests. When everyone takes a test, you have (in many cases) a million tests to grade. This system can blast through the data and can actually perform better than an underpaid, overworked employee who has 1-2 minutes to grade a paper (and who grades hundreds of papers per day).
Re:Missing feedback by Hamster+Of+Death · 2004-05-20 06:56 · Score: 1

Glad to see we agree, for standardized tests they've been doing this for years with the scantrons etc.

As for papers, essays and the like, I think this just won't work.

Antidisestablishmentarianism! by Bluesman · 2004-05-20 05:21 · Score: 4, Funny

By creating a vernacular consisting of elongated words and sophisticated verbiage, obviously indifferent to definition but simultaneously observing grammar regulations while eschewing colloquialisms, perhaps students may increase individual chances of achieving substantial academic acclaim.

If this works anything like the writing level indexes you find on word processors, it should be easy to fool.

--
If moderation could change anything, it would be illegal.

Re:Antidisestablishmentarianism! by Anonymous Coward · 2004-05-20 09:28 · Score: 0

Congrats. - You have recieved a Flesh-Kincaid grade level of 12.0!
Re:Antidisestablishmentarianism! by Anonymous Coward · 2004-05-20 10:36 · Score: 0

Have you been reading research papers on Education lately?

Sending the Wrong Message to Students by grantls · 2004-05-20 05:21 · Score: 0

This to me is a really scary idea. I'm a grad student in English at Indiana University who will begin teaching freshman composition in the fall semester. One of the dangers of humanities computing that I've heard spoken of countless times by composition instructors is that their students believe grammar check results without questioning them; their grammar skills are such that they have problems understanding why the grammar check was wrong when examples are shown to them. This inability to see the difference is due both to poor preparation before college (I'm not just blaming the education system here, but the students as well) and too much trust in computers to do the job for them. The kind of grading program used by Indiana high schools is not only a bad idea for its inherent inability to do its job, but for the message it sends to the students: If code can write (which grammar checking is a first attempt at) and code can grade, then what is the student's role in their own learning process? If this program is producing nearly-identical results then that is indicative of a algorithmic grading on the part of the human graders, which indicates that this mentality has a broader reach than just those in the student subject position. (Or it indicates that the brain is a computer, which many humanitities professionals would debate - I don't have a position on this.)

Re:Sending the Wrong Message to Students by Anonymous Coward · 2004-05-20 06:11 · Score: 0

Heh. I don't study English (although I work in natural language understanding). What the hell do you mean by??

indicates that this mentality has a broader reach than just those in the student subject position

Of course I disagree with using a simple checklist to algorithmically mark an essay as a general principle, but is it that simple? Such a checklist would be simple to program into a computer.

For plain jane essay topics student outputs generally follow the same mold. In that case why not just create a checklist to simplify the workload? Indeed if students could first learn what A-grade essay looks like for menial topics, they may easily step to writing well in relation to novel ideas.

If you pick up an essay at random about a randomly selected thesis, then you feel a 2004 level computer can't evaluate the writing. In the coming years computers may well be able to automatically Google for information relevant to the topic and assemble their own ideas. We should expect computers to gradually return good answers to Ask Jeeves. Once that happens computers will participate in conversations and evaluate prose credibly.

Patriot Act Junior by Kaenneth · 2004-05-20 05:24 · Score: 2, Interesting

I wonder how long until the FBI is linked to this system?

Grammar, 90%
Spelling, 95%
Patriotism, 80%

also:

I'd love to see famous writings graded by this system.

Politcal Correctness Algorithm? by random+coward · 2004-05-20 05:25 · Score: 1

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

Every essay I have ever had scored had over 60% of the grade based on political bias of the scorer matching the conclusions and assumptions in the essay. So what is the PCA(Politcal Correctness Algorithm) they implemented that allowed the results to be virtually identical to those of the indoctrinated^h^h trained readers?

Reality: human graders trained by machine (GMAT) by j_dot_bomb · 2004-05-20 05:26 · Score: 1

e-rater is used on the GMAT. What do you think happens when the human grader disagrees too much with the machine ? They get fired.

Too Uniform by Roger_Wilco · 2004-05-20 05:27 · Score: 3, Insightful

My writing style is somewhat peculiar, though I can't exactly say how (or even approximately how). Partially as a result of this, my marks in English class over the years of high school ranged from C to A, depending not on me, but on who the teacher was. If the teacher happened to like my style, I got a good mark.

This is annoying, but at least each year there was a different teacher, who may like my style. If the marking is computerised, it will not change; if your writing doesn't fit what the computer likes, you're screwed; likewise, if it does like it, you might never learn to express yourself more creatively (ie you'll be punished for trying to write in a manner different from what you usually do).

There are possibilities in this technology, but I suspect that it will be a long while before the eccentric aren't labeled as poor writers.

Re:Too Uniform by Surt · 2004-05-20 05:53 · Score: 1

There are many colleges and businesses who are interested in knowing if you can write a boring, non-eccentric essay. A computerized test for this is exactly what they want.

--
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking

Babelfish by Waffle+Iron · 2004-05-20 05:27 · Score: 3, Funny

I will not trust computer grading until I see a computer-translated document that isn't laughable.

To illustrate my point, I'll restate it. [English -> German -> English]:

I do not trust the computer, which arranges, until I see a computer-translated document of this laughable isn't.

That's about how well a computer "comprehends" language today.

Re:Babelfish by Anonymous Coward · 2004-05-20 10:51 · Score: 0

Hey, that's pretty good by normal computer translation standards.

To illustrate my point, I'll restate it. [English -> Japanese -> Klingon -> Etruscan -> Welsh -> English]:

How elegantly the whelk sitteth upon his tissue.
foreigner paramount tyrannize graphotype milestone

Whoever approved this should be fired by Retired+Replicant · 2004-05-20 05:27 · Score: 3, Insightful

A computer can check spelling and even grammar to a certain extent. However, it cannot evaluate factual accuracy, strength of argument. Even with spelling, the computer is not likely to catch improper use of homonyms. I can guarantee you that it will be possible to create a piece of writing that is utter crap that would get an A+ using this or any other possible computerized grading system. Unfortunately, there are probably many teachers out there who make poorer graders than this system does. The answer to the problem of poor-quality teaching is not replacing teachers with computers; the answer is a combination of better teacher pay and putting higher standards in place for our teachers via competency testing.

Re:Whoever approved this should be fired by CrimsonAvenger · 2004-05-20 06:05 · Score: 1

homonyms
Am I correct that you meant synonyms?
Actually, I would assume the purpose of this is to remove personal bias from the grading system. It will not work, of course, until we can teach a computer English (or such other language as you wish to teach).
Until a computer can understand the difference between "Time flies like an arrow" and "Fruit flies like a banana", computer-grading of essays isn't here yet.

--

"I do not agree with what you say, but I will defend to the death your right to say it"
Re:Whoever approved this should be fired by Anonymous Coward · 2004-05-20 06:12 · Score: 0

"Time flies like an arrow" and "Fruit flies like a banana"

You must have had a math teacher tell you this joke...
Re:Whoever approved this should be fired by CrimsonAvenger · 2004-05-20 07:07 · Score: 1

No, my wife did, when she was doing her graduate work. A wonderful example of the difficulties of English.

--

"I do not agree with what you say, but I will defend to the death your right to say it"

Essay grading is harder than science grading by sustik · 2004-05-20 05:27 · Score: 3, Interesting

I believe that (English) essay grading is harder than grading science exams based on problem solving (no bubbles please), at least if essays are about content and not just grammatical correct sentences.

I say this because there is an objective criteria for grading the solution to a physics or math problem: correctness. For essays I do not beleive that we (and the current state of AI) can come up with an exact criteria like that. You might determine whether an essay is too different from essays which were written by experts, but cannot a very different essay to be just as good?

To my knowledge the AI programs can solve physics problems which are limited to some well defined domain (for example: http://www.cs.utexas.edu/users/novak/cgi/isaacdemo .cgi though that is from 1977...) I am not aware of programs grading physics problem solutions.

I will accept an essay grading program after they grade solutions to math and physics problems.

I conjecture that some writers would feel offended if their essay did well according to the program: they might think it means they are too conformist and conservative and not novel in their approach...

Matyas

From the web site. by Bluesman · 2004-05-20 05:28 · Score: 3, Informative

>If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site.

Submit "and" essay? I guess they haven't run the software on themselves.

F.

--
If moderation could change anything, it would be illegal.

Not always so simple by Anonymous Coward · 2004-05-20 05:30 · Score: 0

Wow, you just showed how to crack SHA1! It takes a file as input and generates a hash as output. All I need to do is reverse the process to get a file that generates that hash! Duh! It's so simple. Why haven't all these Ph.D's thought of that?

Then again, SHA1 took a lot of research to create it such that it would be hard to reverse. Maybe not such a bad idea for a grading program.

Re:Reality: human graders trained by machine (GMAT by j_dot_bomb · 2004-05-20 05:30 · Score: 1

By the way why would the company do this you might ask ? Heres one reason, the GMAM grading rules say that if the scores of the human and machine (e-rater) differ too much then a third grader (human) must grade it. Thats more cost for the company.

The evolution of machine intelligence by mcrbids · 2004-05-20 05:32 · Score: 1

We all remember the Twilight Zone episodes about the Earth being dominated by cold, unthinking, uncaring machines that take over and establish an Orwellian rule, or the Terminator who speaks of "Skynet" becoming self-aware.

Yet this is not how machine intelligence has been evolving. Think Google, think Visa's fraud detection system, think the learning DSPAM which simulates learning so effectively.

Machine intelligence will not replace human intelligence - like the cerebrum augmenting the "lizard brain" at the base of the brain, our extrasomatic information technologies and machinery augment, enhance, and extend the capabilities of our cerebrum.

What's interesting is that the evolution of machine intelligence has hit a point where it surpasses, or will soon surpass, the thinking performed in our collective cerebrums.

We are becoming the Borg of Star Trek tales, armed with our cell phones, Internet access terminals, (I write this from a Hotel 2 blocks from Disneyland, in Anaheim, CA with my Linux laptop - where are you?) telephones, pagers, walkie-talkies, and other inter-personal communication devices.

And I find no end of fascination in watching the birth of this machine intelligence, and watching it become so tightly integrated with our humanity.

I can now perform in minutes research that just a few decades ago would have taken hours or days. This is the result of machine intelligence, and it will get far better, not worse.

Sit back and enjoy the ride - it's going to be a fun one!

--
I have no problem with your religion until you decide it's reason to deprive others of the truth.

Funny if it wasn't so scary... by renzema · 2004-05-20 05:32 · Score: 1

Text from the website...

If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site.

Bush asks you to keep the faith by Anonymous Coward · 2004-05-20 05:32 · Score: 0

It's kind of sad.

Tradeoffs by dtfinch · 2004-05-20 05:33 · Score: 1

Pros:
Every student is treated equally. Emotion and prejudice are taken out of the equation. No more "I got a bad grade because the teacher didn't like me or got PMS the last day of grading."
Faster grading. Might saves money. Grade a small sample of papers by hand, combine your results with that of others grading similar papers, or papers from previous years, and let the program grade the rest.

Cons:
Papers much better than the norm may be penalized if they are unlike anything in the training set.
A student can just write what the kind of language that's expected, with all the right buzzwords, and probably get a good score without any research of other hard work. Though I don't have the software to test this theory. "George Washington freed the slaves from Abraham Lincoln in 1812 in the battle of Gettysburg Address against the British indian pilgrims."
There will be disputes. Students and parents may very well spend as much time arguing with teachers over the results as teachers save in reduced grading time.

Computer Comprehension by Dareth · 2004-05-20 05:33 · Score: 1

Since when did computers learn to comprehend human language? I imagine a nice round of perfect syntax essays on various subjects. If you are supposed to write an essay on former president Bill Clinton, could you compare him to the goatse.cx guy in a syntaxically correct manner and still get a good grade?

And if it does work, could we use it to auto-moderate slashdot. Since I got goatse in here, I must be an automatic -5 Troll!!!

--

I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling

Re:Computer Comprehension by Krach42 · 2004-05-20 07:40 · Score: 1

The computer doesn't comprehend the human language. It just compares the usage and environment of words.

The syntax of the essay plays a lower role in the grading, that the evironment of the words that you do use.

Meaning AdLibs would turn out poorly, even though syntactically correct.

--

I am unamerican, and proud of it!

Another solution in search of a problem by CarrionBird · 2004-05-20 05:33 · Score: 1

Meanwhile our schools continue the slide into the abyss. Just great! Feh!

--
Free Mac Mini Yeah, it's

Unbiased yes, but intelligent, no by hode · 2004-05-20 05:34 · Score: 1

I'll agree with you that this system of grading is unbiased, but that is the most prominent problem with it. While the computer has no preconceived notions about what you are arguing in your paper, it cannot create any either. The system is only concerned with grammar and spelling. A computer cannot possibly determine if an argument is well formed, flows nicely, and leaves the reader questioning their previous beliefs.

If we all have impeccable grammar and spelling, but all we can talk about is what happened on Sponge Bob Square Pants, what has education done for us?

English is a proper noun, too by Anonymous Coward · 2004-05-20 05:34 · Score: 0

Doh!

Re:English is a proper noun, too by drakaan · 2004-05-20 06:37 · Score: 1

what about when you're using the "english" that means adding spin to a billiard ball?

--
"Murphy was an optimist" - O'Toole's commentary on Murphy's Law
Re:English is a proper noun, too by Jaysyn · 2004-05-20 08:24 · Score: 1

Or you could be President Bush.

Thanks, I'm here all weekend.

--
There is a war going on for your mind.

Waiting by sielwolf · 2004-05-20 05:35 · Score: 1

The big question is, will other states begin to emulate Indiana by tossing human grading?

Actually everybody else's heard of an Indian computer that will grade the same paper at a tenth of the cost that will be released in a few months.

We must stop the outsourcing of jobs that can be handled by capable AMERICAN computers! When will this end?!?

--
What is music when you despise all sound?

Funny how well this stuff worked out by SuchiRu · 2004-05-20 05:36 · Score: 1

I was part of the Pilot classes used for this. I go to a "gifed and talented" school by the name of The Indiana Academy, and our entire class was used to test this procedure. The thing was taht there were so many problems with the pilot that no one, not even the person the testers sent, could get the program to work correctly. This was last year by the way. Watch out for this stuff. There are some things computers simply should not be used for.

the triumph of mediocrity by KMonk · 2004-05-20 05:36 · Score: 5, Insightful

Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.

Re:the triumph of mediocrity by whyde · 2004-05-20 07:52 · Score: 2, Interesting

This was the point I was going to make, too.

Nobody wants to read 5-paragraph themes all day long, even if they do get the point across. They are just a means to an end.

One of the best English teachers I've ever had would point to the use of alliteration, clever turns of phrase, humor, novel word choice (not just synonym-madness), and other completely subjective facets of writing as some of what makes the written word worth reading.
Re:the triumph of mediocrity by RDPIII · 2004-05-20 10:44 · Score: 1

You would be correct if the purpose of the test was to distinguish excellent writers from merely very good writers in a creative writing course. However, the issue is quite different. The developers of the grading software know quite well that automatic grading cannot detect mixed metaphors, clicheed expressions, etc. That's why automatic grading is typically used in grades 6 through 10 (it also doesn't offer enough guidance for complete beginners) and only in "low stakes" situations.

Let me repeat that for you: automatic essay grading won't (ever, most likely) be used to score creative, literary works. But it can certainly help advanced beginners, who are struggling with subject-verb agreement (questions like whether to write "the purpose of companies are" or "the purpose of comapnies is"), proper spelling, too much repetition, etc. Simple, trivial stuff for professional writers, but important stuff to point out to beginners.

any truly excellent writer is likely going to do badly

Not really. Because what students are writing are five-part high school essays, not highly creative works of fiction. Simple stuff, like thesis, three supporting points, conclusion. You know, for kids. An excellent writer would realize what's expected of her and write accordingly.

Writing is not mathematics.

Well, you can also say "X is not mathematics" for almost all values of X. That doesn't mean that X doesn't have commonly accepted structures and regularities. High school essays form a cohesive text genre in terms of their structure. So do cooking recipes, job ads, you name it. Sure, kids are graded on how well they can conform to the expectations, but that is true whether or not computers do the grading.

--
Marklar: marklar

As someone procrastinating grading right now... by Dr.+Spork · 2004-05-20 05:36 · Score: 4, Informative

I don't think that automatized, high-school-level grading is an all-bad thing. We can call it unfair if we like, but as someone who grades a lot of stuff I can tell you that I'm nothing like fair. I don't always even know how to distinguish a B- from a C+, and I just go with my gut, which, as far as I can tell, is much like flipping an internal coin. If we looked at human grade assigners as an algorithm, we would find a whole lot of stuff wrong, even among those of us who try hard to be fair (cover author names, compare close grades for consistency, keep a constant mood, that sort of stuff).

But I think that if a computer grading program which is no worse than humans could be devised, it would be a great learning tool. A lot of people make it to college as borderline illiterates. I'm not kidding. I read a lot of their crap. That's because their HS teachers were too overworked to grade their writing, so they didn't assign much. If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

Sure, the first grading applications are going to make a few serious errors. This is the first stage of every application when a computer is asked to interpret rich data. Early voice recognition sucked. Now it sucks much less, and it will just keep getting better. Same with OCR, chess software, machine translation, etc. So the right debate to have is about when this will be good enough for school use, and not whether. I'm prepared to admit that the answer to the right question is "not yet" (I'm sure how deep the current problems go), but I fully support working on this system until it works right.

Re:As someone procrastinating grading right now... by winwar · 2004-05-20 06:04 · Score: 2, Insightful

"That's because their HS teachers were too overworked to grade their writing, so they didn't assign much."

With all due respect, that is a failing of the teacher, or in other words, the difference between an excellent teacher and one that is average. My best high school teachers had no problem assigning an essay a week. Yes, they had a lot of grading, but they realized it was important and the best way for the students to learn. Any teacher unwilling to do the same is not worthy of being considered a good or excellent teacher.

Granted, a computer grading program could make it easier. In the hands of a good teacher, I don't have a problem. It certainly won't be any more biased than a person. But if the teacher wasn't willing to put forth the effort in the first place, how is this program going to make it better? Are they going to check all of the papers to make sure the program didn't screw up, use the extra time to emphasize advanced writing skills, etc.? Or are they going to treat it like an assembly line? I know which one I would place money on...
Re:As someone procrastinating grading right now... by dfay · 2004-05-20 08:26 · Score: 1

Fairness is highly overrated. Sure, if the test has a huge impact on the student's future, (SAT, AP, etc.) fairness is important. In fact, it's always important. But as long as you are in the ballpark, I think that it is of secondary concern. The primary concern, (and it's also why I think grading should be looked at as less of a chore and more of an opportunity,) is the feedback you give the student. I learned more about writing from my High School english teacher than anyone else. I also learned more from the papers she gave back to me than I learned from anything she taught in the class setting. The feedback given is an opportunity to take some time one-on-one with the student (well, actually, with the student's work) and help them out on an individual basis.

Granted, kids (and parents) would say that fairness is the most important factor. However, if you took them aside ten years later and asked them, I think they might place your constructive feedback higher on their list. I think that all teachers get an inflated opinion of fairness because of course that is what their students complain about. Well, even if you could somehow be perfectly fair, they would still complain; it's in their nature.

Anyway, just MHO and IANAT, so I don't speak from experience, other than as a student.
Re:As someone procrastinating grading right now... by Idarubicin · 2004-05-20 08:36 · Score: 3, Insightful

If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.
Unfortunately, the system described here doesn't return any such useful feedback. The Indiana system returns a grade from a six point scale. No comments, no criticism, no hint that the evaluation is meaningful.
Incidentally, what's this about "teaching instead of tedium"? Grading essays by evaluating construction, insight, and creativity should be part of the teaching process. Perhaps this is something that should be addressed earlier in the education of these students - if they're reaching college as "borderline illiterates" there is a problem - but grading in general is a part of teaching. If I were a student, I'd want to know that a human being - at some point - had bothered to look at the work that I did.

--
~Idarubicin
Re:As someone procrastinating grading right now... by DrVomact · 2004-05-20 08:45 · Score: 1

In one of my previous mini-careers (or "careens", if you prefer) was teaching philosophy to undergraduates (never got a tenure track position, that's why I'm here with you geeks). Yes, 95% of the essays sucked, and if I could have had a computer grade them for me, I would have been tempted to chuck them into the hopper. No doubt, the computer would have picked up on incomplete or run-on sentences, lack of paragraph breaks, and would have satisfyingly whacked the people who randomly sprinkle commas throughout their writing as though punctuation were a condiment, like pepper.
Probably, the computer would have given the same number of Bs and B+s as I did. But what about the precious few, the shining jewels I found buried in the muck? What about the guy who offered a unique insight on Descartes' dream argument, or the girl who timidly wrote a criticism of a passage in Bertrand Russell's Problems of Philosophy, but then said she must be wrong, "because I'm not as smart as Bertrand Russell". Would the computer have given those people an A? Would it have made encouraging comments in the margins? Heck, some of these insightful people even made spelling errors, or wrote more than 5 paragraphs.

While I agree with the previous poster that way too many kids graduate from high school lacking even the rudiments of writing ability, I don't think the answer is to have computers grade their essays. Why? Well, for one thing there is more than one way to write a good essay. The best writing doesn't fit into a mold; often, it breaks the rules and succeeds brilliantly. I'd like to see a software algorithm that can recognize such writing. My prediction is that computer grading will become yet another way the educational establishment enforces mediocrity and groupthink. I suppose it will be an intelligence test, of sorts: the smart kids will figure out how to play the system, just like smart kids have always done OK. Creative individuals who don't have that kind of analytical manipulative skill will just be out of luck.

--
Great men are almost always bad men--Lord Acton's Corollary
Re:As someone procrastinating grading right now... by Dr.+Spork · 2004-05-21 11:05 · Score: 1

Great comment. I totally agree. I teach philosophy as well, and I just had a great semester with some impressive papers. No, I couldn't give my precious papers over to a machine application, because I know we don't have anything that could grade them. You're right, merely formal criteria won't do it.
But I'm really intrigued with the idea of helping to write a semantic evaluator for philosophy papers, one that could see common conceptual mistakes, misinterpretations, etc., and then comment on them. I almost do this myself. I stated printing feedback comments because very similar mistakes are repeated over and over, so I explain the problem once in detail, and paste it in wherever it applies. Now, if a computer could spot that sort of problem and paste in similar comments, we'd be getting somewhere. Yes, this will require huge advances in natural language processing, but they are just a matter of time.
Yeah, if the grade is just a number and no explanation, that's as good as useless pedagogically.
Re:As someone procrastinating grading right now... by DrVomact · 2004-05-21 19:06 · Score: 1
Well yes, 90% of all the comments will be on the order of:
1. This doesn't make sense
2. Can't tell what you mean here
3. So, why do you think so? Give some reasons
4. You really need an argument for this!
5. Give some examples of this so I can tell what you mean
6. No, [insert philosopher] is stating is opponent's view here, not his own
I'm not at all sure that it is possible to write a "semantic evaluator" that will even make comments of this sort, let alone the far more complicated responses you have to make when someone actually gets past the preliminary misunderstandings. Heck, the next step would be to write something you can have an interesting philosophical discussion with, and that would count as passing the Turing test, no?

I won't say it's not possible to write such a program--I don't know enough to say that. In fact, I can't even imagine it. (Unless I'm in science fiction mode, of course. Then I can imagine just about anything.)
--
Great men are almost always bad men--Lord Acton's Corollary
Re:As someone procrastinating grading right now... by Anonymous Coward · 2004-05-22 06:47 · Score: 0

winwar wrote: My best high school teachers had no problem assigning an essay a week.

Lucky you. When I went to high school, the English teachers had 30 students per class and taught six classes per day. If each assigned a paper every week, that would be 180 papers to grade per week. If the teacher spent 15 minutes reading each paper, it would take 45 hours per week to grade them all. That's on top of the time actually spent teaching class (22.5 hours) bringing the teacher to 67.5 hours of work per week. That doesn't include any time used for preparing for class, faculty meetings, etc., or grading any other assignments (such as grammar exercises) during the week.

One of my friends is a retired English teacher who assigned a lot of papers, though not every week. She had her husband and collage-age daughter who lived at home help grade them and sometimes would even press dinner guests into service.

The Computer Will Give Jibberish an A by Anonymous Coward · 2004-05-20 05:37 · Score: 0

My sister is a teacher in New Jersey. She and her fellow teachers taught the children how to write essays in nothing but jibberish that give A's every time. As an English MA student, this personally offends me. Without a doubt, it makes electronic voting and spam look benign. I don't know how any schoolbooard in their right mind could approve a machine like this? What is the matter with people? Somebody must have driven a truckload of money up to the school and dumped it on the lawn. Maybe in 100 to 200 years we'll have a machine that can grade papers, but right now we sure don't have one.

Google Cache Link by gato_mato · 2004-05-20 05:37 · Score: 1

Follow this link and you can skip the registration of the NYT website - Got to love Google!

http://www.google.com/search?hl=en&edition=us&ie =U TF-8&q=http://www.nytimes.com/2004/05/19/education /19indiana.html&lr=&sa=N&tab=nw

Gato

Ridiculous - Can't beat the human factor by Anonymous Coward · 2004-05-20 05:37 · Score: 0

How is a computer program supposed to "detect" writing that is particularly inspired, or insightful? What about cleverness, humor, etc?

Or do they assume all teachers are idiots incapable to discerning the above types of writing nuances in the first place, and thus this program works at the same level as their less than stellar intellects??

All it does is compare to human-graded essays by Anonymous Coward · 2004-05-20 05:38 · Score: 0

Q. What's the technology used in e-rater?
A. e-rater is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using NLP for the past 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information. e-rater uses NLP to identify the features of the faculty-scored essays in its sample collection and store them-with their associated weights-in a database. When e-rater evaluates a new essay, it compares its features to those in the database in order to assign a score. Because e-rater is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which e-rater 's database is created.

Q. What is an e-rater advisory?
A. e-rater will generate an advisory if it has difficulty scoring or identifying some or, all, of the writing sample. Scores are reported with low confidence and summarized in a flagged message to the writer. Currently, e-rater will flag brevity, repetition, anomalous responses, or responses determined to be off-topic.

http://www.ets.org/erater/faq.html

I took this test by tundog · 2004-05-20 05:38 · Score: 5, Informative

I live in Indiana (no, NOT India) and took this test. Being a techie, I figured I'd try to fake out the system. This test works out to be 10% of the final grade and since I had a 98 going into the test, I figured I could afford to gamble a little, figuring if it back-fired I could blame it on a computer error since every one would figure the kid with a 98 MUST be telling the truth.

I almost wimped out. I wrote about 80 percent of the essay (about influence of pop-culture on society - and silly me I always thought society influences pop-culture but anyway). I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion. I reoreded the paragraphs, copied the one I felt was the best written and pasted it into the body 3 times.

Guess what I got.....6/6 (six point grading scale which is pretty messed up because a 5/6 is an 83%). Hopefully they won't audit mine....

--
All your base are belong to us!

Re:I took this test by Mikkeles · 2004-05-20 06:46 · Score: 1

'I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion.'

I've seen several references to 'five paragraphs' in the various comments. Would someone explain what this means and, more importantly, what five paragraphs has to do with writing an essay?

--
Great minds think alike; fools seldom differ.
Re:I took this test by nexthec · 2004-05-20 06:51 · Score: 1

Google is your friend.
Re:I took this test by miyako · 2004-05-20 07:20 · Score: 2, Insightful

The 5 paragraph essay is the cookie-cutter bottom line least common denominator for education these days.
I would estimate 90% of the essays given in the US educational system today are 5-paragraph essays.>br? Essentially, the 5 paragraph essay is a mold consisting of an introduction, 3 body paragraphs each focusing on one supporting fact, and a conclusion, and the teachers give students the ingredients in the form of a topic (definitely something that won't require too much thinking),and a style, The students generally mix these ingredients in their heads for about 30 seconds, vomit them into the 5 paragraph mold, and are graded on how aggressively the spell and grammar checked the paper.

--
Famous Last Words: "hmm...wikipedia says it's edible"
Re:I took this test by Mikkeles · 2004-05-20 07:24 · Score: 1

Thanks (I think ;^)

What retard came up with this piece of crap? Montaigne must be turning over in his grave!
(I've been 35 years out of high school; I'm glad to see that low standards have gotten even lower.)

--
Great minds think alike; fools seldom differ.
Re:I took this test by nexthec · 2004-05-20 08:04 · Score: 1

Sorry, didnt mean to be snarky, I just was to lazy to copy and paste ;-> And yeah, it's probably the most singly usless writing format ever, except for determining the students ability to A) follow instructions, B)judge if the student completed atleast some of the reading/research assignment C)keeps the teacher from having to actually read the contents ;->
Re:I took this test by cmpalmer · 2004-05-20 08:05 · Score: 1

Yep, I've got a nice printout provided by my fourth grade son's English teacher describing this exact outline.

I will admit that this does give them some form to follow and helps focus their thoughts, as long as they are taught that this isn't the only way to do it. The proper analogy to me is that there are many kinds of poetry forms, but if you are asked to write an Elizabethan sonnet, you have to follow the rules. Therefore, I don't mind the teacher saying you must write a five paragraph expository essay following this form, but I hope she's not saying, "This is the only way to write an essay...".

Oh yeah, the final essay has to be handwritten, but I always tell my son to compose it on the computer (sometimes I type if he is in a hurry), let me (and the computer) proof it for spelling and grammar, then print it out and write it out by hand.

He tends to follow the rules too closely and I have to prompt him to change it around a bit. Most of the time his first draft looks like this:

This is my introductory sentence to get your attention. This is my topic sentence. To support my topic, I will be talking about a, b, and c. They all are important to my topic.
A is very important. Reason 1. Reason 2. Reason 3.
B is also very important. Reason 1. Reason 2. Reason 3.
Finally, C is very important. Reason 1. Reason 2. Reason 3.
This is a restatement of my topic sentence. Remember that a, b, and c are all important to my topic.

--
-- stream of did I lock the front door consciousness
Re:I took this test by meshko · 2004-05-20 08:35 · Score: 1

So was your whole essay just one paragraph repeated 5 times or did you have 5 different paragraphs, one repeated three times?

--
I passed the Turing test.
Re:I took this test by Mikkeles · 2004-05-20 08:45 · Score: 1

Oh! I didn't take offence to your response. The 'I think' remark is in reference to the bad taste in my mouth that I got after reading the page you indicated and the other response to my question. My thanks was honest.

--
Great minds think alike; fools seldom differ.

nothing to separate one from being just average... by Anonymous Coward · 2004-05-20 05:38 · Score: 0

what are the chances that works by famous authors / poets would pass this test? i.e. Shakespear or Robert Burns, etc.

What makes them unique is thier writing style - all this algorithm is doing is seeing how close you are to the norm (which for the most part is dull boring literature). Teachers generally are use to looking at the norm and not the truly great people who have talent... this is just a cheap way of getting your mark in class. If your satisfied with a 65% then this is probably going to sound great to you.

If you want to be noticed for outstanding work this is not going to give you that... prove me wrong otherwise. A computer has no ability to be swayed by a good argument - only that the argument was 'technically' correct.

Optimizing by cshark · 2004-05-20 05:38 · Score: 1

I think it's only a matter of time before kids figure out how to optimize their papers to fool the machines. Kind of like the way people optimize web pages to do essentially the same thing. They're both looking to get good grades.

--

This signature has Super Cow Powers

The obvious answer is... by null+etc. · 2004-05-20 05:39 · Score: 1

The obvious answer is to let the students contest the results of computerized testing, if so desired. An English teacher would then grade the results. This would have the following consequences:

Students who don't really "try" to get a good grade wouldn't bother to contest the scores, especially as the reputation of the computerized grading system rises.
Students who unjustly receive a grade lower than expected, due to computerized testing, have the opportunity to rectify the situation.
The schools, and developers of the system, gain the opportunity to continually assess the consistency between computer-based grading and human-based grading. The special "contested grades" would be generated by the students who most likely would "break" the computer grading algorithm.

Besides, we all know how poor American education is comparitively. This solution at leasts lets us take advantage of the 80/20 rule to streamline those who care to aspire to higher levels of education.

at least the system will be applied uniformly by Anonymous Coward · 2004-05-20 05:39 · Score: 1, Interesting

As a high school student, in spite of being an an honor student, getting accepted at a top Liberal Arts college, scoring well on SAT's, and taking AP English, I recieved a 53% percentile on my human-scored state writing exam. After a little investigating, I found that these tests were graded by the hundreds, if not thousands, by teachers trying to supplement their meager incomes. I'm sure that it's pretty hard to read carefully and make sure that all exams are graded fairly and equally when there is no accountability, and no way for students to challenge the results. At least this way, all students in the state will be graded by the same algorithm.

So what about output? by petabyte · 2004-05-20 05:39 · Score: 1

The first thing that came to my mine and probably many other /.ers is why not have a computer program that uses the same grading system to output the essay in the first place. I mean, why keep the kids when its much more efficent to just have computers do that work.

I think this just sends the message "you aren't important enough to have someone evaluate your work" and thats a very dangerous message to send. Sure I had alot of teachers I didn't like in school, and there were plenty of "group work" sessions I thought were totally pointless but nothing says that more than "we're going to have the computer grade it." It pushes school even more from being about education to being about grades and getting into the expensive college. I think thats both scary and sad.

Oh and for the record I'm not a teacher. :)

Re:So what about output? by petabyte · 2004-05-20 05:42 · Score: 1

The first thing that came to my mine

As you can see my real reason for posting was my concern that the computer would fail me.

Software used by Unnngh! · 2004-05-20 05:40 · Score: 1

I found the script they were using to grade these essays on the six-point scale, in case anyone's curious...

$score = int(rand(6));

print $score;

Don't let it get out that the code's been leaked...

Shouldn't be the only scoring by Ra5pu7in · 2004-05-20 05:41 · Score: 1

According to the web-site information on how e-rater works: "It cannot read or respond to an essay as a teacher would." and "Some of these writers have submitted essays that have tricked e-rater into giving a score even though the essay does not make any sense. The individual words in these "challenge essays" are grammatically correct, but they are strung together in such a way that they create nonsense sentences."

The real point behind the e-rater seems to be giving a general idea of where the author may need improvement. It does not appear to be a replacement for a teacher reading the essay. In fact, it would seem to be more useful for allowing a student to have their rough draft reviewed and determine whether it needs major rewriting or just fine-tuning.

(Did anyone else imagine "Trained Readers" being given rewards when they scored a test the same as the e-rater?)

--
I was taking one day at a time, but then several days got together and ambushed me. (from a Rhymes with Orange comic)

Re:Shouldn't be the only scoring by IWannaBeAnAC · 2004-05-20 05:51 · Score: 1

Hmm, I would have thought it would be the other way around: get a human to review an early draft and see if the basic ideas and structure are OK, use the e-rater as feedback to fix the grammar and whatever.

Re:Indiana? I thought it said India by Anonymous Coward · 2004-05-20 05:41 · Score: 0

Racism? What racism?

But does it... by Anonymous Coward · 2004-05-20 05:41 · Score: 0

Does this program offer contructive criticism?

Does this program identify plagarism?

Does this program identify hints that the student might have problems not related to educational?
This and more teachers (should?) provide.

how does it grade.. by bigattichouse · 2004-05-20 05:41 · Score: 1

the classics? I imagine James Joyce and other exeprimental writers would fail miserably.

--
meh

Why bother teaching the humans, then? by word+munger · 2004-05-20 05:41 · Score: 1

Once we teach the computers to write, what will we need humans for? I've blogged about this here.

--
ScienceSeeker.org

ETS can't construct a valid sentence by Animats · 2004-05-20 05:42 · Score: 1

From the ETS site:

"If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site."

Automatic essay grading should not be used until the grading program has enough intelligence to construct counter-arguments to the essay.

One problem by mysterious_mark · 2004-05-20 05:43 · Score: 1

As a former college instructor, I do see a serious problem with this. Even though reading and grading reports, essays etc is very time consuming it does allow the instructor to see how the students thinks and writes, and it allows the instructor to provide feedback to the students. By using computer grading, the problem of assigning a grade is solved, but it seems to me it largley undermines the whole point of the writing exercise in that it eliminates a very important aspect of the student/instructor relationship, and degrades the learing process. After all the goal of a writing exercise is to allow the student to learn, not just to assign a numerical value to the student's work. Currently I write educational software, however it seems the purpose of automation should be to free up time and resources such that the student and instructor have more interation, rather than this type of solution which replaces the student instructor interaction. Mark

Paper Hacking???? by Anonymous Coward · 2004-05-20 05:43 · Score: 0

I'm just waiting for the day students can write essays that exploit buffer overflows...... :-p

Computers can't grade "interesting"! by Luscious868 · 2004-05-20 05:44 · Score: 2, Insightful

A computer can not replicate certain aspects of the grading process. Sure they can grade spelling and grammar and probably certain aspects of your writing style but there is plenty of important aspects of writing that they can not grade you on.

For instance, does your essay really grab the reader? Anyone here who reads technical documents knows what I'm talking about. There are some writers that, no matter how dull the subject, can make their work interesting and fun to read. A computer can not possibly grade one on that. I have a good friend who's a high school English teacher and occasionally I'll read some of the things written by his students. I've come across plenty of papers that are grammatically correct, have perfect spelling and are fairly well written from a syntactic and stylistic point of view, but are just plain boring to read. Then I'll move on to another paper, about the same subject, which is interesting and actually fun to read.

That's just one example of something a computer can not possible take into account when grading an essay. The bottom line is that a computer will never be able to grade you on certain subjective things, which although they are subjective and therefore open to a certain amount of interpretation depending on the person doing the grading, are nevertheless still very important aspects of good writing.

With spelling and grammar check, almost any average student can churn out a paper that is going to be mostly correct; however it still takes a good writer to produce something interesting. In my opinion, an interesting paper with a few minor spelling, grammar or syntactic errors is just as good as a boring paper with no spelling, grammar or syntactic errors.

Re:Computers can't grade "interesting"! by Anonymous Coward · 2004-05-20 05:52 · Score: 1, Interesting

With spelling and grammar check, almost any average student can churn out a paper that is going to be mostly correct; however it still takes a good writer to produce something interesting. In my opinion, an interesting paper with a few minor spelling, grammar or syntactic errors is just as good as a boring paper with no spelling, grammar or syntactic errors.

Exactly! That's why a writer or journalist has an editor. A good, interesting writer who makes a few minor spelling, grammer or syntactic errors on occasion is going to have no problem finding work while someone who never makes that type of mistake but who's writings are just plain boring never will. That's why the pro's have editors, to catch the small things and offers suggestions on how to improve what's there. A good editor can take a good work with some errors, eliminate those errors and offer suggestions on how to make the piece great. A good editor can not, however, take a boring piece and make it great. The work, ultimately, has to come from the writer. An editor can help improve a work, but he can't transform it, because that would make him a co-author of the material. Ultimately a good writer has to produce interesting and engaging material. Grammar and spelling and syntactic errors can be fixed. If you are incapable of producing something interesting and engaging, there isn't an editor on the planet that can help you.

gigo by FiniteLoop · 2004-05-20 05:44 · Score: 1

All i can tell is Garbage In Garbage Out

What happens when... by Doverite · 2004-05-20 05:44 · Score: 1

Your Windows grammar checking software and spellchecker is a different version from the schools Windows grammar checking software. Cause everybody knows how compatible different versions of windows progs are, let alone throwing in things like OO.o!

--
You can legislate morally you can't legislate morality

An objective norm by Andrew+Sterian · 2004-05-20 05:45 · Score: 1

This is actually a good idea. It is not a substitute for having a human read your essay for actual content rather than style. In fact, the company's FAQ states that you can easily fool the software by writing nonsense with perfect style.

The good part about it is that as a professor I can point to a bad essay and convince the student that it's not only bad according to my opinion but that it's bad according to EVERYBODY'S OPINION! Students so often believe that (a) professors are nuts and are just out to get them, or (b) they grade too harshly. Having an objective tool to back up a grade is a good thing.

What are we grading here?! by njfuzzy · 2004-05-20 05:48 · Score: 1

Okay. I can see a program being written that can scan a text document, and decide if it is good English. It could check grammar, spelling, broadness of vocabulary, style, etc. and rate the text on how good the language of an essay it was. That at least seems doable within the realm of algorithm.

How exactly would this program be able to tell whether the essay was actually a good essay, though? After all, the function of an essay is to use facts, logic, and rhetoric to prove a thesis. What program could truly decide if an essay had proven its thesis, whether the facts it uses were true, sources good, logic sound, etc.?

The degree to which this removes critical thinking and subjective quality from the entire process scares me. I think this is a Bad Thing.(TM)

--
My Photography - http://ian-x.com
The Deathlings (comic) - http://thedeathlings.com

Thing is by Sycraft-fu · 2004-05-20 05:49 · Score: 1

Human graders aren't fair either. I have fairly good English skills, but nothing great, most notably my spelling sucks. However I tested in to honours English in university, actually receiving a perfect score on the entrance essay by two different graders, and having a high SAT verbal score. Then there was a friend of mine who is a much better writer than I am. He took AP English in high school (I wasn't good enough to get in that) though opted not to take the AP test. He tested in to REMEDIAL English, getting a very poor score on the essay, though having a good SAT verbal score. Now you talk to anyone that knows our writing, they'll tell you he is the better writer. Not according to the professors in charge of grading entrance essays apparently. This was the same essay, the same year, taken on the same day, so it's not like there was a big difference.

Unfortunately, since English isn't a subject that is objectively gradable like Math, grader bias WILL be there. Actually a computer has a better chance of being fair, since it will apply the same criterion every time. Doesn't mean it might not have bad criterion, you need to work to develop one that doesn't, but unlike a human its criterion won't change with time.

Problem by LoneWlf · 2004-05-20 05:49 · Score: 1

There is a huge problem here. I spend X amt of time slaving over an essay, then they chunk it thru a computer?!?!? Where's the love? I mean, writing has been/is/will be done for the joy of writing as much as for the joy of reading. I do not approve of this system as it moves us more in the direction of statistical automatons, away from our humanity...

--
-LoneWolf-

It is by will alone I set my mind in motion.

Grade summary: by slow+train · 2004-05-20 05:49 · Score: 1

--Sentence number 0004 ends in a preposition. This is a situation up with which we should not put.

How well did they test it? by ahfoo · 2004-05-20 05:52 · Score: 1

I happend to know a bit about this as I work with ETS products regularly.

If you're interested to know what their own research shows, you may find this paper an eyeopener.

The paper is a PDF called "Stumping e-Rater" commissioned by ETS, the developers of the product.

Re:How well did they test it? by ahfoo · 2004-05-20 06:03 · Score: 1

As a follow up for people too busy to read the PDF or who don't like PDFs, I'll give away the plot. Basically, it's trivial to create false positives.
It is important to notice the difference between a false positive and a false negative. With a false negative, there's a powerful motivation to address the issue, but with a false positive if nobody makes any noise evberybody is happy. So, you have to be really careful about how well you monitor for false positives. Just because nobody complains, doesn't mean it is working.
My /. user journal has some more info on my personal interaction with this product.

Hmm... by a+whoabot · 2004-05-20 05:53 · Score: 1

So, this is just an algorithm that checks the essay against a theoretical model of a "pefect essay". The concept is ironic, perhaps. For grades, students of literature, of the language, are pushed to conform to this algorithm, not to create something ingenius, striking, or artistic.

This just confirms that the American public school system is nearly entirely about social education--grades your level of conformity to the social algorithm in general--rather than academic education, which would certainly purely try to grade your ability to create in relation to knowledge. And, to say that this move to computer grading is a move towards the hyper-conformity of students, I suppose, would be overdetermining it. The setup has long since been about conformity. Bush can say "no student will be left behind" because those who do not have the capacities who would have otherwise been "left behind", don't need to have the capacities anymore, as long as they socialize acceptably. And the number of iconoclasts are far fewer than the number of those who are below average intelligence. That's the reason they are able to virtualize marking, because it makes no difference anymore; the teachers are already "computerized", the criteria, and content, all reproductions. The reason to bring in computers is purely economical.

Computerized Story Posting by alatesystems · 2004-05-20 05:56 · Score: 2, Funny

Slashdot First With Computerized Story Posting

Now with Computerized Story Posting, the artificial intelligence "seeks out" stories that have either been long archived or just posted the previous day and then posts them as new material. The program then ignores what is stated in the FAQ and disregards all emails stating that the story is a duplicate. This program is also known as "chrisd".

Other features include "mis-classification into the wrong topic", "making up stupid-titles-that-go-into-the-dept", and the most difficult, ignoring stories that should be posted.

Chris Benard

A third source: TurnItIn.com-style relationships by Flexagon · 2004-05-20 05:56 · Score: 2, Interesting

If it doesn't already, I would expect a service like this will eventually include plagiarism detection, due to marketing pressure if nothing else. This is something that human graders do, at least over the space of papers they grade and works they remember.

But if plagiarism detection is added, then the grading service would have to make and retain some encoding of each graded paper, a derivative work, in its database.

Once that happens, the grading service also becomes subject to all of the issues already raised with services like TurnItIn.com, already discussed here.

I also found this comment from ETS's site rather strange, to say the least:

It is important to remember that e-rater is an embedded real-time application; it is not software.

How To Write An Essay by dpbsmith · 2004-05-20 05:57 · Score: 5, Insightful

A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.

It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.

The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.

The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.

In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.

--

"How to Do Nothing," kids activities, back in print!

Re:How To Write An Essay by BillyBlaze · 2004-05-20 08:05 · Score: 1

I'd love to see somebody with access to the trial submit that and see what it gets. But if I were a computer, you'd lose points for lack of an attention-getter and transitions. Add "Did you ever wonder of what a good essay always consists?" to the first paragraph, and "First," "In addition," and "Finally," to the second, third, and fourth paragraphs respectively.

IN SOVIET RUSSIA... by Anonymous Coward · 2004-05-20 05:57 · Score: 0, Troll

computer grades YOU!

Hmm.

This is moronic by woodhouse · 2004-05-20 05:59 · Score: 1

The state of AI currently is such that even simple understanding of english isn't possible. Just the term "Artificial Intelligence" is highly misleading. This program will likely do nothing more complex than a search for certain key-words and phrases. It won't have any understanding of the answers, and it will not be grading work fairly.

The people who okayed this system either have no understanding of the limitations of AI, or they don't care and want to save money. Either way, law suits will be coming.

tricking it by enbody · 2004-05-20 06:03 · Score: 2, Interesting

Yes, you can trick it. From the e-rater article:

"Experienced writers, teachers, and writing assessment specialists have tested e-rater to determine the extent to which it "understands" the content of essay responses. Some of these writers have submitted essays that have tricked e-rater into giving a score even though the essay does not make any sense. The individual words in these "challenge essays" are grammatically correct, but they are strung together in such a way that they create nonsense sentences."

That observation shouldn't be surprising because earlier it says: "An e-rater score will be most beneficial to students who make a good faith effort at using it to improve their writing skills."

The program works (grossly oversimplified) by mimicking the grading of humans on essay samples.

Grading computer programs is no big deal... by Otto · 2004-05-20 06:04 · Score: 1

My professors used custom scripts to help them grade programming assignments. In that case, you can do things like unit testing.. feed in inputs, expect certain outputs. If your program doesn't produce the desired output, instant fail. Simple enough. It doesn't have to do anything requiring non-boolean judgement routines, simple comparisons will suffice in many cases.

Grading essays is a whole other level of sophisication. While it can easily check for simple things such as spelling errors and even errors in grammer or word order, no computer program can grade based on the content or subject matter. One of the things English classes teach is how to think coherently, how to write a paper to convice someone of an argument and such. How to present an idea. It's not all about the grammer.

I see nothing wrong with using computer grading as an aid to the teacher. But to give it the whole grading process is silly.

--
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.

Can we run literature through it? by michaelmalak · 2004-05-20 06:04 · Score: 1

Obviously anything slightly creative is going to get nicked. The best way to demonstrate it would be to run respected literature through it. I'm sure prose-as-poetry would really get a failing grade, such as Christopher Alexander's The Timeless Way of Building or the book of Proverbs from the Bible.

Re:Can we run literature through it? by king-manic · 2004-05-20 06:09 · Score: 1

Proverbs from the Bible

Considering it was written in hebrew I think it should fail a computer scoring program for english.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."

Stupid by cubicledrone · 2004-05-20 06:05 · Score: 1

Computers cannot evaluate the quality of writing. Given the "grammar checker" in word processors only seems to be set to gripe about "passive voice" (and is usually wrong), I don't think there is much else useful about having a computer read essays until, of course, a computer can write a five-page paper on the difference between prose and journalistic style.

Oh, and we can't forget that in the future, teaching will be viewed with as much contempt as every other educated profession except law and business. "Why should you be paid more than minimum wage? All you do is use a computer all day." or "why should we learn to read? The computer can read for us."

--
Business isn't willing to pay for products, innovation and careers, so we get brands, mortgage commercials and layoffs.

Like we always say.... by ogar572 · 2004-05-20 06:09 · Score: 0

there is more than corn in Indiana. Not much though.

Can you submit a paper multiple times? by cacheMan · 2004-05-20 06:10 · Score: 1

The only thing blocking this before was that the teacher couldn't grade fast enough for 25 kids to submit 4 drafts of a paper. Is the grading system available to the kids during the writing process? I think that would be great. What excuse would a kid have for not getting an 6, er, 'A'?

If this is so good, why don't we use it everywhere? Use it in business before sending out a memo, or even an email for that matter. Use it in newspapers. I've seen tons of grammer errors in AP news reports.

I just hope it's better than Word by wcrowe · 2004-05-20 06:11 · Score: 1

I just hope it's better than Microsoft Word's grammar checker. It's worse than no grammar checker at all.

--
Proverbs 21:19

google: how to write "A" grade papers by holy_smoke · 2004-05-20 06:12 · Score: 1

All this will teach students is how to use tools from the internet to write their "A" papers for them.

*rolls eyes*

--
Is the juice worth the sqeeze?

Depends on the purpose of gtrading... by Dr.+Zowie · 2004-05-20 06:13 · Score: 4, Insightful

If grading is intended as a motivator to encourage each student to perform his/her best, then more effort should yield a higher grade. Likewise, if grading is intended to reflect the student's ability to perform in a real-world situation, effort should probably yield a higher grade: folks who work hard tend to do better than folks who are marginally smarter but don't work hard, in real-life situations. But if grading is intended to reflect only the quality of the work that was submitted, then sure -- effort shouldn't count at all.

This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.

As a college instructor, I tend to use a strict grading protocol -- and then "bump up" a few of the students. If someone comes in to my office every week and really struggles to understand the concepts, but the computer tells me that they earned a "C+" -- they're likely to find a "B-" on their transcript. But if someone who's smart enough to get an "A" blows an exam from being hung over, that person gets little or no sympathy.

Re:Depends on the purpose of gtrading... by king-manic · 2004-05-20 06:18 · Score: 1

This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.

Thats true. But being part of one group I resent the other. Working hard is important but contrary to what we're taught, working hard is never enough by itself. Working hard will get you into the middle class, no matter what your startign point is and what abilities you have. But going aboev middle class requires talent in something. Working hard alone never gets you any farther.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Depends on the purpose of gtrading... by hesiod · 2004-05-20 06:55 · Score: 4, Insightful

> If grading is intended as a motivator

A grade is used to show how well you know a subject. If I knew a subject completely before starting the class & wrote the same level of paper as someone who studied his ass off, we deserve the same grade. He should not be given a better grade than me just because he waited for this particular class to learn a subject. Why am I judged differently because I took the initiative to learn the information earlier than I absolutely had to?

No, schools are not there to make you feel good about yourself (that's obvious), they are there to make sure you know the minimum information to pass a class. That'a a D. Then, if you know more, you get a better grade. The amount of work you put into it is irrelevant. In fact, if you put more work into it than you should have, it means you are not doing well, and once you get into the "real world," where you have strict deadlines, you don't have the option of getting paid more just because you worked harder for the same result. The exact OPPOSITE is true, in fact.

School should help people prepare for life. If someone is given a grade they did not deserve, they are being improperly trained how to work.

A problem, however, is with the PARENTS. Many students are C students, that's all there is to it. But they get all high & mighty towards the school if they see their child work very hard for a low grade. They figure their child isn't good enough if they don't have all As, but that they deserve them just for hard work.

If life had a payrate based on how hard you worked, vs. your productivity, I would start working as an astrophysicist. I wouldn't get anything done, since I know nothing about the work, but I would sure as hell work my ass off. Do you think anyone wants an employee like that?
(I mean the lack of knowledge -- everyone wants a hard worker, if they know enough).
Re:Depends on the purpose of gtrading... by Dr.+Zowie · 2004-05-20 07:45 · Score: 1

No, schools are not there to make you feel good about yourself (that's obvious), they are there to make sure you know the minimum information to pass a class. That's a D.

Certainly schools aren't there to make students feel good about themselves -- they are there to help students learn. By motivating people who otherwise might not be interested (through fear of a bad grade or desire for a good one), grades help many students learn by providing a goal to focus on. This aspect of grading is particularly obvious in elementary schools: nobody really cares about your grades in primary school (up to 11 years of age) once you've graduated from high school; the grades provide a check to your parents (and, later, to you) on how you're doing. They also help you to learn something about cause (working in class, or goofing off) and effect (via grades). It's arguable that this use benefits from a "sliding scale" for grades, in which every student is graded according to some estimate of innate ability and students who have to work hard -- but nonetheless achieve real learning -- earn much better grades than smart loafers.
Grades also have a different use: indicating to the world your fitness for a particular task. This use definitely does not benefit from a strongly "sliding scale", because that defeats the purpose of having a relatively objective measure of someone's ability. Nevertheless, grades that reflect diligence and commitment -- but only to the extent that they affect each person's ability to perform a task or understand a subject -- actually help this use of grades.
Re:Depends on the purpose of gtrading... by Colazar · 2004-05-20 08:13 · Score: 1

You are confusing two different kinds of grades. A final grade in a class should be used to show how well you know a subject. A grade on an individual paper can mean anything the instructor wants it to. The instructor's job is to get you to learn *as much as you can* in a class. If they think they can motivate you by grading you down on a paper for arbitrary reasons, then they should. If they think they can do that by grading you up on a paper for arbitrary reasons, then they should.
In the interim stages people need feedback on different things, and for different reasons. These grades are instructive and motivational. Only the final grade needs to be in any way comparative from student to student.
Or maybe I'm weird--I thought the purpose of classes was to learn stuff, not to reward behavior of one kind or another.

--
He decided to just watch the government, and kind of scale it down to size, and run his life that way. --Laurie Anderson
Re:Depends on the purpose of gtrading... by Anonymous Coward · 2004-05-20 08:52 · Score: 0

A grade is used to show how well you know a subject.
That whooshing sound was one of the major points of the GP flying past you: "What is the purpose of a grade?" is a question subject to some debate. Sorry, you don't get to short-circuit that debate with a bald assertion.
Re:Depends on the purpose of gtrading... by mandalayx · 2004-05-20 09:03 · Score: 1

whew, one of the few "Insightful" comments that were actually Insightful. kudos.
Re:Depends on the purpose of gtrading... by JimBobJoe · 2004-05-20 09:29 · Score: 1

Certainly schools aren't there to make students feel good about themselves -- they are there to help students learn.

I find this sentence really funny, because it's easy to take ad absurdum.

"Students who don't finish the test in 15 minutes will be savagely beaten."

By motivating people who otherwise might not be interested (through fear of a bad grade or desire for a good one), grades help many students learn by providing a goal to focus on. This aspect of grading is particularly obvious in elementary schools...

Actually the point of grading in elementary schools has never been all that clear. I believe it to be a ranking mechanism purely.

Perhaps the negative motivation thing works at the high school level, but the educator Maria Montessorri figured out very quickly that grading at the elementary school level had an overall depressing effect on kids performances.

There are many schools who do not use letter grades at the elementary level (particularly Montessorri.) My recollection is that letter grades for 1-6 is actually a fairly new phenomenon (last fifty years or so.)
Re:Depends on the purpose of gtrading... by Mornelithe · 2004-05-20 12:20 · Score: 1

Why would anyone care about a grade on a random paper if it had nothing to do with the final grade?

"Oh, I got an F on this paper that doesn't count for anything and was graded by picking a letter at random."

"Oh, I got an A on this paper that doesn't count for anything and was graded by picking a letter at random."

"I got all A's on my papers and ended up with a D in the class, but Bobby got D's on his papers and got a B in the class."

I don't understand how any of those situations would motivate the student to do well or learn the material.

--

I've come for the woman, and your head.
Re:Depends on the purpose of gtrading... by Niet3sche · 2004-05-20 21:31 · Score: 1

If grading is intended as a motivator to encourage each student to perform his/her best, then more effort should yield a higher grade. Likewise, if grading is intended to reflect the student's ability to perform in a real-world situation, effort should probably yield a higher grade: folks who work hard tend to do better than folks who are marginally smarter but don't work hard, in real-life situations. But if grading is intended to reflect only the quality of the work that was submitted, then sure -- effort shouldn't count at all.

We're in the West here. Process Philosophy is out; outcomes are what matters. You have someone who is brilliant, and someone who just doesn't get it. Now, while the dull person who works their butt off certainly gains the benefit of the doubt (e.g. if they're borderline, but I can tell they've put in quite an effort, then I'll give them the higher of the 2 grades) and also receives my appreciation of working and trying to better themself, it is not the job of educators to turn out individuals who "work hard". Rather, it is the job of (certainly primary school and undergrad) educators to turn out a product that is able to achieve, and able to perform. Now, while it doesn't matter in the case of an infinite amount of time being allotted just how two reports get finished, the real world does not work in this way. In short, the A students typically are that way because they either are time-management wizards (God bless 'em), or exceedingly bright. All else being equal, the bright person working in-domain can pull off a more robust and cogent report, exam, what-have-you than a dull person given the same time table.

You know, I started out disagreeing with you, but now understand where you're coming from. It depends upon the context of the situation. However, I will certainly stand behind my earlier assertion that effort, while noble and honorable, will only take you so far - in postgraduate work, raw intelligence and/or inquisitiveness certainly has a place. Look at folks being groomed for future faculty positions at Research I institutions - effort and hard work are wonderful, but in the absence of the more cerebral nature of their work (knowledge generation - "publishing", in other words), then they're pretty much useless to the serious Research I institution.

With all this being said, though, I'll also tack on that Academics is a weird arena in which to work. I was flatly told that, "your job is not to teach students. If you can do this, that's a nice bonus, but your job is to produce (publish)" - they're right, of course, but it is a bit odd to have this so bluntly stated.

Anyway ... the point of the reply is this: I think the focus shifts from hard work in primary and undergraduate education to that of raw talent (coupled, hopefully, with hard work!) in graduate programs. I say this with a good deal of confidence in reviewing just the nature of course "structure" and "deadlines". At the graduate level, these are flexible if you produce a product that is worth the wait.

PS: Anyone else think the No Child Left Behind mandate will end up killing us all?
Re:Depends on the purpose of gtrading... by hesiod · 2004-05-21 00:49 · Score: 1

> I thought the purpose of classes was to learn stuff, not to reward behavior of one kind or another.

Isn't motivational grading just rewarding a behavior? If I can get an A with no effort, but I get a B because I didn't put more effort than necessary into it, then you are punishing because of a behavior. If you mark a B paper to an A because of hard work, you are rewarding a behavior -- working hard.

It's not a great explanation, but I feel that equal rights extends to individuals just as much as it extends to minorities or women. Same pay for the same job, same grade for the same paper. It may be slightly unfair to compare less-than-perfect students with a minority, but I believe it gets the point across.
Re:Depends on the purpose of gtrading... by hesiod · 2004-05-21 00:52 · Score: 1

> you don't get to short-circuit that debate with a bald assertion.

I've never heard that debated, a grade has always meant "this is how well I know this subject." Sorry, you don't get to short-circuit reality with a claim that someone is debating it. People still debate that the world is 6000 years old. It doesn't mean that these people aren't completely wrong just because they still cling to their ignorance. Or, in the case of grades, the parents still cling to the idea that their child is perfect, so they try to redefine terms to make their child look smarter than they are.
Re:Depends on the purpose of gtrading... by Colazar · 2004-05-21 07:23 · Score: 1

I understand what you're saying, but I think you're making mistaking the form for the function.
All a grade is, is a form of communication. Your final class grade is communication with the outside world (the registrar, your parents, anyone who reads your transcript) to tell them what your mastery of the subject matter of the class is. For this grade I absolutely agree with you, grades need to be equal for equal results. Anything else would be completely unfair.
But a grade on a paper is communication between the instructor and the student, and in a lot of ways is just another form of comment. (Really, I always found the comments to be more valuable than the grade, anyway.) It doesn't have anything to do with any other paper that was turned. (Most people aren't going to be directly comparable anyway.) It should give a general indication of how I'm doing overall, but my final grade should not just be a blind average of all my papers. And for some people there is a big motivational difference between, say, a C+ and a B-, even if the "objective" difference between them is slight. So if you've got it, use it.
IF you want to put it in a real world context, thing of management. A manager's job is to get the best results out of their employees. The way of *managing* the employees is going to be different for each, depending on what motivates each person individually. But the actual *reward* (ie pay, final grade)that they get should depend entirely on what they actually produce. (Another similarity is that you probably only have the freedom to use this approach in classes/departments of a dozen or less. Once you get larger than that, you probably don't have enough knowledge about the individuals involved to use it effectively.)

--
He decided to just watch the government, and kind of scale it down to size, and run his life that way. --Laurie Anderson
Re:Depends on the purpose of gtrading... by hesiod · 2004-05-21 08:12 · Score: 1

> my final grade should not just be a blind average of all my papers

Unfortunately, if you don't base a final grade on the previous grades given, it becomes wholly subjective again, and is therefore not a valid metric. There's enough problem today as-is, when different teachers of the same class give very different grades for the same work. I doubt that is not peculiar to the school I went to. Students dreaded getting certain teachers because they graded harder/unfairly(/gave better grades to girls with short dresses -- and that was a woman). This is another reason that electronic grading might be a good idea -- standardized not only across students, but across teachers as well -- which was the argument to begin with.

How the heck did I get on-topic again?

Boobies! Hee hee... took care of that little problem!

Not that crazy by Ferante · 2004-05-20 06:14 · Score: 1

Having been a physics TA at a large university when I was a grad student, I have graded hundreds and hundreds of astronomy essays and physics tests. It doesn't take long even to recognize the major points and typical pitfalls and develop a very mechanized grading algorithm, even for an essay question. English essays at the undergrad level probably aren't much different. I would rather avoid having the grading process entirely dependant on AI, but I can see this as a valuable tool for a human grader. It would increase fairness and uniformity and allow the human grader to focus on things the computer can't understand.

You hit the nail right on the head by Anonymous Coward · 2004-05-20 06:14 · Score: 0

Computers, at this time, just can't take certain subjective things into account that are important in writing. Of course, a bad teacher won't do it either, but at least with a human being doing the grading there's a chance that he or she will focus, fairly, on some of the subjective things that are important in writing. You've pointed one of them. Any good high school English teacher takes into account how interesting and engaging a paper is into account where appropriate. It's a very important aspect of writing that seperates the good writers from the great ones.

If this was sports there would be an outrage! by logicnazi · 2004-05-20 06:15 · Score: 1

This suggests a very distorted notion of the purpose of grades. Perhaps this is an appropriate philosophy for very young students where the only purpose of the grade is encouragement but it is simply not usefull for college students.

What is the prupose of grades in college? Why not just allow students to take all classes P/F (and this works fine from a learning perspective as the first year of my college showed)? Quite simply because the transcript is supposed to represent the students *competance* in various areas. Prospective employers and graduate schools need to be able to look at a transcript and evaluate the abilities of a particular individual.

Do you want a company to hire an engineer to build a bridge who tried really hard but nevertheless simply doesn't understand the physics involved? I would prefer, as most people that we pick the individual who had the best track record of knowing his shit regardless of the effort level. This is true in all areas, grades serve as an indication of mastery of the subject matter. Effort does not equal mastery.

In fact the argument can be made that those who put in alot of effort deserve a lower grade for the same work. As a prospective employer which is more valuable to me, the student who was only able to acheive that level of understanding after a great deal of effort or the student who pulled this off barely twitching a muscle. If both of them are going to work 40hours/week for me (or whatever other number I require) then the lazy student is going to be far more valubale. I don't in fact support this policy but it is far more than enough to realize we shouldn't give a special benefit to effort. If we wanted to recognize effort we should make two grades (one for effort one for ability) but we have already made our choice about what a grade represents when we decided to make only one of them.

All of this is ignoring the extreme practical problems with a system that rewards effort. Speaking as both a TA and a student I will testify that it is almost impossible to measure effort in a meaningfull way. Unfortunatly the first tendency of teachers is to credit students who are always in class or talking to them as being more dedicate. However, this is grossly unfair to many students who might learn differntly and encourages disgusting brown-nosing which interferes with actual learning.

For instance as an undergrad I found that lectures did me no good in mathematics. I spent a great deal of time on these clases learning from the book myself but I never showed up in class...it simply would have been a waste of time for me. Regardless of what you think about giving a bonus for effort certainly we shouldn't discriminate against those who put in their effort in a non-flashy way, by reading on their own and not spending all their time asking the teacher questions. Often these are the students more interested in learning as opposed to ones who just want the grade.

This brings me to the final problem. If you institute a system which gives better grades to stupid but diligent students things will get crazy. Already as a TA I have to deal with alot of students kissing ass and begging for better grades, if they find out that my opinion of them being diligent affects their grade it gets even worse. Valuable class time we could have spent learning would be used up by students asking questions they already know the answers to just to appear interested/diligent. Most likely smart students would start faking being stupid and diligent to get the grade bonus.

Finally I would offer the analogy. Why don't we run the olympics or college sports on this effort basis. Sure I might suck really bad at running but I am clearly at a disadvantage so my 6minute mile should count just as much as someone elses 3min mile. This is certainly absurd in sports so why would you suggest it in academics?

--

If you liked this thought maybe you would find my blog nice too:

If it's not software - what the heck is it? by crighton · 2004-05-20 06:16 · Score: 1

From the e-rater website "It is important to remember that e-rater is an embedded real-time application; it is not software."

So they're creating chips just for this? When the marketroids can't get the tech right, it makes me really nervous.

Not software? by wcrowe · 2004-05-20 06:17 · Score: 1

From the website: "It is important to remember that e-rater is an embedded real-time application; it is not software."

Huh? How can it NOT be software? I smell snake oil.

--
Proverbs 21:19

MOD PARENT UP! by Luscious868 · 2004-05-20 06:19 · Score: 1

The parent is dead on. While I can understand why a teacher would want to use something like this, grading an essay is, and should be for exactly the reason you stated, a subjective process in some respects. There are certain things a computer will probably never, at least in our lifetimes, be able to do and do effectively. This is one of them.

Teachers who use a program like this are just plain lazy. Teachers are paid to teach, grade work and give feedback. It's a tough job, we all understand that, but that's what you get paid to do.

Hack the system! by edunbar93 · 2004-05-20 06:20 · Score: 1

I'd have to wonder how this would go over:

Name: &f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f&f &f&f&f&f&f&f&f&f&f&f&f

cat "100:Thomas A Andersen" > /home/instructors/grades

Allthough writing atll those &f's would be kind of awkward.

--
"No problem. I have the capacity to do infinite work so long as you don't mind that my quality approaches zero."-Dilbert

I scored 5/5 on the AP English exam... by MichaelCrawford · 2004-05-20 06:21 · Score: 2, Interesting

...and I like to write.

I just signed up for a userid so I can take the exam online, but after submitting my info it said I may have to wait up to two days to get an account.

Curious that they can grade essays with a computer but it looks like they have to have a human pass out the user ids.

Anyway, I'll see if I can submit one of my articles to the exam, and will post here how I did. Since I have to wait for my user ID, you'll have to look back here later to see how I did.

--
Request your free CD of my piano music.

THAT's not the big question.... by Anonymous Coward · 2004-05-20 06:21 · Score: 0

The big question is:

Will students now start to have their computers write their essays?

I've seen a system like this before... by mark-t · 2004-05-20 06:21 · Score: 1

And you know what? It actually works! Much better than anyone would have imagined!

They even work better than a human marker because a human marker can make mistakes based on personal bias or feelings that he or she may happen to have at the time that they are marking. This does not happen with a computer.

Computerized marking ends up being fairer to the student over the long haul than human markers tend to be. By "fair", of course, I mean that the student gets what they really earned, not that they tend to get better marks.

It might appear that the real danger with using computers to mark is that software might contain bugs which can cause the work to be graded unfairly, but in fact in the real world, human markers are more likely to grade unfairly than computers are, even when one is taking into full consideration the frequency of software faults!

--
File under 'M' for 'Manic ranting'

Other products by crighton · 2004-05-20 06:21 · Score: 1

I've been looking into this for a while now.. We're looking into something similar for low-stakes feedback to students during the writing process. Final evaluation will be done by humans, though...

I saw a presentation about 10 years ago at UC Boulder about one of the underlying technologies behind these kinds of systems. The resulting spin-off Knowledge Analysis Technologies has developed a product that uses a statistical technique called latent semantic analysis. From what I understand, it builds a model based on words appearing in proximity to each other over a large set of exemplars. For thos interested in the underlying ideas, there a site here with lots of research and background.

Wrong Emphasis! by logicnazi · 2004-05-20 06:24 · Score: 1

So unlike many of the people replying I don't think having a computer grader in some classes is such a bad thing. Having been a TA myself I am well aware how subjective and inconsistant human grades can be. Even if this computer grader isn't wonderfull at least it is going to be very consistant (and will provide proof against lawsuits).

Still, I think it is entierly inappropriate for a high-school class like this. In high school we should be emphasizing reasoning ability and consistancy of argument not grammar and punctuation. Spelling and punctuation should be emphasized in Jr. high perhaps but the skills students really need in college are reasoning and analysis. These skills help them in all areas in life, poor punctuation can be fixed by a machine (well at least as well as a machine can grade punctuation).

This machine simply can't check logic and consistancy. However, this might just be a lost cause as I'm not sure if high school english teachers can do so either and their hands may already be tied as they have to teach to the new SAT and AP exams. Unfortunatly basic logical skills are lacking in much of our population and it wouldn't surprise me in the people we hire to grade these exams are no better.

--

If you liked this thought maybe you would find my blog nice too:

Re:For grading grammar, a computer would be unbias by Anonymous Coward · 2004-05-20 06:24 · Score: 0

Sure, a computer will be far more fair; after all, it can't understand any significant ideas, so it has to ignore most of the content of your paper. What could be more fair than that (albeit pointless for learning to really write)?

Something's not right here by Rocketboy · 2004-05-20 06:24 · Score: 1

Having been born and raised in Indiana, and having raised three kids here, I have a hard time believing that Indiana is first to implement any kind of worthwhile technology. If this is the first use of this thing, it's got to be so horribly screwed up that no one else would touch it with a three-meter keyboard cable.

Indiana's government and social philosophy is apparently still living in the 19th century, operating under the assumption that 90% of the kids in school are going to spend their lives farming in the same county in which they were born. Hopeless incompetance doesn't even begin to describe the public education system in this state. And the fools in charge can't figure out why Indiana loses more college graduates (moving to other states) than just about any other state in the union. If you have half a brain you probably don't want to be here. Nice place to retire to, though.

Sure, there a few exceptions -- a couple of excellent high schools, some colleges. But the vast majority of Hoosiers don't appear to comprehend the role of education or technology in life, beyond making their DVD players work. I'll bet there are few states in the Union with more appliances blinking 12:00 than Indiana.

rb

Computer Feedback could actually help teach by tomboy17 · 2004-05-20 06:26 · Score: 1

I know most of the thinking here will be (predictably) about the fairness of tests and the ability of students to find loopholes (i.e. bugs), but as an English teacher, I wish someone would work on similar software for teaching rather than testing.

There are lots of basic grammatical errors students make repeatedly -- whatever English teachers do to explain these errors has a pretty high rate of failure.

I would welcome a computer program to give students instant feedback on grammar, sentence complexity ,etc. Human minds learn incredibly well by trial and error: a good program could help students figure out through repetition what they struggle to learn from explanations.

I used Word's grammar checker for the passive-voice recognition alone in order to help wean myself from Academic-style overuse of the passive voice years ago. Eventually, I learned the kinds of patterns that led me to use unecessary passives and the kinds of revisions I needed to make my writing more solid. But I don't think I would have gotten as proficient as these revisions without the program to help highlight the "errors."

Ccomputers could provide students with feedback that could ultimately be a powerful teaching tool used in tandem with teacher's giving lengthy narrative feedback about the ideas in student writing (I don't believe there's a program that can do this yet). Programs could also allow students to practice writing much more without being limited by the number of papers their teachers can read.

I would love to see an open-source project providing schools with sophisticated writing-feedback software. It's time some of this technology actually goes into creating better teaching tools in classrooms rather than more ETS once-a-year yardsticks.

What was not mentioned ? by fygment · 2004-05-20 06:29 · Score: 1

The article takes pains to point out all the worries there are. What was not mentioned was:

a) computers don't tire - so expect more consistent grading on punctuation, grammar, etc. for the first paper and the last;

b) computers don't hate - some students are smart but behaviourally challenged which prejudices graders. Now they can be assessed fairly on the quality of their work not their attitude (or perceived attitude).

c) computers can spot plagiarism - rather more easily than humans. Especially tired overworked humans.

d) computers can keep ahead of cheats - unless a student has access to the grader, it would be a long time to come up with a scheme to fool it. As the system evolves over time it could presumably always be a step ahead.

It can only be a good thing as we get humans out of the grading loop. Of course, the brown-nosers will have a hard time of it. Sucking up to a computer?

--
"Consensus" in science is _always_ a political construct.

AP Essay Rubric by jmichaelg · 2004-05-20 06:38 · Score: 2, Interesting

Fine and dandy if you're writing for humans who are reading for content. On the other hand, the AP Biology essay rubric assigns points based on how many keywords appear in a response. It doesn't score the essay for sense.

A friend of mine who teaches Biology said that she saw some pretty bad essays which she would have given a poor grade to because the english was atrocious but she had to follow the grading rubric and give high scores to because the keywords were present.

Re:AP Essay Rubric by jci · 2004-05-20 08:12 · Score: 1

I loved the AP Bio test. I wrote with no right-hand margin and with fillerless sentences.
In pretests had a high enough keyword count for a 5 in somewhere between half and two-thirds of a page!
I can say at least that my sentences were coherent and did make sense though.

Leaked source code to e-rater! by The+boojum · 2004-05-20 06:40 · Score: 1

sub e_rater { my $essay = shift; my $len = length ( $essay ); return 5 if $len > 4000; # Too windy return 6 if $len > 3000; # Just right return 5 if $len > 2000; return 4 if $len > 1000; return 3 if $len > 700; # Too short return 2 if $len > 300; return 1 if $len > 100; return 0; # Nothing there }

Try it out by NitsujTPU · 2004-05-20 06:42 · Score: 1

This service lets you use the program to simulate the analytical writing section of the GRE.

http://www.ets.org/scoreitnow/

Dammit! by Clark_Griswold · 2004-05-20 06:44 · Score: 1

First they send all our tech jobs over there, now they're grading our kids papers!!!

Allright, Hoosier daddy?

--
-- Mace only makes me hornier.

How long will it be until someone finds an edge? by signingis · 2004-05-20 06:44 · Score: 1

e-reader-whacking anyone? With no human oversight in this it's quite possible that someone will be able to turn in complete crap and get an A. Not that this ever happened with human oversight.

--

I prefer a void in conversation to a vacuous one.

Zero for this. by triso · 2004-05-20 06:49 · Score: 1

From the Web site: "Developed at ETS and grounded in more than a decade of research, e-rater is designed to score essays based on an analysis of writing features as reflected in holistic scoring rubrics."

The ending of that sentence gets a zero from me.

Qualitative v. Quantatitive. by Capt_Troy · 2004-05-20 06:49 · Score: 1

Seems like this might be able to do a good job grading the quality of the writing (i.e. correct grammer, punctuation, spelling, etc) But it seems like a human reader would have to evaulate the quantative value of the essay. Does it contain the correct information? Did the writer stray into different topics?

I, for one... by ajlitt · 2004-05-20 06:50 · Score: 1

... welcome our new cybernetic education overlords.

As so many have mentioned, this should be easy to crack. Do the same thing as the spammers do to get around filters: get a copy of this program, get a bunch of random text from the web, get a Markov chainer to generate random text from the input of all the essays, feed the output to the test program. You can batch the whole thing such that after N iterations you replace the raw input text with essays that did well through the screening program. Keep repeating this outer loop until a good looking paper comes out. Voila!

Here's surprising research from Stanford by Anonymous Coward · 2004-05-20 06:52 · Score: 0

I'm a student at Stanford. Last year, I was taking a class from an education professor who also researches psychmetrics and statistics. He's also on the board of the GRE.

He told us that computers will increasingly be grading essays. In fact, he told us, they can already do so with 96% reliability (i.e., their grading is the same as a human grader 96% of the time).

We were shocked and didn't believe him, but he told us the key: Human graders grade to a rubric. If computers are given that rubric and "taught" representative samples, they can do the same. Fascinating. I spoke to him today and he said new work in this area is even more encouraging than is yet seen in the commercial world.

Re:Here's surprising research from Stanford by Anonymous Coward · 2004-05-20 08:55 · Score: 0

The sad thing is that mechanical essays are graded on conformity to a rigid structure rather than based on quality, clarity, and ability for others to really understand your idead don't real test anything.

If this is the "future" of education, I am shocked and saddened. If all we ask of our students is to robotically write essays, then we will just have a society full of robots.

Just because someone from Stanford endoses this practice does not make it a legitimate one. This is one of the many examples of how standardized testing is nothing but a ploy to sort students based on how mechanized they are - learn a proces rather than actual knowledge.

I went to high school in Indiana in the early 90s by frinkster · 2004-05-20 06:54 · Score: 1

In some of my English classes we had to feed our essays into a computer program that rated them and gave them a score based upon the "grade level" that our essays were written at/for. All of my teachers understood the limitations of the software and used it in an appropriate manner.

An example that I can remember was an assignment to write an essay describing how to do a chosen task that a 5th grade student would be able to understand. Once the software agreed that the essay was not too complex and that the word usage was appropriate, the teacher gave it a quick read to make sure it made sense. For such an assignment, the software worked very well.

Both of my parents are educators and as such I am aware that the teachers around the state and the administrators in Indianapolis are not always on the same page. I wouldn't be surprised at all if the intended use of this software includes being given a cursory read by a human or even to be nothing more than a grading aid to a teacher, yet the state mandated all essays to be graded solely by the software.

Such things happen when high-level administrators are politicians whose only educational experience is that of a student. Indiana is not alone in this regard.

Sanity Check by gr8_phk · 2004-05-20 06:56 · Score: 1

Perhaps a human should look over the paper first. Look over it quickly to verify the overall flow makes sense. If so, let the computer grade it. If not, then revert to manual grading. Treat the computer as an aid, not the authority.

I'm not surprised by melikamp · 2004-05-20 06:56 · Score: 1

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

I've graded an upper division Geometry class in Cal state (kind of similar, because I was looking at proofs in essay form), and I find it very easy to believe that a computer can produce similar grades.

Many of you indicated that this new grading method is unfair and/or easy to trick. I'm not surprised, because, IMHO, the old method was already unfair and easy to trick. Unfair, because a letter grade (or 6 point scale grade) is providing the worst possible kind of feedback. Brilliant flashes of insight + imperfect grammar may give you B- . Being ultimately average in every respect, but reasonably prepared may give you B+ . The positive reinforcement mechanism is broken: people are awarded for the time spent, or sometimes even for how carefully they copy and paste, and never for their actual excellence.

Now, the automated grading takes it one step further. A live teacher could be subjective in grading, by trying to reflect, for example, how student was improving after she was provided with feedback. Automation deals away with it, and we are left with this rigid system which only rewards an adherence to standards.

Grammar and Spelling by Anonymous Coward · 2004-05-20 06:56 · Score: 0

I'm confused. How is "subject and idea"-oriented English geared toward girls? Do you mean that they began requiring students to actually use language rather than require rote memorization of rules for grammar drills? How odd that the boys' grades would slide when that change was made. I wonder if other curriculum changes were made at the same time might also be responsible? I don't know your gender, but if you are male, how do you explain having done well when most male students did not?

You seem like a bright individual. If you feel that your grammar and spelling are sub-par, I encourage you to work on it on your own. When I was in grade school and high school, I couldn't spell my way out of a paper bag. Now I am much better. Constant use of the spell checker is what helped me most. I would get quick feedback on what I had done wrong and could quickly correct it. Each time I would correct an error, I would look closely at the correct spelling and try to remember it. An even better way is to try to fix the word yourself until it is spelled correctly, ignoring the word processor's suggestions. Do not use word processor functions that automatically fix spelling errors; they will just reinforce your bad habits. The spelling errors in your post are almost all careless errors that a little attention to detail would have eliminated.

Grammar is harder. Good writing is harder. I recommend reading examples of good writing and trying to notice what makes it good. Practice writing. You might even consider taking a class.

BTW, if you need to work on usage, you might consider subscribing to "Usage Tip of the Day" found on Oxford Press' website. It has tips on American English usage, but they are probably helpful for Canadian usage as well.

Re:Grammar and Spelling by king-manic · 2004-05-20 07:22 · Score: 1

Thank you for your tips.

Girls and Boys are build very differently mentally and physiologically. Girls will learn gramar and structure on their own through iteration and enjoy idea and subjected oriented lessons more. Boys require constant re-iteration of basic principles to pick up the gramar. A few years ago (20) my school district saw girls were doign poorly ina number of areas and revised the ciriculum, the girls startign doing much better. However the boys had a general slide in their grade in english. In the first year after the revision and for every year after the boys did worse then the girls and worse then the averages from the years before the change. Basically it's atrributed to the system teaching to the strengths of girls while the boys have to tag along. Some boys will do well because they are talented. And they would do well in almost any system. But the averages have dipped for boys.

--
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
Re:Grammar and Spelling by Anonymous Coward · 2004-05-21 03:26 · Score: 0

It sounds as if before the curriculum change, teaching was aimed towards the boys' strengths and the girls had to tag along. It is unfortunate that no method could be found to teach to the strengths of both boys and girls.

However, I'm still confused by your explanation of how this affected you. You complain that you were not taught grammar and spelling because the school focused more on the ideas that you were expressing rather than the way they were being expressed. You also state that boys learn grammar and spelling through repeated drilling while girls either learn it by osmosis from exposure to language or just don't need as much drilling. Under the new (20 years ago) system, you explain, the boys' grades went down while the girls' grades went up--apparently because it is not effective for teaching grammar and spelling to boys.

This makes sense, except for one thing: you say that even though you did not learn grammar and spelling, you received good grades on your ability to express ideas and were not graded down for not being able to spell or use proper grammar. I can only conclude that you were not graded on grammar and spelling. Based on your explanation that the curriculum was not effective in teaching grammar and spelling to boys, if you were not graded on grammar and spelling, why would the boys' grades go down and the girls' grades go up?

It seems more likely to me that the problem is not just that the new curriculum is not effective at teaching grammar and spelling to boys, it is also not effective in teaching them to express ideas (even when grammar and spelling don't count).

In any case, it sounds as if the curriculum needs reworking. I hope you have expressed these concerns to the schools where you were educated in the hope that they will do better with the current crop of students.

P.S. It is "grammar" not "gramar," "English" is capitalized because it is a proper noun, and "subject-oriented" is hyphenated the way you have used it in the sentence (though I can't guess what you intend it to mean) while "reiteration" is not.

school teachers by Anonymous Coward · 2004-05-20 06:57 · Score: 0

Because we know that school teachers don't really grade on thought and content anyways

Well, if computers can read papers... by glwtta · 2004-05-20 06:59 · Score: 1

Lets see if I can reduce the amount of work I put in my slashdot posts:

School the actual 56 was it didnt ahhere to their simply 5 paragraph introduction 3 body and conclusion i did horrible my second time i wrote the article actually i meant the former not the latter indiana is however according to my office every week and really get the first thing that came to my mineas you can skip the registration of the students know that they know will give them the good thing your windows grammar checking software and use it everywhere use it to automoderate slashdot since i like 7 of the computer makes and theyll.

Tell their students what style will get good grades personally ive always felt it is undoubtedly a deep ai problem to how many people did you send a dollar i sent too many yum abiguity although arguably using too here wouldnt answer the question is will other states begin to understand analogy you have been any worse if the gradeovac was inspecting my papers so that we can figure out how to use the erater site erater learns to score essays on a good grade wouldnt bother to contest the results of computerized testing have the following comments are owned.

By whoever posted them we are watermelonwatermelonthis student is treated equally emotion and prejudice are taken out of 10 or something like that you can write using a computer program check that a computer can obviously not grade essays fairly so it gives its favorite students as without reading least favorite students as without reading least favorite students fs and the gold every time 1 reply beneath your current threshold there are easy ways to scam it too bad i dont think its only a matter of time on these clases learning from the rubberstamp high school version of.

Hmmm... looks like there isn't enough variation in your typical slashdot thread for a trivial Markov generator to produce anything interesting.

--
sic transit gloria mundi

You're mistaken about a few things. by Anonymous Coward · 2004-05-20 06:59 · Score: 0

Given your inability to spell ("fiurst"?) or capitalize (sentences should start with caps; "human", "essay", and "university" are not proper nouns), I doubt very much that you ever belonged in "English Honors".

That's not all that's wrong with your post, of course:

First paragraph: The second sentence is a fragment. The third is missing an article or two and its end punctuation is missing entirely. The next sentence is a real mess: I'd recommend commas before and after "though" and after "exam"; they'd save the reader considerable effort in decoding. The sentence after that is even harder to decipher: "For some reason" does not belong where it is; it sounds like you're wondering why your essay was graded, rather than wondering why it got the grade it did (I'm not wondering).

Next paragrah: The first sentence needs a semicolon after "English". "Why else would they be working at a University" should have a question mark at the end.

Conclusion:

You are illiterate in English. I doubt that you are capable of communicating effectively even in speech. If a dean took you out of a remedial class, that's professional malpractice on his part. Oh, and that 720 verbal SAT score? Nope. You're lying about that. Standardized tests mean less than people think they do, but they're not entirely arbitrary. You're probably lying about the rest, too.

Your entire post is incoherent bullshit.

Re:You're mistaken about a few things. by Anonymous Coward · 2004-05-20 07:11 · Score: 0

rofl...
Re:You're mistaken about a few things. by sTalking_Goat · 2004-05-20 08:00 · Score: 1

Post as a real person if you'd like a response.

--
My days of not taking you seriously are certainly coming to a middle...

Redundancy all over again by Anonymous Coward · 2004-05-20 07:03 · Score: 0

Are you talking about the idiot above who asked "is it open source" and got modded redundant?

Well, it was redundant. That kind of Stupid Slashbot Remark has been redundant around here since 1998, at least.

It couldn't've been more redundant if he'd asked for a Beowulf cluster of Natalie Portman, for God's sake.

mod +5 Funny! by mrchaotica · 2004-05-20 07:07 · Score: 1

"Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort."

HA HA HA HA HA HA HA!!!

I agree with the rest of your post, though : )

--

"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz

function gradePaper(char *text) returns by Anonymous Coward · 2004-05-20 07:10 · Score: 0

return random(100);
end function

How long before... by Eminor · 2004-05-20 07:14 · Score: 1

And how long before someone writes a program that spits out nonsense essays that generate high grades?

A commentary from the LA Times by Anonymous Coward · 2004-05-20 07:14 · Score: 0

Reposted here because their registration process sucks. I mean, c'mon. Household income as a required field? What genius came up with that idea?

The Lobotomized Weasel School of Writing

By Crispin Sartwell

[Crispin Sartwell teaches political science at Dickinson College in Carlisle, Pa.]

The other day, our 16-year-old son, struggling with his homework, asked his mother this question: "Do you know how many paragraphs an American history essay is supposed to have?"

The answer, of course, is one. Or seven. Or 700. Whatever.

But that is not what he has been taught; he's been told there's a correct number. Once I was working with him on an essay and he told me we needed exactly three arguments. No more, no fewer, although he did not know yet what they might be.

Today's educational establishment is making actual illiteracy look good, like an act of humanity and rebellion. Writing, which ought to nurture and give shape to thought, is instead being used to pound it into a powder and then reconstitute it into gruel.

The thoroughly modern grade-A public-school prose style is not creative or interesting enough even to be wrong. The people who create and enforce the templates are, not to put too fine a point on it, people without understanding or imagination, lobotomized weasels for whom any effort of thought exceeds their strength. I recently read one of the many boilerplate descriptions of how students should write their essays. "The penultimate sentence," it said, "should restate your basic thesis of the essay." Well, who says? And why?

The teaching of writing as a machine procedure gains momentum by the day. In Indiana this year, the junior-year English essay will be graded by computer, and similar experiments have been tried in Pennsylvania, Massachusetts and Oregon. The SAT and the ACT are planning to test the new computer-grading software as well. That is a reductio ad absurdum of the entire idea of learning. If this is knowledge, then truth and beauty reside only in ignorance.

Vantage Learning, which makes the writing-assessment software called Intellimetric, claims that it "shows more reliable and more consistent results across samples than human expert scorers." Of course "reliable" entails "accurate," and I daresay there is no way to establish that without begging all possible questions.

More to the point, perhaps, machines are cheaper: It costs perhaps $5 for a human being to evaluate an essay, $1 for a machine. And while it takes five to 10 minutes for a human to score an essay, the computer can apparently do it in two seconds.

The actual procedures that the software employs are presumably proprietary. But the dimensions that Intellimetric evaluates are these: (1) focus and unity; (2) development and elaboration; (3) organization and structure; (4) sentence structure; (5) mechanics and conventions.

One can imagine the way a computer assesses such things: The repetition of a given word, for example, helps constitute unity, and the penultimate sentence had better recapitulate the introduction in pretty much the same, recognizable terms. There are to be three "supporting" paragraphs, and the relation of the body of each to its "topic sentence" might again be assessed by word repetition. "Development and elaboration" might, for example, be proportional to the length of words, or of sentences.

The only real argument for the quality of the software is that it is "more reliable and accurate" than human evaluators. But the human evaluators have already transformed themselves into Intellimetric software: These are the military sheep -- their minds both rigid and woolly -- who invented and enforce the mind-numbing five-paragraph essay form.

Every child in the United States, more or less, is being taught to write and to think in this way. I teach these kids when they reach college. I try to tell them that the idea that there is some specifiable way to write an essay is just hoo-ha

Essay Grading & Mail.app: Latent Semantic Anal by Krach42 · 2004-05-20 07:19 · Score: 1

Having just glanced over the article that talks about how Mail.app's filter works.

Yes, this is Latent Semantic Analysis. Though they call it "Vector Representation".

LSA more relies on collapsing the dimensionality of the vectors down, so that words that are similar begin to be represented on the same dimensions.

Read up on that Mail.app article to get an idea of how LSA works.

--

I am unamerican, and proud of it!

s/Analysis/Anal/ by Krach42 · 2004-05-20 07:22 · Score: 1

I didn't think it'd be nice to leave it as "Anal"

--

I am unamerican, and proud of it!

Re:s/Analysis/Anal/ by Anonymous Coward · 2004-05-20 09:10 · Score: 0

I didn't think it'd be nice to leave it as "Anal"

You do realise that 's/Analysis/Anal/' means "replace Analysis with Anal", which is apparently the opposite of what you wanted?
Re:s/Analysis/Anal/ by jimbolaya · 2004-05-20 12:31 · Score: 1

You know what? I'm such a geek, because that's EXACTLY what I thought, without a moment's hesitation.

--
There ain't no rules here; we're trying to accomplish something.
Re:s/Analysis/Anal/ by Krach42 · 2004-05-21 14:22 · Score: 1

CRAP! I've been out of regex's too long...

plus, I wasn't thinking... gr....

--

I am unamerican, and proud of it!

stop throwing your garbage into our dimension by waspleg · 2004-05-20 07:25 · Score: 1

thank you

--indiana

oh and btw we're called hoosiers by waspleg · 2004-05-20 07:27 · Score: 1

not indianans

and not *all* of us have all our money in nascar hardees collector cups either

Re:oh and btw we're called hoosiers by SmittyTheBold · 2004-05-20 09:32 · Score: 1

What the hell is a "hoosier" anyway? And saying "a person from Indiana" is not an acceptable answer. Some of my immediate family just emigrated to your fine state and they couldn't answer the question for me.

--
± 29 dB
Re:oh and btw we're called hoosiers by Anonymous Coward · 2004-05-20 10:51 · Score: 0

I just call them 'hosers'.
Re:oh and btw we're called hoosiers by Anonymous Coward · 2004-05-20 21:22 · Score: 0

There are so many different explanations for it, that I think noone really knows for sure.

According to legend, it's a bastardized way of saying "who's there", but I dunno about that.

Your Structure... by Anonymous Coward · 2004-05-20 07:31 · Score: 0

Judging from the structure of your post, I'd say you probably got what you deserved.

Poor diffusion in the autograder "hash" by tepples · 2004-05-20 07:31 · Score: 1

All I need to do is reverse the process to get a file that generates that hash!

Cryptographic hash algorithms such as SHA1 are designed to have good diffusion, meaning that a text's hash value bears no resemblance to any other measurable quality of the text. On the other hand, the hash values that this automated grader produces are designed to measure specific desirable qualities of the text, and stochastic generation of a text with biases toward the known intended qualities may lead to results.

Re:Poor diffusion in the autograder "hash" by Glonoinha · 2004-05-20 08:57 · Score: 1

There we go again, taking my perfectly funny post and not recognizing it for humor.
You think I would learn, but I don't.

That the output from the grading program effectively is a hash (actually it is effectively a CRC number with six or seven values) and that there is no way to reconstitute an A quality essay from a single digit 'grade' is exactly why my post was funny.

--
Glonoinha the MebiByte Slayer

maybe they should try by waspleg · 2004-05-20 07:38 · Score: 1

asking students if they would like to write

and if they would let them write about what they want

i bet that would improve the substance and quality of handed in essays exponentially as well as cut out 95% of the existing ones by virtue of them not being written at the last second on something no one but the professor cares about

Cheating by Thieron · 2004-05-20 07:43 · Score: 1

How long until someone breaks the system and is able to write a paper that is nothing more than keywords, etc that the grading program look for. Until there is true AI, essays are simply too complex for computer anaylsis. Different people use different writing techniques to get to points and make conclusions, how is software going to be able to predict all the possibilities. I'm not saying this isn't a neat idea, I'd love to know some of the specifics behind the logic in the software, but right now I see this a something can be broken badly as essays start being written for a computer audience and not a human one. Thought I guess at least the computer won't get tired and blurry eyed while grading. Nothing like hoping your good papers where the ones read first and those late nighters were left until the poor TA grading the paper had been through several hundred papers and couldn't tell a load of BS from real work... not that I even BS'd a paper (or my thesis, no, never)....

Spelling national socialism by tepples · 2004-05-20 07:45 · Score: 1

Given your inability to spell ("fiurst"?)

If spelling errors involve adjacent letters in the QWERTY or Dvorak Simplified keyboard layouts, the correct term is "typographical error" ("typo" for short) not "spelling error."

And what is a "Next paragrah"? Did you mean "Next paragraph"?

It seems that one almost can't have a spelling-national-socialist comment without a typo or spelling error.

Re:Spelling national socialism by Anonymous Coward · 2004-05-20 07:56 · Score: 0

It seems that one almost can't have a spelling-national-socialist comment without a typo or spelling error.
Correct. I proofread that post obsessively... but if it's a spelling flame, one always gets through. This is a law of nature.
"Spelling-national-socialist"? I like that! LOLWTFBBQ, as it were.

Let's not forget about... by Squidbait · 2004-05-20 08:02 · Score: 2, Insightful

The postmodern essay generator

Zen notion of Quality by addie · 2004-05-20 08:04 · Score: 1

I would be surprised if a great number of slashdotters haven't read "Zen and the Art of Motorcycle Maitenance" by Robet Pirsig. The book is about many things, but most of it boils down to the subject of "quality". That is, when two pieces of rhetoric are placed side by side, how does a human being determine which one is of the better quality? The fact is, there are so many factors that go into determining this, it almost becomes a philosophical question. If student A and student B both write papers on the same subject, using the same source material, and are both competent writers, what is the difference in the end results? One is going to be of a higher quality than the other for probably non-quantifiable reasons.

"In a sense ... it's the student's choice of Quality that defines him. People differ about Quality, not because Quality is different, but because people are different in terms of experience. ... if two people had identical a priori analogues they would see Quality identically every time."

I'm not sure I've made my point very well, but the fact is we can't have computers grading exams. That's very frightening for the education system. Teachers need to be more involved with their students, not less. And how, as an English teacher, would I be able to tell the differences between four students when they all score 5/6 on a computer graded essay? It's ridiculous.

Everyone needs to place more value on our teachers, it's insulting to suggest a computer can take their place.

Good writing isn't mechanical by servognome · 2004-05-20 08:10 · Score: 1

A mechanical grading system will show you know the mechanics of writing, but not that you can write well. Grammar, sentence structure, punctuation are tools for writing well, but mastering those tools doesn't mean you can combine them into a coherant and well structured essay.
The biggest problem for many technical students isn't the mechanics of writing, its being able to write affectively. To draw in the reader, and to be able to write to your audience to clearly communicate ideas with a certain sense of flow. Higher level writing bends and breaks rules to be more effective.
Writing is an art, like music. Its easy to play the piano if you just look at it as a series of key presses at specific times; but when an audience hears music, its just as important how to transition between one key press to the next, moving between loud and soft, fast and slow, creating imperfections so you are not mechanical.

--
D6 63 0D 70 89 81 BB 8E 7B 7C 5F 5D 54 EA AB 73

What this really shows... by Funkeriffic+Toad · 2004-05-20 08:21 · Score: 1

is that Indiana's teachers SUCK if their grading can be emulated by a machine.

Sucks when people mis-moderate... by Ieshan · 2004-05-20 08:24 · Score: 1

Man. It sucks when people mod your funny posts as insightful.

Although I guess it sucks in the other direction too, Bush must be fed up with that already.

This won't work out... by wicka_wicka · 2004-05-20 08:25 · Score: 1

I live in Indiana and go to high school. I've tried the computer grading - trust me, it sucks. It wouldn't find my brother's introduction and wouldn't grade mine, saying it was "offtopic" or something.

--
hi

Robots will teach robots. by hoggoth · 2004-05-20 08:28 · Score: 1

Way to teach creativity in writing!

My first writing assignment in English 101 was to write a brief story. We were to follow strict grammar rules. I purposely opened the story with a single word sentence that broke any rules of grammar:

"Dawn."

I got an 'A' because my writing COMMUNICATED something to the reader.
I'll bet I can write a perfectly grammatical writing sample that makes absolutely NO sense at all and get an 'A' from this program (perhaps with a little trial and error first).

--
- For the complete works of Shakespeare: cat /dev/random (may take some time)

Scan-Tron by MrBlackBand · 2004-05-20 08:31 · Score: 2, Funny

Anyone remember those Scan-Tron tests we had to take every once in a while? I hated those because unless each circle was filled out perfectly it would mark it wrong. Whenever I pointed out errors in grading to the teacher he or she would just tell me to fill it out better next time. Grrr.

Of course it was fun to mess with cheaters. If I noticed someone was copying off of my work I would make a point to put down all wrong answers. Then I would pretend to check over my work. The person who was cheating off of me would usually just take their test up to the teacher right away. When they sat back down I would make a big deal out of erasing every single one of my answers and doing the whole test over.
Their reaction was always priceless.

--
"It is difficult to get a man to understand something when his salary depends upon his not understanding it."

Re:Scan-Tron by wicka_wicka · 2004-05-20 10:11 · Score: 1

We still use the ScanTrons, my Biology teacher checks over all of them before he gives them back to us to make sure the computer didn't make any errors.

--
hi

Confused by the american system by grahamsz · 2004-05-20 08:31 · Score: 1

What I can't understand is that teachers and tutors have this kind of weight in the american system.

In britain, every one of my exam papers since about age 12 was marked anonymously. Many of them were even traded with other institutions so that in a small class the teacher still couldn't pick you out by your style or writing.

This seems like the only fair way to grade work since it's very hard for teachers to be completely objective - and for critical exams, that's not fair on students.

Your teacher does still mark your day to day classwork - but a subset of that is marked externally as well so that a comparison could be made. I've had a grade increased because my immediate teacher was unreasonably harsh on me.

Re:Confused by the american system by Anonymous+Custard · 2004-05-20 09:18 · Score: 1

That's interesting; I never knew it worked that way in Britain. I'm sure a handful of USA schools do that, and for national standardized tests (college admission tests) you certainly don't know your graders, but it's definitely not widespread.

--
$8.95/mo web hosting

I had computerized grading over 20 yrs ago! by Anonymous Coward · 2004-05-20 08:36 · Score: 1, Interesting

I should have patented it. My high school issued report cards that were printed on fan-fold forms from a dot-matrix printer. They paid "student helpers" to feed the printer. I also paid the "student helpers" for the leftover blank forms at the end of a print run. Our high school administrators decided they would save on postage by handing out the report cards in homeroom and announcing this fact in the local newspaper.

Using my TRS-80 Color Computer and DMP-100 dot matrix printer, I offered an alternative scholarship program. For a $10 fee, I would print a report card that was identicial to the real ones, except for the grades. My "clients" would take their real report card, pencil in their new grades, and my computerized grading system would do the rest. Each kid would go home and say that he forgot his report card in his locker and would bring it home tomorrow. I would deliver the new & improved report cards the next day and all was well.

The "offical" grades remained unchanged, so it was up to each client to avoid flunking courses that would prevent graduation. Anyone who failed a mandatory course was ineligible for my "service". One client tried to blackmail me into providing the service for free, but I said, "Just try and get someone to believe that report cards are being manufactured in a student's house."

The only disappointment I had was when some kids decided to publish an underground newspaper. I wanted to take out an ad, and they refused.

amazing conclusion by meshko · 2004-05-20 08:38 · Score: 1

I'm sorry, but the only thing I can conclude from this is that the trained graders are morons.
There is no fucking way an essay can be graded by something that doesn't understand human language. That's an axiom.
Computers are not capable of understanding human language yet. That's an axiom.
Can you make a different conclusion?
Well, I guess they might not be morons, they could just be overworked. I can easily imagine the sweatshop with 3 minutes on an essay, so that graders, hundreds essays a day... either way system sucks.

--
I passed the Turing test.

Reg Free Link from Google by FelixCat · 2004-05-20 08:38 · Score: 1

Registration free link from Google here: http://tinyurl.com/233vm

I think I can help clear this up a little by tenaciousdRules · 2004-05-20 08:38 · Score: 1

I work for a state department of education. Although my job is not to analyze current trends in student assessment tools, I may have some info to share from a Computer Science standpoint. Firstly, it should be noted that all states have statewide writing assessments for certain grade levels. The lovely "No Child Left Behind" act ensures that students in grades 3-8 and 10-11 will be tested in many subject areas by 2006, with the results having financial ramifications (poor performance = less latitude with federal funding and more federal mandates). All states test some of these grade levels in Reading Comprehension(sp?), Math , and in many cases Writing. So, if you take even a small state, let's just randomly pick one... Connecitcut. CT tests about 140,000 4th, 6th, and 8th graders in writing currently. Those tests are graded by a private contractor in another state. To facilitate this mass grading, 800 'educated' raters are hired and trained. One problem with human raters is consistency across a cohort. In pilot tests (and the real thing), individual raters were observed giving different scores to the same essay twice and if identical essays were given to two different raters, the scores quite frequently differed to a significant degree (How significant? Well the grade given is 1-5, 5 being complete mastery, and what one rater gives a 3 another would give a 5!) That is a huge problem. I have had an opportunity to watch some computer-based grading in action and although it may lack the ability to score based completely on creativity (which sometimes humans may feel compelled to do), these algorithms are amazing. A whole lot of the technology is based in AI and Semantic Search. Papers can be graded not only on grammar, but also on relevance to topic and mastery of subject matter (I would fail). It is said to be a successful grading if a human tester can maintain 80% consitency from paper to paper. These computer based ratings are way over 90%. What does that tell us? Humans are smart enough to write programs to help them do things better. That is the whole point of computers anyway, isn't it?

--
--Always, I mean never..., No I mean always check your references.--

Want to change the world? Here's how! by werdna · 2004-05-20 09:06 · Score: 1

All you need to do is obsolete the software. Produce an open-source version of the system -- make it available to the students.

Hell, make an open-source word processor that will show the student his grade before he submits the paper!

See how fast they recede from relying on automation.

Want to know how the system works, or how it was built? That's what open records act statutes are for!

Make this a nightmare for them, and watch them squirm. It would be fun and, more important, we might make a pedagogical tool out of their laziness -- a tool students can use to IMPROVE their writing!

It is NOT cheating to use a grading word-processor, any more than it is cheating to use a calculator.

the incumbent system by epine · 2004-05-20 09:44 · Score: 1

The point was that this WON'T do a worse job than the real teachers have been doing. Teachers are under a lot of pressure and generally are lucky if any of the essays submitted have content worth grading. Generally, 90% of the grade assigned by a teacher is determined by three factors: spellings with vowels, sentences with verbs, and paragraphs with topics. If this system doesn't reward content, then maybe we should reconsider the incumbent system.

Ah, No Child Left Behind by balor1eye · 2004-05-20 09:48 · Score: 1

It's wonderful to see that my home state of Indiana is so gung-ho to embrace the 'economics over education' policies of our current administration. Indiana only lags behind two-thirds of the rest of the states in education; why should we bother increasing the skills and workforce of teachers when we're failing? Instead, let's accept the status quo and save money by reducing the number of teaching jobs available and have computers raise our children! Computers and robots have been successful parents and teachers in movies and we all know that mass media has raised a generation of exceotional individuals. I say we completely abolish schools and give that money to ISPs and Christian cable networks since that is how George Bush wants to have our children raised. Can you imagine the savings that would occur should we implement such sweeping changes? Just imagine... all those do-nothing teachers forced to find real jobs.

It's all about collision by tepples · 2004-05-20 09:54 · Score: 1

There we go again, taking my perfectly funny post and not recognizing it for humor.

That's probably just my autism acting up.

I agree that there exists no way to reconstitute a particular A quality text from just a grade or any other hash value, as a hash value is necessarily a lossy compression of a text. However, my point relates to a different problem of finding a collision, roughly defined as any text with a known hash value. MD4 is already analytically broken, and MD5 is heading that way. Because of the nature of the autograder hash (better English means a higher hash value), one could probably find a collision using a bit of chatbot tech.

AI? by Lord+Ender · 2004-05-20 10:00 · Score: 1

I clicked this thread hoping for a discussion of the AI involved, and it is just filled with discussion of the social implications. Since when did slashdot stop being about tech and about social and legal issues?

Oh yeah, it's been like that for severl years now.

--
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.

Too easy to coach for. by Yartrebo · 2004-05-20 10:04 · Score: 1

This program could probably be fooled by giving certain pointers to kids. It's probably very sensitive to spelling/grammer (so have the kid run it through the grammer/spell check and accept the first choice proposed ... guaranteed to be gramatically correct).

If the program really was as good, some of its algorithms could be used for improving grammer and spell checks, since as part of the grading process it would have to identify faults and how to fix them (if it can't fix them, it couldn't handle having more than one fault in a paper). Since such grammer checkers don't exist, I believe that it's a crap program that mostly cares about spelling and grammer and perhaps average word length.

Shows why Indiana is a failure... by Roger+Keith+Barrett · 2004-05-20 10:13 · Score: 1

This is the type of crap that shows Indiana is such a failure in Education.

Before you fly for the -1, Flamebait mod, let me say that I actually LIVE here and I grew up with this type of idiocy. It doesn't surprise me a bit that an AI can grade an essay on a standardized test as well as a human reader because the human graders were just awful. I took the ISTEP writing test some 15 years ago now and I remember how dumb and incoherent it was. When it comes to this state they are always more interested in style over substance. It's so bad that they litteraly downgrade anyone that uses any creativity at all.

It's not a surprise that this state has so many problems when they teach not how to do things right, but only how to make it LOOK like you are doing things right.

--

Why don't you embrace your slashbotness instead of living in a dreamworld?

Phi'd by tepples · 2004-05-20 10:34 · Score: 1

I wish. I live in northeast Indiana. I just got out of college with a B.S. in computer science, and I still can't get interviews even on those few relevant job postings that show up on CareerBuilder.

Open Source? by triolus · 2004-05-20 10:40 · Score: 1

Is this paper grader going to be open source? I hope so.. I'd be able to tell exactly what conditions I need to meet in order to make an A. Hell, at that point, I could probably script the entire process of writing papers and never do any work again! =)

No Child Left Behind... by Anonymous Coward · 2004-05-20 10:41 · Score: 0

No child left behind..

I've always interpreted that as

We won't fail a kid no matter how stupid or lazy he is.

Re:No Child Left Behind... by wicka_wicka · 2004-05-21 00:12 · Score: 1

Yes, that's exactly what the program is. My mom works at an elementary school and there are 5th graders who can't add. There is a girl at my high school who needs a calculator to do 22*3. One girl in my Geography class didn't know what country Baghdad is in. Another go confused because only the U.S. had A-bombs in WW2, but "why did Japan set them off twice" (if you don't follow, she thought Hiroshima and Nagasaki was done by the Japanese themselves). It's pretty sad.

--
hi

Lazy Teachers by Anonymous Coward · 2004-05-20 10:50 · Score: 0

What? A 3 month vacation isn't enough ?

Next - AutoTeach - Students just view powerpoint slide shows all day while teachers sit at home and cash paychecks!

Of course, this new education technology must be funded by increasing school taxes 18%, thank you !

proper english? by xpyr · 2004-05-20 11:09 · Score: 1

from the article:

"With paper and pencil we've spent decades figuring out what's going to go wrong and how to deal with it," Mr. Bruce said. "With online we just don't know where all the problems are."

With online? You mean with doing it online. And you call urself an english teacher.

--
My Gawd WTF...

Creativity by b0s0z0ku · 2004-05-20 11:26 · Score: 1

You know - some of the best and most creative writers of history didn't use perfect grammar or punctuation. Dare we discourage a new generation of potentially brilliant writers by having a computer slap them with a zero due to a ,misplaced comma or a cApital Letta in the wr0ng place? At least many humans can recognize brilliance when they see it and forgive minor flaws as a result. I doubt that computers have that trait yet.

-b0s0z0ku

In Indiana.... by Anonymous Coward · 2004-05-20 11:58 · Score: 0

I attend an Indiana High School, and even though my school isn't part of the program, I feel obliged to provide some commentary:

Our english department is crap. Plain and simple. Our high school teachers blame recent clases' awful english grades on elementary teachers not teaching the basics of grammar. And yet, they don't feel obliged to pick up where the elemenary teacher's left off, which is about 5th grade level for the majority of my grade. Ask my classmates what a gerundive is and you'll get a more twisted expression than if you kicked 'em in the gonads. Many of the teachers are also not fit for teaching, and have less command of the language than the immigrant Indians that are destined to replace them some day. I don't live in a hick town, either! The school has 2000 students and is right in the middle of suburbia.

The only reason they're trying to get computers to do the job is that many of the English teachers aren't able to do their job and can't be fired!

Re:How To Write An Essay (Insightful 5?) by Anonymous Coward · 2004-05-20 13:03 · Score: 0

Obviously this post was graded by a coffee maker... Score:5 Insightful. Any humans would grade this post as Score:5 Funny; or even better, Score:6 ROFL/Witty.

Try training the "coffee maker" to read beyond the perfectly formatted introductory paragraph, the three body paragraphs, and the closing paragraph.

This is the kind of subtle humor that no current A.I. can achieve understanding.

Too easy to manipulate.... by kc8jhs · 2004-05-20 13:10 · Score: 1

While in english comp 120 as a freshman, my professor one day showed us all the features of Word that are usuaully unused. Statistics such as the sentences per paragraph, words per sentence, and characters per word, gave you a basic idea what was happening, but to really understand what a computer thinks of your writing you have to look at the readability statistics: Percentage of passive sentences, Flesch Reading Ease, and Flesch-Kincaid Grade Level.

Using statistics, in conjunction with setting the grammar to it's strictest setting, and style as technical, helped me write one of the best marked research papers I've ever written.

This was my technique:

Use the strictest grammar settings available

Check the spelling, and thus the readability statistics after typing each sentence. This enabled me to understand better, how to write in order to manipulate those statistics, and thus hopefully arrive at a more scientific sounding result

The best part about it all, was that it was easy to do, and it worked! In all honesty though, I discussed the technique I used at length in my portfolio for the course, which featured that paper.

-Mikey P

Re:Indiana? I thought it said India by jimbolaya · 2004-05-20 13:10 · Score: 1

Yeah, but in India, you can get 120 racists for one dime.

--

There ain't no rules here; we're trying to accomplish something.

If this software is so good by Wolface · 2004-05-20 13:18 · Score: 1

How come we haven't seen wordprocessors with grammatical and logical "checkers" ?

The ones that try to catch grammatical errors are sucessful 1/3 of the time.

Not to talk about the coherence error correction on an essay.

AP Grading Algorithm by Anonymous Coward · 2004-05-20 13:39 · Score: 0

I can see why ETS is spearheading this; their current manual approach is already highly algorithmic. Back in college (in Indiana strangely enough), one of my profs was a grader for ETS on the AP US History exam. The sheer number of essays they have to grade is staggering: hundreds of thousands in the span of a little over two months.

According to this prof, the graders generated a rubic containing the key points that the essay should address. These points were supposed to be discussed in the first two paragraphs. The strength of a paragraph corresponded with the number of points addressed. The algorithm for grading an essay is as follows:
1. Read the first paragraph
2. Categorize the essay as High or Low based on the rubic
3. Read the second paragraph
4. Sub categorize the essay as upper or lower based on the rubic.
5. Assign a score [1 - 5] based on the category:subcategory classification:

High:upper = 5
High:lower = 4
Low:upper = 2
Low:lower = 1
A three is assigned if there is any uncertainty surrounding the intial partition.
The prof said that the average grading time for an essay was a little over one minute.

This can easily be done by computer by autopr0n · 2004-05-20 13:59 · Score: 1

Just have the machine store the previous grades of the student, and give the student some sort of bonus based on how much better their work is from normal. On the other hand, there''s no way for the machine to be able to guage how much effort they put into it, as a teacher might.

--
autopr0n is like, down and stuff.

the goal is to grade more? outsorce to India! by Anonymous Coward · 2004-05-20 14:22 · Score: 0

in Inida people with humanities degrees work in call centers if they can find any work at all. There should be enough of them to keep every student writing 24 hours a day at very reasonable prices. Surely the comments of a human grader are of more relevance than those of a computer. Plus it would be harder to beat the system and there will be somebody to punish if things go wrong. How do you discipline a script?

You can't see the forest for the trees by MeowChow · 2004-05-20 15:02 · Score: 1

I took the liberty of comparing the score of your comment, an essay of sorts, to the score of a random computer-generated essay (courtesy of the postmodernism generator). Care to guess which one faired better?

Your score:

5 Overall 90.67 2.067 5 5 Content 88.882 1.8882 5 5 Creativity 88.647 1.8647 5 4 Style 80.343 1.0343 4 4 Mechanics 81.919 1.1919 4 5 Organization 92.523 2.2523 5

The score of the random computer-generated essay:

6 Overall 100 3.2384 6 5 Content 88.882 1.8882 5 6 Creativity 99.973 2.9973 6 6 Style 100 3.4543 6 6 Mechanics 100 3.6429 6 6 Organization 100 3.4075 6

Your post, though interesting, is, by its own standards, inferior to a randomly generated essay, which apparently approached perfection itself!

Looks like you didn't deserve that "+5" interesting after all. If only we used such software to moderate Slashdot comments as well as to grade papers!

By the way, this comment is only scoring a 2 overall. Maybe I shouldn't hit submi...

Re:You can't see the forest for the trees by clifyt · 2004-05-21 00:15 · Score: 1

I was mentioning to friends just last night that we actually used something similar to the PostModern Generator to create texts that are rated in a test environment to see where things could be improved.

Sadly, the text in PMG is actually pretty good. We aren't grading for content. At least in this model. It is grading for correct sentence structures. It also uprates bigger value words, unfortunately. PMG uses its share of twenty dollar words.

Again, this is a 10 year old model. I explained in detail what it rated for. Anyone that read my description and tried to understand what I was saying as opposed to just making an ignorant point to prove they were more right than those that have been doing this for a good deal of the responders life. Of course fucking PMG is going to be better than 90% of most incoming high school students in the fact that I'm sure all words were spelled correctly, and one has to understand correct sentence structures and otherwise or the whole snotty bit of the PMG gets discarded. Who ever wrote that program knew what they were doing and had a firm grasp of the English language.

I'm sorry if this sounds defensive, but I get a little irked when people don't read my warning and explanations of what this looks at and then puts far more advanced text in there and claims its bad because they need to prove their worth. Almost any college grad that didn't sleep though their classes could write a 5 minute impromptu essay that maxes this out, easily. I made that obvious in my original comment.

It needs just one more feature... by Log+from+Blammo · 2004-05-20 15:22 · Score: 1

This program could also be linked to a web-searching robot for fact-checking and proper attributions. It could detect plagiarism within seconds, and generate a probability that any given statement is correct.

And then it could be run on Slashdot in lieu of metamoderation.

--
"This quote is a product of the Frobozz Magic Quote Company."

Quoth the computer... by Anonymous Coward · 2004-05-20 18:09 · Score: 0

Actual output:

5 replies beneath your current to get a long boots.

For essays consisted in the stupid not actuational is was the same learn to be paragraphs, and you hands of work the idea of past Journal: Wednesday a process. Perfect in high schools. We've bet I could summar, do instead of an "A" blows and I just a higher 90% of how a comments, how how the this

standardized tests by formfeed · 2004-05-20 19:28 · Score: 1

That's the problem with all standardized tests. Instead of testing the actual subject, the tests check for certain "markers" that are correlated to skills in the subject area. People with good English skills most likely have a larger vocabulary than people with poor English, people good at math might be able to solve weird math puzzles in short time... You get the picture.

What happens next, is that people are getting trained for the markers. Especially bad, if this replaces learning the original subject. Learning stupid word lists doesn't replace writing and reading; and cramming past tests instead of trying to understand it might be the best way to deal with these tests, but you certainly don't get anything out of it.

Computer graded essays make this situation even worse, the content doesn't matter anymore, as long as the essay sounds intelligent.

Interesting by Anonymous Coward · 2004-05-20 19:39 · Score: 0

Check out what's touted as "good writing."

Try e-rater

To see how e-rater works, select a sample essay on a demo topic and submit it to e-rater for scoring. Several seconds later, you will see the e-rater score assigned to that essay, along with descriptive text associated with that score point.

If you would like to try out e-rater, you can obtain an ID and password and submit an original essay for scoring on the CriterionSM Web site.
Using e-rater

The following essays are samples that reflect various scores on a six-point scale (6 is the highest score; 1 is the lowest). If you wish, you may submit these essays to e-rater for scoring on the Criterion website to see whether e-rater 's score reflects your own judgment. These essays are reproduced exactly as submitted by the writers. They are randomly presented and are not in score point order.

Essay Topic: Often in life we experience a conflict in choosing between something we want to do and something we feel we should do.

In your opinion, are there any circumstances in which it is better for people to do what they want to do rather than what they feel they should do? Support your position with evidence from your own experience or your observations of other people.

Essay Sample A:

In every situation, people have a choice to make. Sometimes, one chooses what they want to do and other times they choose what they feel they should do. In most circumstances, it is better for the individual to put away their wants, and do what they feel should be done.

Going to college may not be what some want to do, but it is definatly what they should do. It is a serious choice to make. The decision they make is vital and will affect the individual's life forever. The most important thing to remember in this situation is that, what one wants now, may not be what they want later in life. By going to college they can make their decision later. The person would have an education to fall back on and have more of an ability to get a good job. If one chooses not to continue their education because of what they want, they may be faced with difficulties down the road.

A choice many teenagers are faced with is whether to listen to their parents. Teenagers are filled with things they want. The problem is that those wants and desires change daily. A smart individual will set priorities for themselves and do what they feel they should. many parents try to guide their children in the right direction. Although most know that they should listen to their parents advice, they don't want to. In the long-run it would benefit them to set their desires aside and do what they feel they should.

It is a tough decision to make; what you want to do and what you should do. The decision is left up to the individual but in most circumstances it is better to do what should be done.

Essay Sample B:

Many people are different because they are raised however their parents want to, but in my occation my parents raised me to respect other's especially your elders. This experience I had had to do with and old lady, she wanted to cross the street but she could not get down to the street because the concrete was to thick and she could not step down.

She had some trouble, especially since she was carring her big bag that old people carry, I looked at some people that past right by her and didn't even pay attention to her and since I was there I decided to help her even though I didn't want to, but I felt that I should because that was the way I was raised.

So I went up to her and asked her if she needed some help, she said yes, took her hand to help her down and walked her across the street after all doing this felt good even though I didn't want but I did. In that occasion, I know that many people should of done it but they didn't want to, why because probably they didn't care, but in my opinion peo

Slashdot Mirror

Indiana First With Computerized Grading

524 comments