Automated System Developed To Grade Student Essays

RougeFemme points out this story at the Times about software that can be used to grade student essays and offer almost instant feedback. "Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the 'send' button when you are done and receiving a grade back instantly, your essay scored by a software program. And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade. EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."

Comments

This is horrid

[swm]

One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.
- go to the site
- get a problem
- solve the problem
- type in the numerical answer
- right answer? go on to the next problem
- wrong answer? try again
The web app allowed maybe 0.5% margin for rounding error, and you got 5 tries before it failed you on that problem.

It sounds reasonable in the abstract, but in practice it was utterly wretched.
All learning is, at some level, an interaction--a conversation--between student and teacher.
Even if it is nothing more than a red check mark or a red X on a homework paper,
you have communicated some thing to some person and gotten some response.
You don't realize how important this is until it is gone.

With nothing but a machine to talk to, it stops being about learning.
It is just about satisfying the machine by whatever means necessary.
In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google.
This would reliably return the general formula for solving that problem;
plugging in the numbers that the web app had generated for your instance of the problem would then yield the correct answer.
By the end of the school year, I was telling him that if he didn't want to deal with the web app, he should use google to get his grade,
and if he wanted to learn physics, I would teach it to him.

Automated essay grading is going to be even worse.
There is no point writing prose unless a human is going to read it.
When I want to talk to machines, I write code.

Writing songs, that voices never shared...
-- Paul Simon

Re:This is horrid

[Anonymous Coward]

I went through the same system and it taught me all sorts of useful things unrelated to my actual physics curriculum, like
1/2 != 2/4
0.5 != 1/2
x != x+1-1
x^2 != x*x

Re:This is horrid

[reve_etrange]

I had the same experience in university calculus and physics. Even for problems with one right answer, there are typically many (even infinite) ways of expressing that answer. Even something as advanced as Mathematica or Maple can be fooled, and the websites in question are no Mathematica.

Re:This is horrid

[Anonymous Coward]

yeah, been there with my HS kid at university courses. Profs outsource the homework to... the publisher of the BOOK! (It's another story about specialized university editions that kill the used textbook market) Grad student TA's don't even have to do anything anymore. I remember one particular instance of her typing in an answer of 123,456 it was the correct solution... but the computer saw it as two numerical entries eg 123 and 456 after a half hour I told her to fork on it.... cmon, even I could parse that much 20 years ago in high school. At the end of the course there was a note from the TA that said "were told not to worry about our TA assignments and focus on our own studies"

This model is horrid, a completely wrong answer gives the same feed back as one that is rounded improperly.

I remember another instance when trying to get the teachers up to speed with online portion of the new math curriculum at the high shcool (new text books get adopted every so many years)... I read all the manuals and things were still not working properly, so I actually called... Support agreed quickly that there was a problem... the first thing they asked for was the usernames and PASSWORDS for the teacher accounts involved. Refused any help when I didn't provide account passwords.

Follow the dollars in the end, universities are big businesses as well. One has to wonder what their purpose will be in a short while if online education (or more likely qualification) really takes hold.

Re:This is horrid

[sm284614]

There's nothing wrong with using automated marking where it's actually saving time, as long as the results are used to further learning. The simple idea (employed by the Khan academy and others) is to get students/pupils to practise at home, a system tracks what they can do and what they struggle with, and the teacher-pupil/student-professor contact time is used to address any issues. Grading simple, repetitive tasks is a waste of time if it can be automated; it's the feedback that matters. I assume that this paper marking thing is able to generate reasons for the grades it gives, and some of that is probably also a little useful.

Re:This is horrid

[RougeFemme]

I'm currently tutoring my daughter in statistics for the same reason. She's in college and while she's flipping through her homework appliication and her e-textbook, I'm flipping through my old statistics books, plus a couple of study guides I picked up. Also, sometimes the homework application is simply wrong. (Doesn't every tool/program have at least one bug?) My sister, a teacher, uses one - mandated by the community college where she teaches. Occasionally, she has to override the application so that she can mark correct problems that the application marked wrong. The students alert her, she checks and then overrides when the application is clearly wrong.

Re:This is horrid

[Anonymous Coward]

I agree on the essay grading; I haven't seen anything suggesting that real NLP is applied to actually ensure the essay even makes sense, which is a rather low bar for grading an essay. On the other hand, as you say, it seems like computer grading should work for physics.

As a student, the system you describe sounds terrible. As a researcher vaguely involved in those technologies (on the problem generation side, not the UI side, although they certainly interact), I am wondering how those problems can be solved. Obviously, the system you describe didn't teach much (I guess your son had to know enough to figure out how to plug in the numbers), so it failed at its intended purpose.

You suggest the problem is the lack of communication. I assume there is no mechanism for partial credit or otherwise identifying where the student went wrong due to the student only being given a chance to enter their final solution. Along with being able to grade answers where a student shows their work, there is work on modeling incorrect variants of the correct procedure in order to be able to figure out which incorrect procedure the student followed (which in physics/chemistry would probably often including rounding to different amounts in different places, which may or may not be considered incorrect), which then can be used to give proper feedback to the student either in the form of an explanation of why their answer is different from the correct answer or in the form of a simpler problem for them to solve that isolates the error.

Of course, giving useful feedback is quite different from grading understanding. The 5 retries system you mention sounds like it was designed to ensure the student would try 3 or 4 times at most before cheating like you describe, but zero retries seems overly strict for grading (and infinite retries makes no sense for grading). The real solution is to not grade homeworks on correctness because that's silly; give the students tests in class to find out if they actually understand the material: homework is for learning the material.

Re:This is horrid

[icebike]

One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.

Something like what you described is indeed no more than filling out yet another web page, hoping to win a prize.

That doesn't mean that grading compositions would be the same, since by their very nature they have to accept human language in and assign a grade to it by some hocus pocus. The game becomes figuring out the limits and parameters of said hocus pocus. Allowing them to revise their paper is a way of training them on the dimensions of said hocus pocus, and probably not all that useful to either the student or the system.

I suspect that after a couple hundred thousand compositions are graded, everything looks like plagiarism. And I suspect that is part of the process here.

Re:This is horrid

[Anonymous Coward]

Even something as advanced as Mathematica or Maple can be fooled, and the websites in question are no Mathematica.

Which is really just pure laziness on the part of the designers of the system. They could have used Maxima which is probably about as good (or at least, not a lot worse than) Mathematica or Maple for simplifying expressions and is open source.

Re:This is horrid

[sgt scrub]

That sounds like he has a pretty good teacher. If it were me, I'd be fine with the link Google produced. :P

Re:This is horrid

[TapeCutter]

Maths is all about simplification. Examples 1, 3 & 4 are poor answers because they haven't been fully "simplyfied" (which is more than just a character count), example 2 doesn't have enough context but it could be "correct" in the context of a lesson about decimal points. OTOH I doubt that was the logic behind the software's answer.

Re:This is horrid

[Anonymous Coward]

These have gotten better over time - WebAssign can now handle symbolic answers and manipulate numerical answers into canonical forms for comparison. Depending on the level, students are often expected to express answers in simplest form, so the grandparent's 2/4 as 1/2 and x+1-1 as x would not be acceptable. Similarly, if asked for a polynomial, x^2 and x*x really would be different things. There are great horror stories for math/science implementations, but the worst of them were from early versions from a decade ago. Similarly, this solution may not be great initially, but probably comparable to AP rubric grading where you check for whether certain items are included, rather than a holistic quality. Give it another decade and this could be a great system.

Re:This is horrid

[Anonymous Coward]

Is this really unique to technology though? Even if you have a 99.9% accuracy rate in grading, you are likely to miss mark one question on one student's paper per class each week or so.

Re:This is horrid

[TsuruchiBrian]

In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google. This would reliably return the general formula for solving that problem

Isn't that what physics is? Applying the right formula? I don't think much is gained by having students memorize formulas anyway. In real life you just look up general formulas on google/wikipedia/wolfram and apply them to your specific problem. I suppose it's useful to be able to derive certain formulas as a method to gain greater insight into patterns and ways of thinking, but I don't think this depth is commonly required in high school.

If it were my kid, I'd rather he knew how to google formulas and apply them to specific problems than have a bunch of formulas memorized. People naturally memorize things if they use them enough. Rather than having kids memorize specific formulas, I think it's more useful that they memorize the best way to find specific formulas (i.e. google, wolfram, etc). That's much more efficient.

I have a bunch of physics formulas memorized because I used them a lot. The ones I don't use a lot I just look up when they are needed. If I forget one, it's not a big deal. If I forget how to look up formulas (not sure how that would happen) I would be screwed.

Re:This is horrid

[TsuruchiBrian]

Math is about simplification, but simplicity is subjective.

1/2 is simpler than 2/4, but not if you have something like this: 2/4 * a + 3/4 * b + 1/4 * c

maybe all this would be simpler as: (2a + 3b + c)/ 4 or maybe not it depends on the application...

Re:This is horrid

[EvanED]

When I want to talk to machines, I write code.

I've thought a bit about how I would like to grade a typical upper-division project-based CS course, like your typical OS class, where there are assignments of the form "okay, now go program such-and-such". (As opposed to theory courses, or more open-ended project classes where the students can pick what they want to do or different people do different projects.)

What I'd kind of like to do is have a submission server which would do three things that each contribute to the final grade. First: before the submission deadline, people could submit at any point. It would build the project and run a set of tests which was released to the students, reporting full results and that portion of the grade. (Students could submit multiple times without penalty, and it would use either the final or highest submission. Not sure which.) Second: before the submission deadline, it would do the same thing with a set of hidden tests, except not report information about what the tests were, just giving a number or letter grade. If students can't figure out what they're doing wrong, they could come talk to me. Third: after the submission deadline closes, it would run another set of secret tests to get the third component of the grade, and report it to the students at that time. In addition, there would be a fourth portion of the grade which was not automated. I'm not totally sure what the mix would be between them.

(I've also thought about making students submit enough test cases to get good coverage on either their or my solution or both, which would be a fifth category.)

Any thoughts?

Re:This is horrid

[Anonymous Coward]

Who even grades homework?
All my teachers ever did was check if you'd actually done it, if that.
Maybe they'd have you write one of the problems on the board.
Homework is there to help you.

Re:This is horrid

[__aaltlg1547]

Isn't that what physics is? Applying the right formula? I don't think much is gained by having students memorize formulas anyway. In real life you just look up general formulas on google/wikipedia/wolfram and apply them to your specific problem. I suppose it's useful to be able to derive certain formulas as a method to gain greater insight into patterns and ways of thinking, but I don't think this depth is commonly required in high school.

If it were my kid, I'd rather he knew how to google formulas and apply them to specific problems than have a bunch of formulas memorized. People naturally memorize things if they use them enough. Rather than having kids memorize specific formulas, I think it's more useful that they memorize the best way to find specific formulas (i.e. google, wolfram, etc). That's much more efficient.

I have a bunch of physics formulas memorized because I used them a lot. The ones I don't use a lot I just look up when they are needed. If I forget one, it's not a big deal. If I forget how to look up formulas (not sure how that would happen) I would be screwed.

The purpose of EDUCATION is to teach the student how to understand and think about the material.

Re:This is horrid

[Scarred Intellect]

I've got that for Chemistry AND Physics. It's horrible. I've entirely missed problems because I couldn't figure out how it wanted the answer represented.

I mirror your sentiment exactly:

The teacher couldn't be bothered to assign and grade proper homework.

The kicker is that teachers who DO assign real homework have TA's or graders to grade the homework, all they actually grade are our tests.

Oh yea, and I have to pay extra for this online bullshit. It's required in a more in-depth way than textbooks (I literally can't pass the course without it) but isn't paid by the school, it's paid by me alongside my already-ridiculously overpriced textbook.

Yes, I have to pay for the teacher to be lazy.

Re:This is horrid

[kermidge]

After reading over thirty of the posts, many worthy, I return to my first conclusion after reading yours: you said enough to cover the absurdity of what eDx is trying to do. The only possible useful thing I could see for it at first blush would be to act as a rough filter for very basic errors - and it'd still be making mistakes. At a time when there are plenty of sufficiently literate people un- or under-employed, let them grade essays at least as a first filter. One needn't be an expert on a subject to decide if an essay is at least readable, somewhat self-consistent, and maybe even find glaring logical flaws.

Yet I have to give 'em props for trying to do the presently impossible.

Re:This is horrid

[commodore73]

Totally different. My college essays could have been much better if they had allowed me to type.

Re:This is horrid

[TsuruchiBrian]

That's not the ultimate purpose. The ultimate purpose is to solve problems. Having a useful way to think about problems is a good tool for solving them. Afterall, that is the way we judge whether a particular way of thinking is valid or not, or whether someone really understands something. We judge understanding of a problem by whether someone can solve a similar problem. We judge whether a "way of thinking" is correct by whether it leads to correct solutions to problems.

Re:This is horrid

[Anonymous Coward]

Automated grading for essays won't be worse; it already is worse. It exists now and it is easily the worst invention in the field of education. There are far too many ways to gang the system. Furthermore, the "artificial intelligence" looks for structure and not content. This encourages the student to write using only one type of sentence and discourages the idea that people can write just as well as one another without writing in the exact same format.

Re:This is horrid

[Anonymous Coward]

Simplified isn't subjective. The first one would be the correct answer as the last one requires an additional step. Although 2/4 is never going to be acceptable for full credit in any class I've taken.

Plus, it's unlikely that you would have all those variables in an answer that's being graded by a computer. By the time you're dealing with that, you have a lot more than just 1 answer to worry about.

Re:This is horrid

[Anonymous Coward]

Grading simple, repetitive tasks is a waste of time if it can be automated; it's the feedback that matters.

Maybe it is assigning simple repetitive tasks that is the waste of time.

Re:This is horrid

[PopeRatzo]

I graded undergrad essays of the children of privilege for well over a decade before progressing to their graduate essays.

Overall, I don't think automated grading is going to make a difference. Since most of the students don't care, I can't imagine there's much harm in having a grader who doesn't care. The notion of "caring" in higher education has been made obsolete, along with live lecturers. And I don't really blame the students gpt mpy vstomh, because under the stress of the huge debt that our system of higher education places on them, I can understand why they'd just want to move past these essays and onto the more socially important business of making money to pay their loans. I'm just glad I retired before the rise of the industrialized, for-profit schools.

The tiny percentage that did care was enough to motivate me, but motivation won't be a problem for the grading machine. And it's unlikely that the automaton will try to diddle the co-eds.

Re:This is horrid

[Opportunist]

Your son learned something important: Rote learning is a thing of the past, and being able to know where to find the answer is more important than having it, and finding someone else to do the work for him is easier than doing it himself.

He's going to do great in a management position.

Re:This is horrid

[Pulzar]

That's not the ultimate purpose. The ultimate purpose is to solve problems.

The *ultimate* purpose of education in science is to solve problems we have no current solutions for. They are not solved by looking up the formula, but by developing your own formula based on your understanding of how things work.

I don't need to look up the formula that allows me to calculate the acceleration of a body of known mass when known force is applied to it, because I understand their relationship. I also understand the relationship between velocity, time, and acceleration, so I can create further formulas based on these two sets of relationships that might've not been obvious at first.

If I've just looked up the final formula, I've skipped the important steps that give me the underlying understanding of physics, which will allow me to create new formulas to solve new problems.

Re:This is horrid

[_KiTA_]

I think that particular app, which I forget the name of, is very popular nowadays in Intro Physics classes. It's outrageously annoying. Hilariously, all the answers and solutions guide are available via a quick google search, something my classmates were quick to tell the rest of us.

The scary thing is, the people who make these books, who re-release the book every 6 months so you can't buy them used, are trying to push for the same kind of automation... of lectures. You'll have a "teacher," but they won't be an expert in the field. They'll have some basic IT training to ensure you can get to the website. The website will have a variety of web videos and automated homework assignments, and you will be graded based on how many videos you watch and how many automated assignments you complete.

And with the push to for profit schools... yeah. Expect McEducation to show up sooner rather than later.

Re:This is horrid

[Jstlook]

I went through that type of system for a Chemistry class. After class, the entire class would wander down to the computer lab and do the homework together. We'd get a question, find the [book] answer, then have each person try to obtain the correct [computer-identified] solution on the first try, trying various syntax adjustments each time. Of the three chances we got, someone usually got the right syntax before everyone had failed the question the second time.

Best benefit? Getting a group of people in the same place to research, debate, and agree on a single answer, then be open-minded and organized enough to shape the solution to fit the constraints given.

Re:This is horrid

[Anonymous Coward]

This assumes that universities are about teaching and real learning these days.

Newsflash: It, like everything else these days, is a business. And this is the business of the hamster wheel. Students in, frustrated, tired and empty-pocketed students out. Oh, and student opinion doesn't count. No matter how bad materials, professors, and overall degree quality gets, those impacted the most can do absolutely nothing about it (try transferring as a 3rd year CS without incurring an additional year of school, or more debt). Because universities know that there will always be another batch of idiots ready to pay to jump on the hamster wheel next Fall, and they know that students are in many cases locked in investors -this isn't going to change.

Re:This is horrid

[Anonymous Coward]

Couldn't your son discuss the problem with classmates, teaching assistants, tutors, or the professor during his office hours after the first incorrect answer, prior to attempting it again?

At least this (sounds like CAPA (Computer Assisted Personalized Approach) or LON-CAPA (Learning online CAPA)) has students trading equations back and forth to cheat instead of the same numerical answers.

Re:This is horrid

[Anonymous Coward]

Typically, these problems are addressed at the beginning of the course, where the format for acceptable answers is extensively discussed. It's normally trivial to put the answer in the desired format.

Re:This is horrid

[Anonymous Coward]

Also depends on what you are trying to communicate. If it's "pure math" then (2a + 3b + c)/4 might be simpler, but if it's a problem where you have to.. say find how much components a,b, and c you need for exactly one unit of something else the first one would be way better as an answer. Ok, bad example, but you get the point.

Re:This is horrid

[Anonymous Coward]

You are probably thinking of WebAssign, which many textbooks reference.

If an instructor uses CAPA or LON-CAPA, they make up their own problems, and the online solution guides would be less valuable, but still valuable if you look up both the corresponding question and solution from other textbooks presenting similar material (treating it like a form of Schaum's guides, which would be more useful, as they show the derivation of the solution).

Re:This is horrid

[tlhIngan]

My discrete math course did that - it had a bunch of web pages and tools you need to use to solve the problems. They never worked right at all.

OTOH, I can see the system being used in those massively online open courses (MOOCs) where a lecture can admit easily 3000+ or more students in a term. This seems like a way for them to have homework in the course and to do tests so it closes the loop. The prof then gets back a summary of how many people get a question wrong, etc.

So it has its uses and if we're going to embrace online learning, we've got to embrace automated testing and evaluation as well, because evaluating 3000+ answers and essays is no longer feasible. And I'm sure the students doing these MOOCs would like to know how well they do as well - while they may not be paying too much money, they are investing a lot of time.

Re:This is horrid

[tbird81]

Repetition is how we learn. We learn by repeating things more often. We tend to remember things that repeated better, and repeating an action tends to get it into our brain's more permanent memory.

Re:This is horrid

[Anonymous Coward]

- go to the site
- get a problem
- solve the problem
- type in the numerical answer

Just from this, I can already see that it probably gives students the types of problems that require little to no thought to solve. Simple and easy to generate problems. Problems that test for rote memorization at best.

Re:This is horrid

[Anonymous Coward]

Repetition is how we learn.

But how much a person has to repeat something before they 'learn' it depends on the individual; there's no one-size-fits-all number. Don't force me to complete nonsensical repetitive busywork if I already understand the material (Oh, but that wouldn't fit into their one-size-fits-all approach to education!).

Furthermore, this is exactly the problem I have with so many schools: too much useless busy work which forces students to answer problems that only test their memories, and not their understanding of the material. I doubt a student would come to understand why the Pythagorean theorem (as a simple example) works if all they're doing is answering countless problems that require them to use it but require nothing other than the memorization of a certain procedure to solve. Too often we focus on mere repetition to help people memorize things, but focusing on why things work can not only make the material much more memorable (because it becomes more than simply memorizing facts and procedures), but it also helps people solve more complex problems that don't just test for whether or not someone memorized something.

Re:This is horrid

[cheekyjohnson]

Repetition is how we learn.

Really? I tend to forget things I don't care about even if I try to drill the information into my brain; it's a temporary solution for me at best. However, things that I actually care about will simply be retained in my memory in no time at all, and I never give myself any silly repetitive assignments.

Maybe we should be focusing on making things more meaningful to people?

Re:This is horrid

[Anonymous Coward]

I'm not sure what you meant, but
0.5 is most certainly not equal to 1/2. The latter is mathematically exact value, while 0.5 is not exact. E.g. 0.51 could become 0.5 when rounded.

regarding the difference between 1/2 and 2/4, I would accept only first one as an answer in a math test. While latter is also correct, it can be simplified. If you don't require the most simple form as an answer, then the answer to 5+4 could as well be 5+4.

Re:This is horrid

[Anonymous Coward]

Repetition is how we learn.

No, learning is about understanding, not about being able to repeat what the teacher said.
Being able to list important dates in the past is not the same as understanding why the events occurred.
Teachers tend to grade students after what can be repeated because that is easier to measure, but that doesn't mean that they are measuring learning.

Re:This is horrid

[oreaq]

Simplified isn't subjective.

What is the objective defintion of "simplified"?

Re:This is horrid

[Anonymous Coward]

There isn't, see this recent question on MathOverflow (a Q&A site for professional mathematicians): http://mathoverflow.net/questions/126519/is-there-a-mathematical-definition-of-simplify

Re:This is horrid

[gtall]

"All learning is, at some level, an interaction--a conversation--between student and teacher."

I don't believe this, maybe it is because I always preferred independent courses where I rarely saw the teacher. I think teachers can at best organize and properly thread the material, but they cannot "teach" you in the sense of making you understand. Understanding a deeply personal notion, you have to work out the relationships for yourself in a manner you can grok. That cannot be "taught" because no one else has quite your perspective.

I vigorously despise on-line teaching tools, oddly enough. They are brittle, prone to direct you as only a machine can. This latter is the problem. If your approach is outside the program, your are SOL. With a teacher, you can some patient (hopefully) interaction that may very well validate your approach to a problem or at least show you things you should also consider. I would think the problem would only get worse for English programs rather than something more formal like math. Physics to me is between Math and English, it is very connected to the world in a way math and machines (in the abstract sense) are not.

Re:This is horrid

[gtall]

Math isn't about simplification and simplification is relative rather than subjective. Math doesn't really give a rat's ass about simplification, it cannot because there is no objective standard, it depends on the form of the solution you need. Learn Laplace transforms sometime. Some problems get simple depending upon which domain you express them in, however if you need the solution in other domain for other considerations, then you learn to live with the complexity and math still doesn't give a rat's ass which form you need.

Re:This is horrid

[gtall]

We must have different real lives. In the Universe in which I live, very few problems can be simply looked up for formulaic solutions. The real world is kind of like that, complicated. And if physics is just applying formulas, could you please tell the physicists that so they can quit their jobs and start some gainful employment somewhere?

Re:This is horrid

[dave420]

Homework is never a discussion between the student and the teacher, unless the student in question is home-schooled. If the teacher was using this in class in lieu of lessons, you'd have a point. Homework and examinations are not about interacting with the teacher, but reviewing what's been learned in the lessons, and evaluating the level of understanding of the student on the subject at hand.

Re:This is horrid

[Anonymous Coward]

I can absolutely, positively, 100% guarantee that I will never, ever use an automated system to grade essays. If you're so lazy that you won't do your job to evaluate and provide useful feedback on a student's essay, including the possibility that they've figured out another way to answer the question that you hadn't thought of (but which is correct), then what the hell are you doing in a teaching position?

This program is so obviously a disaster waiting to happen that I don't understand why people would even try to to use it. Even if it gives valid feedback, it's going to be pretty basic. For something structured like math, physics or chemistry, maybe you can get something useful with the occasional incorrect evaluation that will require human intervention. For free-form language? It's going to be a joke from the start. It's snake oil. This system will carefully teach the students how to game the program instead of writing better essays that make sense.

Re:This is horrid

[Anonymous Coward]

Yeah, maybe in a math class, but a physics class it shouldn't matter that much. The physics profs I had just wanted to see that you understood the concepts and applied the correct formulas (within reason of course).

The problem with that online physics coursework (its probably just one website that a large percentage of the courses all use) is that you can't demonstrate your thought process and show your work... So you either get it right or get it wrong - no in between.

Re:This is horrid

[moeinvt]

Are you sure that was written by Paul Simon? I've never heard it, but it looks like it would be terrible to sing. None of the verses even rhyme, some are ridiculously long compared to others and there's no chorus.

Re:This is horrid

[jalopezp]

0.5 is most certainly equal to 1/2. What are you, a physicist?

Re:This is horrid

[nedlohs]

You didn't have such a submission system? The one we used was, and I'm not kidding, written in the 60s: http://www.cs.tufts.edu/~nr/comp150fp/archive/jb-hext/hext-automatic-grading.pdf

Obviously it's had a bunch of changes since then, but it did basically what you describe. I see it is still being used:

; givelist
Name Sufx Category Until Cse Description

COMP3520_Assignment1_2013 .tar Ontime Fri Apr 26 23:59 Assignment 1
INFO5990_AssignmentOne .doc ontime Tue Apr 9 18:00 Article Precis
comp5114.project_p1 .zip OnTime Thu Apr 18 17:00 Project-Part 1 (COMP5114, Semester 1, 2013)
comp5114.project_p2 .zip OnTime Thu May 30 17:00 Project-Part 2 (COMP5114, Semester 1, 2013)
comp5425.part1 .zip OnTime Wed Apr 17 17:00 Part 1 of COMP5425 Assignment, 2013S1
comp5425.part2 .zip OnTime Wed May 29 17:00 Part 2 of COMP5425 Assignment, 2013S1
comp5426_Assignment1_2013 .tar ontime Fri Apr 19 23:59 COMP5426 Assignment 1 - Parallel Collapse Sets of Integers
; submit f.tar COMP3520_Assignment1_2013

Name: COMP3520_Assignment1_2013
File suffix: .tar
Description: Assignment 1

Due dates and their categories are:

                Currently -> Ontime Fri Apr 26 23:59
                                          Late Mon Apr 29 09:00

Is this the assignment you wish to submit ? [yn default y] n
submit: Submission has been cancelled at your request
;

Hopefully they've made some updates in the last decade since we playing with the submit was a common past time when I was using it as a student (and I assume for my students when I was using it while teaching). When you have a system that takes a programming assignment, builds it, and runs it that's not unsurprisingly easy. "I wonder if I could just do system("(whoami;pwd;ls -l) | mail someaddress"); in my program" being the fist obvious thing. "I wonder if I could fork a child and have it make a tcp connection to X:Y and hook a shell to it" being a quick second when an email arrives...

Re:This is horrid

[jemenake]

One of my kids had something like this: not for English, but for physics. The teacher couldn't be bothered to assign and grade proper homework. Instead, he fobbed the kids off onto a web app.

That's why they're called "teachers" and not "graders". If you want people who grade papers all day, then have the job listing say "graders wanted".

All learning is, at some level, an interaction--a conversation--between student and teacher. Even if it is nothing more than a red check mark or a red X on a homework paper, you have communicated some thing to some person and gotten some response. You don't realize how important this is until it is gone.

With nothing but a machine to talk to, it stops being about learning.

I disagree. I'm learning Spanish and I have apps which let me practice my conjugation. It gives me a verb, conjugation, and tense, and then I have to produce the proper conjugation. I can practice them for hours. That is much, *much* more time that I would dare ask a human to sit there and quiz/grade me.

It is just about satisfying the machine by whatever means necessary. In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google.

Correction: For your son, it stopped being about learning and became just about satisfying the machine. Apparently, he doesn't get that learning is a two-part process: 1) learning the "concept" (which requires either a teacher at a blackboard or reading it in a book or whatever), and 2) practicing it over and over in order to etch it into your brain until you really grok it. Now, your son's teacher really might suck ass at part #1 (lots of physics teachers do), but that would be a separate issue from their decision to automate part #2... or your son's (apparent) notion that the homework problems are not a whetstone upon which he is meant to hone his skills, but rather just a barrier between him and getting to go play on the XBox.

Automated essay grading is going to be even worse. There is no point writing prose unless a human is going to read it.

Possibly. But, if I were a grader for a composition class, I would want to slash my wrists every. goddamn. minute. I think back to the stuff I wrote in my essay classes in college and, internally, I'm horribly embarrassed. What absolute drivel... and I thought I was being all deep and insightful. And I was just one dude in a class of 35... and one class out of 3-4 that prof was teaching. My hunch is that, when the profs grade those, they actually do turn their brains into automaton mode and just flag punctuation and grammatical errors. They've probably gotten really good at just spotting common errors without even giving the text much attention from their higher brain functions. Y'know how you can drive to work not even remember stopping at all of the stop-lights and stop-signs on the way, because driving is so automatic that it's almost a reflex? My hunch is that that's what the profs are like when grading a mountain of poorly-developed blather from students. So, moving the grading to a computer isn't changing what the grading process is actually looking at; it's just moving the process to something which is better at it.

Re:This is horrid

[jameshofo]

ERRR: Unable to properly parse comment text, Please insert $500 to re-take this class.

Re:This is horrid

[Anonymous Coward]

Although 2/4 is never going to be acceptable for full credit in any class I've taken.

Well if it's physics then the problem is there are no labels, so it's wrong no matter how you express it.
And if the question is asking you how many quarters you have, 2/4 is correct while 1/2 is not, even though mathematically they are identical.
Or as was already stated, the context is what matters to determine which expression is the correct one- just being the most mathematically simplified is only the most correct when you've been asked to provide the most simplified answer.

Re:This is horrid

[Anonymous Coward]

The *ultimate* purpose of education in science is to solve problems we have no current solutions for

No. That is one purpose which usually only applies to research scientists. For most people, the purpose is to allow them to perform a job. As long as they have the tools to do the job, they don't need the understanding required to develop new tools.

They are not solved by looking up the formula, but by developing your own formula based on your understanding of how things work.

There really isn't much difference between looking a formula up from a book, and looking it up from your memory. You're still just looking it up.
The reason we have students memorize formulas is so that they can do the work, and the goal is for them to develop the understanding behind the formula while doing so. Most people won't, and those people turn out to be the rank-and-file workers. Some people do, and those people go on to be researchers and developers of new ideas and methods.

If you really have a good grasp on the underlying physics, you don't need to memorize any formulas- you can simply derive everything as needed. But that's incredibly rare. Most people who have a good understanding still memorize the formulas to save time and prevent repeating work of others.

Back on topic- the article is crap. Professors don't usually grade papers, that's done by teacher's assistants and grad students. Once you get to the higher levels of a study where a professor IS grading the papers/exams, they're operating at a level where this mickey-mouse kind of program simply can't hold up against a human.

Re:This is horrid

[Anonymous Coward]

Yes, I have to pay for the teacher to be lazy.

If you're still in high school, then tell your parents to take you out of whatever private school they're paying to send you to.
If you're in college, then you don't HAVE a teacher. You have a professor. His job is to profess, not to teach. The homework is there for your benefit, and frankly speaking the only grade which should even matter is your final exam. Sadly, many profs don't do this any more because there's so much pressure from the Administration to pass students, so homework scores are used to allow idiots to shore up their test scores with homework.

Re:This is horrid

[Anonymous Coward]

When I want to talk to machines, I write code.

As will the students who find a way to hack the grading software. Little Bobby Tables should love this.

Re:This is horrid

[Musc]

> Really? I tend to forget things I don't care about even if I try to drill the information into my brain;

But how do you know what you are going to care about later?
It is useful to be able to take in all of the information presented to you and learn it, just in case you need it later.
Of course, if you prefer to learn on the job instead of learning ahead of time, then you could just pick things up as you go and never learn anything you don't care about. For some reason this isn't standard, must be some reason for formal education that we aren't thinking of...

Re:This is horrid

[TsuruchiBrian]

Most people who "do physics" are not physicists. Most people who use phsyics are just applying already discovered formulas to specific problems (i.e. engineers), rather than discovering new formulas describing physical laws. In fact most people are using physics that was already discovered more than a century ago (e.g. Newtons laws, thermodynamics, etc).

Yes discovering new physics is important. But this is not what most people who do physics are doing. The person we are talking about is a high school student. Maybe I should have said "High school physics is just applying formulas". Maybe we need better terminology for the kinds of physics physicists "do" and the kind that engineers and high school physics students "do"

Re:This is horrid

[cheekyjohnson]

But how do you know what you are going to care about later?

I don't. My point was that drilling information into my brain isn't an effective strategy for me.

It is useful to be able to take in all of the information presented to you and learn it, just in case you need it later.

No thanks. Waste your time if you wish, but I go by the estimated probability that I'll need it later rather than a vague "just in case." Furthermore, I make an effort to understand it rather than just memorize it, which is pretty much what I was talking about there.

then you could just pick things up as you go and never learn anything you don't care about.

If someone truly doesn't care about something, I very highly doubt they'd remember for long. It seems like an exercise in futility to me.

must be some reason for formal education that we aren't thinking of...

One might come to your mind, but it would just be a colossal waste of time and money for me.

But none of this matters because I was only talking about rote memorization. Rarely does merely memorizing the material help someone for long, so I suggested that we make the material seem more meaningful (like exploring why something works, among other things) so that it will be more memorable.

I used to just memorize material way back when I was in public school. The result? I forgot over 90% of it. That, to me, is a waste of time. So no, completing repetitive assignments isn't necessarily helpful.

Re:This is horrid

[Anonymous Coward]

As long as they have the tools to do the job, they don't need the understanding required to develop new tools.

If you want them to be able to do anything that's even slightly complex and not spelled out for them, you want them to understand the material.

The reason we have students memorize formulas is so that they can do the work, and the goal is for them to develop the understanding behind the formula while doing so.

You do not need to memorize the formulas to understand them, and you do not need to memorize formulas to do the work unless the "work" is poorly thought out.

Most people won't, and those people turn out to be the rank-and-file workers.

With a teaching method like that, are you really surprised? Seriously, that's exactly the problem people are pointing out!

prevent repeating work of others.

It's not about repeating work; it's about understanding the damn formulas.

Re:This is horrid

[TsuruchiBrian]

Math is about simplification. I never claimed that there is a universal metric of simplicity. In fact I actually said the opposite in my post. There is on objective definition for simplicity that applies in specific cases which is this: If a particular formulation of a problem reveals the answer and another doesn't, then the former is simpler.

For example, doing gaussian elimination on a matrix is just reformulating a problem in different ways. When you have finished, you have a formulation which makes finding the answers to some questions trivial. Different problems have different "most simple" configurations, so there isn't a universal standard for simplicity, but for individual problems it is just a fact that certain configurations have answers that are more readily exploitable, in that they require fewer computations to arrive at a given answer.

Re:This is horrid

[TsuruchiBrian]

It's not about repeating work; it's about understanding the damn formulas.

So when you teach a High schools student about gravity and you give them the formula F = Gm1m2/r^2. What exactly are they supposed to understand beyond being able to apply this formula to determine the gravitational force between to objects?

There really *isn't* currently much of a deeper understanding to be had. The formula itself bestows pretty much all of the understanding humans currently have about gravity.

How would you fundamentally understand this particular formula any better than a kid who can google the formula, plug in the numbers, and get the right answer? I am not saying you can't, but there doesn't seem to be an obvious answer.

Besides, having a kid who can plug numbers into formulas and consistently get the right answer is already pretty good for the American school system. When you get to college you are still essentially doing the same thing, except plugging in numbers becomes mundane and figuring out which formula to use is a bit harder.

Re:This is horrid

[TsuruchiBrian]

Yes, and many of the unsolved physics problems are simple applications of existing formulas, that engineers need to be able to google and plug numbers into.

If you want to build a bridge or send a rocket into space, a bunch of physics needs to be done to make that possible. None of it will include coming up with new formulas describing yet unexplained physical laws.

99.9999% of the physics problems that are solved every day are easy (e.g. how much weight can this bridge support, how much fuel does this rocket need) . Why? Well it's because the hard problems are hard and they take longer (e.g. Resolve quantum mechanics and relativity).

Yes the hard physics is important. The hard physics of today will be the easy physics in 100 years. The ultimate goal of Newtons laws when Newton discovered them was to be able to do all the currently mundane applications of his laws that make rocket launches and massive suspension bridges possible.

It took hundreds of thousands of years of human civilization to get to figure out Newton's laws, and then it was relatively trivial to teach them to every future engineer. While it is exciting to discover the way the universe works, this knowledge is useless until you use it.

Re:This is horrid

[Anonymous Coward]

There really *isn't* currently much of a deeper understanding to be had.

We aren't even teaching people why the formulas work, and this is not just about physics at all. Even for something like your example, there is some understanding to be had.

And hell, you could say that about anything, not just physics. Downplaying the value of actually understanding the material is just disappointing to hear from a Slashdot user.

My TA had that 35 years ago

[gewalker]

Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.

Re:My TA had that 35 years ago

[mrchaotica]

Did he at least flip through to make sure there weren't a bunch of blank pages in the middle?

Re:My TA had that 35 years ago

[Anonymous Coward]

This page seditiously left blank.

Re:My TA had that 35 years ago

[Shadow of Eternity]

We already have that today, you can guess any of the AP/SAT/GRE essays' grades with phenomenal accuracy just by looking at how long they are:
http://www.nytimes.com/2005/05/04/education/04education.html?_r=0

Re:My TA had that 35 years ago

[Anonymous Coward]

Use thicker paper?

Re:My TA had that 35 years ago

[camperdave]

Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.

I guess that beats throwing them down a flight of ten stairs twice.

Re:My TA had that 35 years ago

[Anonymous Coward]

Hell, when I was a TA I knew of an even simpler method employed by some of my colleagues: give everyone a 100. It's fair and takes less time than measuring with a micrometer.

Alternatively, as actually happened in one case that I found out about after it was far too late to do anything (and, ironically, in an engineering ethics class), one of my fellow TAs didn't bother grading the final essays, instead just averaging each student's previous essay scores, ranking those averages, and then distributing scores between 90 and 100 along a bell curve on which those rankings sat. I was rather pissed when I found out about that one much later, though the students were glad from what he had said, since all of them received grades that were higher than what they had been averaging.

Re:My TA had that 35 years ago

[demonlapin]

That's interesting in its own way, but a much more interesting comparison would be between the essays' lengths and the respective SAT Verbal scores of their writers. I would bet that they are also correlated quite closely.

News flash: when presented with an essay topic, smart people spend a few minutes planning and then proceed to write voluminously about the subject, because they are fluent writers. Dumb people start muddling along, lose track of where they are, and stop when they've stated (though not proved) their main point, because they're not. Fun game: ask a room full of people to write nonstop for five minutes on any topic(s) of their choosing, then compare word counts vs IQ/class grades/whatever.

If you're a HS student reading this (and I imagine there are a lot of you who are): practice writing. Practice writing. Practice writing. It's important. It's probably the most valuable skill you will ever acquire for dealing with people you don't meet with face-to-face. Bad writing is universally considered a sign of low intelligence. It takes a lot to overcome the negative impression that bad writing gives, and you often will not have the opportunity to try - when given a stack of 100 resumes for two positions, guess how the initial winnowing occurs? Toss anything on colored paper, anything written in a funny typeface, and anything with grammatical or spelling errors. I cringe today when I read some of the stuff that I wrote in HS, but it's grammatical and correctly spelled, even if the verbiage is ponderous (and occasionally verges on purple prose).

Re:My TA had that 35 years ago

[DragonWriter]

News flash: when presented with an essay topic, smart people spend a few minutes planning and then proceed to write voluminously about the subject, because they are fluent writers. Dumb people start muddling along, lose track of where they are, and stop when they've stated (though not proved) their main point, because they're not.

IME, smart people write concisely and to the point of the prompt, while dumb people write voluminous, rambling, redundant, and unfocussed walls of text.

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." --Antoine de Saint-Exupery

Re:My TA had that 35 years ago

[Anonymous Coward]

The length thing always annoyed me when I was in school because, sure, I could type an however long essay in 30 minutes no problem, but there was no way I could hand write very much that quickly and standardized tests were one of the few times I actually hand-wrote anything.

Re:My TA had that 35 years ago

[thoromyr]

I use essay questions for tests and I teach in a technical field. My instructions note that a good answer is complete and succinct. Many students don't know the word "succinct" and end up thinking it means "short" or "brief". Most college students lack the language skills to be complete and succinct on a timed exam. Heck, most probably can't do it even if it were not timed. Consequently, good answers tend to also be long.

It isn't that length correlates to quality (you always get someone who thinks they can bury their ignorance in a pile of words), its that brevity is a strong indicator for a lack of quality. Significant sections of most answers could be deleted without detracting from the answer (it isn't succinct), but it takes skill and/or time to pare down the verbage. As noted, they lack the time and unless they have good skill to compensate succinctness just doesn't happen.

OTOH it /is/ a pain to grade. Much more time consuming than using traditional multiple choice exams.

With respect to your statement, it isn't that "smart people" write concisely, it is that skilled people with time and motivation to do so write concisely. The brevity of your post betrays the lack of quality in your contribution.

Re:My TA had that 35 years ago

[Anonymous Coward]

Ah yes- but we shan't forget that the essential is invisible to the eyes.

Re:My TA had that 35 years ago

[Anonymous Coward]

Just because someone is dyslexic and writes poorly doesn't mean they have a lower IQ than you.

Re:My TA had that 35 years ago

[Shadow of Eternity]

Fluency != length. I spent most of my time in college learning how to waffle my essays up to the minimum page count. If anything it's more likely that the more intelligent someone is, and more importantly the better of a writer they are, the more capable they will be of getting the point across in a smaller amount of writing.

Re:My TA had that 35 years ago

[Shadow of Eternity]

And the final sentence of your post betrays that you're so biased to believe that length DOES correlate directly to quality and intelligence that you started with a lack of quality as a foregone conclusion and worked backwards from there to justify it, playing semantics with the definition of "succinct" in the process.

Your second paragraph is particularly ironic given that you clearly state there's always "someone who thinks they can bury their ignorance in a pile of words" while doing exactly that. If brevity is a strong indicator for a lack of quality then by definition length correlates to quality. Your argument is as if I had tried to claim that size does not correlate to mass but then gone on to say that a lack of size is a strong indicator of a lack of mass... ie it STRONGLY CORRELATES with a lack of mass. The only difference being that you hid that inherent contradiction beneath a pile of words seperating the two halves, making it harder to immediately recognize.

P.s. Succinct: "Briefly and clearly expressed."

Re:My TA had that 35 years ago

[demonlapin]

A tiny fraction of people who write short essays are capable of producing succinct prose on the first edit in 25 minutes from concept to completion. But most people who write short essays in 25 minutes do so because they can't think of anything to write.

Re:My TA had that 35 years ago

[demonlapin]

These are tests in which students have 25 minutes from start to finish. Sure, there are some 17-year-olds who can write perfectly succinct prose on a first edit in that time frame, but the vast majority of short answers are the result of running out of ideas.

Re:My TA had that 35 years ago

[demonlapin]

No, but they're pretty well correlated. There are exceptions to every rule, but that doesn't make the rule generally wrong.

Re:My TA had that 35 years ago

[DragonWriter]

I use essay questions for tests and I teach in a technical field. My instructions note that a good answer is complete and succinct. Many students don't know the word "succinct" and end up thinking it means "short" or "brief". Most college students lack the language skills to be complete and succinct on a timed exam.

To the extent that's true, its because most instructors, despite expressing a desire for succinctness, don't create sufficient incentives for students developing these skills. Probably the best thing that ever happened to my writing are the instructors I had who set what seemed, at first, very short length limits on essays/papers and strictly ignored anything beyond the limit (even if it was reached mid-sentence), while grading just as stricly on completeness as other instructors. (Though experience in competitive improptu and extemporaneous speech, and team and individual debate, certainly also helped.) Saying succinctness is desired isn't enough..

But, even so, succinctness isn't binary; even while there may be some general lack of skill in honing responses down to the minimum necessary, smarter people are going to, all other things being equal, produce more succinct responses, with fewer non-sequiturs, non-communicative self-contradictory constructions, etc.

It isn't that length correlates to quality (you always get someone who thinks they can bury their ignorance in a pile of words), its that brevity is a strong indicator for a lack of quality.

"Brevity is a strong indicator of lack of quality" means the same thing as "length correlates to quality", so this is self-contradictory.

Significant sections of most answers could be deleted without detracting from the answer (it isn't succinct), but it takes skill and/or time to pare down the verbage.

That this is true of "most answers" does not suppor the assertion that "smart people [...] write voluminously on the subject", rather, it supports the contrary argument that relatively unskilled writers tend to write voluminously, due to the verbiage you reference.

OTOH it /is/ a pain to grade. Much more time consuming than using traditional multiple choice exams.

How hard it is to grade is a non-sequitur, but, in any case, multiple choice eams aren't any more traditional than essays.

With respect to your statement, it isn't that "smart people" write concisely, it is that skilled people with time and motivation to do so write concisely.

It is both, both because clarity of thought itself contributes to conciseness, and because smart people acquire skills faster making "smart" and "skilled" are correlated here.

The brevity of your post betrays the lack of quality in your contribution.

Conversely, the length of your post is due to the things that detract from its quality: non-sequiturs, meaningless self-contradictions, etc.

Re:My TA had that 35 years ago

[thoromyr]

Your lack of reading comprehension is amazing.

""Brevity is a strong indicator of lack of quality" means the same thing as "length correlates to quality", so this is self-contradictory."

Only for the logic impaired. And the rest of your post is even worse. Grade: 1/10 points -- kinda topical but attempting to hide lack of comprehension behind large words that are poorly understood.

Re:My TA had that 35 years ago

[thoromyr]

What is it with logic failures and lack of reading comprehension? You could try again, and this time pay attention to what is being said. Such as the point about skill and time constraints. Perhaps you can't grasp the difference between a write-at-your-own-pace essay and a timed exam, but the information is all there.

AI has not come far enough for this

[Anonymous Coward]

To actually grade an essay well you need to analyze its logic as well as its sentence structures. Computers cannot do this yet! Human writing assumes all sorts of "common" knowledge in order to convey a point concisely, in addition to omitting a completely detailed logical argument to leave out obvious steps (often steps which are related to the aforementioned common knowledge). It could be done one day, but that day is not today.

In the meantime, expect these systems to be gamed pretty hard.

Re:AI has not come far enough for this

[dgatwood]

Computers suck at even the most basic grammar checking. I once decided to try a bunch of online grammar checkers to see if they would be useful at providing a sanity check for my novels. I concluded that they report so many bogus mistakes that it simply wasn't practical to use their output at all. To test them, I fed them a block of content, some with intentional errors that the grammar checker should have caught, others with deliberately (or accidentally) tricky bits that should not have produced any errors.

• Upon seeing that, Joseph resolved to stop. Several grammar checkers thought "seeing that" was used idiomatically, and suggested replacing it with because. Upon because, Joseph resolved to stop. Yes. Much better.... Oh, and some others suggested that "Upon" is archaic.
• “Time to impact: seventy-six hours, fifteen minutes, twelve seconds,” the computer intoned. Oddly, several checkers suggested that "twelve seconds" was a fraction and should be hyphenated. Ugh.
• It's simple, really. There must be some mistake. Several spell checkers suggested "their". Others said that "must be" is passive voice. Uh, no, not every use of "to be" is passive construction.
• This isn’t your class anymore. Some checkers reported an agreement problem with "class". Huh?
• The room was dark, its plant-covered landscape shimmering green in the light of their headlamps. At least one checker suggested replacing "in the light of" with "considering". Eek!
• Joseph climbed up first. Several spell checkers suggested that "climbed up" is redundant. Apparently, their editors have never climbed down something.
• One checker even called "chided" archaic, but did not comment on the highly offensive swear word that I placed elsewhere in the sentence.

And so on. Heck, my phone doesn't even know the difference between "its" and "it's" and tries to auto-correct me into looking like I failed first grade English. And these folks expect me to believe that computers can feasibly help students learn to write better papers? Give me a break. Maybe in thirty to fifty years (*) we'll get there, but....

* Which many grammar checkers would probably suggest changing to "thirty-two fifty".

Re:AI has not come far enough for this

[camperdave]

Did any pick up that the comma between "dark" and "its" in the fifth example should really be a period?

Re:AI has not come far enough for this

[Anonymous Coward]

One checker even called "chided" archaic, but did not comment on the highly offensive swear word that I placed elsewhere in the sentence.

Swear words are not a grammar problem, and a grammar checker should not complains about them ... unless they are used ungrammatically. A style checker ... well maybe, depending on what the relevant style rules say about "offensive" language.

Re:AI has not come far enough for this

[Anonymous Coward]

Why would they? "Its plant-covered landscape shimmering green in the light of their headlamps." would be a sentence fragment. (There's no verb. "Shimmered" would be a verb, but "shimmering" doesn't count with respect to making it a complete sentence.)

Re:AI has not come far enough for this

[Anonymous Coward]

That would be one of the mistakes they didn't make.

Re:AI has not come far enough for this

[cervesaebraciator]

Archaisms are also not a grammatical problem. Even so, the checker chided him for chided, so why shouldn't he expect it for swearing?

Re:AI has not come far enough for this

[Anonymous Coward]

Joseph climbed up first. Several spell checkers suggested that "climbed up" is redundant. Apparently, their editors have never climbed down something.

Not to nitpick, but the 'up' is redundant. There's no need for 'up' paired with climbed. 'Climbed down' is a different matter - and in fact, nonsensical. You can't actually 'climb down' something.

Re:AI has not come far enough for this

[dgatwood]

According to Merriam Webster, yes, you can. That has been common English usage for at least as long as I've been alive.

3: to go about or down usually by grasping or holding with the hands <climb down the ladder>

Oxford English Dictionary agrees (#2), and uses the same example. So does Random House. And so on. So maybe you can't climb down something, but I hope your house never catches on fire, or else you're in serious trouble.

Re:AI has not come far enough for this

[dgatwood]

The second part is a nominative absolute not a clause, so no, the comma is correct. That said, the construction is unusual enough that I've never been taught it in any English class, and I had to look it up just to figure out what it was called.

Perhaps the most famous example of this construction appears in the U.S. Bill of Rights (2nd amendment).

Re:AI has not come far enough for this

[Tom]

Maybe in thirty to fifty years (*) we'll get there, but....

We thought we'd be there in 10 years. That was 1984. No, not the book, the year that Cyc was started.

So, it's already been almost 30 years since we first thought we'd be there. I'm sure we'll look at this problem again in another 30 years and think again that we'll be there really soon now.

Re:AI has not come far enough for this

[Anonymous Coward]

Maybe he is feline. They can only climb and need help to get off trees. Or he lives in moms basement and there is no further down.

Re:AI has not come far enough for this

[mcvos]

I don't care how they do it. If you can submit more than once and see the result immediately, I'd use it as the fitness function to develop my own AI that writes the perfect assay. It doesn't have to make sense, it just has to please the machine.

Nice distillation of the school learning process to its very core, and once they find out, everybody learns a valuable lesson.

Re:AI has not come far enough for this

[Anonymous Coward]

So maybe you can't climb down something, but I hope your house never catches on fire, or else you're in serious trouble.

...I live in a single-floor dwelling, you insensitive clod!

Re:AI has not come far enough for this

[nukenerd]

It will be interesting to run some of the works of the great writers through this software. Joseph Conrad is well known for some quirky gramatical constructions. Proust is notorious for his long sentences and paragraphs. And would it make sense of Shakespeare's "This fell sergeant Death is strict in his arrest" ?

I expect all three of them would fail their exam. Oops, I started a sentence with "And".

Re:AI has not come far enough for this

[mjr167]

He probably has also never climbed over or around an obstacle or through a tunnel.

I have come to the conclusion that most people suck at writing. I was proof reading a paper for a graduate level course and my team members had liberally sprinkled commas throughout the text in, seem,ingly random places. They also decided that it was not necessary to place spaces following a period and that it was necessary to hit enter at the end of every line instead of letting the line wrap automatically... You don't want to know what they did with the poor words.

Re:AI has not come far enough for this

[camperdave]

It just looks wrong, somehow. Perhaps, because rooms don't normally have plant covered landscapes, the two parts of the sentence don't "connect" cognitively.

Re:AI has not come far enough for this

[Anonymous Coward]

Eye went through university, running. All my paper's through automated grading system's fore fore years came and out my grammar fine. Thankyou very much.

Have a computer write your submission too

[hawguy]

Seems like it's a small step from this to having computer algorithms that automatically write your paper for you too - then you can let it go through thousands of submit-edit-submit cycles until the scoring computer gives you a perfect score.

Kind of like the guys that came up with software to generate nonsense scientific papers and actually had a few accepted at conferences and journals.

Re:Have a computer write your submission too

[korgitser]

I wonder how these would do:

the postmodernism generator http://www.elsewhere.org/pomo/
the math paper generator http://thatsmathematics.com/mathgen/

Re:Have a computer write your submission too

[Anonymous Coward]

Or at the very least, you could crank out a bad paper really quickly, and then let the machine evolve it for a while until it gets a high score on the marking algorithms.

Re:Have a computer write your submission too

[lgw]

Much of modern finance actually works this way. Press releases about mergers and such are written by computer to be read by computer and fed into algorithmic trading. The relationship to nonsense is left as an exercise for the student's computer.

Re:Have a computer write your submission too

[fermion]

This pretty much already happens. The question is are you in school to learn or to get a grade.

One problem with teachers grading papers in today world is that you have no idea if the work is original. Reports for all the common topics are all over the place, and it is easy to adjust and mix and match so the common tools will only catch the laziest students. Solution to textbook problems are all over the place as well. The is why any decent science or math teacher will give computer generated tests. Any student who knows the material will be able to generate a solution. Any student copying off the internet and trying to adjust to the current problem is likely to have less success, especially in a time contraint situation. Since such tests are not multiple choice, common test taking strategies that allow students who know nothing to pass tests do not work.

The system at hand will be an invaluable tool for those who are in school to learn. In principle it will provide more feedback and allow more drafts than would be possible otherwise, The prof or TA and then focus on improving a draft that polished enough to work on the high level skills.

And who cares about those that simply use a program to write a paper that can pass the software to grade the paper. Those people would not have taken the assignment seriously anyway, and would not have learned. It is a myth that through grading, or threats, or whatever, you can make people learn. If it were so there would be many fewer people in the world who can't write an essay or solve an equation. Through good teaching we can get people to memorize a fact or process. That does not in any mean they have learned anything, or, after the passage of time, create anything useful.

Re:Have a computer write your submission too

[Opportunist]

Oh, that's it! I was wondering why I don't understand it, but it's actually not in a human readable format, it just disguises as English.

Re:Have a computer write your submission too

[Anonymous Coward]

You can bet your ass I won't hire a single person from any place that uses automated grading.
Remember: The one doing the grading always has to be more intelligent than the one being graded.

Hell, right now I'm not hiring anyone who didn't already have a large self-interest in the topics *before* entering any school. And the value I assign to graduation reports is *negative*. Meaning: If you were top-notch, that only means you were a top-notch *mindless automaton drone* with an attitude of a yes man. And I need people who can *think for themselves*. Which usually are bad at being such a drone.

So bad grades is more likely to be a good sign. You just have to treat people right, and they will blossom into something the system would never have allowed them to be: An individual. Somebody who cares. And who tells me, as a boss, when I am *wrong*.

Sample Admittance Essay

[milbournosphere]

Why I want to goto Harvard By P Q Student Up up down down left right left right B A

Re:Sample Admittance Essay

[SGDarkKnight]

"Up up down down left right left right B A select start"

there, fixed that for you

feedback...

[retchdog]

``Your grade is C. To improve your grade in the future, you need to do the following:

use 25-30 words per sentence; include more words from the wordnet entry for the topic of your essay; avoid simplistic or run-on sentences as measured by number of noun and verb phrases detected by our proprietary NLP tokenizer.

As a helpful reminder, our preparatory guides are available as a subscription service and include 100 practice submissions per week; only $29.95 per month."

Re:feedback...

[reve_etrange]

The sad thing is that's how ETS already grades essay questions, only the "tokenizer" is an underpaid college graduate.

Re:feedback...

[JMZero]

If their use was kept secret, systems like this would likely perform well most of the time; the correlations these systems are based on are probably pretty steady.

Once students get any information about the system, however, it's doomed - and in any case it's unlikely the system will give real, useful assistance in improving skills beyond what you'd get from a grammar checker.

Re:feedback...

[retchdog]

Good point.

Re:feedback...

[Anonymous Coward]

It's already doomed as it really can't understand anything. I could arbitrarily string words together in a structurally correct manner, and merely choose enough relevant words to satisfy the algorithm. Beyond that, I'm sure it just looks for a few other common essay tricks such as restating the question and briefly indicating the main points in the first paragraph; spending the next several paragraphs expanding on those points; and finally using the last paragraph to form a conclusion while touching on key points that were previously discussed. The rest could just be inane gibberish, such as "Many leading scientists agree that blue monkeys advance cunningly in their appropriation of computer linguistics, but do not present improvements in algorithms."

Re:feedback...

[retchdog]

You're being overly generous. Everything you describe after "Beyond that" is beyond the state-of-the-art in NLP to do consistently.

Re:feedback...

[Rich0]

Once students get any information about the system, however, it's doomed - and in any case it's unlikely the system will give real, useful assistance in improving skills beyond what you'd get from a grammar checker.

Yup. I'm sure essay-writing programs optimized to various brands of essay-checking programs will be on the market soon.

Just feed it a "topic" and it will do the relevant google searches to find correlated words, and string them together into a bazillion sentences that satisfy the grading algorithm.

Grading is about feedback

[FailedTheTuringTest]

Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student. Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student. Can a computer explain how to refine a research question or structure an argument? Sadly, many lecturers don't in fact give good feedback, but we should be looking for ways to enable lecturers to give better feedback, not accepting poor feedback as the norm.

Re:Grading is about feedback

[DragonWriter]

Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student.

Since being able to grade well requires the ability to make the exact same distinctions required to identify the feedback that would need to be given, I would be very surprised if software that could do one could not also do the other.

I'd also be surprised if current software was able to do either.

Re:Grading is about feedback

[RougeFemme]

Grading is about the grade. Learning is about the feedback. Unfortunately, more and more, the educational experience is about the grade or standardized test score rather than learning. . .and learning to love learning. . .and learning how to learn. Kids don't have to show how their work in math anymore; all the teachers care about is the answer. We shouldn't be surprised at this latest development - well, not too terribly surprised.

Re:Grading is about feedback

[Roger W Moore]

Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student.

Not always: there are two types of assessment. Formative assessment where the aim is to let the student know what they understand and what they need to work on. This is what you describe. There is also Summative Assessment where the aim to to assess what the student actually knows. Usually I try to get some of both - for example although a midterm is mainly aimed and finding out how much the students have learnt I'll also spend a lecture to go through the exam to give detailed solutions and feedback to students so they know what they did not understand and can learn from the experience. However since final exams occur after the end of the course these are pretty much entirely assessing a student's knowledge and have little educational value (although I post usually solutions online and will explain things to students if they come and ask).

Re:Grading is about feedback

[__aaltlg1547]

I suspect that these automatic graders would give the same grade to an essay if all the sentences were rearranged, or if the nouns were randomly switched around.

For example, the previous sentence might receive the same grade as this one:

Sentences suspect that these automatic nouns would give the same essay to a grade if all the graders were rearranged, or if I were randomly switched around.

In my defense, I fixed the articles to make the rearranged sentence conventional in terms of article use. But that could be done automatically.

Now let's mess with things on the next level by changing some of the verbs that automated understanding probably wouldn't object to, but a human reader would.

Sentences expect that these automatic nouns would imagine the same essay to a grade if all the graders were expunged, or if I were randomly spotted around.

The original Google search yielded 23800 results. The second sentence yielded 8160 results and the top site was about grammar (and several of the top results were about autism). Impressive given that the sentence breaks no grammar rules despite being nonsense. The last sentence yielded 454 results.

So maybe the situation is not quite so bad as one would think. Google, at least, seems to distinguish between nonsense sentences and normal language, but the search results can't be used as a measure of sentence quality. "Oh my God!!!" turned up 384 million results, whereas the previous sentence was only worth 1.2 million.

Re:Grading is about feedback

[clifyt]

"Grading is not, or should not be, about the grade"

I worked with Dr. Shermis back in the mid '90s on this (one of the professors quoted in the article) and we had our own software we were working on.

One of the very first things we had agreed upon was that grading and rating are two very different things. And when we worked on our software, it was designed to give back several scores that summarized why there was an overall score that was given.

Beyond this, *MOST* work in this area are not based around how to refine a research question or structure an argument. It was based around how to rate writing. From here, verbage could be created based around all the factors explaining why a particular metric was used as opposed to simply providing the metric. I left this field before it got that far, but its been a decade and I'm quite certain that this has happened because there were enough engines out there that could work in collaboration with one another to provide both quantitative and qualitative feedback (at the time, too many researchers were working against each other and either closely guarding their secrets or blatantly looking down their nose at the works of others because it was approaching the idea with a different vision).

That said, the #1 way to become a better writer? Write more. Even without feedback, you become a better writer simply by exercising these skills. With plain metrics? You become a far better writer simply by seeing that the computer thinks your mechanics could be improved. Maybe work on your creativity (this was by far the hardest to calculate), maybe see how improving one area weakens another as there were proven links between different areas that were almost always inversely proportional to one another except in the hands of talented writers.

And the problem with writing more? Instructors, especially writing instructors, are overwhelmed as it is. I would GLADLY grade 100 undergrad psych essays as opposed to a technical writing course. In our studies, one of the ideas was to allow more writing assignments, while giving more quality to feedback by the instructor...allow the instructor to read every 3rd paper for instance and give in depth analysis. Or maybe give more time to students that were struggling with basic concepts by catching them earlier. Almost all of this was designed to find ways to help the instructors do more with less, which is the reality of higher education these days. And when we did this, students actually got far more support than they did without our software.

Will their be lazy professors that just phone it in? Probably...but most instructors I've worked with have been passionate about their jobs, and the ones that weren't? They got this way because they were overwhelmed with the lack of resources provided to them.

BTW...every time I write about these subjects some jackass mentions my writing style. I'm not writing for publication, I'm writing for conversation (and most likely not proof reading nor thinking much more than a few words ahead...the opposite of the academic writing I use to do).

Re:Grading is about feedback

[zAPPzAPP]

I'd suspect your grammar and autism hits come from your keywords: nouns, sentences, essay, grade
These combined suggest a certain context.

As you switch them around, you lose hits that came from common verb-noun combinations and are left with only those from keywords.

New market for SEO companies

[Narrowband]

Watch as we move from "search engine optimization" to "grading engine optimization," as students look for AI solutions to write their papers, freeing them for other tasks.

The death of teaching.

[Anonymous Coward]

I've before argued that most instructors today could be replaced with a video for lecture, a FAQ database for questions and a guy with a stopwatch for tests. I used to say proctor, but that implies the individual with the stopwatch is educated and I don't believe this is necessary anymore than a McDonald's fry jockey.

Anyway, this technology just makes it more true. God help you if you're just a little different from the majority somehow.

*sigh*

Not just ivory, but pointless too

[Anonymous Coward]

And just what, pray tell, are the goddamn professors supposed to do at university? Collect honorariums and edit their book?

Re:Not just ivory, but pointless too

[DragonWriter]

And just what, pray tell, are the goddamn professors supposed to do at university?

The same as what they get hired and paid for now, research.

You notice the common statement is "publish or perish" not "educate or expire", right?

Re:Not just ivory, but pointless too

[MightyYar]

Teach more students? Current university education certainly does not scale very well.

Re:Not just ivory, but pointless too

[mark-t]

I thought professors were hired to... you know... teach stuff.

If a professor isn't actually giving feedback to the students about their performance, then a person might as well just try to learn everything from books and not bother with classes at all.

They had these back in 1991 too

[GoodNewsJimDotCom]

My friend wrote a story about his cat that was grammatically correct,and used big words, but made little to no sense. The auto-grader program told him he was approaching PHD level English. So he took his paper into school and showed it to the English teachers who reviled at it. He was like,"Show's what you know, the computer told me I'm university level."

Re:They had these back in 1991 too

[Anonymous Coward]

Ah, that must have used the Flesch-Kincaid tests. I had a professor in college who used to required that you post the results of those at the end of your paper and would base a good chunk of your grade on them. What a waste of time that class was as more and more people discovered the literal magic formula for a good grade.

Re:They had these back in 1991 too

[DragonWriter]

While I am rather skeptical about the quality of AI essay grading, I'd be very surprised if software in this area hasn't advanced since 1991. I mean, software in most other domains certainly has.

Re:They had these back in 1991 too

[GoodNewsJimDotCom]

One thing is for certain, it won't be able to understand context of the article. For AI to understand what you're talking about, it basically needs to be able to imagine it. For a computer to imagine stuff, it'd need a 3d model of the situation and a huge library of nouns, verbs, adjectives, adverbs. The day we have true AI, I might think we'll have the capabilities of natural language interpretation. Until then, the best it can do is weigh sentence structure "quality".

Re:They had these back in 1991 too

[Ksevio]

I know MS Word has a simple analysis that can tell you stuff like that. WordPerfect probably had something similar.

Re:They had these back in 1991 too

[TapeCutter]

IBM's Watson is capable of these tasks as demonstrated by the Jepordy stunt. Of course Watson is one of a kind and requires 20 tons of air conditioning equipment, but it is doable. The most interesting part of the stunt was when they gave Watson the ability to "hear" the human answers (both right and wrong), just that one bit of information basically wiped out all the wrong answers that arose from Watson not "understanding" the category.

Re:They had these back in 1991 too

[Anonymous Coward]

Watson may still fail at writing that plays with current affairs.

I was 13 years old, in Secondary one (Singapore), and the first Gulf war just started.

We were supposed to do a free form essay at the beginning of the school year. I wrote a story from a dog's point of view and how it was confused over the "Golf war" thingy that was going on. It even made fun of it's human masters as to why it's masters seem to be throwing a fit over a simple game of Golf and why were there some issue involving a bush (Bush was US president).

My teacher marked me as making a bunch of mistake earlier on in the essay until she got to the end, when she corrected her marking and gave me a glowing response.

Lets see Watson or any other current marking AI getting such references. Or grading you for creativity.

Re:They had these back in 1991 too

[HungryHobo]

yep, sounds like your friends school was using an awful system.

however it should be noted that these systems aren't new. and there are very very good ones. they're already used for checking *teachers* work in the UK.

For exams a teacher marks a paper to give you your grade. then a program like what is described in the OP marks it. If they agree then it stops there. if they disagree much then the paper goes to a second senior human being who also marks it.

if the senior human being disagrees with the other human then they sort out why they gave different marks.

if the humans agree but disagree with the machine then the paper is added to the training set for the AI

typically the program agrees with the humans more often than another randomly selected pair of humans.

but that makes perfect sense since something sitting at the middle of the normal curve is more likely to be closer to any given element than 2 elements are to each other.

it cuts down on the work for senior teachers and allows more comprehensive double checking of teachers work.

Re:They had these back in 1991 too

[Anonymous Coward]

Of course Watson is one of a kind and requires 20 tons of air conditioning equipment, but it is doable.

...which, of course, means that in 30 years this capability will fit in your pocket and run for a whole day on a battery.

Re:They had these back in 1991 too

[Anonymous Coward]

Here, spend some time on this site and kill three birds with one stone...learn some new words, help feed starving people, and test your current level of vocabulary, etc., etc. :)

http://freerice.com/

Some Things Never Change

[skywire]

Every era has its snake-oil salesmen and their marks. Sadly, in this case it will not be the customers who suffer, but their hapless students.

What, what?

[Anonymous Coward]

Professors need more time for other things now? I'm sick of hearing this "for other things" crap we keep pushing. The reality is we're trying to push them out of their jobs via automation. The "other things" you're referring to include "standing in line for your welfare check" and "filling out job applications".

Part of the "joy" of grading is that you get feedback on how effective your teaching approach is for a given class. If you take the grading process away from the professor then how the heck are they supposed to evaluate their own performance? Not to mention the fact that the system can be gamed. Now we also get multiple tries at a test as well? The educational establishment really has gone all the way to fuck and buggery. The only good thing I can see in this is that it makes the grading more consistent. That's good...because people aren't different, no no no. No, all people are the same. You fill their heads with the same words and you'll get the same exam solution. Probably even the same handwriting. We should just make all of human existence a multiple choice question - the answers can be "Pay tax" and "Go to jail".

Computers have solved zero problems here, and they've created far more. This software will be written, casually proved to be barely adequate, and then shipped off to India for maintenance. We are now taught and graded by an underpaid maintenance programmer in India. Congratulations.

*facepalm*

Re:What, what?

[DragonWriter]

Professors need more time for other things now?

Well, yeah, research, which is what they mostly get paid for. (Though, really, at many universities, the people who this is freeing up to do more of the primary focus—again, research—is likely to be more often graduate students working as graders/TAs, not professors.)

Congratulations, you have been admitted...

[Anonymous Coward]

And you have been awarded 2500 extra HP.

Grades grammar not content. A.I. not ready yet.

[doug141]

"A director of writing at MIT Les Perelman says that because these robo-graders work according to an algorithm, it is not hard to find out what it values and thus beat the system. He found that if you write long essays with big words, even if they are nonsensical, you will score high. The algorithm does not like short sentences or paragraphs or sentences that begin with ‘and’ or ‘or’ nor is it enamored of sentence fragments. In other words, all the little rules that good writers will break to create a particular effect will cause your essay to be marked down.

Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically. The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.

For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants. “The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”

E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it. He could."

http://freethoughtblogs.com/singham/2012/05/03/how-to-fool-a-computer-grader/

Re:Grades grammar not content. A.I. not ready yet.

[dantotheman]

Where the hell are teaching assistants receiving...

extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures

???

Re:Grades grammar not content. A.I. not ready yet.

[Anonymous Coward]

I'm pretty sure that was sarcasm and was meant to show how the algorithms don't actually do any factchecking :)

Re:Grades grammar not content. A.I. not ready yet.

[RobKow]

The point of that sentence is that it is obviously counterfactual, yet E-Rater gave him a top score for the insightful essay in which it was contained.

Re:Grades grammar not content. A.I. not ready yet.

[Anonymous Coward]

"The most interesting feature of the algorithm is that it doesn’t care about substance or even truth."...

Well there you go - most university faculty do not care about substance or truth either, so in this respect I suppose the algorithm is pretty accurate.

Re:Grades grammar not content. A.I. not ready yet.

[dantotheman]

Ahh, misread that part. This makes much more sense.

Re:Grades grammar not content. A.I. not ready yet.

[dwhitaker]

But this is precisely why it makes this system useful AND should please teachers. If the system works well for grammar and ( hopefully programmable) essay structure, then the teachers can focus on the content, style, and finer points of writing. A computer can correct to/too/two, and if it frees up more time for the expert that is being paid to grade in depth, this is a good thing.

Re:Grades grammar not content. A.I. not ready yet.

[RougeFemme]

Reportedly, long essays with big words also help your SAT writing score. And in my state - which shall remain nameless - it definitely seems to help on the standardized writing tests.

Re:Grades grammar not content. A.I. not ready yet.

[Lemmeoutada Collecti]

So all you really need is x (where x matches the length of the assignment) occurrences of the word buffalo, with the first one capitalized and the last followed by a period. A grammatically perfect sentence, with the maximum possible length results.

Re:Grades grammar not content. A.I. not ready yet.

[high_rolla]

Get ready to see SEO (Submission Engine Optimisation) specialists cropping up and making a fortune off lazy students.

Or should that be ESO (Essay Submission Optimisation)

And new websites such as LOLEssays where you can see examples of the most ridiculous essays people actually got top marks for.

Re:Grades grammar not content. A.I. not ready yet.

[cervesaebraciator]

The point of that sentence is that it is obviously counterfactual [...]

Ahh, misread that part. This makes much more sense.

A perfectly normal, human mistake. Now you understand the sarcasm. Had the computer been able to fact check, well, let's imagine its reaction.

Re:Grades grammar not content. A.I. not ready yet.

[Anonymous Coward]

A director of writing at MIT Les Perelman says that because these robo-graders work according to an algorithm, it is not hard to find out what it values and thus beat the system. He found that if you write long essays with big words, even if they are nonsensical, you will score high.

This is true of humans as well, as many academic journals have shown.

Re:Grades grammar not content. A.I. not ready yet.

[Murdoch5]

Your right! We had to use a system like this in College where after the essay was done we had to submit to a service called turnitin, basically it checked for plagiarism and correct citing / sourcing of work. The problem was it didn't work and ended up causing more trouble then it solved.

For one class in particular we had to write an essay about critical thinking and source it based off what sourcing method our department used, In my case we sourced using IEEE and so I told the professor that. She looked up at me and said "IEEE isn't a sourcing / citing method! Use APA or MLA", the next day I came back to her with proof that IEEE is valid and it's what my department used and hence she had to let me use it on the essay.

Before I handed the essay in I had to "turn it in", the problem was the site didn't understand IEEE format and didn't understand much of the wording I used throughout my essay as it was heavily technical, the service gave up marking my paper and just gave it a prefect 0 score as in "So clean and un-copied it truly is an example of wonder and amazement", sure enough the professor was pissed and talked with me about it. She had to manually mark my paper and check it herself, only to find that it was okay and I didn't copy it, it took her an extra few hours but she became a professor knowing the work load so I did't care.

My point is these systems don't work, there horrible. They can't mark a paper correctly, it shouldn't matter if I write the paper from the stand point of an engineer or the stand point of an abstract artist, if the service can't adapt it's broken.

Re:Grades grammar not content. A.I. not ready yet.

[neurovish]

But this is precisely why it makes this system useful AND should please teachers. If the system works well for grammar and ( hopefully programmable) essay structure, then the teachers can focus on the content, style, and finer points of writing. A computer can correct to/too/two, and if it frees up more time for the expert that is being paid to grade in depth, this is a good thing.

That is exactly what my last 3 English teachers did. They would mark improper grammar and spelling errors, but generally did not care as long as the content was effectively presented. These courses were all generally above the required university-level writing courses though, so it wasn't like we were turning in a bunch of youtube comments. At that point, our teachers let us into the secret of English writing...grammar really doesn't exist as long as your writing is understandable.

My theatre teacher took a different approach though and would make us re-write a paper that had grammatical errors. He also did not point out what they were. In those instances, we had to rely on our classmates to check our work.

Any computer can be beat

[alen]

Any algorithm can be beat
The all you have to do is fill your paper full of the right keywords

Re:Any computer can be beat

[Psychotria]

Score: 0 out of 6.

Re:Any computer can be beat

[Anonymous Coward]

College is exactly like this. That's all that matters in every single interaction you have. Use the right keywords, you pass. These days people have no use... Oh what's the right word again, yeah, "time" for questions or reactions. Soon enough, civilizations will so thoroughly poison everyone with useless terminologies that we'll be programmed to behave at the drop of a hat. We can look forward to the glorious leaders taking control of our daily lives. So much for the low hanging fruit. Turns out, that fruit is what the devil wanted us to eat in the first place.

This won't last

[Xcott Craver]

No matter how sophisticated the algorithm, the set of strings that get graded an A is bound to contain some weird and illegible elements. They probably won't be too hard to find by inspection of the algorithm and its training data. It will only take a few widely publicized examples of meaningless essays with a high auto-grade to cast doubts on this method of grading, no matter how effective it is in the common case.

Artificial Intelligence Software

[Anonymous Coward]

// My version of this software
MIN_GRADE_FLOOR = 50;
STUDENT_FEEL_GOOD_PERCENT = 0.03;
score = randbetween(MIN_GRADE_FLOOR , 100);
score = score * (1 + STUDENT_FEEL_GOOD_PERCENT);

"ways to enable lecturers to give feedback"

[Anonymous Coward]

You're absolutely correct, this should be the goal.

You ain't seen nothing yet

[govett]

With strong AI software, people could be tested continually, in real time. When the software detects a deficit, it could supply the relevant information for memorization. Ultimately, of course, the strong AI will become our brain, at least the rational part.

Our Machine Overlords

[dragon-file]

Before computers, teachers actually had to do things by hand. But computers made things easy so we gave every teacher a computer.

Even that was to hard and so teachers turned to AI to grade their students papers.

Soon going into work was to hard (especially with hangovers from all the free time spent partying) and so they made robots to give lectures in their place.

Then the students couldn't be bothered to come in so they just started recording the lectures and making them available online.

Then learning things was to labor intensive and for that matter so was work so we let robots and computers do everything.

The year is now 2635 and we spend our days watching Jersey Shore season 628. Obesity is the new norm and from birth every waking moment is 'free time' but life is better. Right?

Re:Our Machine Overlords

[Githaron]

Can you have a Jersey Shore if obesity is the new norm?

Re:Our Machine Overlords

[dragon-file]

I imagine so.... they would be the most physically active of the obese people performing activities such as getting off the couch to check the fridge and making out.

I really have no idea what happens on that show. I loath and despise it.

submissions

[Anonymous Coward]

Perhaps /. should send all story postings to this and give editors (and contributors) a grade based on the results...

On the other hand, as this is not likely to change any behaviour, perhaps that says something about the value of grammar on the internet...

This one had me checking the date for April 1.

[uCallHimDrJ0NES]

I thought maybe it was a delayed feed from a hoax.

Anything profs can use, students can use

[Anonymous Coward]

Imagine this post is an essay, and Slashdot has a little widget showing a letter grade on the right side. It starts out showing an 'F' and as I type more and more, and as my karma whores and whores, it moves up the scale until I have an 'A'. Then I hit send.

If this technology ends up in colleges, the whole essay writing process becomes moot... or does it? I suppose it's conceivable that at this point the essay writing is just a game. The exercise is tantamount to the professor saying, "get to level 6 on Super Mario College Essay" and if you do that you get an A.

I guess it boils down to a question of whether or not there's really any human skill in writing the essay, and whether or not that skill should matter. If algorithms can do it in school, then algorithms can do it for the media. There's no point learning journalism. There's a point in learning computer science to write the essay-writing algorithm... but only until the algorithm is good enough to take the place of humans. Then there's no point in that either.

I guess I could have gotten the same grade on this essay if I had simply cut to the case and said... "I for one welcome our robotic essay writing overlords".

free time?

[v1]

The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."

There are probably at least several good jabs to make at this so I'll try to just address the best one.

First thing that comes to my mind is 'free them up to do what?" Education is adding more and more distance between the student and the teacher without throwing this into the mix. In a perfect world, teachers would be there to teach and that's it. But it's become more a problem of time management, lecture halls full of hundreds of students per teacher, and throw in the odd paper publish and grand write here and there. I think what we need to be doing is not looking for ways to further skew the student-to-teacher ratio, but to dial it back down a bit instead. Get me back to the good 'ol days where your prof knew your name.

Well, maybe two. "Artificial Intelligence beats Real Stupidity."

Re:free time?

[cervesaebraciator]

Get me back to the good 'ol days where your prof knew your name.

That would be nice. But, of course, back in those days a person like me wouldn't have had a shot at higher education, much less a Ph.D. No, I'd probably be on my way to black lung or some similar ailment by now, working in mines acquired by some guy who could afford to send his kids to a low student-to-teach ratio private school. The democratization of education has its costs as well as its benefits.

Free professors for other activities

[the eric conspiracy]

Would you like fries with that?

Seriously though I don't think writing to what an algorithm wants is a bad thing if the algorithm wants the right stuff.

It's not as if students don't write to the algorithm the professor uses even now. The only difference is where the algorithm is stored and how flexible it is.

That is Nothing..

[wisnoskij]

At my university we have AI that continually checks our essays as we write them. It even points out the specific mistakes, gives suggestions to fix them, and allows us to rewrite that section of the essay.

We call this advanced AI MS Word.

No such thing as 'AI'

[Anonymous Coward]

AI literally means "semantics automatically arising from syntax", and is thus a clear nonsense. It is the brain-dead rubbish believed by idiots that a convoluted and complex system will magically gain meta functionality.

So what does 'AI' really mean? It means computer systems that draw from databases that store some form of extracted Human knowledge. Usually the best so-called AI systems are using statistics in some clever and non-obvious way. Take this story...

I assume it actually draws from an April 1st joke, but let's assume it is real for the moment. Take a large collection of student essays, each graded by a first class teacher. Now, run a stats engine on the PATTERN (not meaning) of the syntactical form of each grade-group essay. Look for things like sentence length, paragraph length, number of sub-clauses, and occurrence of different reading-level-group words. Do a crude analysis on grammar and punctuation.

Now see if there is a strong correlation between a grade an essay earns, and its statistical form. If such a correlation exists, propose the absolutely dreadful idea of reversing the process- ie., use the same stats methods on ungraded essays, and apply a grade according to which grade-group the stats best correlate to.

Sad to say, betas will think this 'method' makes sense, just as betas tend to think 'pyramid' schemes are worth joining. To make it easy for you dumb-dumbs to understand why such a system is a terrible idea, here's the problem. Once such a system is applied to grading, two things happen. Firstly, essays that refuse to follow the dumb rules the AI system has extracted from the original essay pool will be mis-graded. Secondly, the dumb AI rules can be discovered, allowing 'essays' to be created automatically that will be nonsense to Human readers, but will score the highest grade from the 'AI' grading engine.

To make it clearer to you dumb-dumbs- it is impossible to create an essay auto-grading computer program that has any merit whatsoever. Sadly, dumb-dumbs will carry on falling for impossible compression schemes, fool-proof strategies for winning at roulette, and 'Human thought process like' AI computer programs.

PS Google creates its so-called 'intelligent' algorithms by employing vast numbers of Humans to enter new rules into their computer systems daily. Patterns of search requests are inspected by Humans, and enhanced by Humans identifying what the users were really after. AI, in the true meaning of the word, does not and cannot exist (on a Turing complete computer- and no, there is no magic other form of 'better' computer, or magic other form of maths that cannot run on a Turing complete computer).

Re:No such thing as 'AI'

[Anonymous Coward]

AI literally means "semantics automatically arising from syntax", and is thus a clear nonsense.

All these intelligent individuals researching AI for years... what a bunch of dummies! Listen to us ACs instead! Humans are special snowflakes, and nothing can comprehend things like our magical brains can; deal with it.

Re:No such thing as 'AI'

[oreaq]

Has one of those intelligent individuals that have been researching AI for years managed to define the word "intelligence" yet?

Automated Grading AKA Spellcheck???

[wisnoskij]

So basically they feed the essay into Google spellchecker and count the number of underlined words?

Rewrite the test?

[PPH]

How many times?

I can envision an essay writing 'bot making multiple submissions with some AI to optimize subsequent submissions based on the .

I know it sounds like an awful lot of effort to build, but its time better spent than writing a two page essay on Dickens, IMO.

Better than you think

[Roger W Moore]

It sounds reasonable in the abstract, but in practice it was utterly wretched.

No, the abstract does not sound reasonable: as with most things online you can always find bad ways to do it. I'm a physics prof working as part of a team to develop an open source, algebra capable question and content system. However even the current capabilities of something like Moodle (which is Open Source) is far in excess of what you describe. You can type in multiple "answers" to a problem and have the student get feedback and a partial grade if they get the problem wrong in a way that you managed to guess. Obviously if they find a new way to get it wrong then they will not get feedback though.

Commercial systems go even further with the student having the option to click on a help button which can break the question into steps for the student to complete in rder to guide them through to the right answer. This can be configured to give a grade penalty at the choice of the instructor - this is one of the features we want to add to an Open Source solution.

However even with current Moodle capabilities you can build a system that, I would argue, is better pedagogically for many physics problems (those with numerical or symbolic responses) than paper-graded assignments because, with an online system with some feedback and multiple attempts the student is encouraged to keep trying until they figure out how to get it right. This encourages them to think out the solution themselves whereas with a paper assignment they get one try and are then given the answer. To make this work though you need some means for students to come and talk to you and/or TAs to provide some help towards getting the right method. So you still need the student-teacher interaction but computers can provide a first line of contact and so let a teacher help more students.

That being said I find it exceedingly unlikely that this EdX system can work for written responses beyond checking that their english is good. For physics how can it possibly know that the statement "the Higgs boson has a mass of 140 GeV/c2" is wrong and "Dark Matter does not interact with photons" is correct? To be able to grade it will have to know a huge amount of information about a massive range of topics - and looking this stuff up on Google is not an option given all the crazy people and their wacky physics theories which they stick on a web page.

Re:Better than you think

[matmota]

Is it the Algebra module for Moode? http://docs.moodle.org/24/en/question/type/algebra, https://tracker.moodle.org/browse/CONTRIB/component/10326, https://moodle.org/mod/forum/discuss.php?d=98670

I worked nine years on such a system, called "ActiveMath" http://activemath.org/ then and now "Math-Bridge" http://math-bridge.org/, where I designed and implemented precisely the exercise system that does the answer evaluation: http://matracas.org/escritos/#exercise_system_report

There you can apply different tests to the student's answer, and one common use was to first check for the exact correct form, like "1/2", with a "syntactical" comparison of the expression tree parsed from the textual input. You would get the "Correct!" feedback for that one.
Then you could compare it "semantically" with the expected answer, which sent the expression to a Computer Algebra System for simplification in a specific context (set of simplification rules, depending on the task), so if you answered "2/4" you would get the feedback "That's correct, but not fully simplified. Please give the irreducible form.".
The exercise author can include any number of such classifier expressions to catch different forms of the correct answer, different half-done answers, and wrong answers, giving adequate feedback for each.
Feedback is not just text, but a complete "exercise subgraph" that could be entire sub-exercises intended to correct the misconception corresponding to the wrong answer given by the student.

Re:Better than you think

[Anonymous Coward]

You're using the words "open source" when you actually seem to just mean "freeware".

Re:Better than you think

[nedlohs]

It's licensed under the GPL, you have very strange distinctions between "open source" and "freeware".

Re:Better than you think

[Roger W Moore]

Yes - the Moodle algebra question plugin was written by me and has been ported to newer Moodle versions by others (one of the things I love about Open Source!). However I've now teamed up with others around our campus and we are putting together a more math-capable engine with far better SAGE integration. Thanks for the links to your projects - they look interesting!

That BS again

[gweihir]

This is now the n-th time something like this has been on /.

The answer is still the same: This is a very bad idea. Students will learn how to game the software instead of how to write well. No software can grade whether the reasoning is sound, the images vivid, the prose well readable. But what students will learn is that writing essays is not important, after all it is not even worth bothering a human being to read and grade.

Re:That BS again

[Etherwalk]

Now assume you grow up somewhere where the teachers can't read and write proper English. We can call this place "public school in a non-educated community unless you're lucky." This kind of thing could be useful. The only problem is that people would not be smart enough to correct it, but it might still raise the bar.

Re:That BS again

[gweihir]

I doubt it. I think if things are that bad, everything is lost anyways. And "we do not even rate a real teacher" could well destroy what little motivation is still there.

If we actually had true AI, or at least some reasonably faked one like Watson, with a real competent team to teach it and look at unclear cases, this could maybe help some. But this software is not even nearly in that class.

"Freeing professors for other tasks"?

[msobkow]

Any professors I've ever known or been taught by had their grad students doing the grading, anyhow.

Besides, what exactly are the professors being "freed up" for? Isn't their JOB to TEACH?

Oh, yeah, I'm thinking old school. Nowadays a professor's job is to find corporate grants...

Re:"Freeing professors for other tasks"?

[Anonymous Coward]

Nowadays a professor's job is to find corporate grants...

That is totally not true! My adviser, for example, gets government grants!

Re:"Freeing professors for other tasks"?

[Etherwalk]

Depends on the school. Good, small undergrad schools use little or no TA grading.

Re:"Freeing professors for other tasks"?

[Anonymous Coward]

I use TAs for grading, but for multiple choice, fill-in-the-blank, and similar types of questions. I also use them for labs, which are quite structured exercises. For long essays I alwyas do it myself. Doing less would not be doing my job.

I'm totally with you. I can't see how this program would usefully free up any time.

Re:"Freeing professors for other tasks"?

[supercrisp]

"Isn't their JOB to TEACH?" Not completely, sometimes barely at all. At an R1, the typical humanities appointment is 25-40% teaching, 50% research, and the balance to service. Some faculty may only teach one class a semester, if they're administrating a department or subdivision of a department, or if they're running a onerous committee, like a hiring committee. At a teaching school, your "main" job is teaching, but you're still required to produce some token level of research and serve the university in other ways, such as by working on committees, being a public figure, and other stuff that you might not consider right away. So, at my job, at a teaching school, about 70% of my time goes into teaching. The rest goes into mandatory requirements to publish, present papers, do committee work, assist developing colleagues, and perform community service. (Note that in my annual performance review, I'm only allowed to indicate that teaching was a maximum of 60% of my effort, and this at a teaching school. This may be atypical, but I suspect it's not.) Now, in the sciences there are faculty with no teaching requirements. And in the humanities, at R1 schools, faculty get a year or a semester off periodically during which time they are expected to complete a research project, typically a book.

Re:"Freeing professors for other tasks"?

[Anonymous Coward]

How do you think a lot of state universities and colleges even continue to even operate? Take away grants and watch your tuition rates skyrocket.

Careful what you wish for..

Re: freeing professors for other tasks

[Kwyj1b0]

What other tasks? The only task of the professor is to teach students. Anyway, this is probably at least third submission about automated grading systems over last 6 months, and it really is getting tired.

To any sane person it is obvious that the idea is retarded, so why do we keep discussing it?

Really? The last time I checked, no one got denied tenure for poor teaching. The faculty are there to bring money to the university - if they manage to teach well, that is an added bonus. However, they could be great teachers (I've known a few) who get fired because they didn't bring in hundred-thousand/million dollar grants every year of their assistant professorship.

While the idea is retarded, you seem to be missing the danger. When the university/faculty see the benefits of doing this, they will do it in a heartbeat. It frees up the teachers to spend more time on research and grant applications. It frees up teaching assistants (who are often hired as TAs to save grant money - they still have to do research). Everyone except the students (who have already paid for the class) benefit.

Creationism vs. Evolution seems retarded too - but because some people saw some idealogical/political benefit to pushing it through into the classroom, it did.

also short answers?? How does it know the answer?

[Joe_Dragon]

also short answers?? How does it know the answer? and what if there is more then 1 answer / a wide range of part answers.

Also what about questions on the level of how do you do X in X os and you fail as the way you put down to get to control panel is not the way the gradeing software is looking for.

Re: freeing professors for other tasks

[Anonymous Coward]

The main task of a professor is to do research, not teach. Sometimes, that is actual education research, research on how to teach.

Good for the filler / big lecture classes t online

[Joe_Dragon]

Good for the filler / big lecture classes to move to a full online / test only setting and maybe they can pass on the lower costs to the students.

It may just lead to people to gameing the system but what does that give them? More time to work on there core classes?? I think that it is better to say do the min to pass art history so I have the time to work on the classes I want to work on.

Where can I find more information on this?

[RobertinXinyang]

I read the article and went to edX. At edX I signed up; but, I can not find out about this system. Quite Frankly, I am a teacher and I need my students to be writing more; however, I do not have the time to grade all of their papers so I have been assigning more objective homework that I would like.

A system like this may work as a first pass filter to do the bulk of the grading, allowing me to focus on identifying common problems and developing lessons based on common errors rather than tying myself down with a huge stack of papers. This would also benefit the students by providing them with more consistent grading and feedback.

This may not be what I go with; but, I would like to have a look at it. That takes me back to my question, can anyone point me to somewhere that I can get more information on this?

Re:Where can I find more information on this?

[gl4ss]

who is this Frank?

the kind of first pass filter you should rely on this is the type of filtering you could do by glancing at the essay... having your students write just for writings sake you could just ask them to write comments on slashdot and base the grade on moderation it receives, if the course you're giving is actually about beating automatic tests with choice list of words and simple enough structure that the auto-grader doesn't go on the fritz then by all means go ahead and use the auto grader. I'm sure their sales will gladly contact you for a trial.

Re:Where can I find more information on this?

[thesandtiger]

For elementary and early high school, this would be a useful tool, since the content isn't as important as learning the basic forms.

Even so, it would still need to be just a part of the toolkit - have the machine grade the grammar, have the teacher then read through and judge the content without wasting time proof reading or spell checking.

Re:Where can I find more information on this?

[RobertinXinyang]

Right, there is no magic bullet to education. There are just a variety of tools in the kit.

SCIgen v2.0?

[Anonymous Coward]

Come on, this is slashdot and noone has pointed http://pdos.csail.mit.edu/scigen/ yet?

All you need is to feed it with a new richer syntax tree generator, for getting an undisputed and bold 10!

Grade this!

side effect of big classes. also Multiple choice

[Joe_Dragon]

side effect of big classes. also can drop tests down to a Multiple hidden choice test. It's like a Multiple choice test without knowing what the choices are.

smaller classes and team based work maybe even apprenticeships / more of a tech / trades schools level testing where it's more about real skills and not about test cramming taken to the next level where you now just need to know the buzz words.

They'll work the bugs out

[rsilvergun]

Face it, we're all going to get replaced by Expert Systems. They talked about this in the 80s and you didn't believe. 95% of us follow pretty simple patterns. There's damn little that most of us can do that a machine can't. Sure, there are exceptions. But most of us don't qualify, we just think we do.

You're being replaced. The real question is how are you going to deal with it? What do we do when 95% of us are completely unnecessary?

Re:They'll work the bugs out

[scamper_22]

This is absolutely true.

Worst of all, most people can see this applying to 'manual' labor jobs. Heck, many people are so dismissive and even appear to think it moral that manual labor jobs are done. Ever hear people talk about auto workers or manufacturing jobs or low-wage warehouse work.

There's a huge class of people who rant along the lines of "Can't stop progress. You should have gotten an education to better yourself. You don't deserve as much money because I went to school for 4 years..."

Yet, as you rightly point out, 95% of people have jobs that could be very easily automated to a large degree if you were to look at results.

At its most basic level, most of financial services/accounting could be pretty easily replaced by computer systems. As many people say, our tax system can be automated to a large degree. Some countries already prefill tax forms. Simplify the tax code and the entire process becomes null and void. Let's not even get into the vast majority of the underlings in financial services who basically just apply a few formulas they learned blindly. Some of the top people might have some judgment though. But most anyone else just ends up with simple work... or at best... they are a sales person.

Even something like education. We are talking about something that is taught to pretty much every single child. Although a very personal relationship, on the whole, most classes teach the same thing. A grade 9 math class is a grade 9 math class. You could pretty much record the lesson and play it back. A teacher relegated to classroom control. Just put in perspective of the numbers... and you as a student are not that unique... and neither is your teacher.

Even many skilled jobs... like a family doctor.
Most spend their ACTUAL time with patients in a very simple pattern. The process could generally be automated in terms of diagnosis and scheduling tests and prescriptions.

But most 'educated' jobs have protected themsevles in one way or another, but its changing as well.

Much like the automated car, people have this perception of enhanced human intelligence, but we all *KNOW* that on average, when the automated car becomes reality, it will be better than 95% of human drivers. Sure you as a person might be able to react better in that 0.0001% case that the computer was not programmed for. But compare that against all the times a human is not behaving better than the regular tasked computer.

 

Re:They'll work the bugs out

[LihTox]

Isaac Asimov wrote about this in many essays on robotics and computers, and saw this as a plus: we may get to the point where we don't NEED everybody to be working 40 hours a week, so people would be free to pursue their own activities. And if by saying that these people are no longer "necessary" than you're buying into the mindset that a person's essential worth is tied up with their work and how they contribute to the economy. Asimov, unfortunately, underestimated how powerful that mindset is. The thought of paying everybody a living wage whether they work or not is abhorrent to so many people (in America, at least), even if there isn't any work that needs doing.

You do this at Berkeley

[Anonymous Coward]

In the engineering courses, some teachers will assign you work to submit online and you have to type it in exactly as it should be written by the application's standards. The problem of course is that the teacher doesn't want to bother with false positives so they give you 3 chances to correct your work. If you don't get it right then you get nothing for that problem. What was strange is that there were often bonus questions as a sort of backup which they wouldn't count for your grade other than improve upon it if you missed other questions. Say they assign you 5 questions, you may only need to answer 2 correct to get 100%, the rest doesn't improve your grade. It's a stupid system and it certainly takes away from the 1 on 1 education that students need. The US has a strange concept of having a lot of students per class, I thought it was ridiculous and they still have the nerve to charge an arm and a leg for it.

Hard to believe

[Anonymous Coward]

You are familiar with the email "Help desk" at web sites. That's where our email questions are read by software scanners that look for key words and automate a response unrelated to what we wrote about. Well guess who thinks that's a great paradigm for grading exams? Harvard and MIT, that's who.

On college exams, instead of handing in a blue book and getting a grade from a TA a few weeks later, click on a 'send' button when you're done and receive a grade instantly, your essays scored by software.

Then, instead of being done with the exam, imagine that the system would immediately let you rewrite the test to try and find more of the keywords it's programmed to reward.

EdX, an education disgrace founded by Harvard and the Massachusetts Institute of Technology to offer EasyA courses on the Internet, has just introduced this system and will make its automated software available free on the Web to any misguided institution that wants to use it. (Prediction at UVa: Dragas Yes!, Sullivan No!)

Like help desk software, this uses artificial intelligence to grade essays and written answers by keyword searching, preparing future faculty for writing unfathomable distorted prose making simple ideas incomprehensible.

Among the many losers in this rotten plan: the TAs who now read and grade undergraduate exams can be let go.

US Leads in Race to Bottom

[EmagGeek]

This has got to be the dumbest fucking idea I have ever seen - and it's completely unsurprising where it is coming from.

is not?

[Anonymous Coward]

is not the point of grading an essay written by a student the job of a "TEACHER"?

Now a robot can replace a teacher?

ok then why write.

Why not just observe Asimov's three laws and tell all kids that they are "NOT WORTHY" of Robot-dom.

The Actual "Essay"

[efitton]

http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html

Anecdote time.

[Anonymous Coward]

In my time at college, I had a business teacher who used a similar system for homework.

It was horrible. No, let me try that again. HORRIBLE.

You'd be given the problem and two chances to upload and submit an answer. The software was extremely finicky, however, in what it considered the proper answer - it allowed for no deviations at all. (That's the problem inherent with such a "tool" - it completely ignores the human element in learning.

In the end, I formed a group of classmates that would each take a turn at submitting the answer. I'd use my first chance, everyone learns from my "mistakes," next person would refine the answer and use their first chance, then the next person would further refine the answer and use their first chance... until we reached the "right" answer, at which point we used our second chance to submit it.

What a bull**** way to educate students.

Record High Student Debt Load

[Fieryphoenix]

And this is what higher tuition is buying?

The policeman's Beard is Half-Constructed

[Opportunist]

aka, "your lazy bum grades me with a computer, my lazy bum lets a computer write one".

How long until such a system is being gamed? Quite frankly, I wouldn't be too surprised if within weeks, students notice what the computer "wants" to "read" and write their essays accordingly.

Not that it would change too much from today, where you learn what your professor wants to read and write for your audience...

Essay grading machines have been in use for years

[milkasing]

All your GRE essays are evaluated by a machine and have been for years -- the e-rater. http://www.ets.org/research/topics/as_nlp/writing_quality/
The rating is also done by humans. It works well in practice and ensures that essays are graded fairly. If there is a significant discrepancy between the two ratings for a essay, that essay is examined further by another specialist. It prevents students from being victims of someone having a bad day at the office, and also does not encourage writing an essay to beat a machine.
The significance of the EDX news is not the concept of automated grading, it is that that such software is now free and opensource.

Needed: better teachers

[Animats]

I read the article and went to edX. At edX I signed up; (semicolon should be a comma) but, (misplaced comma) I can not (should be "cannot") find out about this system. Quite Frankly,(lower case!) I am a teacher and I need my students to be writing more; (Run-on sentence. Use a period.) however, I do not have the time to grade all of their papers (comma here) so I have been assigning more objective homework that I would like.

Grade: D

Re:Needed: better teachers

[RobertinXinyang]

If I really care about what I am typing, I type it in word first. My students know I am the worlds worst typist when I type on the overhead; that is why I make an effort to use class material that I have written in advance. If I take the time to write well, which I am definitely not doing now, I can write, and have written, material suitable for publication

I will also add that I am a business teacher, I wrote well enough to make it through an MBA program (oddly enough, I was even called on to help edit for the ESL students). However, for all the amusement your jackass answer provided, it did not address the question. -- Grade F

Students fire back with....

[Anonymous Coward]

Students fire back with automated systems to write essays.

Correct Grammer

[Anonymous Coward]

Panda eats shots and leaves. Be on the lookout for one bad Panda.

Re: freeing professors for other tasks

[Anonymous Coward]

Depends on whether the college/university is a "Research 1" institution or not.

Looking to the future

[Loki_666]

with the prevalence of advertising and corporations dominating everything, you can be sure that soon the answer to all questions will be "Pepsi"

http://homepage.smc.edu/nestler_andrew/pepsi.jpg

Re:Good for the filler / big lecture classes t onl

[Patman64]

and maybe they can pass on the lower costs to the students.

BWAHAHAHA

Computers

[sgunhouse]

I've probably been in this longer than anyone - in 1986 I was working with a teacher (High School Biology) who had networked C-64s in his classroom. Of course back then the questions were all multiple choice (we couldn't give it enough intelligence to evaluate expressions), and yes he did the semester tests himself.

If used properly, there is nothing especially wrong with doing assignments or quizzes on computer. That being said, you know there is going to be a tendency to misuse them. They'll assign more work or have to handle more students, and start depending more and more on the computer ...

It's hard to imagine one grading essays except on structure (grammar, spelling, etc.) as it even tends to be hard for humans to grade essays. But then again, I'm a Math and Science guy, so what do I know.

Kertrats

[rossdee]

My nephew was developing something like this, he's studying at Carnegie Mellon

GMAT is done this way

[Anonymous Coward]

MBA's are scored by computer for both quant, qual and essay... and we rule the world... if it's good enough for the elite why not the plebs? :-)

Hello

[Anonymous Coward]

Hello

simple cure

[sister_miriam]

1. Ask friend from other school to act as class representative (This is important, He must be unaccountable.)
2. Get professor's thesis and score it.
3. At class let representative tell in an authoritative voice:
"Professor Warnsdorf, your thesis (Wave with the thesis.)
got 27/100 points. Therefore YOU are not worthy to teach English."
4. Then entire class will leave the room.
5a. Most of time prof will come to senses and stop this idiocy.
5b. If he continues to assert that there is nothing wrong with online testing quit the school.
There is substantial chance that continuing this "education" would misdirect you more than to teach you.

Symptom of something more fundamentally wrong

[jandersen]

The future of education looks bleak when this is the kind of crap that goes on, but I think it is only yet another symptom of what is going wrong with our education system, and it has been a problem for a very long time.

This narrow focus on exam results means that it is rather hit or miss, whether the student actually learns anything of use. However, it only reflects the wider problem, that most teaching seems to focus on results only. Take mathematics: the typical study material goes "definition, theorem, proof, theorem, proof, ..."; so the focus is on the grand results - the theorems. When I left university, with quite good grades, I found my knowledge amazingly useless; it took me years to realise that I had learned the wrong things - that the useful things are not primarily in theorems, but in the methods used to prove the theorems. A good teaching staff will help you understand this point; I don't think an exam robot can.

Are you sure?

[Anonymous Coward]

The question is are you in school to learn or to get a grade.

That's a false dichotomy, at least under the current higher-education regime. Try to advance to graduate school if you only care about learning to the detriment of your grades. There was one student in my CS program who claimed she deliberately failed several courses because she didn't believe she deserved to pass (or so she said; she was fairly smart so it seemed plausible). That really put a wrench in her ability to advance when she "deliberately failed" courses that were pre-reqs for the next year of school. That's expensive to do, both in terms of time and money.

Besides, I found that the bulk of my computer engineering course content was useless filler that I would never use again. In those cases, I focused on the grade. Furthermore, I found that the material I self-taught that was outside the context of the curricula was more useful long-term... e.g. I taught myself how to use current microcontrollers on my own while my coursework forced me to work with (literally) 8086 hardware. I could have done that outside the context of college... and in the decade since college I have never encountered another 8086. So: if I could handle the advanced, modern hardware on my own, did I really need a "foundation" in 8086 in order to be able to "grow and understand" the later technology? Hell, most of the stuff we interacted with was made before I was born, and had been supplanted by far better systems (really? who uses discrete TTL instead of at least something based on CMOS anymore?)

Get the grade and move on, because higher education revolves around that, and—at least for the first few years out of college—employers care about your GPA. Teach yourself the really important stuff, because that will give you the auto-didactic skills to be able to remain relevant beyond age 40 in this age-focused industry.

How about automatic grading of /. comments?

[davide marney]

Some immediate feedback along the lines of, "you have a 50% chance of being ignored because this is a me-too comment", or "you have a 80% chance of modded down because you are expressing an unpopular opinion".

Great for MOOCs, real life...

[Aonghus142000]

I've been taking classes through Coursera (another MOOC, like EduX) since they opened, and I, for one, am thrilled to see this system rolled out. My very first experience was with an English course, and while the subject matter (fantasy and science fiction) was fascinating, and the professor a compelling teacher, the grading system left something to be desired.

The format of the twelve-week course was simple. Read work (Everything from Grimm's Fairytales to Doctorow’s Little Brother,) write a 300-500 word essay on the reading, (Just try to expand on a theme in Dracula in 500 words. Go on, I dare you,) and submit. Then read and grade five other student's essays, and receive your grade from the five other students who read your essay.

Many of my fellow students were knowledgeable and insightful, but I also got accused of plagiarism for pointing out that Frankenstein was a treatise on the dangers of technology, and had my grammar attacked in broken English.

If I were in an actual University setting, where I had paid money to be there and the grade actually meant something more than my vanity, I would be very leery of this system. In a MOOC, where I’m taking the class simply for the love of learning, this system represents a quantum leap forward.

For those who are interested in seeing firsthand how a system like the one described in the article works, try plugging something you’ve written into this: http://www.paperrater.com/free_paper_grader/

Preparedness

[Anonymous Coward]

So one make up any idoitic lie, fluff it up with unrelated but pretty sounding words, and the system will give you a pass?

If the point of education is to get kids ready for the real world, it seems like it should work just fine.

analyze form more than content

[peter303]

"Form"- spelling, grammar, sentence structure, pragraph devlopement, etc.- should be half of an essay's grade. Especially due devolving literacy standards these days of students who spend all therir time on digital devices and testing.
Only a human can understand if the essay is on topic, clever, humorous, special, etc.
Computer assisted grading will cut the necessary number of graders in half. This would save a lot of cost for SAT essay grading. Please the human grader could concentrate on the more interesting task of grading content.

run slashdot posts through this grader?

[peter303]

That could be an interesting execise. And its free. I probably would not get a good grade.

What are professors here for?

[tdalbo92]

I was just thinking to myself this morning, "Gee, I wonder if there's a way we could separate professors even more from their students" Of course professors are crucial to research in their fields, but it'd be really nice if the university actually cared about its customers.

Automated system that will write essays for you

[Anonymous Coward]

Soon there will be automated systems that will write essays for you so the the automatic grade system can grade it. We will save so much time doing nothing

It may be useful as augmentation

[jameshofo]

Well I may be the odd ball out in that "this really could work" but not for a final or something that actually matters, but you could do it right at some points. Lets say you take a few prescriptive words of an event like what you input into an ad-lib then get the student to create the sentence structure around that. Sure there's probably about 10 problems you could think of off the bat that would make this infeasible. But what if it was just a practice mock-up before you took more advanced classes? Or as side assignments there's room for this in the junk assignments that are given out to make someone practice, practice, practice.

Its a long way off from replacing a teacher, and may never if you can't properly articulate life experience in a mechanism like this. It would be a constant effort to keep up with culture too. When does a tweet not refer to a bird? Or at what point does "I shot an elephant this morning, wearing my pajamas" refer to the zoo, africa or a picture being taken inside your apartment in Manhattan?

A possible problem ...

[porky_pig_jr]

A nd then - oh horror - you'll find out that your grader used a different software to grade your essay. *Quite* different.

Versant

[mattr]

I was surprised the other day when helping another friend prepare for a test on English speaking proficiency (for foreigners to join the U.S. government). One of the applicable tests is Versant, which does this. It is a verbal test. You call a phone number, enter your ID and then do tasks including: read sentences, give one or two word answers, create sentences by reordering and linking three phrases, retell a story with as many as possible of the characters, actions and situation; and finally give your opinion about a concept (like, "Do you think children should be able to decide when and how much to study?"). The system grades you not on individual questions but gives you a general score in areas like vocabulary, pronunciation, and fluency. The problem is it is very rapid-fire so you can't really pass it unless you are close to a native. The grades can be viewed on the web in a few minutes.
On one practice test I typed what he should say live during the test, which gave him an idea of how much work he has cut out for himself to achieve a passing score.
http://www.versant.jp/e_about.html

Turing??

[jamessnell]

Since I doubt this thing can pass the Turing Test, I'm skeptical. A cool notion and significant effort. But still, seems pretty far fetched. I hope it works well.

Conditioning is not education

[yusing]

The message a college using this is sending you is: go somewhere else. Go somewhere where people's first concern is about you getting an education, where that isn't just ancillary to their "real" mission but where it IS their mission. Where helping you isn't a time-limited inconvenience, but the core mission of the institution.

I've spent a lifetime working with machine learning, and the very idea that any machine today can even begin to comprehend natural language is pathetically laughable. You'll be expected to adapt your responses to the extremely low common denominator the machine represents, and the result is you'll be conditioned to retard your own thinking processes. THEY can't pay YOU enough to settle for such nonsense.

Machine learning

[Anonymous Coward]

While reading this I thought that it would be interesting to see if we can write some artificial intelligence thingy that maximizes the text score given a number of iterations. It would be quite cool :).

Ok, but there's a mighty big assumption in there

[Anonymous Coward]

That's assuming that the professor actually *uses* an algorithm to evaluate student's work. It's an open scientific question whether consciousness and understanding even are algorithmic in nature, and many highly respected scientists such as Roger Penrose suspect they may well not be.

I know it was just a figure of speech, but that's kind of the point at issue here. Either the complexity/capability of the computer's algorithm is many many orders of magnitude inferior to the professors, so as to render the professor's understanding incomparable in practice, or the professor is not even using an algorithm and it's incomparable in principle.

Either way the automated grading sucks, (even if the professor really is an automaton at a fundamental level ;)

Christ! Have we fallen so low?

[Anonymous Coward]

"Form"- spelling, grammar, sentence structure, paragraph development, etc.- should be half of an essay's grade.

When I studies humanities it was assumed that your grasp of form was up to scratch. Hand written essays were the norm in those days, so a few spelling mistakes were forgivable, but bad grammar or incoherent paragraph structure would simply get you failed regardless of the content. The grade was based on demonstration of knowledge and quality of argument alone, assuming you cleared the bar with regards to form. Hopeless cases when it came to writing ability were soon sent down.