Slashdot Mirror
The Promise and Limits of 'Learning Analytics' (shar.es)
jyosim writes: College students always pay attention in class and do all the readings, right? Ok, they probably never did, but today's professors can actually find out how much each student pays attention in a lecture and how much time they spent on readings, thanks to so-called "learning analytics." Some colleges are experimenting with using the data to re-engineer courses hoping students will learn more and retention will improve. Professors get "dashboards" and sophisticated charts, changing their role in the classroom. MIT is an early adopter, assigning post-docs to help professors interpret this new data. As the article on the new Re:Learning project notes, though, "How much can big data actually reveal about something as personal and subjective as learning?"
"Analytics." Heh.
Most professors are bad at teaching.
It's great that they are doing this, trying to diversify learning styles and measuring their import, but for some subjects this is next to impossible to do. Engineering subjects aren't terribly amenable to this. These courses boil down to "Can student do calculation X to accomplish Y?" An essay describing the process is not useful. True/false or multiple choice grading are poor options if an instructor really wants to diagnose why the students were unable to get the result. In these classes, the student must do the work out and illustrate the steps in the process.
Left MS Windows for Linux Mint and never looked back!
Vote for Bernie in 2016!
The biggest problem with data is that it's very easy to make the wrong conclusions if you're not trained. Interpreting data is not naturally intuitive. I've seen some rookie elementary statistics mistakes among administration, let alone paradoxes like Simpson's paradox.
I once sat through a presentation that talked about student improvements over time. A professional survey company was hired and seemingly had pretty decent methodology. However, the administrative staff presenting the data didn't understand what "statistically significant" means and glossed over the fact that many of the improvements were not statistically significant.
I don't think they were trying to hide the fact they weren't statistically significant because they left the little asterisk in all the slides. I genuinely got the impression they didn't understand what it meant.
I promise to learn analytics.
You must have gone to college before the rise of "social justice".
Things are very different now than they were even just a decade ago.
It's extremely risky for a professor to give anything less than a high grade to a student.
Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.
Also suppose that the professor goes ahead and gives the student this well-earned failing grade.
If the student's skin color differs from the professor's, then all the student has to do is claim that the failing grade was given because the professor is "racist".
If the student's gender differs from that of the professor's, then all the student has to do is claim that the failing grade was given because the professor is "sexist".
If the student happens to be a homosexual and the professor isn't, then all the student has to do is claim that the failing grade was given because the professor is "homophobic".
If the student happens to be a transsexual and the professor isn't, then all the student has to do is claim that the failing grade was given because the professor is "transphobic".
If the student happens to have a different religion than the professor, then all the student has to do is claim that the failing grade was given because the professor is "intolerant".
If none of those apply, then all the student has to do is claim that they were "raped" by the professor.
Thanks to "social justice" and it's twisted philosophy where the "victim" cannot be questioned, and the alleged "perpetrator" is surely guilty without any sort of investigation being allowed, any professors giving bad grades are likely to lose their careers, if not much more.
So the professors do the only sensible thing: they just give out A's and sometimes B's, even if the students deserve much lower grades.
That makes it much harder for a student to come back with a false accusation that will be treated as fact.
Would entry exams not ensure people who can and want to study get in and have the skills to keep learning for a few more years?
Its not the course.
The selection of people been allowed to attend should ensure a limited number of places go to people who put in some effort to show they had some study skills in past academic settings.
"personal and subjective as learning?"
Years of testing, standardized testing and good education should have sorted that issue out a few years before consideration entry into an advanced higher educational setting.
If the retention rate is low adjust the ways people can be accepted until an academic balance is restored so more of the study ready students get accepted.
Most people admitted should be able to write up their work, present their work as requested and pass exams on time depending on the course as required in a that kind of educational setting.
Domestic spying is now "Benign Information Gathering"
Because the reality of a test is this, if you already know something you probably don't care to much to repeat it.
The exception to that rule is if the repetition is for your personal gain. The truth of college is that it probably isn't for your personal gain.
MIT is an early adopter, assigning post-docs to help professors interpret this new data.
It's the post-docz fault, really. If they want to be employable they need at LEAST some post-post-doc education / research experience.
Surely then the old professors will retire / die and let one of the 35-year-old youngins fight over a single assistant professor position.
If they play their cards right by the time they're 50 they'll be an associate professor, and if they publish, publish, publish they'll make tenure and become a full vampire^w professor.
...and time consuming!
He seems to have put a lot of effort into it. But if everyone put that much time/effect into just teaching I imagine getting as much info. Half the info only seems to apply to him, noone actually does multiple delivery methods. The one example of finding a bad question shouldn't need big data....teacher doesn't read his own tests? How about when he corrects them and sees they all got the hard one right. PErhaps spending the time teaching instead of farming out as much of the workload as possible would get you similar results!
Interesting as a research project tho.
Doesn't mean that academics have any time or incentive to do anything based on it.
Academics are not rated on their actual teaching performance, they are rated on a) grant money brought in, c) research money brought in... y) pass rates, z) student satisfaction surveys. Note the complete absence of whether students actually learn anything as an evaluation criteria.
But, then, the universities that employ them aren't really rated on teaching quality either. While there is good teaching happening in the Ivy League and other top-rated institutions, there's also a hell of a lot of coasting on the smarts and work ethic of the students they select.
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
What appears to be being talked about doesn't even really count as analytics. Actual educational analytics requires that the student work out problems on a computer (possibly also using scratch paper), so that it can be noted in detail which particular steps are not being understood and then that addressed.
Mind you, while I've heard of this being done, I've never actually witnessed it. It sounds like a good idea, but one that would require an extreme amount of time from the teacher/professor unless it was implemented via an AI. And the AI that can really do that hasn't been invented yet.
I think we've pushed this "anyone can grow up to be president" thing too far.
What we read here is that the recent failures in our education system are not due to Government intervention, policy changes, and a forced influence of what we label progressivism. It's all because the professors don't have the right data to work with, so we need to fix it with statistical methodology. I call bullshit!
This is an attempt to market bullshit, because people are finally starting to grasp the fact that our education system (including so called Universities) has become bullshit Historically here is how we tested whether or not a person paid attention and learned something. We sat them down and give them a test. Every course I had in college required 6 blue books per semester. You had 2 tests and a mid term, then 2 tests and a final. You brought your bluebook, a pencil, and professors could let you bring in other materials used during the course if they wanted. Some courses were very easy to test this way, like Calculus, Chemistry, and Symbolic Logic. Some things were not "easy" to test this way, such as Ethics. Professors had assistant professors and associates for the more complex to grade, like English Lit which always seemed to have more aids working than students (probably for extra credits). They all had questions that if you paid attention to the materials you could answer.
Saying we now need statistical hullabaloo is laughable. Not even close. We need to get Universities to actually teach people instead of coddle them, and we need professors and Deans willing to do the same. College has become a joke on the public in order to grow government and make a few people rich. Even if you want to teach as a professor, the Deans will not defend you because they fear a student's opinion more than care for the Universities reputation. Yes, that's right. A history teacher can't teach history if it "offends" a student because feelings" now trump facts in college. That's fucking scary!
Before I get off my soap box, I do want to tell people that the generalization fallacies don't work. That one bad professor never causes an educational institution to collapse. In fact the market usually removed the professor within a pretty short time as nobody would sign up or pay for their courses.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
This is where we are at now in the USA :( Hopefully we enjoy the ride down.
Probably not very much, it is something that is too individuated. That actually goes for most things when it comes to big data, it's a red herring.
The problem is the attempt to correlate specific activities with learning, when in fact they are not the same thing. Collecting data on how often a person hits the nail with a hammer has no bearing on whether or not they can build a shelf. How difficult is it to teach a monkey to use a mouse? Do you somehow believe that the monkey who uses a mouse better than a person learned more history in a class? Even more appropriate, how easy is it to write a program which reports events that never occurred to a remote server?
In the words of Mark Twain, there are 3 kinds of lies. Lies, Damn lies, and Statistics.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
I see the evil work of IBM and their ilk at play.
"Be on your guard. There are older and fouler things than Orcs in the deep places of the world."
The whole theme of TFA seems to be TL;DR.
The students find course materials to be TL;DR:
One of the biggest surprises [a professor] found: Only half the students ever used the home page he had so carefully built for the course. Instead, many students just jumped to the homework, and only clicked to a reading assignment or lecture if they didn't know the answer to a question.
The professors find reading student assignments to be TL;DR, so they plan to read summaries of stats and cool-looking graphs instead:
Mr. Chuang was blunt with his colleagues at the conference about the typical faculty reaction to an older experiment with learning analytics, which flagged struggling students at the fifth week of classes, an approach many colleges have experimented with. "The number one thing our faculty would like is an ease in the burden of teaching so they can go back to research," he said.
But the profs then find the new analytics to be TL;DR, so they need to hire other people to look at them for them:
How did those professors react? "A big Huh, and Why?, and What can I use this for?" explained Mr. Chuang at a session at recent education-technology conference. "I think part of the challenge is addressing that gap of understanding the potential of analytics," he added. To do that, MIT started a Digital Learning Lab and hired postdoctoral fellows who serve as "ambassadors to a revolution" to help professors interpret the numbers.
This all seems rather ridiculous, particularly if you look at the ONE practical example in TFA of something useful they seem to have discovered through all of this complex data analysis (other than that students just don't do anything you don't force them to). That practical example was where a "difficult" problem on a problem set turned out to have an unexpectedly high number of correct answers from students. Why? The problem was poorly written and gave a clue away.
Seriously? That's the best example they can come up with?
I have a modest proposal instead of all of this "analytics" and TL;DR -- try READING.
If students do the reading, and professors actually look at the work they do... or at least have graduate teaching assistants look at that work, then the teaching assistants will discover where the questions are poorly worded and "too many" students are getting a tough problem right. We don't need to hire a new team of postdoc "ambassadors to a revolution" to collect hoards of data just to find errors in a problem set.
And professors should stop expecting college students to do readings or use materials if they aren't directly relevant to assignments or tests, particularly in online courses. Is it really surprising to ANYONE that students in online courses will skip over materials unless they actually need them to complete the assignments? Do people really need "analytics" to discover this? If you want students to use materials or do readings, then hold them responsible for them. Seriously. I remember when I was an undergrad -- I had some lecturers who were quite bad, so just stopped going to class. If I could get everything I needed to do well from problem sets and assignments, why waste my time? On the other hand, I had some lecturers who would spend a lot of time discussing materials necessary for the tests and stuff that was harder to get from readings, etc. -- so I'd go to those classes.
This is really basic pedagogical common sense. But instead, TFA seems to be recommending some sort of data-mining approach for the TL;DR generation, rather than just having everyone (students, faculty, teaching assistants) just spend a little more time paying attention to what's going on. Studies have shown that college students on average spend about half as much time actually studying/reading/etc. outside of class compared to 50 years ago. This kind of stuff doesn't seem to be improving that sit
That's a beautiful, passionate, almost poetic response. Sadly, it is completely wrong. None of that stuff ever happens at any significant rate outside of your fantasy life. In fact, I can't recall having heard of any student who ever made any of those claims, although I'm sure that there must be some, somewhere, that have (personality disorders being as prevalent as they are in the general population).
Actually, students have a pretty finely honed sense of fair play. If you flunk them and they deserve to flunk, you rarely get more than whimpering and sometimes begging. Ditto for most other grades, and grade boundaries. If you flunk them on a technicality, of course, they will be resentful -- and with some reason. But no, students do not in general argue over grades claiming that the professor is biased against them due to race, creed, color, etc, except possibly in the very rare cases where there is some reason to make the argument.
As for learning analytics -- I have to say that it is (also sadly) mostly bullshit. I don't know about soft subjects, but in things like math and physics:
a) It is painfully, oppressively difficult to find a good object instrument to measure "learning" at the college level. And I say this as somebody that has used what there is for upwards of a decade. The instruments themselves are badly flawed and it is impossible to prevent an instructor from teaching to the test if they so desire (indeed, it is difficult NOT to teach to the test if you know what is on it and what weight will be assigned to outcomes in terms of "ranking" teaching/learning performance in the course.
b) There is nothing like standardization of the courses at the level required to build a uniform instrument that might be of some use. In Europe they have such a thing, supposedly, and too bad for them! If Joe gives a wussy, "physics lite" algebraic physics course but Suzie gives a tough, full calculus course covering exactly the same chapters, how do you even compare them. Now imagine comparing them and developing performance analytics when they don't even cover the same chapters from the same book in the same order and with the same basic understanding of the material they are teaching...
Here's a single example of the problems we really do face. I give all of my entering physics students an assessment to determine how much they remember of basic math. A page of algebra. A page of simultaneous equations. A page of differential calculus. A page of integral calculus. A bit of vectors and trig. Nothing difficult as far as calculus goes -- one can manage a typical intro physics course with five -- that's right, only five -- integral/derivative rules on board, plus the chain rule/u-substitution, plus the product rule/integration by parts.
Every student entering the class is supposed to have passed two full semester college calculus courses. Yet the mean score on the assessment is around 50%, with plenty of students scoring as low as 15 to 25%. And these are bright students at a very good university.
Forget "analytics". The problem is deep, not shallow. It isn't going to be solved by improved statistics on more tests.
rgb
Even when the experts all agree, they may well be mistaken. --- Bertrand Russell.
It seems this measures how much time a student spends staring at a page. That maybe informative if all the material is controlled by the monitoring system but very little otherwise. Even the per-page values will be relative: Is the page in a chapter discussing difficult/complex ideas? Does the student have trouble with temporal/mathematical/linguistic/social concepts?
None of that stuff ever happens at any significant rate outside of your fantasy life.
Depends on your definition of 'significant'. It happens enough to frighten people, and the same outcome is achieved.
All bullying works that way.
Analytics like this might be capable of informing professors what doesn't work: if students aren't paying any attention to something, then either the professor should find a way to get them to pay attention to it, or should just drop it from the curriculum entirely.
They won't, of course, provide any information about what actually does work. But at least some things that don't work could potentially be eliminated.
I guess we should shut down all social sciences then. How can science tell us anything about personal and subjective stuff? Obviously big data only works when all data are the same.
Now that Mr. Stewart knows that, he is considering some design changes. One idea is to put course material on the assignment pages, where he knows all students visit.
Please don't! I go to the assignment pages when I want to do an assignment. Don't clutter it with other stuff. This is exactly why people who know what they're looking for don't use the homepage. We don't have the time to listen to each and every one of your lectures, just as you might have noticed that most students in a traditional university don't attend the lectures but still submit homework and pass exams. People have their own ways of learning, please respect that. Video/audio lectures are good to have, but they're also time-consuming, often boring (because they're adapted to slow, distracted learners and people who don't bother reading the textbook) and hard to search.
You must have gone to college before the rise of "social justice".
Possibly, but I was teaching in college after what you consider to be the rise of "social justice". Though quite why social justice is supposed to be a bad thing eludes me. Social justice is what Martin Luther King Jr was fighting for and, well, it worked.
Things are very different now than they were even just a decade ago.
Nope.
It's extremely risky for a professor to give anything less than a high grade to a student.
I failed a few students on my course. I got no pushback. I was told in advance to make sure that the distribution of grades from my students was similar to the distribution of grades the same group got in previous years (and was given the distribution, but anonymised), because otherwise something was probably wrong.
But then again, I don't have a damn clue who I passed and who I failed.
Because the exams are all pseudonymous. The students write their examination number on the paper and anyone involved in the exam has no mapping between examination numbers and names.
The thing is your paranoid whinging about "social justice" (again, social justice is not a bad thing) is completely inane. The much, much bigger problem is that the teaching staff develop a personal relationship with some of the students (not in a skeezy way, but over the years especially with tutoring, practicals and summer projects, you just get to know some of the students), and the anonymity prevents one from accidentally being biased for or against the student when marking.
Doing anonymised marking is simple, effective and prevents all sorts of biases.
SJW n. One who posts facts.
Except outside you fantasy life it doesn't even happen enough to frighten people.
I've certainly heard academics complaining about not being able to fail enough people because the university wants to collect its tuition fees, and that's only possible for students still on the course. I've never heard anyone complain that they couldn't flunk someone because of your hypothetical gender and race bullying.
SJW n. One who posts facts.
You must have gone to college before the rise of "social justice".
Things are very different now than they were even just a decade ago.
It's extremely risky for a professor to give anything less than a high grade to a student.
Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.
You must have gone to college before the rise of "common sense". This is not about "grades", it's about the quality of teaching. No teacher is perfect, and everyone can improve -- analytics are aimed at telling us what we're doing right and what we're doing wrong, and then we (in theory) give better lessons to all our pupils. Grades should improve as a consequence, but the goal of teaching is not "good grades", it's "learning". Grades are only one way of auditing learning -- a suboptimal way, but to date the only practical option that we have.
Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
Same experience here, students have had 2 years calculus and basic physics and they cant do trigonometry...
Martin Luther King, Jr., and the other civil rights activists of a better generation believed in exposing themselves to risk in order to secure an equitable future for their children. They were arrested, they were beaten, some were even killed -- but they won. The crop of whiners in college now (I teach too) wants absolutely no risk. They want no consequences for failure, no criticism of propaganda, no action that involves effort. Worse, they want what is not equitable: they want advantages simply for being a different race or gender, not, as the civil rights generation wanted, to find a place in society based on their abilities. They want safe spaces segregated by race and special bathrooms, they want affirmative action, they want qualified professors fired and less qualified professors hired based on skin color or gender "identity," and so forth. But, they won't actually risk anything to fight for these iniquitous advantages - just demanding something on Twitter is sufficient to scare the administration into submission.
In fact, I can't recall having heard of any student who ever made any of those claims, although I'm sure that there must be some, somewhere, that have
I guess nothing interesting has happened at the University of Missouri recently?
Wow. I guess that's why, despite being a white male, I've consistently failed students who did not do the work. Its disappointing to fail 20% of a class, but it certainly happens.
But my example is nothing. All you have to do is look at the press: every now and then there's a kerfluffle about student cheating and all sections in a class will be failed. And the professors involved are usually white males.
The reality is that nearly any grading scheme is defensible as long as it is applied uniformly and consistently. Other students that complained stopped when presented with the "changing your grade would not be fair to the other students" argument -- I've only had the one who replied with "you should give everyone an A".
That student did seek retribution, but unfortunately for the student I had, naturally, been documenting the lack of work through the grading process and a separate student performance report.
I do have concerns along the lines you have raised. The standard for a "microaggression" seems to be "I was offended" or even "I thought someone else should have been offended". The other trendy label is "triggered" which similarly has no meaningful criteria or bar to reach. Still, I've never heard of either instance being used to argue for grades.
... and where were grades raised in that discussion? Oh, right, they weren't.
The issue raised was a perceived lack of concern about racial issues on the part of the university administration. Every single instance I recall seeing reported was outside of the classroom (and some off of the campus).
"This content is available exclusively to Chronicle subscribers"
About the only pushback happens not because of race, gender, sexuality, religion or other "social justice" thing. Instead, it's money.
Sometimes a professor must take back a failing grade - not because the student didn't deserve it, but because their parents either are high ranking members of society (i.e., rich) and thus threatened to withhold their annual contribution, or the administration is worried that students may drop out, and depriving them of tuition funds.
Forget social justice. Anti-social justice is just as bad. "I'm rich" or "I belong to a rich family" or "My father owns your ass" is basically what it boils down to.
good information for this article. thank you visit my web that discuss about education too education
Again, do you have the slightest bit of actual evidence to support this? I know for a fact that students at Duke who's families have donated millions of dollars to the University have been given F's and expelled for cheating. Also, I do not understand these words: "take back a failing grade". No, professors never have to "take back a grade" that was correctly assigned. Duke has a written policy that grades CANNOT be changed once assigned. Most faculty have no idea who the family of any given student is and whether or not they are big donors -- names mean nothing and nobody comes around and says "Suzy Q comes from the powerful Q family, don't give her a failing grade".
This is all fantasy. It's all the stuff of bad novels, or Animal House quality movies, not reality. Neither I nor anyone I've ever taught with or heard of teaching in other courses has ever, ever, been questioned about a failure. Nor, for the record, are we pressured concerning athletes famous or otherwise that we teach. I won't say there is no sexism or racism at all, but it is a per-individual-teacher thing and is strongly opposed, in writing and in active practice, on pretty much every University community and, by the way, in law.
rgb
Even when the experts all agree, they may well be mistaken. --- Bertrand Russell.