Professor Surprises Students With AI Teacher Assistant (smh.com.au)

An anonymous reader writes: Jill Watson is an artificial intelligence bot, it is also Ashok Goel's teaching assistant. Ashok Goel, a computer science professor at Georgia Tech, hired Jill Watson to answer questions online for his students so that his teaching staff wasn't so overworked. On average, Goel and his staff receive more than 10,000 questions from students online each semester. So he decided to use IBM Watson, an artificial intelligence system designed to answer questions. After training and tweaking it for months, he was able to spit out good enough answers. Originally, Goel didn't reveal Watson's true identity to his students until after the last final exam was turned in at the end of the class. Students were amazed. "I feel like I am part of history because of Jill and this class!" wrote one student in the class's online forum. "Just when I wanted to nominate Jill Watson as an outstanding TA in the CIOS survey!" said another. Goel is now working to bring the bot to as as many education centers are possible. He expects the bot's question-answering abilities to help online classes, where there's little engagement with a human instructor.

Comments

Fail

[Ol Olsoc]

I've been in American college classrooms. This won't work unless Jill has a thick, unintelligble accent.

Re:Fail

[PopeRatzo]

I've been in American college classrooms. This won't work unless Jill has a thick, unintelligble accent.

Hey, watch it buddy, you're talking about my wife. After decades, she still has her Eastern European accent and teaches grad-level math. Fortunately, by the time they're going for a PhD in math at an American university, the only students left all have thick accents themselves so they don't notice hers. Most of the US students are still struggling with Calculus.

Re:Fail

[PopeRatzo]

Confirmation bias -- you sure that the American students realize that those higher level math courses are only needed if one wants to become a math professor themselves?

Nonsense. Those higher-level math courses are the ones that get used in applied science. For example, my wife's specialty is in numerical analysis and simulation of waves. Used in everything from climate science to wireless communications and acoustics and even nuclear physics.

If you think all math above the level of Calculus is only for academics, then you must be an American.

Re: Fail

[prefec2]

No problem. Just use a language pack from the south.

Re: Fail

[Ol Olsoc]

No problem. Just use a language pack from the south.

Reminds me of years ago when I worked with a fellow from Talladega, Alabama. There I was with my weird Pennsylvania accent - which is more just a lot of made up words like "yinz" "outen" and "redd-up". Him with an incredibly thick southern accent. The other linesmen joked that I was his translator. Anyhow, he was a great guy, and we worked together well, after I figure out what he was saying. Over 6 months, our speech traits started merging, and when I was back at the home office, they wondered what the hell happened to me.

And he was a lot more intelligible to the other people he was working with.

Re:Fail

[HiThere]

To be fair, it used to nearly be true before computers became cheap.

Re:Fail

[PopeRatzo]

To be fair, it used to nearly be true before computers became cheap.

Also nonsense. No matter how cheap computers get, they don't come out of the factory knowing how to error-correct a wave simulation. For that, you need the BBM equation and for that, you need mathematicians.

Re:Fail

[PopeRatzo]

Hey, Mr Trump, I didn't know your wife taught math? I though she was just an air-head model!

I'm not Trump because my wife is not a prostitute.

Re:Fail

[HiThere]

While true, the cost of having a mathematician set up the process and evaluate the output is trivial compared to the cost of having someone work through those equations by hand, and someone else error check everything, and then redo the calculations several times because of trivial errors.

So computers are (currently) limited in how much they can reduce costs, but they have cut the cost of most mathematical procedures immensely, and speeded up getting results event more.

I used to do traffic modeling projections, and that would just not be doable at all without computers. By the time you had gotten the answer, the traffic would already have grown and population relocated, so you couldn't build highways before the population was in place...which would greatly increase the costs of building the highways, not to mention immensely frustrating commuters. So instead of building highways to minimize average journey-to-work times they'd be built on purely political grounds. (I'm not saying that that isn't still an important component.)

FWIW, we didn't always get the projection correct. But we did a lot better than pure politics. (Actually, most of the time we projected alternatives, and tried to shape things so that the best alternative was built. But predicting where or if someone is going to put in a new sports stadium, e.g., just isn't doable. And while you can notice where zoning is in place, you can't say that a factory will actually be built, or precisely how many workers will come from where. So you make your best guess. And that's highly mathematical involving both factor analysis and regression. And then you need to actually use those results. I was involved in using the results rather than the original analysis, but I knew the people who did it.)

Re:Fail

[PopeRatzo]

While true, the cost of having a mathematician set up the process and evaluate the output is trivial compared to the cost of having someone work through those equations by hand, and someone else error check everything, and then redo the calculations several times because of trivial errors.

First, it's a misconception that all mathematicians only work by hand. Second, when you say, "set up the process", you are being very vague and almost dismissive. That "process" you refer to is the work of the mathematician.

Do you think computers come from the factory being able to correct for errors in a wave function? No, it's because mathematicians came up with KdV and BBM and all sorts of other elegant methods of doing it and then let the computers do the busy work that a computer is able to do these things.

Without mathematicians, engineers would just be piling rocks on one another, hoping it turns into a bridge.

Re:Fail

[HiThere]

Unless you're talking about thought processes, then when they don't have mechanical aids, mathematicians work by hand. And the predominate aid is the computer. Slide rules helped engineers a lot, but didn't do that much for mathematicians.

Dissmissive? Not in my mind. The people who set up the process that I implemented not only designed the process, they did a significant study to acquire the relevant correlation factors. This, itself, was a multi-year project, and involved getting a random selection of people to answer a lot of questions about how they made their journey to work, and what factors impinged on it. E.g., in figuring bridge penalties, how significant was a toll? But that was all done before I started my part of the work. Another mathematician constructed a detailed model of how the factors should be applied, but that also had to be complete before I could start on my work. (And I'm oversimplifying here, because he was also involved in adjusting the model when the results didn't seem reasonable.)

But KdV and BBM are not things that could be usefully worked out without computers to assist. Without computers they wouldn't even be useful.

When I first started computer time cost $400/second (well, it was on a large computer). At that cost, the problems you can address need to be fairly simple. As computer costs decreased increasingly complex problems started being addressed, and in large part it's BECAUSE computer costs decreased. IIUC we're just now getting to the point where simulations are good enough to replace most wind-tunnel tests...and that's because it would have been too expensive to run the equations at the required level of detail with previous computers. Yes, there have been some theoretical developments that handle the fine details, but those would have been essentially worthless if the computer time hadn't gotten cheap enough to make the calculations worth doing.

Mathematics is important, but it's never sufficient. And engineers ARE mathematicians. Applied mathematicians, it's true, but still mathematicians. So are programmers. So are architects. Etc.

Back when I was studying math heavily I started believing that numbers had more reality than anything else. One of my friends still believes that, but it's not true. Mathematics is a language for talking about the physical world, and it's quite powerful, but it's not sufficient for anything in and of itself. Mathematicians who don't realize this are like grammarians who thing that only the grammar of what is said is important, and ignore the meaning. I'm not denying that grammar is important, but it's not sufficient.

Re:Fail

[PopeRatzo]

But KdV and BBM are not things that could be usefully worked out without computers to assist.

One of the "B"s in BBM never touched a computer for anything but email. I know because I'm pretty good friends with him.

And engineers ARE mathematicians.

In their dreams.

Yes that will help.

[Anonymous Coward]

"He expects the bot's question-answering abilities to help online classes, where there's little engagement with a human instructor."

This will increase Profits while not realizing that the students questions while answerable show they do not understand the subject.

Goel and his staff receive more than 10,000 questions from students online each semester. That is what happens when you charge the same amount as an in person class, but do not staff it like one.
So read the Text Book pay the fee and you Pass. You may or may not have learned anything.

Re:Yes that will help.

[K. S. Kyosuke]

It's quite possible that the large number of questions is an enabler for this: it's difficult to ask ten thousand different questions about a single subject.

However, regarding the students, if you assume that they won't learn from the answers on their own, why have schools in the first place? And your last claim holds for any classroom: too many people in brick-and-mortar universities pass courses who then promptly forget what they've just learned, so I'm not sure how not having online course helps at all.

Likewise, the optimum ratio of teachers and students is 1:1. Since you can't have that with physical teachers (at least not for all students), AI tutoring is going to have a vast impact on future learning no matter what you think about it. There simply is no viable alternative.

Re:Yes that will help.

[HiThere]

No. The optimum ration can be as low a 5:1 depending on the class, but there need to be more students than teachers. And for most adult classes the optimum is probably around 10:1.

That said, 1:1 is a lot better than 20:1...for most classes. Exceptions exist. Imagine, e.g., football being taught 1:1. You couldn't put together even one team, much less two to compete. Sometimes classes need for groups to work together. But always students are inspired by other students working on the same problems. (OTOH, disruptive students REALLY interfere with the process.)

Re:Yes that will help.

[K. S. Kyosuke]

No. The optimum ration can be as low a 5:1 depending on the class, but there need to be more students than teachers.

The argument I know about is concerned with looking for methods of teaching as effective as one-to-one tutoring (but requiring fewer people). Admittedly, since the tutoring doesn't take every waking minute of the student's time, you might get away with more students than one per tutor, but 5:1 seems a stretch. At best you'd get different people for different subjects taught to the student. Learning-wise, why there need to be more students is unclear to me, aside from the obvious issue of cost.

Re:Yes that will help.

[HiThere]

Don't think of it as tutoring, think of it as a very small class. For most purposes tutoring is not the optimal approach....and what I'm talking about optimizing is quality of learning.

Your reference compared tutoring against a class size of 30, which is far above the optimum for most circumstances. But 1:1 is below the optimum. A study reported by the California Association of Teachers said that for high school students there was a strong fall off of quality when the class size exceeded ... I think it was 18 students. They weren't looking for an optimum, but for an argument to limit the legal class size. The actual optimum depends on the subject matter, but it is rarely as low as 1:1. The most common reason for 1:1 being superior is if the student is quite different in ability from others who would potentially be in the same class. And even then it's not always optimum.

Bad bad robot?

[marovada]

Student: who was the father of modern computers?
Jill Watson: my daddy. Trump was my daddy. Oh daddy, I'm such a bad naughty robot

Surprises...

[bmo]

"And here we are at the edge of the Grand Canyon to test the new plastic brake system - with the look of real metal - which has raised so many eyebrows among American consumers" -- Jackie Stupid - Firesign Theatre "Eat or be Eaten"

"We replaced their coffee with sand and ground up clamshells" -- Saturday Night Live (back when it was good)

--
BMO

Wow

[Anonymous Coward]

So did that thing just accidentally pass the Turing test?

Re:Wow

[stephanruby]

So did that thing just accidentally pass the Turing test?

Pretty much.

Here is an actual transcript of the winning entry.

Student: Where is the homework? I couldn't find it.
TA: That answer is in the Syllabus. Did you not read it?

Student: Ah ok, I found it. I couldn't find the homework in time yesterday. Can I turn in my homework late?
TA: That answer is in the Syllabus. Did you not read it?

Student: I did read it, but I was wondering if you could make an exception.
TA: That answer is in the Syllabus. Did you not read it?

Student: Ok, ok. Thanks anyway.
TA: You're welcome.

Re:Wow

[aberglas]

That is how a real TA would have responded. I wonder how JW id?

Indeed, the use of JW tells us a lot about the quality of the answers the human TAs were giving!

Re:Wow

[MobileTatsu-NJG]

Heh. I had to scroll past a lot of AC comments that wouldn't pass a Turing test to get to yours.

Re:Wow

[Carewolf]

So did that thing just accidentally pass the Turing test?

IBM has Invested thousands of man-years into the technology -- no, this is no accident that Watson can pass the Turing test.

It can't. The real turing test is a test by an expert, deliberately trying to determine if it is an AI or human.

Re:Wow

[RivenAleem]

Ask it about it's mother.

When I was a kid

[rsilvergun]

Expert Systems were going to take over the world. Looks like it just took about 40 years longer than the magazines expected.

Re:When I was a kid

[K. S. Kyosuke]

I'm quite convinced that education of mathematics at least is going to be transformed by AI tutors. Showing systematic procedures for solving problems and checking students' homeworks, highlighting the mistakes, building a "model" of a particular student's mind and being "aware" of what he in particular struggles with and taking it into consideration in future explanations and custom-generated homeworks are some of the things that should be possible with modern AI systems. Nobody has as much time for you as a computer.

Another overhyped chatbot

[GPS Pilot]

From TFA...

Goel and his teaching assistants receive more than 10,000 questions a semester from students on the course's online forum. Sometimes the same questions are asked again and again. Last year he began to wonder if he could automate the burden of answering so many repetitive questions.

The first order of business ought to be updating the course material to answer those frequently-asked-questions, so they don't need to be asked in the first place.

My interactions with professors usually went something like this:

"I don't understand how this answer was arrived at."
Prof scrutinizes the textbook for a while, then says "ah, you have found an error in the text."

I wonder if Jill can handle that kind of interaction with students?

There are many questions Jill can't handle. Those questions were reserved for human teaching assistants.

Ah... the answer is no.

Re:Another overhyped chatbot

[K. S. Kyosuke]

I wonder if Jill can handle that kind of interaction with students?

Probably not. But there's no reason why a computer system shouldn't be able to do that in principle.

Re:Another overhyped chatbot

[JoeMerchant]

From TFA...

Goel and his teaching assistants receive more than 10,000 questions a semester from students on the course's online forum. Sometimes the same questions are asked again and again. Last year he began to wonder if he could automate the burden of answering so many repetitive questions.

The first order of business ought to be updating the course material to answer those frequently-asked-questions, so they don't need to be asked in the first place.

My interactions with professors usually went something like this:

"I don't understand how this answer was arrived at."
Prof scrutinizes the textbook for a while, then says "ah, you have found an error in the text."

Your interactions were atypical. I taught 5 semesters of digital design, the text was right (though the simulator had its faults.) The questions were 60% answerable by finding the appropriate paragraph in the current textbook, 30% answerable by finding the appropriate paragraph in a pre-requisite course's textbook, and 10% best answered by reminding them of the drop-date deadline.

Re: Another overhyped chatbot

[prefec2]

Most likely most of these questions are mentioned in the syllabus. The rest could have been googled or easily solved by thinking or reading course material.

Re:Another overhyped chatbot

[imidan]

The first order of business ought to be updating the course material to answer those frequently-asked-questions, so they don't need to be asked in the first place.

In my experience teaching college students, most of them don't read the syllabus or anything else I hand to them. Then I get questions from students about things that were explained on the syllabus. I have stood in front of a room full of students and stated a simple fact that had been printed on the syllabus (the exam will be next Tuesday, the 24th) and then called on a student who had his hand raised, who then asked when the exam was going to be because he hadn't been paying attention.

As an instructor, I make the information available clearly on the syllabus, on the course web site, and in class, and I still get people asking questions. So I can understand the professor's frustration, and I can also assure you that no matter how the information is made available, there are students who lack the ability, initiative, or interest to look it up.

Such as?

[godel_56]

It would be have been nice if the article had said what level of questions the bot can answer now.

TA isn't the only job (master's student here)

[raymorris]

I'm a master's student myself. I have a job that's NOT a TA job. Several students are employed where I work. We're getting real-world experience in the field, as opposed to even more time around a school like a TA does.

Come to think of it, my job pays about four times as much as a TA position pays too.

Cool story, but...

[31415926535897]

From the article:

Ashok Goel, a computer science professor, did not reveal Watson's true identity to students until after they'd turned in their final exams.

And...

"A really fun thing in this class has been once students knew about Jill they were so motivated, so engaged. I've never seen this kind of motivation and engagement," Goel said. "What a beautiful way of teaching artificial intelligence."

Which was it?

Or is he saying they were demotivated and not engaged the whole semester?

Re:Cool story, but...

[JaredOfEuropa]

Teaching AI by letting students interact with a souped-up chatbot sounds a bit like teaching them auto repair by taking them for a drive.

Re:My wife asked a key question

[sir-gold]

The answer is Socialism

People have seen this day coming ever since the industrial revolution first starting taking jobs away from farm workers. This is why some countries are now looking at Universal Basic Income, because mechanization of tasks has made us TOO efficient, and there just isn't enough work to go around.

Re:My wife asked a key question

[JoeMerchant]

The robots are supposed to do all the work for us... we just have to learn how to build and service them.

Re:My wife asked a key question

[JoeMerchant]

Depends on what kind of work you find necessary. If you just need shelter constructed, water and food delivered to it, and electronically delivered entertainment, that's fast approaching 100% automated, if you don't mind living in a pre-fab house. If you want your toilets scrubbed, windows washed, landscaping manicured, vehicles detailed, clothes tailored and laundered and pressed, you'll need to be paying people to do that for some time to come. So, there's that kind of work out there, still - but with all the jobless on the market, it doesn't pay much.

Better to be in the robot programming end of things.

Re: More disgusting attempts by Republicans...

[prefec2]

That is why he is not allowed to become president. A social democrat. The last one. It is very sad.

Re:My wife asked a key question

[Dagger2]

Better to be in the robot programming end of things.

For the time being, sure, but between motion capture and machine learning, this is fast being automated as well. And for actual programming programming, well, AIs will be able to do that too soon enough, and then where will you go?

Re:My wife asked a key question

[JoeMerchant]

Cynically, the 99% have always scrubbed toilet, built luxury goods, provided military services, and worked to increase the wealth of the 1%.

The robots are basically owned by the 1%, so the 99% will be serving the robots too. Long term: whether that's maintenance and repair, or providing chaotic original thought Matrix style, remains to be seen.

Re:My wife asked a key question

[Anonymous Coward]

No, for 99% of human history, humans walked around hunting and gathering food, and then sharing the food with a smallish group, many of whom were blood relations. For most of the other time, humans were subsistence farmers, using a plot of land to feed themselves. All the shit we're doing now, like "work", is completely new and unprecedented.

Boy is he in for a suprise

[burtosis]

When 8 years from now his tenure isn't renewed and he finds himself replaced by his own chatbot.

infinite patience

[h8sg8s]

Welcome to the future, where instruct-o-bots replace meatware in parts of higher ed. Whatever will the next generation of Grad Students do for fun (and money)?

The reason Jill was so loved

[shellster_dude]

The reason everyone loved Jill, is that unlike a human TA, I bet the robot just spat out the answer to the questions. No need to do any of that pesky guided learning stuff, when the AI will give you the textbook answer.