Jeopardy-Playing Supercomputer Beats Humans

An anonymous reader writes "Ok, this was just a practice round. But in a short demonstration today IBM's Jeopardy-playing supercomputer, a whiz by the name of Watson, thoroughly bested two talented human contestants. IBM has been working on this artificial intelligence project for years to prove that a computer can be programmed to understand conversational speech and wordplay. In today's demo, Watson seems to have proved the point: it started out on a roll in the category 'Chicks Dig Me,' about women and archaeology. The real man versus machine face-off (in which the same contestants compete for a $1 million prize) will be taped tomorrow, and aired in February."

Comments

"The pod bay doors cannot be opened."

[MrEricSir]

"What is the mission is too important for me to allow you to jeopardize it?"

Re:Soon, no more call centers

[Carnivorous Vulgaris]

They will give the AI a heavy Indian accent, because it's what callers expect.

Re:Soon, no more call centers

[Anonymous Coward]

Call center employees aren't allowed to be smart. They have scripts that they must follow. They're reduced to a very simple algorithm, executed by human beings only because there are still people who prefer talking to a other people over interacting with a machine, and because speech recognition software is still not ready to deal with what some people call speech.

A Rising Tide

[bughunter]

Best quote from the article:

Jennings says it’s worth noting that humans built the thing. Whoever wins, we win.

Truly. Although it sounds threatening to some, the practical applications of the natural language parsing technology will ultimately benefit everyone.

Until, that is, you dial your bank's customer service number from a noisy restaurant, and try to talk to Watson to ask him why your Visa was denied.

(Rutter's quote was a nifty Skynet allusion, but its syntax was mangled by the reporter/editor, so it comes in second best.)

So what this is saying...

[TheL0ser]

So the AI triumphed, and they're calling it a huge success? I wonder how the programmers feel about this. Pretty satisfied, I'd imagine.

Re:So what this is saying...

[Imagix]

"triumph". Did they make a note of their "huge sucess". Find it hard to overstate their satisfaction?

Re:Soon, no more call centers

[Wyatt Earp]

I worked Comcast call center for a while in '09. No scripts at all, when I started work and asked for something to follow, to figure out what I was expected to say, I was told with a smile "we don't do any scripts here, good luck!".

The only guideline I had was "Get their name and phone number, don't trust the system to give you an accurate phone number."

HUGE amount of secrecy surrounding this

[Faizdog]

So my neighbor works at the IBM facility where this is taking place, but in a completely unrelated function(it's a huge complex with a lot of people). He said that everyone is taking a forced day off on Friday when they will be taping the actual show. There's only going to be a small amount of the very top IBM brass there (supposedly even the head of this facility won't be allowed in). And that this is a HUGE secrecy issue (I'm guessing so that the results aren't leaked before the broadcast date).

My neighbor works with semiconductors and so works with a lot of dangerous chemicals and stuff. According to him, they've all been told to make sure that all their hazardous materials have been safely stored, and that (I have trouble believing this) even the IBM emergency response/hazmat teams have been told that they aren't allowed onsite and not to respond to any alarms that may be issued. That's a fairly dangerous decision if true, I'm doubtful but my neighbor stands by his statement.

Anyhoo, this is a pretty big deal apparently. More so from the Jeapordy people's end I'd guess since I don't think IBM has anything related to this project that they'd be that paranoid about keeping secret.

Re:HUGE amount of secrecy surrounding this

[afidel]

The local fire warden won't care about their secrecy if he gets alerted to a hazmat issue.

Re:HUGE amount of secrecy surrounding this

[Bigbutt]

Sounds like a lame Shadowrun mission.

GM: You discover that a big hush-hush project is underway Friday.

Street Sam: *rolls dice* 15 successes

GM: A little tidbit from the 'net. Emergency teams have been told not to respond to any alarms.

Street Sam: Excellent. A cakewalk. I could do this in my sleep.

[John]

When do they get the question?

[SeattleGameboy]

In the article, they mention that the computer gets the question as text. Does anyone know exactly when the computer receives the question? Does it receive when the human host starts talking or when the human host completes the question? If it is when the host starts speaking, the computer is getting at least several second head-start on the humans.

Re:When do they get the question?

[Hatta]

In the article, they mention that the computer gets the question as text.

Well that's cheating. With all the work that went into the natural language processing here, would it have been so hard to slap an OCR module in there?

Re:When do they get the question?

[MozeeToby]

The humans also get the question as text, at the exact same moment Watson does. That's the way it's always worked on Jeopardy!. They see the question as text the same you do when you watch the show on TV. The best competitors read well a head of Trebek and have an answer ready the instant they're allowed to buzz in, which as after Trebek finishes reading the question. Watson has the exact same advantages and disadvantages as any contestant, except that he can read the text basically instantly.

Re:When do they get the question?

[WMD_88]

Also, unless I'm mistaken, one doesn't have to wait for the entire question to be read. You can jump in early if you think you know the answer.

You're mistaken. The clickers to ring in are shut off until Trebek is finished reading out loud. (Jeopardy was probably the first quiz game to do it this way.)

Re:When do they get the question?

[bughunter]

In the article, they mention that the computer gets the question as text. Does anyone know exactly when the computer receives the question?

Well, remember this is Jeopardy, so the contestants receive the 'answer,' and must supply the 'question.'

And in the interest of even-handedness, I suspect that Watson is provided the text version of the 'answer' at the same time that the text of the 'answer' is revealed to the human contestants.

When I watch Jeopardy, I seldom wait to hear Trebek read the 'answer' aloud before I start figuring out the 'question.' It's available on the screen as text, and I can read much faster than Trebek speaks. (And I assume this is a key players' strategy in the live game, too.)

I just wonder how this all works for those "video answers" provided by the blonde cutie or the academic-looking bro...

Re:When do they get the question?

[FrootLoops]

Humans don't have to listen to Trebek; they can read ahead if they want. TBH he reads pretty slowly and if you can tune him out you might be at an advantage. A computer might be able to "read" faster than a human, but I don't see that as unfair.

Re:When do they get the question?

[HarvardAce]

Also, unless I'm mistaken, one doesn't have to wait for the entire question to be read. You can jump in early if you think you know the answer.

You are mistaken. You cannot buzz in until Trebek has finished reading the question. One of the reasons that Ken Jennings was so successful is because he had the timing down perfectly such that he was reliably the first one to buzz in when he knew the answer.

Re:When do they get the question?

[thomst]

In the article, they mention that the computer gets the question as text. Does anyone know exactly when the computer receives the question? Does it receive when the human host starts talking or when the human host completes the question? If it is when the host starts speaking, the computer is getting at least several second head-start on the humans.

It shouldn't matter much, because Jeopardy's rules lock the buzzers out until Alex has finished reading the question - and that lock-out period is determined by a human producer, who sits at a table off-camera, listening to Alex, and who has a button of her own that enables the buzzers.

Should a contestant try to "buzz in" before the producer pushes that button, his/her/its buzzer is locked out for three seconds - and any attempt to buzz in before that penalty period expires locks your buzzer out for an additional three seconds.

So, yeah, Watson may get to begin parsing the whole question a little early, but, typically, the human contestants get to begin working on it while Alex is still in the process of reading it, too - they just have to anticipate how it will end.

Given the speed of silicon vs. wetware, I agree that it will make a difference - but the real question is whether Watson has to determine that the buzzer is enabled by use of a light sensor (human contestants are notified by a ring of lights around the game board - which home viewers never get to see), or whether it gets notified electronically when the enable switch is activated. I say that, because, at least in my own experience, the ability properly to time the use of your buzzer is an enormous factor in whether you'll do well as a contestant or not.

When I was in the contestant pool in 1991, during the taping of the episode before I was chosen to compete, a four-time winner who was just a monster on the buzzer went up against two newbies. One of them, a little man from New Jersey, obviously became more and more frustrated as the game progressed, when he was unable to buzz in against the Monster, who completely dominated the game (the Monster was a word processor from New Mexico who played videogames as a hobby, so it wasn't surprising that his timing with the buzzer was extra-super good to begin with - and he'd had the non-trivial advantage of four previous games in which to hone his timing). Twice during Double Jeopardy, the New Mexico Monster declined to buzz in, which permitted the little man from New Jersey to do so. Both times, the question was a difficult and obscure one, and both times the little man from New Jersey failed to supply the correct answer, so, when Final Jeopardy came around, there was an empty podium where the little man from New Jersey had been, and, predictably, the Monster became a five-time champion.

Boy, was I glad I didn't have to go up against HIM.

Re:Soon, no more call centers

[OrangeTide]

maybe you should RTFA.

the unique challenges it poses to its contestants: the breadth of topics; the puns, metaphors, and slang in the questions; the speed it takes to buzz and answer.

Speech processing that can deal with the context heavy language of Jeopardy is a pretty big test and I think means we're just a little bit closer to a general purpose natural language speech recognition system.

Word play

[wiredlogic]

The ability to handle Jeopardy's style of word play is very impressive. I have to wonder if Watson can handle it in all the varieties that is is used on the show and whether the categories are cherry picked to match its abilities. Ideally the writers won't know that their answers are going to be used for the big game and the categories will be picked at random from a pool (minus audio and video clues).

Re:Now to use it to win the lottery

[SnarfQuest]

Just buy some of those "how to win the lottery" books, and program in those rules. You should be able to win every single one. Once you determine the winnig number, make sure you buy hundreds of winning tickets! Mortgage your house so you can buy as many as possible!

Re:Now to use it to win the lottery

[blair1q]

Really?

Quiz shows are designed with a targeted IQ in mind.

The people who write and select the questions know the percentages of how many people in their player population will know the answer.

Winning the game is dependent on the random distribution of the selected set of questions falling within the percentage of things you know.

Which means that it's not just as skill or talent game. You also have to be asked the right questions, and since you don't control that, it's luck.

Re:What? They didn't even videotape the demo?

[Samalie]

(Shitty) video at http://www.zdnet.com/blog/btl/ibms-watson-wins-jeopardy-practice-round-can-humans-hang/43601?tag=content;selector-blogs

Re:Soon, no more call centers

[coinreturn]

Actually, it doesn't do any speech processing - it does text processing - a HUGE difference. This is a shortcoming in my opinion. It should have to process Alex's voice, not clear text, though I guess it could do OCR on the screen instead. I bet they didn't have any Audio or Video Daily Doubles. In the show, audio and video clues are not uncommon.

Re:Buzz in Times

[MozeeToby]

Watson only buzzes in when he is confident that he knows the answer, which is apparently about 50% of the time. Of the 50% that he does buzz in, he answers correctly 80-90% of the time. If how the measure the confidence is accurate (and doesn't produce a lot of false negatives) it's likely that Watson would end the game in the red if he just buzzed in every time before he was sure of the answer. And according to the article, Watson has to physically push the button on the buzzer to buzz in. That probably doesn't delay him by much, but you're already talking about man vs machine when it comes to answering the questions, you might as well say that the humans have an unfair advantage because they speak English fluently.

Re:Soon, no more call centers

[parliboy]

Except that 1) Before And After is also a category on Jeopardy and 2) it was a category in run-throughs for Watson:

In a test tourney, Watson hit the bullseye on a question about clothing a young girl might wear on an operatic ship. The answer, pinafore, is also found in the title of the Gilbert & Sullivan opera H.M.S. Pinafore. And the computer was also successful with a before-and-after Jeopardy question about a candy bar and a Supreme Court justice, Baby Ruth Bader-Ginsberg. But earlier in its career, when asked, "What does a grasshopper eat?", it responded, "Kosher."

Source: http://www.philly.com/philly/blogs/storm/Jeopardy_A_Computer_Takes_on_Ken_Jennings_and_Brad_Rutter_.html

Re:Soon, no more call centers

[AchilleTalon]

Yes, definitely, you should RTFA.

This is very exciting news and for now, forget replacement of call center peoples, this machine is composed of 2800 Power7 cores, which renders it very expensive compare to the typical call center person. But this accomplishement, is a major step in the AI field and open it to many many exciting applications in the future which is not now too far.

Re:Soon, no more call centers

[BagOBones]

Engadget has a video of the match that includes the visualization of the response... In several cases it did not have an answer by the time the other contestant buzzed in and in some cases it was close to a tie.

In an interview it was stated it takes about 3 seconds on average to answer a question, which is actually kinda long when some of the humans will be able to predict the answer as soon as they have read or heard only part of the clue.

somebody call Ray Kurzweil

[Wes42h]

It sounds like soon we will be able to put another "a computer will never be able to..." on the scrapheap.

"A computer will never be able to beat a human at Jeopardy!"

Re:Verizon, Fedex already there

[PopeRatzo]

Interactive voice menus are probably singlehandedly the worst innovation in menu systems ever.

But tell me it's not entertaining to listen to someone else dealing with an interactive voice menu. It's a great one-sided conversation:

"Yes"

"Yes"

"Billing"

"I said BILLING!"

"Question about charge."

"Question about charge"

"Problem with charge"

"Jesus Christ. YOU CHARGED ME FOR SOMETHING I DIDN'T BUY!"

"No."

"No.

"NO NO NO NO! YOU FUCKING MACHINE! GIVE ME A HUMAN BEING!" (Followed by insane button mashing)

It's like an old Bob Newhart routine, if Bob Newhart had Tourette's Syndrome and anger management issues.

Re:Soon, no more call centers

[bws111]

Think for a moment about all the stuff you just hand-waved away, and you will begin to appreciate the problem. Let's take your "chicks dig me" example. You correctly identified that "chick" could mean baby chickens or females (it can also mean any young bird or a small child). "Dig" has a bunch of meanings, both as a noun and a verb. "Me" you just brushed off as "no relevance", but there are two problems with that: first, how is a computer supposed to know the word has no relevance, and secondly, it is VERY relevant to the category. Because of the wordplay, "me" is not referring to a person, it is referring to a LOCATION that was EXCAVATED by a FEMALE, so your answer had better either be an archeological site or a female archeologist. And it just keeps getting harder from there.

Re:Soon, no more call centers

[spazdor]

"TELL US HOW YOU CHEATED." "Okay fine, there's a chess-playing Turk concealed inside the cabinet."

Re:Soon, no more call centers

[Jerf]

You misunderstand in two ways. First, I wasn't hypothesizing. I was actually going from an earlier article about how the machine works. Second, this is a real computer, not a magical computer. It is doing something very hard and it takes many seconds to process a question and come up with an answer. When I say it is possible for a human to flat-out beat the computer, I mean that Alex can read the question, the lights can flash to let a human buzz in, and a human can have the correct answer ready while the computer is still churning away and doesn't even have a hypothesis ready. In fact the human is faster than even that implies because typically a Jeopardy contestant, especially at the championship skill level, already has the answer long before Alex has stopped speaking and now the competition is about who can buzz in correctly; Alex is reading for the benefit of the folks at home.

This computer is state of the art, and consequently and quite expectedly, it is barely capable of playing Jeopardy, basically. Next year, of course, for the same money you could build a machine 50% better for the same price, and next year, 50% better again for the same price, and so on... but this year it's just barely able to compete.

Re:Verizon, Fedex already there

[mjwx]

To be fair, I get the same with people, the machines aren't really any dumber.

Hold music fades...

Me: OK, I have a problem with my application.

Them: I can see your application.

M: Can you see how my name is misspelled.

T: Yes.

M: Can you fix it, it's just two transposed letters

T: I'll need you to send in your original birth certificate with a copy of your credit card, with passport, drivers license or other photo ID in addition to a urine sample, proof of noble linage, papal bull, letter from your doctor and bill less then three months old with your current address on it.

M: OK, can you cancel my application.

T: I'll need you to send in your original birth certificate

M: No I want to...

T: with a copy of your credit card, with passport, drivers license

M: No.

T: or other photo

M: Stop.

T: in addition to a urine sample, proof of noble linage, papal

M: SILENCE

T: ...

M: I would like to cancel my application.

T: I'll need to transfer you to another department.

Cue the hold music.