Replacing the Turing Test

mikejuk writes A plan is afoot to replace the Turing test as a measure of a computer's ability to think. The idea is for an annual or bi-annual Turing Championship consisting of three to five different challenging tasks. A recent workshop at the 2015 AAAI Conference of Artificial Intelligence was chaired by Gary Marcus, a professor of psychology at New York University. His opinion is that the Turing Test had reached its expiry date and has become "an exercise in deception and evasion." Marcus points out: the real value of the Turing Test comes from the sense of competition it sparks amongst programmers and engineers which has motivated the new initiative for a multi-task competition. The one of the tasks is based on Winograd Schemas. This requires participants to grasp the meaning of sentences that are easy for humans to understand through their knowledge of the world. One simple example is: "The trophy would not fit in the brown suitcase because it was too big. What was too big?" Another suggestion is for the program to answer questions about a TV program: No existing program — not Watson, not Goostman, not Siri — can currently come close to doing what any bright, real teenager can do: watch an episode of "The Simpsons," and tell us when to laugh. Another is called the "Ikea" challenge and asks for robots to co-operate with humans to build flat-pack furniture. This involves interpreting written instructions, choosing the right piece, and holding it in just the right position for a human teammate. This at least is a useful skill that might encourage us to welcome machines into our homes.

Comments

Re:When it's funny?

[Sarten-X]

The most common underlying basis of humor is subverted expectations. We expect people to behave according to the norms of society, we expect people to act to the best of their intelligence, we expect misfortune to be avoided, and we expect that words will be used according to their common meanings.

Subvert any of those expectations, and you have various kinds of humor. How funny a particular joke is perceived to be is related to how strongly the viewer is attached to their expectations. Since a computer is only an expert in the things they've been explicitly exposed to, it's very difficult to subvert their expectations. Watson would be familiar with all of the meanings of each word in a script, for example, so it would have a difficult time identifying the usual meaning that a human would expect from a situation, and would therefore likely fail to notice that when a different meaning was used, it was an attempt at humor.

As another example, consider a military comedy, like Good Morning, Vietnam. Much of the humor is derived from Robin Williams' fast-paced ad-lib radio show contrasting the rigid military structure, and the inversion where a superior at the radio station is practically inferior in every way. A computer, properly educated in the norms of military behavior, might recognize that the characters' behaviors are contrary to expectations, but then to really understand the jokes, the computer must also have an encyclopedic knowledge of pop culture from the period to understand why Williams' antics were more than just absurd drivel.

Finally, a computer must also understand that humor is also based largely on the history of humor. Age-old jokes can become funny again simply because they aren't funny in their original context any more, so their use in a new context is a subverted expectation in itself. Common joke patterns have also become fixed in human culture, such that merely following a pattern (like the Russian Reversal) is a joke in itself.

Algorithms simply haven't combined all of the relevant factors yet to recognize humor. Here on Slashdot, for instance, a computer would need to recognize the intellectual context, the pacing of a comment, the pattern of speech, and even how much class a commenter maintains, in order to realize when someone is trying to be funny.

Poop.