Clever Clues Clobber Crossword Computer

← Back to Stories (view on slashdot.org)

Clever Clues Clobber Crossword Computer

Posted by Soulskill on Tuesday March 20, 2012 @09:16PM from the blithe-banter-bamboozles-brainy-bots dept.

Hugh Pickens writes "Steve Lohr reports that an impressive crossword-solving computer program called Dr. Fill matched its digital wits against 600 of the nation's best human crossword-solvers, finishing only 141st at the American Crossword Puzzle Tournament in New York. 'I wish it had done better,' says Dr. Matthew Ginsberg, the creator of Dr. Fill and an expert in artificial intelligence. Dr. Fill typically thrives on conventional crosswords, even ones with arcane clues and answers; it solved one of the most difficult puzzles at the tournament perfectly. But the computer does poorly with clever clues based on puns or jokes, because humans and machines solve the crosswords very differently. Humans recognize patterns based on accumulated knowledge and experience, while computers make endless calculations to determine the most statistically probable answer. The computer program is literal minded, and tends to struggle on puzzles with humor, and puzzles with unusual themes or letter arrangements. Take this clue from a 2010 puzzle in The Times: Apollo 11 and 12 (180 degrees). The answer is SNOISSIWNOOW, seemingly gibberish. A clever human could eventually figure out that those letters, when rotated 180 degrees, spell MOON MISSIONS. Humans get the joke, while a literal-minded computer does not. 'Occasionally, Dr. Fill just doesn't get it,' says Ginsberg. 'That's my nightmare.'"

70 comments

Min score:

Reason:

Sort:

He named it a pun, but it sucks at puns. by Anonymous Coward · 2012-03-20 21:24 · Score: 2, Funny

Can it manage ironic clues?
GIGO by Jimbookis · 2012-03-20 21:27 · Score: 2

It's not Dr Fill's fault for not getting it. Clearly it's Dr Ginsberg who is not getting it.
My favourite "not real" crossword clues by whyloginwhysubscribe · 2012-03-20 21:27 · Score: 5, Funny

If I see someone doing a crossword I usually say "I was stuck on a crossword the other day - the clue was 'very busy postman'". Eventually (sometimes it takes a while) they ask "how many letters" at which point you can say "hundreds!"

I'm such a funny guy...

Oh - another one is to say "seven up is lemonade"...
1. Re:My favourite "not real" crossword clues by Anonymous Coward · 2012-03-21 01:31 · Score: 2, Funny
  
  Perhaps unsurprisingly, one down is justifiable homicide.
2. Re:My favourite "not real" crossword clues by Inda · 2012-03-21 04:04 · Score: 2
  
  I went to a funeral the other day. The deceased was a crossword compiler.
  
  He was buried 6 down and 4 across.
  
  ^^ you can have that one for free :)
  
  --
  This post contains benzene, nitrosamines, formaldehyde and hydrogen cyanide.
I'd luck to congratulate by mapkinase · 2012-03-20 21:29 · Score: 5, Interesting

I'd luck to congratulate submitter on a clever title. Does not happen very often here.

--
I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
1. Re:I'd luck to congratulate by mapkinase · 2012-03-21 02:44 · Score: 1
  
  *like. If somebody thought that this was some kind of word play and modded up because of that (that's the only explanation of modups I have: my typo somehow made my comment clever beyond my understanding), it was not.
  
  --
  I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
2. Re:I'd luck to congratulate by Anonymous Coward · 2012-03-21 03:32 · Score: 0
  
  I'd luck to congratulate submitter on a clever title. Does not happen very often here.
  And it still hasn't happened, unless by "clever" you mean "something that a 12 year old would think was clever if he came up with it himself".
3. Re:I'd luck to congratulate by DaFallus · 2012-03-21 03:53 · Score: 1
  
  I have to respectfully disagree. I find these over the top alliterative titles to be incredibly contrived and annoying.
  
  --
  No one cares what your captcha was
  
  Houston TX, USA
Computers have no sense of humor... by dargaud · 2012-03-20 21:31 · Score: 1

...film at 11. And extended debugging session afterwards.

--
Non-Linux Penguins ?
1. Re:Computers have no sense of humor... by SnarfQuest · 2012-03-21 06:06 · Score: 0
  
  ...film at 11. And extended debugging session afterwards.
  Is that 11 across, or down?
  
  --
  Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
2. Re:Computers have no sense of humor... by dodobh · 2012-03-21 11:25 · Score: 1
  
  Non ex transverso, sed deorsum.
  
  --
  I can throw myself at the ground, and miss.
Poor example? by John+Pfeiffer · 2012-03-20 21:48 · Score: 1

Would the Apollo example really trip up a decently-written program though? I mean, my first thought was "Well, what if it had a fallback routine where it tries anagrams of possible answers?" so I have to imagine someone smarter than me has thought of that. I guess there's some limitation I'm not seeing...
On second thought...what am I still doing awake at 5:48am commenting on a post about crossword-solving computers?!

--

Friend: "The NIC is misconfigured..." Me: "No prob, I'll just telnet in and fix it." *Silence*
1. Re:Poor example? by N1AK · 2012-03-20 22:08 · Score: 1
  
  You could and chances are that each time it hits something new like this it can be improved a little and any clue about rotation/flipping etc will check for this kind of skullduggery. However just building in functionality to look for something that looks like the word if viewed backwards and upside down isn't automatically something you'd add to a crossword computer.
2. Re:Poor example? by Anonymous Coward · 2012-03-20 22:13 · Score: 0
  
  SNOISSIWNOOWI is not an anagram of MOONMISSIONS, so your suggestion wouldn't help.
3. Re:Poor example? by million_monkeys · 2012-03-20 22:19 · Score: 1
  
  I don't think it's that simple. Of course you could write the program to include answers made of letters rotated 180 degrees. But how often does an answer like that happen? I'm not a big crossword person, but i'd guess it's pretty rare. Maybe 1 out of 10,000 - if even that much? So while you might get that one clue right, meanwhile you've expanded the set of possible answers to include numerous elements with an extremely low chance of ever being correct. In addition to the added expense of considering them, you may inadvertently get wrong answers from including those options.
  Whether those issues would be addressed by making that search a fall back option depends on how the program works. For that particular clue, the program may have come up with another answer, in which case the fall back routine would never have been called. Or the fallback might produce false answers that are worse than not having an answer for a given clue.
  I guess the point is, you can add custom routines to account for every possibility you're smart enough to think up. And for common tricks, it's probably worth doing. But for what are essentially one off edge cases, whether those extra routines will actually help isn't clear. They may make things worse.
4. Re:Poor example? by dkf · 2012-03-20 22:23 · Score: 3, Funny
  
  Nor is it a word in any conventional sense, nor comprised of words. The only place it ever appears is in relation to discussion of this particular clue. The correct clever human response to such thing is to punch the setter in the face; they have broken the formal compact of crossword setting by using a non-word/non-phrase as an answer.
  
  --
  "Little does he know, but there is no 'I' in 'Idiot'!"
5. Re:Poor example? by NekSnappa · 2012-03-20 23:12 · Score: 1
  
  The correct clever human response to such thing is to punch the setter in the face; they have broken the formal compact of crossword setting by using a non-word/non-phrase as an answer.
  Exactly. I like doing crosswords, but it hate it when they do stupid shit like that. I really hate it when the answer is two words and there's no indication of it.
  
  --
  I want to shoot the messenger!
6. Re:Poor example? by fuzzyfuzzyfungus · 2012-03-21 00:18 · Score: 2
  
  TFA does make it sound like crossword puzzles are the paper equivalent of the most horrible, broken, 'adventure'/'puzzle' games of the 90s. The ones that were only solvable by either having a direct mind-meld with the developer and gaining mystic insight into "puzzles" or by brute-force-clicking every single pixel on every single ill-drawn background to interact with the entire set of interactable objects in the gameworld in all possible orders...
  
  There is a fine line between 'subtle clue' and 'inside joke with a population of one', and it is entirely possible to cross it.
7. Re:Poor example? by mwvdlee · 2012-03-21 00:57 · Score: 1
  
  Exactly. If not, then what's to stop somebody from using "bunch of random letters" as a clue to a bunch of random letters?
  OTOH, without such back-assed clues, a brute force attack would probably be more efficient at solving a crosswords than some AI.
  
  --
  Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
8. Re:Poor example? by bws111 · 2012-03-21 00:58 · Score: 3, Interesting
  
  Here is one that just appeared this week (LA Times, I think):
  Clue: Hail Answer: DANTESINFERNO
  Clue: Poe Answer: FLATBROKE
  Clue: What you need to get the above two answers: SOUTHERNDRAWL
  Not sure how you make a routine to come up with those answers.
9. Re:Poor example? by makomk · 2012-03-21 02:55 · Score: 1
  
  There is no formal contract. Also, if you look at the information on that particular puzzle that was linked in TFA, it appears the crossword setter pretty much gave the game away as to what the trick was in one of the other clues.
10. Re:Poor example? by operagost · 2012-03-21 04:43 · Score: 1
  
  That clinches it... not going to bother doing a crossword again!
  
  --
  
  Gamingmuseum.com: Give your 3D accelerator a rest.
11. Re:Poor example? by John+Pfeiffer · 2012-03-21 09:12 · Score: 1
  
  Christ! Those are... Those are just ridiculous... I guess what I didn't understand is what crosswords have become!
  Now I get what they mean by requiring some cleverness... I mean, there're PEOPLE who wouldn't get those! It makes a lot more sense now.
  
  --
  
  Friend: "The NIC is misconfigured..." Me: "No prob, I'll just telnet in and fix it." *Silence*
Not cryptic though by ISoldMyLowIdOnEbay · 2012-03-20 21:52 · Score: 1

All decent crosswords in the UK tend to be of the cryptic kind, rather than just needing a thesaurus most of the time. Writing answers backwards wouldn't be allowed, though, as the answer has to be an actual word. Here's one that a computer might struggle with.... V? (6,2,7) Answer: Centre of Gravity
1. Re:Not cryptic though by digitig · 2012-03-20 22:46 · Score: 1
  
  That would be considered an exceptionally poor cryptic clue, though, because one of the "rules" is that there should be some allusion to the -- or an -- actual meaning of the answer, however misleading. A better version of the clue would be "V is a source of strength".
  
  --
  Quidnam Latine loqui modo coepi?
2. Re:Not cryptic though by netwarerip · 2012-03-21 00:21 · Score: 1
  
  The computer would definitely struggle with that one because you spelled 'Center' incorrectly. :)
3. Re:Not cryptic though by CastrTroy · 2012-03-21 00:32 · Score: 1
  
  Not in the UK. Or Canada, or Australia. Actually I think the Americans are the only ones that have it wrong.
  
  --
  
  Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
4. Re:Not cryptic though by Col.+Bloodnok · 2012-03-21 00:59 · Score: 1
  
  A clue definition of '...V' is acceptable, if it follows on from a previous related question or answer. Even then it may seem related, but isn't. This will throw one off the scent, but the answer will often be part of an overarching theme. A theme often allows weaker definition cluing. Several other clues might include tangential references to science, but there will one clue that is often refered to by clue number only, elsewhere in the puzzle - e.g. 'Amphibian is working by force' (6) - hinting at the overall theme 'Newton'. I would also be on the look out for oblique references to the biblical character of 'Isaac'.
  Themes are particularly popular with Araucaria in the Guardian. He loves to bend the conventional rules and it is expected that he will. Which brings up another reason why this AI would struggle with UK cryptics - a priori knowledge of the setter's style, or even the house style.
5. Re:Not cryptic though by Col.+Bloodnok · 2012-03-21 01:14 · Score: 1
  
  Other one AI might struggle with:
  Lisping girl of legend (4) - (ans: 'Myth')
  How about some Cockney Rhyming slang?:
  Beehive in North London? (4, 6) - (ans: 'High Barnet')
6. Re:Not cryptic though by 91degrees · 2012-03-21 01:50 · Score: 1
  
  A really good thesaurus might help with that second one.
  
  Use of cockney isn't uncommon so it would make sense to include both of those in the definition for "hairstyle".
7. Re:Not cryptic though by operagost · 2012-03-21 04:46 · Score: 1
  
  Whoosh.
  
  --
  
  Gamingmuseum.com: Give your 3D accelerator a rest.
8. Re:Not cryptic though by tlhIngan · 2012-03-21 05:13 · Score: 1
  
  The computer would definitely struggle with that one because you spelled 'Center' incorrectly. :)
  There is a nasty one inside the crossword app on the Nook Color (not sure if it's in other Nooks, but I'm guessing it is) where the answer is spelled "CENTRE". The problem is the down answers really want "CENTER" to make any sense (one of the down ones was "TENT" which became "TRNT").
  Not sure if it was a typo or not. And the puzzles have no identifier so you can point it out.
9. Re:Not cryptic though by Anonymous Coward · 2012-03-21 16:35 · Score: 0
  
  Not difficult to hide the Cockney though.
  Intoxicated Walford do?
  To be fair, I made the Beehive one up in 30s, and it lacks a
10. Re:Not cryptic though by digitig · 2012-03-22 05:07 · Score: 1
  
  I'm not sure, but I seem to remember that it was Araucaria who laid down the general rules that most British setters are nowadays expected to follow. In which case he probably feels somewhat at liberty to bend them :-)
  
  --
  Quidnam Latine loqui modo coepi?
Cultural differences by Anonymous Coward · 2012-03-20 21:58 · Score: 0

Different cultures apparently have different rules for crossword puzzles. AFAIK, Finnish crossword puzzles would require that each answer must be a valid, independent word (in singular or plural). Moreover, in Finnish crossword puzzles, half of the clues are graphical.
1. Re:Cultural differences by mwvdlee · 2012-03-21 01:03 · Score: 1
  
  It probably has a lot to do with the frequency of letters in a language. I can't think of any reasonable way to fill out all the 4x4 blocks in your puzzle with actual words in my native language, Dutch. I'm guessing the distribution of letters in Finnish is much more skewed.
  
  --
  Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
Looks like by Anonymous Coward · 2012-03-20 22:31 · Score: 1

the computer needs help from Joe Piscopo
My favourites by troon · 2012-03-20 22:37 · Score: 1

I'd like to see if Dr Fill manages these two:
HIJKLMNO (5)
___ (2, 3, 4, 1, 4)

--
Ydco co ,df C erb-y go. a Ekrpat t.fxrapev
1. Re:My favourites by 91degrees · 2012-03-20 22:49 · Score: 1
  
  HIJKLMNO
  
  Yes, I remember seeing that one. One of those clues that filled me with delight when I got it.
  
  ___ (2, 3, 4, 1, 4)
  
  To Not Have A Clue?
2. Re:My favourites by dkf · 2012-03-21 01:52 · Score: 1
  
  HIJKLMNO (5)
  That reminds me of this one:
  ABCDEFGHIJKMNOPQRSTUVWXYZ (4)
  
  --
  "Little does he know, but there is no 'I' in 'Idiot'!"
3. Re:My favourites by Anonymous Coward · 2012-03-21 02:56 · Score: 0
  
  And:
  e (13)
  Answer: "Senselessness"
Redundancy by Hentes · 2012-03-20 22:50 · Score: 3, Insightful

Not being able to guess a few words might not be a problem, skip it and solve the other ones, once there are enough letters in it a computer can easily look up the available words, and if there are more than one even use a nonlinear approach. Even without any clues, a few words can't be that hard to bruteforce.
1. Re:Redundancy by Anonymous Coward · 2012-03-20 23:30 · Score: 0
  
  But all the competitors are already doing that. The computer is trying to win, not draw even.
2. Re:Redundancy by Anonymous Coward · 2012-03-21 00:06 · Score: 0
  
  It's not always that simple. Some years back one of the easiest puzzles at the competition was completely backwards. Every clue was written backwards so it was obvious that all the answers would be backwards -- well, obvious to eveyone but the computer. Just about every human got the puzzle right, but the computer got only one word just by shear luck.
  dom
3. Re:Redundancy by Hentes · 2012-03-21 00:11 · Score: 1
  
  A computer can bruteforce much faster than a human.
4. Re:Redundancy by CastrTroy · 2012-03-21 00:37 · Score: 1
  
  This is what I was thinking. Crossword puzzles should be pretty easy to solve if you can brute force the thing. If you can't solve something, solve all the other clues in the other direction, and you have an answer. Unless these contests don't actually solve whole puzzles, but rather are given a partial puzzle with some parts filled in, and have to answer particular clues. Also, are the contestants allowed to use dictionaries, thesauruses, and other reference materials? Because if they aren't it's even more disappointing how well this computer did, since it would be easy for the computer to contain an entire dictionary, and every crossword puzzle ever published in it's memory.
  
  --
  
  Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
5. Re:Redundancy by Bigby · 2012-03-21 00:37 · Score: 2
  
  A human can cleverly clobber a crossword computer with brute force.
6. Re:Redundancy by Hillgiant · 2012-03-21 02:01 · Score: 1
  
  Not necessarily. The computer likely double checks cross clues to verify any individual answer. If one of answers starts to appear to be jibberish, it will throw a large swath of intersecting answers in doubt. If the jibberish answer crosses a large portion of the grid, the doubt can propagate through the entire puzzle.
  
  --
  -
Nice post by normanlee · 2012-03-20 23:00 · Score: 0

Ya that is right and computer will randomly change the crossword puzzle. The writer has discussed a very good point that we humans recognize and take decision on patterns based on accumulated knowledge and experience, while computers make endless calculations to determine the most statistically probable answer not even knowing what are they doing. Bathroom Faucets
"That's my nightmare" by Anonymous Coward · 2012-03-21 00:01 · Score: 2

Unfortunately for Dr Fill's creator, the problem of how to get the program to work with such unorthodox solutions is the same as getting it to think like a person. At a certain point, all AI questions become the same AI question: this is the very essence of Turing horizon, and all such efforts converge there.
The program he wants to write is, sadly, doomed, as it will be impossible until such time as our species generates a true artificial consciousness with human intelligence, at which point the problem will be trivial - we will have much larger concerns that day.
Witty Watson Won by Anonymous Coward · 2012-03-21 00:02 · Score: 1

The crossword puzzle guy needs lessons from Watson, who clobbered several Jeopardy human champions. That show has clever categories and clues. Watson probably had more impressive computing power, but I doubt that was the issue. The Watson designers clearly had a better grasp of natural language, including humor-filled and storied language.
1. Re:Witty Watson Won by Anonymous Coward · 2012-03-21 05:40 · Score: 0
  
  Watson won, in part, because he was faster than his opponents. The way that Jeopardy is structured, Watson needed only to know a high percentage of the answers and be faster. It could (and did) fail miserably at some questions just like Dr. Fill did.
  If there was a prize for who could fill out 75% of the crossword puzzle first, I'm sure Dr. Fill could have won that easily.
this does not show humans are special by Anonymous Coward · 2012-03-21 00:46 · Score: 0

There is nothing magical about the example given. 180 degrees correlates with rotation. Apollo correlates to Moon missions. Rotation correlates to crew rotation, reversing numbers, rotating around the center... bingo. That fits. Humorous answers follow the exact same rules as any other answers.
different processes? by shadowrat · 2012-03-21 01:02 · Score: 1

Humans recognize patterns based on accumulated knowledge and experience, while computers make endless calculations to determine the most statistically probable answer.
what's different about that? I've often said, "The more I learn about AI, the more I think it lacks any intelligence at all. The more I learn about psychology, the more I believe that humans think just like an AI."

Humans are also determining the most statistically probable answer. They just have a better algorithm for factoring humor into those statistics.
Re:When is a moon mission not a moon mission? by OldeTimeGeek · 2012-03-21 02:02 · Score: 1

No, they're correct - you're just thinking of rotation on the wrong axis. To see how it works, try this: Take the word SNOISSIWNOOW and write it on a piece of paper. Rotate the paper 180 degrees so the lower right becomes the upper left. The part of the letters that were the bottom are now on the top - hence "W" becomes an "M" and the result is MOONMISSIONS.
Re:When is a moon mission not a moon mission? by grandpastackhouse · 2012-03-21 02:12 · Score: 3, Funny

*HSOOM*
Re:When is a moon mission not a moon mission? by Anonymous Coward · 2012-03-21 02:40 · Score: 0

Seems like you're a failed AI.
So Doctor Fill... by Oswald+McWeany · 2012-03-21 03:08 · Score: 1

So Doctor Fill has the same ability to comprehend humour as Doctor Phil?

--
"That's the way to do it" - Punch
Sounds familiar. by Anonymous Coward · 2012-03-21 03:11 · Score: 0

Blaine the Mono was unavailable for comment.
The two puzzles it tanked on by Anonymous Coward · 2012-03-21 04:33 · Score: 0

The first puzzle it tanked on (#2 in the tournament) had every other across line reading backwards. Thus, the answer to "Title in a Joel Chandler Harris story" was RERB, not BRER. I would expect a computer to fail at this.
The other one puzzle it tanked on (#5) had long words containing the trigram ANT split into to three parts, with the ANT portion connecting the beginning and the end by running diagonally through the grid. I'd expect the computer to fail on that one too.
However, it did very well on the straightforward, gimmick-less puzzles, cruising through the final puzzle, which was deemed the hardest by humans.
SNOISSIWNOOW by operagost · 2012-03-21 04:40 · Score: 1

Isn't having a gibberish answer in a crossword puzzle like making up your own words in Scrabble? Doesn't the creation of a crossword puzzle have any rules? No wonder I often do poorly with them; I had no idea that they could be making up nonsense words.

--

Gamingmuseum.com: Give your 3D accelerator a rest.
1. Re:SNOISSIWNOOW by SnarfQuest · 2012-03-21 06:17 · Score: 0
  
  Isn't having a gibberish answer in a crossword puzzle like making up your own words in Scrabble?
  All they need to do is claim it's Polish. Most puzzles in the major papers consist of 50% foreign words anyway. That allows them to make up very long words containing no vowels.
  
  --
  Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
I thought Crosswords... by thatbloke83 · 2012-03-21 06:38 · Score: 1

...were supposed to be composed of REAL words.
WTF is "SNOISSIWNOOW"??
1. Re:I thought Crosswords... by Anonymous Coward · 2012-03-25 17:43 · Score: 0
  
  the clue, and therefore not part of the crossword. ltr.
so it is just plain stupid... by Anonymous Coward · 2012-03-21 08:23 · Score: 0

so it is just plain stupid...
I hate crossword puzzles by dentin · 2012-03-21 12:38 · Score: 1

I can't blame the computer for not doing well on these; a lot of crossword puzzles are a puzzle of "guess what the creator was thinking", and not a puzzle of words and language. Quite frankly, I'm not interested in guessing what someone else happens to be thinking when they write down a clue like "blue, red, and big"; I find that fundamentally uninteresting and of no long term value.
I have the same problem with many Mensa puzzles. A lot of them I can do, but puzzles that require deep and specific information from an extremely narrow field are really not useful tests of intelligence. Just as the "SNOISSIWNOOW" question above requires deep and specific information about an extremely narrow field (the narrow field being "how certain subsets of human minds think"), so to do questions such as "guess the next number in the sequence: 3, 5, 205782654, 6, 308" (which also requires knowledge about how certain human minds think). Neither answer is derivable, and both must be guessed via trial and error from models that few people will have.
The most common response to this viewpoint has been along the lines of "ha ha, you just can't think outside the box". In reality, I'm actually pretty damned good at thinking outside the box. What I'm not particularly good at is thinking inside someone else's box, because if I actually need to know, I can simply talk to them and ask them. It's far more efficient.

--
Alter Aeon Multiclass MUD - http://www.alteraeon.com