Google Brain Researchers Make Significant Progress On Language Modeling (arxiv.org)

← Back to Stories (view on slashdot.org)

Google Brain Researchers Make Significant Progress On Language Modeling (arxiv.org)

Posted by timothy on Friday February 12, 2016 @04:43AM from the like-totally-whatever-man dept.

New submitter integralclosure writes: Using neural networks, Google Brain researchers have significantly improved a computer's ability to model English (achieving extremely low perplexity score on a large dataset). Using the model they were able to generate random sentences, such as the following: 'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.' The sentences are generally coherent and mostly grammatically correct. Advances seem to be a replay of neural networks' dominance in the Imagenet competition.

32 comments

Min score:

Reason:

Sort:

Perplexity by Geoffrey.landis · 2016-02-12 04:54 · Score: 4, Funny

"achieving extremely low perplexity score on a large dataset"
That's quite an achievement, since I have a very high perplexity score on that sentence.
Do they check whether it achieves low perplexity only when the sentence is not, in fact, perplexing?

--
http://www.geoffreylandis.com
1. Re:Perplexity by Noah+Haders · 2016-02-12 05:04 · Score: 1
  
  if you read the article, you would know that the perplexity score is related to how perplexing something is, and the dataset is the number of examples of sentences.
2. Re:Perplexity by gstoddart · 2016-02-12 05:05 · Score: 2
  
  I have the same issue with this one: Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.
  .
  WTF? Sounds like chess.
  
  --
  Lost at C:>. Found at C.
3. Re:Perplexity by Anonymous Coward · 2016-02-12 05:07 · Score: 1
  
  After reading this, http://www.zonalmarking.net/2011/03/01/chelsea-2-1-man-utd-tactics/, the sentence "Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence." doesn't seem like much of an accomplishment. SubReddit Simulator produces as much.
4. Re:Perplexity by Anonymous Coward · 2016-02-12 05:09 · Score: 1
  
  Football, well soccer.
  Stamford Bridge is the home ground of Chelsea. Drogba used to play for them. Dunno who Malouda was but Zhirkov was a Russian player who I don't think ever played for either Chelsea or Barcelona.
5. Re:Perplexity by gstoddart · 2016-02-12 06:17 · Score: 1
  
  Football, well soccer.
  Stamford Bridge is the home ground of Chelsea. Drogba used to play for them. Dunno who Malouda was but Zhirkov was a Russian player who I don't think ever played for either Chelsea or Barcelona.
  Oh, I though that was describing the technical breakthrough. ;-)
  Made no sense at all.
  
  --
  Lost at C:>. Found at C.
6. Re:Perplexity by davec727 · 2016-02-12 09:01 · Score: 1
  
  Florent Malouda was French and did play for Chelsea, as well as Lyon.
7. Re:Perplexity by dsmatthews9379 · 2016-02-12 10:29 · Score: 1
  
  https://www.youtube.com/watch?...
8. Re:Perplexity by Anonymous Coward · 2016-02-12 13:41 · Score: 0
  
  Maybe this AI will replace the editors.
9. Re:Perplexity by Puff_Of_Hot_Air · 2016-02-12 17:37 · Score: 1
  
  Posting to undo incorrect moderation. ^^This looks to be a very informative lecture on the topic, for any other mods who may come along.
Not impressed by Coisiche · 2016-02-12 05:00 · Score: 2

'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.'
Barcelona playing at Stamford Bridge? That hasn't happened for years!
1. Re:Not impressed by Anonymous Coward · 2016-02-12 06:12 · Score: 0
  
  also not impressed that a pile of silicon and software can replace a sportscaster. watch as ESPN races to cut costs ...
Slashdot by Anonymous Coward · 2016-02-12 05:27 · Score: 1

They should hook this up to the Firehose and have it edit Slashdot stories.
Brought to you by the letters... by __aaclcg7560 · 2016-02-12 05:35 · Score: 1

F and U shouldn't be that perplexing than?
How well would it understand this? by presidenteloco · 2016-02-12 05:36 · Score: 3, Interesting

Time flies like an arrow. Fruit flies like a banana.

--

Where are we going and why are we in a handbasket?
1. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 06:19 · Score: 0
  
  Fruits fly to your fly hoping for a banana.
2. Re:How well would it understand this? by Half-pint+HAL · 2016-02-12 07:10 · Score: 1
  
  Time flies like an arrow. Fruit flies like a banana.
  That's an ill-formed utterance though. Clever joke, but not really real English.
  
  --
  Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
3. Re:How well would it understand this? by rwven · 2016-02-12 07:29 · Score: 1
  
  I think it's still a very useful thing to be able to understand though. English is absolutely loaded with ill-formed utterances in order to impart humor, sarcasm, intent, etc. It'll be very important for artificial systems to be able to reproduce and understand things like these.
4. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 07:46 · Score: 0
  
  That's an ill-formed utterance though. Clever joke, but not really real English.
  Ummm... what exactly do you think is "not really real English" about it?
5. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 08:30 · Score: 1
  
  Time flies like an arrow. Fruit flies like a banana.
  That's an ill-formed utterance though. Clever joke, but not really real English.
  Wrong. It's not a joke and it's perfectly legitimate English. Admitedly, it's an absurd statement, but that's the point.
6. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 08:37 · Score: 0
  
  That's an ill-formed utterance though. Clever joke, but not really real English.
  It is technically correct. The *best* kind of correct.
7. Re:How well would it understand this? by davec727 · 2016-02-12 09:03 · Score: 1
  
  That's a HIGHLY advanced function, though. For right now, it's far better to focus on correctly interpreting and generating meaningful, well-formed English. More casual language can come much later.
8. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 09:22 · Score: 0
  
  I kid you not...it took me 10 years to figure that one out...
  LOL
9. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 09:57 · Score: 0
  
  >> Time flies like an arrow. Fruit flies like a banana.
  > That's an ill-formed utterance though. Clever joke, but not really real English.
  Looks OK to me. What's wrong with it? (non-native here)
10. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 12:56 · Score: 0
  
  Because s/he didn't get the last part of the joke.
11. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 16:31 · Score: 0
  
  Time flies like an arrow.
  S + V(+Adv) + O
  Fruit flies like a banana.
  S + V + O
  100% English form (SVO). Less than 25 words per sentence. The Shakespearian pun is used to emit wit.
  What we have here, is a sentence that is 100% English in style, substance and form.
12. Re:How well would it understand this? by Half-pint+HAL · 2016-02-13 07:45 · Score: 1
  
  Ummm... what exactly do you think is "not really real English" about it?
  It's about what's called "generic reference". There are three generally used idioms in English for generic reference: 1) plural, zero article (fruit flies are insects); 2) singular, definite article (the fruit fly is an an insect) and 3) singular, indefinite article (a fruit fly is an insect). But that really only works for the subject of the subject of the sentence -- generic reference in grammatical objects is pretty much exclusively plural, particularly when dealing with the verb like, because we have the difference between "I like" and "I would like": "I like bananas" vs "I would like a banana". On top of that, it gets slightly unhinged by the fact that the generic reference to the fruit flies is in the plural, but the banana is in the singular.
  Oh wait... it is well formed if "flies" is a verb. It's just not ambiguous.
  
  --
  Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
congratulations! by Anonymous Coward · 2016-02-12 07:58 · Score: 0

You built a simulacrum. Does it teach us anything about how the mind works?
1. Re:congratulations! by ceoyoyo · 2016-02-12 09:34 · Score: 1
  
  Yes. Because it's not a simulacrum, it's a greatly simplified model. That it works at all suggests you've discovered at least a few of the important principles that let the vastly more complicated original work.
  Also, you can poke at the model, see what improves it, what breaks it, plot receptive fields, all those things that are messy, difficult or unappreciated in an actual brain.
Fail by wonkey_monkey · 2016-02-12 09:03 · Score: 1

Yuri Zhirkov was in attendance at the Stamford Bridge
it's just "Stamford Bridge", not "the Stamford Bridge".
Try again!

--
systemd is Roko's Basilisk.
1. Re:Fail by Anonymous Coward · 2016-02-12 21:46 · Score: 0
  
  It was the worst of times, it was the blurst of times.
Modeling Language by techdolphin · 2016-02-12 12:45 · Score: 1

"The sentences are generally coherent and mostly grammatically correct," which means nobody will believe they are talking to a person.
I thought they said the program improved language modeling.