Google Brain Researchers Make Significant Progress On Language Modeling (arxiv.org)

← Back to Stories (view on slashdot.org)

Google Brain Researchers Make Significant Progress On Language Modeling (arxiv.org)

Posted by timothy on Friday February 12, 2016 @04:43AM from the like-totally-whatever-man dept.

New submitter integralclosure writes: Using neural networks, Google Brain researchers have significantly improved a computer's ability to model English (achieving extremely low perplexity score on a large dataset). Using the model they were able to generate random sentences, such as the following: 'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.' The sentences are generally coherent and mostly grammatically correct. Advances seem to be a replay of neural networks' dominance in the Imagenet competition.

21 of 32 comments (clear)

Min score:

Reason:

Sort:

Perplexity by Geoffrey.landis · 2016-02-12 04:54 · Score: 4, Funny

"achieving extremely low perplexity score on a large dataset"
That's quite an achievement, since I have a very high perplexity score on that sentence.
Do they check whether it achieves low perplexity only when the sentence is not, in fact, perplexing?

--
http://www.geoffreylandis.com
1. Re:Perplexity by Noah+Haders · 2016-02-12 05:04 · Score: 1
  
  if you read the article, you would know that the perplexity score is related to how perplexing something is, and the dataset is the number of examples of sentences.
2. Re:Perplexity by gstoddart · 2016-02-12 05:05 · Score: 2
  
  I have the same issue with this one: Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.
  .
  WTF? Sounds like chess.
  
  --
  Lost at C:>. Found at C.
3. Re:Perplexity by Anonymous Coward · 2016-02-12 05:07 · Score: 1
  
  After reading this, http://www.zonalmarking.net/2011/03/01/chelsea-2-1-man-utd-tactics/, the sentence "Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence." doesn't seem like much of an accomplishment. SubReddit Simulator produces as much.
4. Re:Perplexity by Anonymous Coward · 2016-02-12 05:09 · Score: 1
  
  Football, well soccer.
  Stamford Bridge is the home ground of Chelsea. Drogba used to play for them. Dunno who Malouda was but Zhirkov was a Russian player who I don't think ever played for either Chelsea or Barcelona.
5. Re:Perplexity by gstoddart · 2016-02-12 06:17 · Score: 1
  
  Football, well soccer.
  Stamford Bridge is the home ground of Chelsea. Drogba used to play for them. Dunno who Malouda was but Zhirkov was a Russian player who I don't think ever played for either Chelsea or Barcelona.
  Oh, I though that was describing the technical breakthrough. ;-)
  Made no sense at all.
  
  --
  Lost at C:>. Found at C.
6. Re:Perplexity by davec727 · 2016-02-12 09:01 · Score: 1
  
  Florent Malouda was French and did play for Chelsea, as well as Lyon.
7. Re:Perplexity by dsmatthews9379 · 2016-02-12 10:29 · Score: 1
  
  https://www.youtube.com/watch?...
8. Re:Perplexity by Puff_Of_Hot_Air · 2016-02-12 17:37 · Score: 1
  
  Posting to undo incorrect moderation. ^^This looks to be a very informative lecture on the topic, for any other mods who may come along.
Not impressed by Coisiche · 2016-02-12 05:00 · Score: 2

'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.'
Barcelona playing at Stamford Bridge? That hasn't happened for years!
Slashdot by Anonymous Coward · 2016-02-12 05:27 · Score: 1

They should hook this up to the Firehose and have it edit Slashdot stories.
Brought to you by the letters... by __aaclcg7560 · 2016-02-12 05:35 · Score: 1

F and U shouldn't be that perplexing than?
How well would it understand this? by presidenteloco · 2016-02-12 05:36 · Score: 3, Interesting

Time flies like an arrow. Fruit flies like a banana.

--

Where are we going and why are we in a handbasket?
1. Re:How well would it understand this? by Half-pint+HAL · 2016-02-12 07:10 · Score: 1
  
  Time flies like an arrow. Fruit flies like a banana.
  That's an ill-formed utterance though. Clever joke, but not really real English.
  
  --
  Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
2. Re:How well would it understand this? by rwven · 2016-02-12 07:29 · Score: 1
  
  I think it's still a very useful thing to be able to understand though. English is absolutely loaded with ill-formed utterances in order to impart humor, sarcasm, intent, etc. It'll be very important for artificial systems to be able to reproduce and understand things like these.
3. Re:How well would it understand this? by Anonymous Coward · 2016-02-12 08:30 · Score: 1
  
  Time flies like an arrow. Fruit flies like a banana.
  That's an ill-formed utterance though. Clever joke, but not really real English.
  Wrong. It's not a joke and it's perfectly legitimate English. Admitedly, it's an absurd statement, but that's the point.
4. Re:How well would it understand this? by davec727 · 2016-02-12 09:03 · Score: 1
  
  That's a HIGHLY advanced function, though. For right now, it's far better to focus on correctly interpreting and generating meaningful, well-formed English. More casual language can come much later.
5. Re:How well would it understand this? by Half-pint+HAL · 2016-02-13 07:45 · Score: 1
  
  Ummm... what exactly do you think is "not really real English" about it?
  It's about what's called "generic reference". There are three generally used idioms in English for generic reference: 1) plural, zero article (fruit flies are insects); 2) singular, definite article (the fruit fly is an an insect) and 3) singular, indefinite article (a fruit fly is an insect). But that really only works for the subject of the subject of the sentence -- generic reference in grammatical objects is pretty much exclusively plural, particularly when dealing with the verb like, because we have the difference between "I like" and "I would like": "I like bananas" vs "I would like a banana". On top of that, it gets slightly unhinged by the fact that the generic reference to the fruit flies is in the plural, but the banana is in the singular.
  Oh wait... it is well formed if "flies" is a verb. It's just not ambiguous.
  
  --
  Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
Fail by wonkey_monkey · 2016-02-12 09:03 · Score: 1

Yuri Zhirkov was in attendance at the Stamford Bridge
it's just "Stamford Bridge", not "the Stamford Bridge".
Try again!

--
systemd is Roko's Basilisk.
Re:congratulations! by ceoyoyo · 2016-02-12 09:34 · Score: 1

Yes. Because it's not a simulacrum, it's a greatly simplified model. That it works at all suggests you've discovered at least a few of the important principles that let the vastly more complicated original work.
Also, you can poke at the model, see what improves it, what breaks it, plot receptive fields, all those things that are messy, difficult or unappreciated in an actual brain.
Modeling Language by techdolphin · 2016-02-12 12:45 · Score: 1

"The sentences are generally coherent and mostly grammatically correct," which means nobody will believe they are talking to a person.
I thought they said the program improved language modeling.