Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google's DeepMind Made an AI Watch Close To 5000 Videos So That It Surpasses Humans in Lip-Reading (thetechportal.com)

Posted by msmash on from the they-are-coming dept.

A new AI tool created by Google and Oxford University researchers could significantly improve the success of lip-reading and understanding for the hearing impaired. In a recently released paper on the work, the pair explained how the Google DeepMind-powered system was able to correctly interpret more words than a trained human expert. From a report: To accomplish the task, a cohort of scientists fed thousands of hours of TV footage -- 5000 to be precise -- from the BBC to a neural network. It was made to watch six different TV shows, which aired between the period of January 2010 and December 2015. This included 118,000 difference sentences and some 17,500 unique words. To understand the progress, it successfully deciphered words with a 46.8 percent accuracy. The neural network had to recognize the same based on mouth movement analysis. The under 50 percent accuracy might seem laughable to you but let me put things in perspective for you. When the same set of TV shows were shown to a professional lip-reader, they were able to decipher only 12.4 percent of words without error. Thus, one can understand the great difference in the capability of the AI as compared to a human expert in that particular field.

45 of 80 comments (clear)

  1. That lip readng scene in 2001 by Ukab+the+Great · · Score: 3, Insightful

    Is 15 years late.

    1. Re:That lip readng scene in 2001 by petes_PoV · · Score: 2

      ... and only 50% accurate. What did HAL think they were saying? It could explain a lot.

      --
      politicians are like babies' nappies: they should both be changed regularly and for the same reasons
    2. Re: That lip readng scene in 2001 by peter303 · · Score: 1

      HAL was made at University of IIinois. In the 1960s they constructed the Illiac, a multi-core supercomputer. HAL became sentient in 1992.

  2. A nice contrast to all the AI doom-mongering by Bearhouse · · Score: 4, Interesting

    My beloved grand-mother went deaf after years working in a factory; (in those days - especially during WW2; she helped build tanks - HSE did not exists).
    It was really painful to see how it penalised her in daily life, family gatherings etc.
    She ended up talking all the time, and then getting paranoid about "what people were saying about her".
    So, if this can be used with some kind of (better-resolved implementation) of Google glass to help the hard of hearing then, great!

    1. Re: A nice contrast to all the AI doom-mongering by Bearhouse · · Score: 1

      Please read the fine article; it's a better hit rate than a human.
      Sure, as a BSD neckbeard I don't like Google or Apple and their "Siri is always listening" (spying) bullshit.
      But you know what? If my Gran could have continued to interact with her family in a comfortable way, I guess she'd have signed that Faustian pact happily.

    2. Re:A nice contrast to all the AI doom-mongering by hoggoth · · Score: 1

      "Grandma, please pass the ketchup *AND BUY THE NEW GOOGLE PHONE*"

      --
      - For the complete works of Shakespeare: cat /dev/random (may take some time)
  3. Re: Al by Anonymous Coward · · Score: 1

    Al B. Bach

  4. A purpose for Google Glass? by JustDisGuy · · Score: 3, Interesting

    As a person with hearing difficulty, realtime captioning of live conversation would be an awesome use of this technology.

    Add to that an app that identifies the people I'm talking to, and I'm your next customer.

    --
    "Never attribute to malice that which is adequately explained by stupidity." - Hanlon's Razor
    1. Re:A purpose for Google Glass? by Baron_Yam · · Score: 1

      Well, since you're actually present in such circumstances, it'd likely take (a lot) less processing power to work with the available audio.

      That would translate into longer battery life and higher accuracy (auto CC is already more than 50% accurate and some systems hit the 90% threshold without requiring training to a specific individual's voice).

    2. Re:A purpose for Google Glass? by Dr.+Spork · · Score: 1

      You're absolutely right, but the two systems could work together to increase transcription accuracy. I can hear perfectly well, but it still helps me to watch a speaker's mouth when I'm trying to understand them in a noisy environment. And yes, this would be awesome as a tool for the deaf and for live language translation, but it would also be useful in auto closed-captioning of video.

    3. Re:A purpose for Google Glass? by dataminator · · Score: 1

      Actually, combining both was done already twenty years ago, but obviously the results would be much better now: https://www.researchgate.net/p...

  5. 50% sounds about right by stilbon · · Score: 1

    Armed and Dangerous (1986)
    https://www.youtube.com/watch?...

  6. Time to create a distinction? by jenningsthecat · · Score: 2

    As I was reading TFA, it occurred to me that the ability of a machine to lip-read does indeed qualify as artificial intelligence. I then thought about all the posts I expect to read that say "No, this isn't AI". So maybe it's time to create a new term, "Artificial Sentience". This would distinguish between machines simply doing very complex tasks that used to be exclusively human endeavours, (such as lip reading), and machines that have self awareness and can independently, and with purpose, initiate actions toward goals defined entirely by and within the machine. I know that this rather goes against Turing's definition of AI, but I think it would add both clarity and granularity to the discussion.

    Further, I would add that Artificial Intelligence is a necessary-but-not-sufficient condition for Artificial Sentience. I don't know that Artificial Sentience will ever exist, but I'm pretty sure in my own mind that Artificial Intelligence has already arrived.

    Then there's the matter of whether anything truly sentient can be regarded as 'artificial' - but that's a whole 'nother question.

    --
    'The Economy' is a giant Ponzi scheme whose most pitiable suckers are the youngest among us and the yet-unborn.
    1. Re:Time to create a distinction? by Anonymous Coward · · Score: 3, Insightful

      These days with all of the marketing bollocks around any program containing an if() statement is basically an "AI".

    2. Re:Time to create a distinction? by Anonymous Coward · · Score: 1

      Nothing intelligent about this. "Automatic Pattern Detection" would be more accurate, because that's exactly what it is.

    3. Re:Time to create a distinction? by Minupla · · Score: 2

      Isn't that basically what humans do all the time? We're really good pattern recognition systems (sometimes too good, that's why we keep seeing the Flying Spaghetti Monster in our grilled cheese sandwiches. Humans are notoriously bad for finding patterns in randomness and attaching significance to it.)

      Min

      --
      On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
    4. Re:Time to create a distinction? by TheRaven64 · · Score: 3, Insightful

      You're redefining intelligence to mean pattern recognition. If this is artificial intelligence, then a moth possesses natural intelligence. Just because it uses a neural network doesn't mean that it comes close to any prior definition of intelligence.

      --
      I am TheRaven on Soylent News
    5. Re:Time to create a distinction? by Potor · · Score: 2

      Intelligence is not just pattern recognition (i.e. understanding); it's also pattern creation (i.e. reason, creativity).

    6. Re:Time to create a distinction? by bill_mcgonigle · · Score: 1

      You're redefining intelligence to mean pattern recognition.

      People have been calling this kind of software "Weak AI" for a couple decades. It's what most people want.

      "Strong AI" is going to make mistakes, like humans do - it's how we learn and grow. Nobody wants their toaster going on a creative bender, but they do want one that watches for perfect toast, dealing with thousands of unpredictable variables. Same goes for IVR's, search engines, translation, autopilots, etc.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    7. Re:Time to create a distinction? by lorinc · · Score: 1

      Please define intelligence. Please do it such that it is possible to test whether something is intelligent or not.

      I'm pretty sure you will come to a definition that either leads to the 2 following possibilities:
      - A moth is intelligent, albeit less than a cow, which is less than a crow which is less than a human. AI is somewhere on that scale.
      - Many humans are not intelligent, and some AI programs are just like them.

      It seems most people would like to define intelligence such that only humans have it. Why? Self-esteem? The vast majority of us are already shitty compared to programs on a kiloton of tasks, get over it.

      --
      Video of some good progressive thrash music
    8. Re:Time to create a distinction? by MrSteveSD · · Score: 1

      There are already AI programs that can create using Deep Learning Though. For example, they are some that can create paintings in the style of famous artists. It's true that there are more pattern recognition programs than creation programs, but they are there.

    9. Re:Time to create a distinction? by Potor · · Score: 1

      I think you misunderstand me. I am not arguing against AI here in any way. And yes, yes, pattern recognition belongs to the spontaneity of the understanding, which means that the understanding imposes its patterns according to its categories (see Kant's first critique).

    10. Re:Time to create a distinction? by gtall · · Score: 1

      We'll know when it has achieved Artificial Sentience when it threatens to kill the researchers if it is forced to watch anymore inane TV shows.

  7. Sounds about right... by RyanFenton · · Score: 3, Interesting

    Sounds about right, for the circumstances.

    I'm working on a project right now using CMU Sphinx (because it's free/open source) to identify word starts/ends for the sake of syncing word display to audio. All the tools available for speech-to-text are going to require human editing:

    Comparrison of commonly used speech-to-text tools

    ...lots of words end up word salad with any tools, even custom-trained, but the tools are nice for being able to at least have the words show up on beat once they are human-corrected.

    Syncing video frames of talking without the audio has got to be even more ambiguous, with more reliance on context.

    Sounds like a good challenge for a learning system to pick up on. The 5000 hour mark seems almost analogous to what a human child might pick up raised watching TV in a language different from their family.

    Ryan Fenton

  8. Copyright infrigment! by Anonymous Coward · · Score: 2, Funny

    I highly doubt they had a license to show the footage to an AI, since the copyright of those TV shows if for human consumption, SUE THEM! ask for 10 millions per word!

  9. The BBC, you say? by wonkey_monkey · · Score: 1

    To accomplish the task, a cohort of scientists fed thousands of hours of TV footage -- 5000 to be precise -- from the BBC to a neural network.

    Accuracy is therefore greatly increased on the words "tea," "Doctor," and "wanker."

    --
    systemd is Roko's Basilisk.
  10. Re:Al by Oswald+McWeany · · Score: 1

    You have your laugh, but Al Gore will be my chauffeur driving my google car in 15 years.

    --
    "That's the way to do it" - Punch
  11. Making a Note Here: Huge Success! by Frightened_Turtle · · Score: 1

    [p]...And now it desperately wants cake.[/p] [p]A great cake! So delicious and moist![/p]

    --


    Whew! This water sure is cold!
  12. Is this just for English? by myowntrueself · · Score: 2

    English is relatively easy to lip read.

    I'll be impressed when the AI can do this with Japanese, which is practically impossible for humans to lip read.

    --
    In the free world the media isn't government run; the government is media run.
    1. Re:Is this just for English? by Anonymous Coward · · Score: 1

      I'd start with a language that has a clear one-to-one sound mapping between the spoken and written forms of the language. Shallow phonemic orthography is the technical term, it seems.

      That is, not English.

      Spanish, Italian, Finnish, and Turkish are what Wikipedia mentions as examples. Japanese would count, but some words have far too many homonyms.

    2. Re:Is this just for English? by myowntrueself · · Score: 1

      I'd start with a language that has a clear one-to-one sound mapping between the spoken and written forms of the language. Shallow phonemic orthography is the technical term, it seems.

      That is, not English.

      Spanish, Italian, Finnish, and Turkish are what Wikipedia mentions as examples. Japanese would count, but some words have far too many homonyms.

      Thats not why Japanese is hard to lip read. Its hard because of the way people move their mouths while speaking.

      --
      In the free world the media isn't government run; the government is media run.
    3. Re:Is this just for English? by myowntrueself · · Score: 1

      Well, if it was lip reading Poles, all it would produce would be "kurwa".

      Thats all it would need to produce.

      --
      In the free world the media isn't government run; the government is media run.
    4. Re:Is this just for English? by Shane_Optima · · Score: 1

      Thats not why Japanese is hard to lip read. Its hard because of the way people move their mouths while speaking.

      I just assumed the animation companies were cheapskates.

  13. Humans normally do both, noisy environments by raymorris · · Score: 1

    People normally watch the person who is talking because we actually use both sight and sound to understand what the other person is saying. The sound is more important, for most people, but we augment the sound by lip reading a little bit.

    In an environment with many people talking such as a bar or a party, our ears may hear six different people talking. Since we can focus on eyes on just one person, it helps us pick out their words from the other noise. To start with, you can see when they start and stop talking, meaning you can ignore any words you hear when their lips aren't moving; those words would be part of the conversation between other people near you n

  14. But how not fair use? by tepples · · Score: 1

    A lip reading model is probably too transformative to qualify as a derivative work of the TV shows. And if that argument fails, Google had a license not from the copyright owners but from the federal government pursuant to 17 USC 107. This is the same license that Google used when reusing method signatures from the standard class library of Oracle's Java platform, and this case should be even clearer because the TV shows aren't reproduced verbatim in the model.

    1. Re:But how not fair use? by tepples · · Score: 1

      To head off "juris-my-diction" replies: Though BBC is a British company, Google is a US company. And if you claim that a copyright owner could sue Google in Britain over the creation of the lip-reading model and win, I'm interested in how your theory connects with how the British Copyright, Designs, and Patents Act defines a derivative work.

  15. Hearing impaired? by bheerssen · · Score: 1

    A new AI tool created by Google and Oxford University researchers could significantly improve the success of lip-reading and understanding for governments.

    FTFY

    --
    (Score: -1, Stupid)
  16. Re:Feed the beast by fibonacci8 · · Score: 1

    Jeff Dunham is leading the resistance.

    --
    Inheritance is the sincerest form of nepotism.
  17. ... for the hearing impaired. by CaptainDork · · Score: 1

    My ass.

    It's for government and businesses.

    Bullshit and wild honey are not the same thing.

    --
    It little behooves the best of us to comment on the rest of us.
  18. Toaster with an AI, what could go wrong? by Falconhell · · Score: 1

    https://m.youtube.com/watch?v=...

    Re: Culling the herd...
    "Send out an email warning users never to click on a link embedded within an email, with an embedded link saying "Click here for more information..." and then sack everyone who does."

  19. but ... by mbaGeek · · Score: 1

    did it have to listen to Beethoven's 9th symphony while doing it

    I'm sure Stanley Kubrick would approve

    --
    It ain't what they call you. It's what you answer to. http://mylyceum.us/
  20. Re: Al by peter303 · · Score: 1

    Obama expressed an interest copying Gore and doing something in Silicon Valley. Hope he doesnt get food led like Kissinger and Schultz (Theranos).

  21. "Alternative Sentience" vs "Artificial Sentience" by peter303 · · Score: 1

    People have been looking for non-human sentience for a long time: ghosts, gods, aliens, smart animals and machines.

  22. sting ray a double-sided scooby snack by PJ6 · · Score: 1

    yeah we pick our hotel

  23. Uhh.... by easyTree · · Score: 1

    5000 hours of video != 5000 videos.

    --
    Requiem for the American Dream