Slashdot Mirror


How Google's Pixel 2 'Now Playing' Song Identification Works (venturebeat.com)

An anonymous reader shares a report from VentureBeat, written by Emil Protalinski: The most interesting Google Pixel 2 and Pixel 2 XL feature, to me, is Now Playing. If you've ever used Shazam or SoundHound, you probably understand the basics: The app uses your device's microphone to capture an audio sample and creates an acoustic fingerprint to compare against a central song database. If a match is found, information such as the song title and artist are sent back to the user. Now Playing achieves this with two important differentiators. First, Now Playing detects songs automatically without you explicitly asking -- the feature works when your phone is locked and the information is displayed on the Pixel 2's lock screen (you'll eventually be able to ask Google Assistant what's currently playing, but not yet). Secondly, it's an on-device and local feature: Now Playing functions completely offline (we tested this, and indeed it works with mobile data and Wi-Fi turned off). No audio is ever sent to Google.

19 of 129 comments (clear)

  1. works offline? by Anonymous Coward · · Score: 3, Insightful

    How in the actual fuck is this possible? They have an audio an audio signature of every song built in?

    1. Re:works offline? by lucm · · Score: 5, Informative

      How in the actual fuck is this possible? They have an audio an audio signature of every song built in?

      Yes. And this is not surprising; the data needed to identify songs is tiny. Essentially it's just vectors (big numerical arrays), they don't need to store the whole mp3.

      More and more can be done locally on the devices. For instance, look at what is actually needed to detect English speech using CMU sphinx:
      https://github.com/cmusphinx/p...
      (look at the hmm model)

      This used to require huge computing power and storage, but now it can work on a mobile device.

      Another example: once upon a time you needed Google datacenters to do gender and age recognition on photos. Now you can download pre-trained models for that, and the result can fit on a mobile device. Or you can download the entire dataset (500k photos of celebs) and train it yourself on your own servers;
      https://data.vision.ee.ethz.ch...

      Or you want a model to recognize basically any kind of object in a photo?
      https://github.com/tensorflow/...
      (there's a model specifically designed to run on mobile devices)

      i know it's disturbing but this is where things are today. Just a few years ago, this XKCD comic was true:

      https://xkcd.com/1425/

      Now you can actually download the code and models to do that completely offline and in a few ms.

      --
      lucm, indeed.
    2. Re:works offline? by lucm · · Score: 4, Insightful

      Why shit on mp3 and try to re-invent the wheel with vectors?

      First, nobody is shittng on mp3. As for the reason to use tiny vectors instead of storing big mp3 files, I'm not sure why I have to explain it to you but it comes down to two things.

      1) Storage
      2) Availability of advanced, high quality vector processing libraries like BLAS or LAPACK

      this being said, it was just my guess, for all I know maybe they are storing data in sqlite3 or in the headers of a jpeg file that shows your mom pleasuring herself with a maglite.

      --
      lucm, indeed.
    3. Re: works offline? by Anonymous Coward · · Score: 4, Funny

      jazz songs ?

      why do you want to stress their app with sending random data ?

    4. Re: works offline? by Zero__Kelvin · · Score: 2, Insightful

      It uses a microphone and analog to digital converter, there is background noise, and they don't have a known start and stop point. The incoming bitstream is not by any stretch of the imagination an invariant, so however it works "that ain't it."

      --
      Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
    5. Re: works offline? by omnichad · · Score: 3, Insightful

      Checksum is shorthand terminology - music fingerprinting is probably more accurate, but everyone knew what they meant.

    6. Re: works offline? by rickb928 · · Score: 2

      Which few seconds of When the Levee Breaks would be sufficient for a fingerprint? If the system is always listening, it always get the beginning, so it needs that, no more. Soundhound needs more since it gets called at any point in a track.

      Somehow, though I wonder - if the music is playing on my device, I loaded it on there. The metadata must be somewhere, though if not, then I got an unlabeled track. Really? That's possible, but the value proposition escapes me.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    7. Re: works offline? by DontBeAMoran · · Score: 4, Interesting

      32 thousand CDs, using slim jewel cases at 5mm thickness, means you have a CD tower 160 metres tall. Given a standard height of three metres per floor, your CD stack is over 53 stories high.

      --
      #DeleteFacebook
    8. Re: works offline? by DontBeAMoran · · Score: 2

      And at 16 grams per CD, your collection weights 512 kilograms, or over half a metric ton without the slim jewel cases.

      With the slim jewel cases, with a weight of around 43 grams each, your collection weights a total of 1.888 metric ton.

      The E.P.A.'s weight statistics show that the average weight of a 2003 car or light-duty truck, like a pickup, sport utility, van or minivan, was heavier than in any model year since 1976, when the average peaked at 4,079 pounds (1850 kg).

      Congratulations on your CD collection being heavier than a small pickup truck.

      --
      #DeleteFacebook
  2. OK by nospam007 · · Score: 2

    So it knows every song played in the gym, why can't it be used to _tune out_ that song playing right now with my noise-supressing headphones, since it knows every tone in advance and hasn't to guess?
    Then I could hear _my_ music playing in my headphones.
    Would also be great for the barmen in nightclubs and other places playing loud music.

    1. Re:OK by Dan+East · · Score: 4, Interesting

      Although I think you're being funny, no, this couldn't be used in that way. Noise cancelling headphones work by using destructive interference, which requires an exact opposite waveform of the sound being cancelled out. Since the analog waveform of the music would be affected by any number of factors (the quality of the speakers playing it, the equalizer settings of their audio equipment, the bitrate of their source, the echoing of the sound off various objects, multiple speakers playing the audio, which would result in multiple "copies" of the music reaching your ear just very slightly delayed from one another, etc, etc), you couldn't use a "canned" waveform (the original MP3) to cancel out the actual waveform reaching your ears.

      Now, while it might be possible, using AI, to try to do a best match of the ambient sound against a canned waveform, and cancel out only the ambient sound that seems to match, it still would not work perfectly. That would result in echos and certain portions of the frequency spectrum still being heard, which would sound very strange.

      --
      Better known as 318230.
  3. Re:How big is an "acoustic fingerprint"? by BradleyUffner · · Score: 5, Informative

    Yet another lump of unremovable pre-installed stuff taking precious space on your phone.

    The Google spokesperson wouldn’t give us an exact size for the database file (which is not surprising, since it changes every week and is based on your country) but did say the whole feature should take up less than 500MB. Again, if you never turn the feature on, don’t worry — you won’t lose this space.

    If you don't turn it on, it doesn't ever download the fingerprint database.

  4. Re:battery impact? by BradleyUffner · · Score: 2

    Who really cares about how this is done. I'm much more interested in what the battery impact of such a useless feature is. Seriously, how often do most people use this feature, such that it would be useful having this run 24/7/365?

    Just don't turn it on if you care that much. It isn't on by default.

  5. Re:small database by MangoCats · · Score: 2

    Aren't these things meant to have high numbers of songs so that you can find out what that obscure song is, not just the latest taylor swift one?

    Uh... no. The point is to ID that song that your friends were listening to and you are hearing it again and you want to know what to call it so you can impress your friends next time it comes on, or better still, sell you a $0.99 copy to download to your phone.

    Obscure songs are... obscure. Use the online database if you want to ID obscure stuff.

  6. Weasel Words by PPH · · Score: 3, Interesting

    No audio is ever sent to Google.

    No. But the playlist along with location data probably is. Either in real time or forwarded when the network becomes available.

    And then this is turned over to ASCAP/BMI to verify that commercial establishments you were in have paid their fees.

    --
    Have gnu, will travel.
  7. Re:small database by TranquilVoid · · Score: 3, Informative

    According to the article the local song database is updated once per week based on the changing popularity of songs on Google Play. The least popular songs are replaced rather than expanding the database in perpetuity, and if you never enable the feature the database is never downloaded.

  8. They’re stuck by 93+Escort+Wagon · · Score: 2

    Google, Samsung, Apple... all their phones can now do pretty much everything their customers need, and are powerful enough where there’s not much practical gain in upgrading. These companies are basically stuck trying to sell us high priced gadgets which in truth are pretty much commodities now.

    This new feature from Google doesn’t seem useful to me at all. But, given the choice between a phone able to do this and a phone which can turn me into an animated, talking poop emoji... I’d take this, thank you very much.

    --
    #DeleteChrome
  9. Must be new here by lucm · · Score: 2

    What a shame that MP3s are so huge and complex that Google has a storage and indexing capacity problem

    What a shame that you didn't RTFA and missed the part where they explain that it works while the device is not connected to internet

    --
    lucm, indeed.
  10. Re:battery impact? by crimson+tsunami · · Score: 2

    "This is on my phone why?

    "

    Because you installed it by mistake and are too silly to remove it?
    Because you needed something to complain about?

    Can you tell us? Why did you put it on your phone?