Slashdot Mirror

← Back to Stories (view on slashdot.org)

Searching Sound

Posted by michael on from the when-grep-isn't-enough dept.

Technology Review has one of their few stories that's not registration-required describing searching audio files for any specified set of sounds. All sorts of interesting applications become possible if you can turn analog audio into a digitally-useful product without massive human intervention.

68 comments

  1. Usage scenario by markov_chain · · Score: 1, Funny
    % grep "nixon.*(fool|idiot|moron)" /usr/mp3/watergate*.mp3
    --
    Tsunami -- You can't bring a good wave down!
    1. Re:Usage scenario by MacroRex · · Score: 3, Interesting

      (paranoia)
      No really, what if they start bugging public places where people talk a lot (bars etc) and run the output through something like this? After acquiring a speech sample from bank/airport/whatever and thus connecting it to a person, it's a breeze to have a global textual log of everything the person says in a public place.

      Of course, the article talks only about deconstructing the audio sample into words, but further analysis is a natural extension of the idea.
      (/paranoia)

  2. Where have they been? by Chris_Stankowitz · · Score: 1, Funny
    "...if you can turn analog audio into a digitally-useful product without massive human intervention.

    Aren't CDs just that? If you really want to make a more usefull digital product, start scouting for some new tallent. American/Pop Idol isn't cutting it. :)~

  3. been there, done that by shachart · · Score: 2, Informative

    I work for a CT (Computer Telephony) company (see comment on story from half an hour ago). My company does soundex, phonex, and some proprietary stuff too, to convert recorded phone calls into the text of the call, regardless of noise, tone, etc. Useful for your friendly government to spy on you. This is really old news.

    --
    Those who can, do. Those who can't, consult.
    1. Re:been there, done that by dentldir · · Score: 1

      Except that Fast-Talk doesn't do speech to text processing at all. It does waveform pattern matching for language patterns without ever passing through a text phase. It actually does a reasonable job of it too.

      I played with The SDK a year ago. I suspect its gotten considerably better since then. In that respect, it is old news.

  4. When you think about it... by Ieshan · · Score: 4, Interesting

    When you think about it, though, government and military agencies must have had this for quite some time.

    Tapping and bugging really does no good unless you've got someone listening all the time - and that's both expensive and impossible. While I realize that someone only has to be listening every time someone makes a phone call with the tapping situation, the outcome is lots more hours of audio then are feasible to search and use.

    If we couldn't have searched audio on a wide scale before, then I find it hard to believe we'd ever be catching anyone by specific phone intercepts. Instead, we'd just be using that sort of thing as evidence.

    I mean, I realize this is a great technology, I just doubt it's as "new" as it seems...

    1. Re:When you think about it... by Chris_Stankowitz · · Score: 2, Interesting
      Tapping and bugging really does no good unless you've got someone listening all the time -

      It was done this way for many many years. It is partly why many investigations took a long time to be fruitfull. There are also laws in some states that do not allow for a tapping to continue if after "xyz time" has passed without any usefull information.

    2. Re:When you think about it... by arvindn · · Score: 2, Interesting
      I'm not sure.

      What's new about this technology is that it does searches without transcription, but instead works at the phoneme level. This doesn't mean that the results are more accurate than if transcription and indexing are used. Its just that the new technique has applications in some cases that can't be handled by the conventional method, like when your model is inadequate, and you would lose information by converting phonemes into lexical form.

      Its not clear how this sort of thing would be useful for the military. My guess is that for the purpose of espionage it would be much better to have the recording converted to text first.

    3. Re:When you think about it... by PerryMason · · Score: 2, Interesting

      The big problem with this sort of technology is that in the past when you wanted to tap someone, you had to have a good reason (good enough to persuade a judge anyway) and you had limits on what you could and could not record/listen to. Now with technology like this and Echelon etc, it becomes possible to monitor every person who makes a phone call or sends an email. In effect you are presumed guilty and have to prove your innocence by not discussing or commiting a crime. One of the fundamental tenets of the western legal tradition is that you're presumed innocent until proven guilty and technology like Echelon turns that right on its head. Its just another example of fundamental rights being subjugated for the purpose of protecting us from 'evil-doers' who will just end up using other methods of communication. Meanwhile its you and I who end up losing our rights.

      Its similar in some ways to the mass DNA testing of populations to find a rapist for instance. Every person submits a sample, but is it likely that the perpetrator is going to submit their's? Every member of the population is presumed guilty until they prove themselves innocent, while the guilty simply refuses the test, and anyone who refuses the test, regardless of grounds, is tarred with the brush.

      I honestly think we're at the beginning of a massive degredation of human rights (particularly privacy) as a result of both technological and global factors. Unless we do something to ensure our fundamental rights, it really won't be that long till 1984.

      --
      "I'm tired of all this 'Aren't humanity great' bullshit. We're a virus with shoes" - Bill Hicks
    4. Re:When you think about it... by Ieshan · · Score: 1

      Its not clear how this sort of thing would be useful for the military.

      Well, you could tap thousands of phone lines and search for phonemes that indicate the high level military commanders, etc. After you "found" one, you could immediately jump in and listen on it, or if the communication is laden enough, hit that phone-system. With the recent military advances and precision weapons, this isn't *that* hard to imagine.

      I mean, with a powerful enough system, you could filter thousands of hours of data at once, ignoring all but the useful stuff. It'd be like audio spy satelites.

    5. Re:When you think about it... by arvindn · · Score: 1
      Well, you could tap thousands of phone lines and search for phonemes that indicate the high level military commanders, etc.

      Hmm.. phonemes just tell you what the sound is, they contain no information about the voice.

    6. Re:When you think about it... by Anonymous Coward · · Score: 0

      Yeah -- because we all know that high level military commanders use unsecure telephone lines all the time....

    7. Re:When you think about it... by zdislaw · · Score: 1

      phoneme: "The smallest phonetic unit in a language that is capable of conveying a distinction in meaning, as the m of mat and the b of bat in English." What phoneme do you suggest looking for to indicate high level military commanders?

      --
      bad sig...no donut.
    8. Re:When you think about it... by Daniel_ · · Score: 1
      Its been out for quite a while. The biggest difference is that the computational linguists that do this kind of thing don't get much attention from places like slashdot.

      There numerous packages/toolkits that can be used to do the same thing. (If your willing to take the time to put the pieces together). One is Praat. Its a (mostly) GPL toolkit for sound and speech analysis.

      --
      The number you have dialed is imaginary, please rotate your phone 90 degrees and try again.
    9. Re:When you think about it... by gl4ss · · Score: 1

      modern way of tapping is that the phone company records all the calls, suitable for listening to cellphones too.

      you just then go through the records.

      if man spent 20 minutes per day in phone.. you need 20 mins to listen through those calls per day.. not much of a chore.

      --
      world was created 5 seconds before this post as it is.
    10. Re:When you think about it... by Ieshan · · Score: 1

      Their names? Sensative information about them? Suspected locations?

    11. Re:When you think about it... by zdislaw · · Score: 1
      Yeah, I was being kind of a wiseass about searching text for a phoneme. A phoneme carries no meaning, as it is a simple sound. Now on the other hand, a particular string of phonemes...

      Oh well, never mind me.

      --
      bad sig...no donut.
    12. Re:When you think about it... by npendleton · · Score: 2, Informative

      See or read "Killing Pablo" and then tell me what you think about catching an individual from an intercepted phone call. The U.S. Government poured top flight resources (NSA and Delta Force) on the problem of helping a Colombian Government military unit find and kill drug king-pin Pablo Escobar. Escobar was killed by this Colombian military unit.

      This technology would help immensely on message analysis. Evaluating messages typically is divided into two areas, signal analysis, and message analysis.

      Signal Analysis is when and where the signal (phone, fax, email, ham radio, etc.) originated and went to. Even if you can't read the messages, the signal analysis may be all that one needs.

      Message analysis means understanding the content of the message. Decrypting or deciphering the message is common problem for text based messages. Voice is much harder to scramble in telephone networks. Once a message is opened, can "Voice matching" quickly and accurately discern who is speaking, regardless of where and to whom signal analysis says the telephone numbers belongs to. Indexing and phoneme transcription clearly helps analysts search for instances and patterns. But this is not a transcription, this phoneme transcription, that reduces the mountain of words in a language to 25 sounds. Search results can bring too many hits, or none, because people are using ambigious pronouns or homonyms, like "He" instead "Pablo" or "Their|There|They're", that ambiguate the meaning for the search tool. Ultimately message analysis requires understanding the way people in an organization think and speak. Indexing and transcription technology can help but not replace people understanding. What does "I dropped off the package." mean to you?

      The other place that phonemes transcription could be helpful is with Foreign Broadcast Information Service (FBIS). The CIA set up FBIS during the Cold War to monitor news services around the world in native languages. FBIS helped monitor trends and propaganda.

      Mac Refugee, paper MCSE, Linux wanna be

  5. RIAA? by Anonymous Coward · · Score: 0

    Is this sponsored by the RIAA so they can detect Metallica
    files named 1.x or 8TyX.jpg?

  6. Does that mean... by Valdrax · · Score: 3, Funny

    ...that I can finally find that one song that goes Wagga-chigga wa! Wagga-chigga wa! Wagga-chigga wa-wa! Thoomp! Meedly-meedly-meedly-meedly! Meedly-meedly-meedly-meedly meedly-meedly-meedly-meedly meeeeeeee!!

    --
    If it's for-profit but free, you're not the customer -- you're the product (e.g., the Slashdot Beta's "audience").
    1. Re:Does that mean... by madmarcel · · Score: 1

      Ehm...maybe...there was/is a research-project
      at (*shame-less plug* :) Waikato University's
      school of CS which might be able to help you out.

      The idea is that you hum or whistle a tune
      into the microphone and the computer will then use
      some fancy pattern-matching (I think) and spit out a song(s) that match. (From memory, too tired to look it up)

      I have not seen this system in action, so I couldn't tell you how good it is. I don't think it works with MP3's though :o
      I think one of my fellow students is working on a related project...or I could just ask the people involved....if I could *find* them :o

      website:
      http://www.cs.waikato.ac.nz

      This is the one I think:
      ---
      McNab, R.J., Smith, L.A., Witten, I.H. and Henderson, C.L. (2000)
      Tune retrieval in the multimedia library Multimedia Tools and Applications,
      10, 113--132. Kluwer Academic Publishers
      ---

    2. Re:Does that mean... by Migrant+Programmer · · Score: 1

      And then Strong Mad comes in on his bass and he's like doo doo doo doo doo doo doooo!

      And then The Cheat comes in on his keyboard and he's like boop boop boop boop boop boop boop boop boop!

      And then I come in with And the dragon comes in the NIIIIIIIGGGGHTTTT!!!

      --
      Bitchslapped. Neat.
    3. Re:Does that mean... by gbpuckett · · Score: 1

      You're not that far off of describing the search engine I'd really like to see, sort of a "Name that Tune" search engine. Instead of inputting a text string, you would be able to put in a musical notation string and have the search engine return a listing of tunes that contain (or begin with, or end with, or repeat) that string. It would be useful for trying to deal (probably unsuccessfully) with those snippets of music that occasionally create an endless loop in your head. It would also help tunesmiths avoid inadvertent plagarism. You could put in the "new" tune you just composed and be reminded that it was actually an old tune you heard once at a party ten years ago.

    4. Re:Does that mean... by Anonymous Coward · · Score: 0

      aaaahhhh....EMAIL!
      chuk-chuk-chuk
      chuk-chuk-chuk
      MEEEEEE!do-de-de-de do-de-de-de (chuk-chuk-chuk)

      Whoa. That was awesome.

    5. Re:Does that mean... by McWilde · · Score: 1

      Here it is. Albeit through massive human intervention.

      --
      Maybe
    6. Re:Does that mean... by absolut_kurant · · Score: 1

      Query by Humming (really)

      http://www.cs.cornell.edu/Info/Faculty/bsmith/quer y-by-humming.html

      --
      Yes.
  7. Voice Control by 2sleep2type · · Score: 1

    This is a step towards full voice control of systems I have always felt the computers will not have truly come of age until they are voice controlled. For general business use all other forms of interaction are a compromise. The future I look forward to is full voice control of systems. Probably via a discreet headset so the box next door to you doesn't start typing your letter. I will then be possible to have truly 'afordant' systems.

    1. Re:Voice Control by Biogenesis · · Score: 1

      I'd still prefer simply having to wave my hand in the general direction of the thing, zaphod beeblebrox style. It'd make for more 'creative' input styles.

      --
      ...just so Google finds it.
    2. Re:Voice Control by s0m3body · · Score: 1

      do you really want your microsoft windows to be talking on you ?

    3. Re:Voice Control by samael · · Score: 1

      I don't know about you, but I can type darn near as fast as I can talk and I can certainly rearrange text faster with a keyboard/mouse combo than I could describe what I wanted using voice.

      Voice as an adjunct to keyboard/mouse would certainly be handy though.

      --
      My Journal
    4. Re:Voice Control by gilgongo · · Score: 1

      I agree - although my father, now retired, spent most of his working life dictating letters to a dictaphone. It's a really impressive skill - he kinda thinks a then, dictates a couple of sentences. Stops, thinks again then does another five or six. Rewinds to review it, maybe changes a bit in the middle, pads out anything that needs it with a bit of silence, then continues.

      The fact is that we're used to keyboards and word processors, clipboards, etc. so we can't see another way of doing it. We also tend to "micro edit" and don't think very far ahead into what were want to say. But that can change.

      For my part, I'd just be happy to have some "meta control" over my PC: saying maybe application or document names ("Word - letter to bank.doc") to start them rather than opening the Start menu or hunting the desktop.

      --
      "And the meaning of words; when they cease to function; when will it start worrying you?"
    5. Re:Voice Control by Kazoo+the+Clown · · Score: 1

      While I figure I may want voice control someday when my eyeballs or fingers don't work so good anymore, I've always figured that's a less-than-optimum workaround needed because the more efficient means of interconnection are impaired. Now if you are a hunt-and-peck typist and didn't find the move to mouse input akin to amputating 9 fingers, perhaps you'll prefer voice I/O. However, you'll be way behind those who have no problem with adapting to technology, just like those were who insisted that automobiles should be controlled with reins rather than a steering wheel.

  8. Google for sounds? by Shiranui · · Score: 2, Interesting

    It would be cool if we're able to actually 'search' for any soundbytes. Even with altered speed / tone.

    Listening to all those techno remixes, I always have a hard time trying to find out where those cute backgound soundbytes came from...only to find out it was a heavily distorted Mozart or a mixed up vocal of JFK.

    1. Re:Google for sounds? by DJ+FirBee · · Score: 1

      Would'nt it be cool to have even more intellectual property laws so that it is even harder to make music in the first place ? Wouldn't it be cool that I could be sued for using something as inconsequential as so and so's snare drum hit in a composition ?

  9. Oh, NO! by Anonymous Coward · · Score: 1, Insightful

    Lets home this doesn't make pointy head bosses think they can store customer information as a blob of speech data...

  10. Re:fr1st ps0t by Pingular · · Score: 0

    Yes, and /. isn't, you insensitive clod.

    --

    When anger rises, think of the consequences.
    Confucius (551 BC - 479 BC)
  11. Oops, here's a link by madmarcel · · Score: 4, Informative

    If you really want to find out how it works:

    Links to PS and PDF files are on this page

    http://www.cs.waikato.ac.nz/~nzdl/publications/

    (They are not going to like what I am about to do to their server ;)

  12. Dialoggle... no... Earggle... by rinkjustice · · Score: 1, Funny

    It would be nice if there was a search engine exclusively for that - instead of typing "linus torvald linux .au", you would navigate to a subdirectory called 'pronounciations', pick the audio format and voila...

    If you can search websites and images, jobs and news articles, sound bytes would be the logical next step.

    --
    SEO Copywriter. Just Say ON
  13. grep "humor" Mod.. no results? how suprising (n/t) by Anonymous Coward · · Score: 0

    (n/t)

  14. One good implementation by emcron · · Score: 2, Informative

    A company called Fast-Talk Communications has a set of tools that they resell for 3rd-party apps for things like searching interviews for specific words that were said. I have actually seen this feature in used some newsroom software made by Dalet Digital Media and it was amazing to see in action. Very fast and accurate

    The research for the fast-talk technology was done at Georgia Tech's Interactive Media Technology Center (IMTC). They've got a page about the corporate spin-off of the technology.

  15. Index or no index? by Psychic+Burrito · · Score: 3, Insightful
    There's quite a contradiction in this text. First they tell us that FastTalk doesn't uses an "index":

    The key to expediting the process was eliminating the need for transcription or indexing or both.

    Then on the second page, they say that some sort of pre-processing is needed:

    (...) the Fast-Talk approach ?processes the speech in such a way that you can later go back and search it very efficiently (...)

    So I see no revolution here... it's just about indexing the phonemes of a audio stream and then searching these, right?

    1. Re:Index or no index? by Polarweasel · · Score: 1

      Stop thinking like a geek!

      By "indexing", they mean creating a list of every place a particular phoneme occurs. Think of an index like you find in the back of a book...

  16. I said "moos" by sin(theta) · · Score: 1
    a system trained by a speaker from Canada would transcribe the sound "hoos" into the word "house."

    But I don't say "hoos".

    --
    Music wants to be free.
  17. RIAA funding forthcoming? by fluffhead · · Score: 2, Interesting

    I wonder if the RIAA will throw money at this type of technology, to help catch "pirates" who might otherwise escape by subtly transmogrifying their shared MP3s. Or maybe it already has?

    --

    #include "disclaim.h"
    "All the best people in life seem to like LINUX." - Steve Wozniak
    1. Re:RIAA funding forthcoming? by Anonymous Coward · · Score: 2, Funny

      Let's see:
      - RIAA mention in title CHECKED
      - pirates word between quotes CHECKED
      - use of the word shared CHECKED

      All you need for a couple of karma points! +3 ??

  18. Animals? by kaoshin · · Score: 1

    This seems like it may be a good tool to use for learning how animals communicate with each other. Just an idea.

  19. Any open-source software that can do like this? by Anonymous Coward · · Score: 0

    I've always been intrigued by this kind of technology (and eventually would like to build my own product). It would be nice to know what system architecture other people have used (and algorithms, etc). I imagine the searching for sounds to actually be the conceptually easy process. I'm more interested in the precise categorization of sounds. Does this turn an audio file into a single stream of phonemes/sounds or maybe a streamof sets of possible or concurrent sounds or ...?

    I'm not saying this product should be open-sourced like some zealots around here, but I do think such a (cheaply) available product could change peoples lives. Imagine being able to search for precise sounds/words in your favorite movie or news broadcasts. (Assuming you have a continuously recording device for a specific channel) Anytime your favorite news topic was mentioned, the whole story could be recorded and saved just by noting where the story transitions took place (This might take a bit of analysis to see what sort of indicators might be used for story transition, but I don't think that it would prove unmanageable.) Furthermore, a gillete/idsoftware/whatever type approach could be used to market such a product. The digital audio to sound categorization software could be free/open source. The strategies to implement certain uses (like that mentioned above) could be closed source.

    (One might be able to accomplish the same feat by the analysis of the Closed Caption signal many news shows provide, so maybe this wasn't a good example, however some shows (like Good-Day here in Dallas-Fort Worth area) dont have a CC.)

  20. Google for ... by Dossy · · Score: 1
    Google
    Web Images Sounds Groups Directory News

    __porn_blowjob_"money_shot"__ [I'm Feeling Lucky]

    I can see it now ...

    -- Dossy

    --
    Dossy's Blog
  21. Soundtrack by mbbac · · Score: 1

    Apple's Final Cut Pro's Soundtrack feature does this. It automatically categorizes inported samples depending on the instrument in them. Then, you can search on that.

    --

    mbbac

  22. Music 'fingerprints' - Polyphonic HMI by scrimshander · · Score: 1

    Has anyone heard about this company Polyphonic HMI (www.polyphonichmi.com) that claims to be able to create a digital 'fingerprint' of music (beat, melody, etc.) and identify potential 'hits'? Anybody know how they're claiming to do it?

  23. findsounds.com by Qender · · Score: 1

    I know it's just indexed by name, but this can be useful.

    http://findsounds.com/