Slashdot Mirror
Full-Text Audio Search
Captain Chad writes "The latest print edition (12/16/2002) of InfoWorld has an interesting article about an audio search program by Fast-Talk Communications. (The article is not yet available on the InfoWorld web site, but the Fast-Talk site has some good info, including a downloadable trial version.) The product works by breaking the audio stream into phonemes, which are the 'basic units of sound in a language.' The search is then performed for a specific sequence of phonemes. This method is faster and far superior to traditional audio searches which convert to text and then perform a normal text search. The author of the Infoworld article, Jon Udell, tried a variety of searches that were surpisingly successful. If this technology is as good as he claims, there is a reasonable chance it will revolutionize the way we store data. Maybe there will even be an 'Audio' tab on Google." Here's the Infoworld article.
/sarcasm
How long before the feds start digitizing all of our telephone conversations and using this technology to google our private conversations?
Yay!
Mass searches of phone calls for "terrorist," "bomb," "Allah," and "Slashdot."
before we have a "video" tab on google? :)
I can't help but wonder what the point of a "Full text audio search" would be. Most songs have their lyrics online, most speaches are already in document form, etc etc. Plus, wouldn't searching through audio files be incredibly computer-taxing? What about the services like AOL that cache almost everything? Would they start having to cache 300meg wav files?
Now, if I could hum a tune into my computer and have it find what song I was humming (for those songs you just can't remember the lyrics too), i would be much happier.
Do the songs need to be converted to this new phoenic format or can you just search the audio? Wouldn't this use a tremendous amount of computing power?
I don't keep a lid on my coffee so when I walk around I look busy -me
I don't know much about the subject, but isn't this the method used to convert speech to text? Sounds to me like it's the only way to do it...comparison of a sequence of phonemes to another, except that the each word in the dictionary is associated with a sequence of phonemes. And that's why you're required to "train" the software with your own voice/accent.
Somebody who knows about the subject, please post and explain the process.
Warning: Opinions known to be heavily biased.
Actually, Google already has a voice search, albeit in beta form.
Now I can finally search for the Free Radio Linux kernel reading by phonemes!
*Splort*
linx ear
Karma: Censored (mostly affected by decency laws)
...can it decode rap and/or reggae? I swear I can't understand 3/4 of those lyrics. Songs could start with
-----BEGIN PGP MESSAGE-----
and I wouldn't be able to tell the difference.
Of course, if I'd read the post properly I'd have known they were talking about searching on audio, not by audio . . . *sigh*
ANALYSIS by John_udell A T infoworld d o t com
.Net, but C# or Visual Basic .Net programmers can use the COM API.) The API supports multithreading so that indexing and search tasks can be parceled out to a set of processors. Non-Windows packaging of the engine, when needed, will be straightforward to produce.
The power of voice
By Jon Udell
December 13, 2002
CHEAP STORAGE MAKES it feasible to save voice recordings of many of our meetings, teleconferences, interviews, and other conversations. In some environments -- call centers and certain sectors of finance and government -- that already happens. But audio surveillance isn't yet routine, and the thorny legal, social, and cultural issues it raises haven't yet been widely debated. That's because, until now, there was no practical way to mine voice data.
As with other forms of practical obscurity, this artificial barrier was bound to topple, and now it has. Fast-Talk Communications' revolutionary phonetic indexing and search technology brings the magic of full-text search to the formerly opaque realms of audio recordings and video soundtracks. If you consider the way in which Google has already become everyone's indispensable "outboard brain," and extrapolate that to all the voice data that exists -- and to the vast quantities that soon will exist -- it's hard to avoid the conclusion that Fast-Talk is one of the most disruptive technologies in the pipeline.
A phonetic search engine
What Fast-Talk sells is an engine and a software development kit, not an end-user product. The kit includes a "technology demo," however, which is a fully functional tool that has changed how I work in a dramatic way. Though I've been a journalist on and off for many years, I had never integrated audio recording into my routine. Finding quotes in those recordings was a painful process, and sending them out for transcription (as my InfoWorld colleagues routinely do) incurred delay and expense. So, being a fast typist, I just captured what I needed live. That technique was stressful, not always accurate, and obviously not appropriate for most people. So when I interviewed Antarctica Systems CTO Tim Bray recently for InfoWorld's CTO Zone (see "Mapping the future"), I used Fast-Talk to record, index, and then search the conversation.
The Fast-Talk engine can work with multiple audio formats, using pluggable "media accessors" to encapsulate them. The technology demo supports only WAV files, which it indexes to create PAT (phonetic audio track) indexes. If you want to search video, Fast-Talk recommends using VirtualDub, an open-source program, to extract the audio track as a WAV file. You can use Fast-Talk's demo to index pre-existing WAV files or, as I did, to index a WAV file while recording. This near-real-time indexing meant I was able to begin searching the index as soon as the 45-minute conversation ended. That was true because Fast-Talk's phonetic technology is orders of magnitude faster than the conventional alternative: speech-to-text translation followed by text indexing.
Like many great innovations, Fast-Talk is simple to describe. Phonemes are the basic units of sound in a language, and North American English has 39 of them. You can look up a word's phonetic spelling in the Carnegie Mellon dictionary (see Kevin Lenzo's Web site at www.speech.cs.cmu.edu/cgi-bin/cmudict). "Dictionary," for example, works out to "D IH K SH AH N EH R IY." Fast-Talk's indexer recognizes phonemes and notes the time of their occurrence. The searcher converts text input to phoneme strings, looks for them, and returns their time-codes. It's as simple -- and brilliant -- as that.
Fast-Talk in action
When my interview with Tim Bray was done, the first segment I looked for was the one where Bray said, "Jean Paoli spent four hours showing me XDocs." The name "Jean Paoli" was, not surprisingly, ineffective as a search term. But "four hours" found the segment instantly, as did "fore ours" -- which of course resolves to the same string of phonemes. "Zhawn Powli" also worked, illustrating what will soon become a new strategy for users of voice-aware search engines: When in doubt, spell it out phonetically. In practice, I find myself resorting to this strategy less often than I'd have expected. And it was fairly obvious when to do so. I guessed correctly that "MySQL" would not work, for example, but that "my sequel" would.
The query language is dead simple, but there's an interesting twist on proximity. In a conventional search engine, proximity means "find a word within so many words of another word." In Fast-Talk's engine, it means "find a string of phonemes within so many seconds of another string of phonemes."
I was unable to find any variant of "XDocs," but I chalk that up to the recording's poor quality -- I was testing an IP phone at the time. There were some dropouts, and "XDocs" came during one of them. The marginal recording quality was, in fact, an excellent test. Like most people, I have no special audio engineering skill and no special recording equipment. To succeed in the real world, Fast-Talk will have to work well with whatever raw material it can get -- and it does. Although it is tuned for North American English, the international nature of our industry made it inevitable that I would push those limits. Sure enough, the accents I threw at it included Ximian CTO Miguel de Icaza's (Mexican), OpenLink Software CEO Kingsley Idehen's (Nigerian/British), and Systinet CEO Roman Stanek's (Czech), with usable results in each case. It's preferable, of course, to have a high-quality recording of a native speaker of North American English. When I indexed a well-modulated phone conversation that Test Center Director Steve Gillmor had with Microsoft's Mark Lucovsky, the results were simply uncanny.
Developers will find Fast-Talk to be a clean, well-documented toolkit. The engine is packaged as a static link library for use in Microsoft's C++ environment, and from other languages by way of a COM (Component Object Model) wrapper. (There's not yet a managed interface for
Call centers are obvious first candidates for the Fast-Talk treatment. "Think about running a support center," says Patrick Taylor, Atlanta-based Fast-Talk's vice president of sales and marketing. In theory, answers to hard questions are written down in a knowledge base. In practice, that rarely happens. "It's compelling to just index everything that's said by the best experts," suggests Taylor, "so you can instantly find where they mention, say, NT kernel error 304."
Clearly, that's just the tip of the iceberg. The implications are both exhilarating and frightening. "This business of recording everything scares the bejesus out of me," says Ray Ozzie, CEO of Groove Networks in Beverly, Mass. With entry-level deployment of Fast-Talk starting at $10,000, routine meetings and phone calls won't be indexed anytime soon. But it's coming, and it is scary. As always, great power brings great responsibility. The genie's out of the lamp, though, so we'll just have to learn to use this new power well.
BOTTOM LINE
Fast-Talk's phonetic searching
EXECUTIVE SUMMARY
With Fast-Talk Communication's revolutionary phonetic indexing and search engine, you can instantly find words and phrases buried in many hours of spoken recordings. It's a major breakthrough that will forever transform voice data.
TEST CENTER PERSPECTIVE
Google has become the "outboard brain" that we increasingly cannot function without. However, while Google is a voracious reader, it can't hear a thing. Fast-Talk's technology promises to remedy that handicap someday soon. It's a dizzying, if sobering, prospect.
Combine this with the Streamed Audio Kernel source, and it's only a matter of time before people leave patches on Linus' voicemail! The great thing is that to patch an audio kernel, you only need a tape recorder.... :P
First I'd like to say that this would be wonderful for NPR to use. *drool*
On a serious note. I really didn't think NLP software was to the point to make this plausible. I've never actually used NLP tools, but what I've heard in the main stream is that while they work they aren't perfect. This is fine for someone starting at a screen while talking or someone who is going to review the transcription, but it seems like it would break any automated system when there is not system of checks in place, since this involves a human.
"Not knowing when the dawn will come, I open every door." - Emily Dickinson
Now you can pretend you care about what they are saying and not just listening for the sweet intro music.
I don't keep a lid on my coffee so when I walk around I look busy -me
So does anyone out there know how well this technology deals with accents and dialects? If so, perhaps we could finally see that 'Star Trek' like universal translator - or at least translate on a large scale media works from the past century into other languages.
... does this spell the end to stenography as we know it?
Of course, noble thoughts aside, I keep thinking how useful it would have been to have such technology in college when I had to transpose long lectures from my chicken scratch.
Hmmm
healyourchurchwebsite.com - WWJB?
... Or imagine Google recording all possible audio streams (TV, radio, ... streets?) and allowing us to search those? All it takes is enough procesors, a bit of wiring...
Now if you record street conversations or all types of public conversations... Do a search on 'bomb'... How appealing is that to big brother.
All right... I'm learning sign language. Now.
I just hope one of those nuisance lawsuits from Tzsvestaeya Zolskovova, the eccentric widow of Sergei Zolskovova, (Russian lunguist who coined the word phoneme) over the use of the term "phoneme" doesn't hobble progress in this fascinating area.
I once wrote a phoenetic search engine for a site that took keywords and broke them down into their soundex phonemes, then stored those. Then, when a search was executed, it would convert the text words into phoenetic pieces and search the database for matches. It was quite accurate, actually. For example, one could search for "olif ghardin" and one of the returned results would be "olive garden".
I guess this is a similar idea. Pretty cool tech.
For editing films. For documentaries in particular, this would be a godsend. Imagine if, in addition to video/audio tracks, you had a simple 'text' track with which you could easily assemble your cuts.
If nothing else, putting the computer to work on the 'condense 100 hours of footage into pieces of paper' stage would be a nice step.
If it prevented just one assitant editor from going insane, it'd be worth it. Do it for the children.
-Brett
A search engine that lets you hum a song and it figures out which one it is.
I can see it now...
.wav to index the page is encrypted in 'southern talk'
meta HTTP-EQUIV="Keywords" CONTENT="slashdot"
meta NAME="Description" CONTENT="News for Nerds. Studd that matters"
meta NAME="Voice" CONTENT="/slashdot.wav"
Seriously though, this would be great, except for when the web developer is from the deep south and the
Someone mentioned it can be used by the government for TIA stuff - agreed, but same with any technology. It has its positive and negative uses. I don't think we are all going to revert to cavemen to get away from it.
Random is the New Order.
If you had a database big enough, you could almost make your own audio episode of any tv/radio show there is out there. With the right equipment, you could have a celebrity say anything, anything at all you want. This sort of program was coming, but with it brings cool and at the same time terrifying feelings from me...
Forget rap and reggae lyrics. How about technical terms using letters and numbers, especially since they often are only intended to have one pronunciation of several possibilities (See Roman Numerals, MAC OSX(="ten"), etc.). I'm wondering how it would deal with those.
We Are Familiar With Elephants By Virtue Of Their Size.
There's one kind of "audio search" I'd really like to see: searches for a song by tune.
I've seen a couple of web sites which offer tune searches, but they all work on the index system used in fake-books: start from the first note, and then from there, say whether the next note is higher, lower, or the same. But this system has problems: a reasonably short search will match a whole lot of songs; it's often hard to tell whether certain extra notes are considered part of the tune; and some songs have an obscure beginning and an easily recognizable theme farther in, and you don't know which one is indexed.
These sites have also tended to only index very well-known tunes - usually, folk songs, show tunes, and a few jazz standards.
One site allows you to send them a recording of you whistling the tune, which seems like an improvement, but it actually just translates it into the up-down-repeat notation.
My ideal music search would be something that would take large quantities of music (let's ignore for the moment where it gets the large quantities of music without pissing off the RIAA) and scan each song for prominent tunes. You could then search these with perhaps the up-down-repeat notation, but also by inputting music notation, for people who know it. The search would have to be key-insensitive, and allow fairly fuzzy matches.
If it could give me the name of that pop song/jazz tune/classical piece I just heard on the radio, it'd be pretty good.
But if it works really well, it'd be a blessing for music composers - they could just search for that tune that just popped into their head, instead of worrying over whether they're subconsciously ripping off another song.
Win dain a lotica, en vai tu ri silota
We have three > 300,000 sqft underground facilities loaded with rows of 2U rack systems with eight 120gig hard drives in each. Every phone call you have had since 1994 is now stored in this massive datasystem. Transferring all the old calls from tapes to the new system was probably the most tedious job I have ever done.
Soundex, which uses the way words sound rather than the way they are spelled, has been widely used by the government and genealogy researchers for the past 60 years. This isn't exactly "new" technology.
/. articles starting to sound like corporate press releases?
Why are more and more
Does it recognize speech, or does it wreck a nice peach?
This sig no verb.
and there are several research prototypes that can do it (check out a paper from Cornell, or just google for query-by-humming)
Direct search using various phonetic representations has been around for many years. All things being equal, it's known to be somewhat better than searching the output of speech recognizer using approximate string matching. But you have to weigh against that that both speech recognition and approximate string matching are being pushed much harder than this kind of search, so you may end up getting better performance using speech recognition and string searching anyway.
The folks down at Streamsage have been working on this for a while now. They are working on an index of NPR, last I heard. They do video and audio; a search retrieves the relevant clips of video. It works really well, apparently. This will be a fantastic boon for universities who have all kinds of lectures on video with no way of knowing where to find the information a student needs.
Actually it is. InfoWorld: The Power of Voice.
Mod me up boys, it's raining in Prince Rupert.
Fuck now what are we going to do?
Could you physically protect conversations by some means? Maybe talking with a little encoder around your mouth? /Half-sarcasm
A speech synthetizer (http://slashdot.org/article.pl?sid=02/12/22/14342 40&mode=thread&tid=106) in order to do a Full-Audio text search?
I'm sure someone could do it IN SOVIET RUSSIA
There are a few papers available for download from their website, but you have to register. Basically, traditional voice recognition parses the audio stream into some meta-form, usually representing phonemes (the low-level "atomic" sounds that your speech consists of). These phonemes are then matched against a dictionary of known words (and the phonemes they consist of) and text is produced.
Because phoneme recognition is not particularly accurate (for example, it's hard to tell the difference between "hard d" as in "Dan" and "hard b" as in "Ban" over a noisy phone line), traditional speech to text systems use several approaches to improve accuracy. One is to improve the accuracy of the basic phoneme recognition by "training" it for a specific voice. Another is to use all sorts of hairy-language-specific grammar / syntax algorithms.
Computationally, it's the matching of the phonemes against the dictionary that's the most difficult, and the larger the dictionary, the less accurate and more CPU-chomping it becomes. In addition, searching the resulting text for specific matches grows less accurate as the search string increases in length, due to the likelihood of a transcription errors.
The cool thing that Fast Talk has done is to store and index the phoneme meta-data, rather than complete the recognition to text. When you enter search words, they break the search string into phonemes and look for matches that way. This has several positive benefits:
1. Computational resources are dramatically lessened, since the "phoneme recognition algorithms" are fast and there's no dictionary matching.
2. The matching doesn't depend on having the right words in the dictionary at input time. It works just as well for unusual proper names and technical jargon as it does for common words, since they're all formed from the same basic phonemes.
3. The longer the search string, the greater probability of an accurate match.
4. No need for accurate search string spelling. It doesn't matter if you know how to spell a word, as long as you can write it down phonetically.
In theory, the system should work for any language, but reality is that different languages do have different sets of phonemes, and I think Fast Talk has only really worked on English. So languages like Spanish that are fairly similar phonetically to English would probably work pretty well, but tonal languages like Mandarin Chinese or those with non-vocal sounds like the clicks and pops of the African Bushmen would require a rework of the phoneme recognition code.
The main downside of their system is that it doesn't actually produce text... which means that you'd need another speech-to-text system if you wanted transcripts, or want the data to be searchable with whatever standard text-based search engine you are using on your intranet. But they appear to be aiming at applications where that's not necessary. One of my favorite ideas is integrating it with a video editing suite and being able to jump to different cues in your video clip library simply by stating the dialogue that's found there.
Of course, one of the most obvious applications is for intelligence and security. So far it doesn't appear that the company is pushing too hard in that direction -- it was founded by an academic group that originally developed the technology for a library project at Georgia Tech. However, I'm betting that's where the real money is, and it's only a matter of time before their ideas are found in your favorite national department of big-brotherhood.
-R
And this doesn't even begin to deal with "Engrish" speakers =]
In the great CONS chain of life, you can either be the CAR or be in the CDR.
My understanding is that most phone calls are digitised already. The connections between exchanges are (mostly) fibre-optic, so the traffic is digital. I can't comment on who's listening in. Surely that doesn't happen without due cause and the appropriate courtroom approvals?
</wide-eyed innocence>
Ironically, I'm posting this over a dial-up which modulates my digital data to analogue. The signal is then digitally encoded at the telephone exchange, with the whole process being reversed as the signal reaches my ISP.
1 - real-time key-word alerting - i.e. having software listen out for key-words on an audio/visual source, and alert someone appropriately when they have detected. TIA, anyone ?
;o))
2 - data retrieval - a phonetic query language - cool !!!
3 - pr0n video spider - just listen out for lots of Ooohs, Aaahs, and the like (sorry, could not resist
$ strings FTP.EXE | grep Copyright
@(#) Copyright (c) 1983 The Regents of the University of California.
How is this different from soundex? For decades, databases of names have been stored in soundex. If your driver's license number begins with letter-number-number-number, it is probably soundex. If you have done any ancestry searching, as I have, you have encountered soundex; this way, if you search for John Smith in 1732, you will find records for Juan Smyth, Jon Smythe, John Smitt, etc.
The benefits of having actual sound? If it's just going to use a soundex-type formula in the core functioning, the sound would just be a gimmick, and a storage-taking one at that. Sure, compression has gotten amazing, but will the sound of Smith really take anything near the same 4 bytes as "S720" ??
Fast-Talk assumes you know the word you are looking for before you search, which is not super useful, except for google style searches, also it is extraordinarily slow (only searches about 10 hours of audio a second, think about how slow that is for say a year of NPR). Check out CallMiner for a much cooler use of speech to text technology. CallMiner uses trending to find trends in large volumes of calls. a real business use.
Congratulations on your first post. I am former Nigerian dictator Mugabe, and I am happy to finally have met such a dedicated (and obviously quite successful) individual as you are.
My need is as follows - I am currently seeking to transfer $50,000,000.00 (FIFTY MILLION US DOLLARS AND ZERO US CENTS) to a dedicated account in the US. My technical advisor pointed to Slashdot first posters being an extremely capable and trustworthy group of individuals capable of coping with such task for a measly 25% commission. After all, the advisor said, how many people DO YOU know that can boast a successful first post? Not too many.
Please leave your coordinates, pictures and bank account numbers as a reply to my post. The money is already waiting on this side.
I am former Nigerian dictator Mugabe and I am currently seeking to transfer $50,000,000.00 (FIFTY MILLION US DOLLARS AND ZERO US CENTS) to a dedicated account in the US.
But since you are not the first poster, NO MONEY FOR YOU. Keep trying.
www.speechbot.compaq.com
As 90% of the data for diagnosis comes from the history-taking (interviewing) the patient, the potential for automating/supporting diagnosis is exciting.
Imagine a system that listens to a consultation in real time, making helpful suggestions for diagnosis based on analysis of the patient and the doctor's phoneme streams! And no tedious data entry, just an unobtrusive microphone.
I've been waiting for this.
You can never eat too much, only cycle too little.
The basic idea of using audio similarity to "grep" short sounds out of audio streams (as opposed to using ASR and text-matching) is quite old - some classic papers based on dynamic timewarping date back to 1977, and HMMs became popular for this application about ten years after that. Papers on this kind of thing appear in conferences like ICASSP - look for keywords like "keyword spotting" or "wordspotting." The phone company wanted to do this for obvious reasons.
Note that I'm not saying the GATech technology used by this company is derivative - I haven't looked at the specifics of this approach.
Is probably already using something like it.
They say the first thing to go is your penis. Well, it's either that or your brain. I forget which...
That is the most ridiculous claim I have ever heard. It is impossible to do.
Now I'll be able to search for particular lines of code.
They say the first thing to go is your penis. Well, it's either that or your brain. I forget which...
"beep +techno" .... Results 1-30 of 27596432189831415926535 displayed, search took 0.91 seconds.
Really, when would a terrorist's conversation actually include the word "terrorist" ? Maybe he would say something like, "Hey Abdul, we need another terrorist in on this bombing." Or maybe: "Terrorist Jim, meet terrorist Mike."
They say the first thing to go is your penis. Well, it's either that or your brain. I forget which...
Italys nose probes them et al. fingering our hose
the fee Cult to longer stained syrups and Hussein marmot pervert sucks eggs rat. Intact, eye amusing into dick tape his pest of flash snot.
- - -
It has no problems at all firguring out those difficult to understand lyrics and has an almost perfect success rate. In fact, I am using it to dictate this post to Slashdot.
"Everything is adjustable, provided you have the right tools"
Abbey Road backwards.
KFG
After reading a posting that someone had probably typed in silence and submitted to slashdot, I posted this reply silently and now you're reading it. Chances are you aren't reading this out loud. Nobody said a word, or even had to hear one. Reading can be an astoundingly efficient way to transfer information.
:) Click that white button and turn it green.
I have yet to meet anyone in good health who prefers getting ten voice mails over ten emails.
What the world needs is fewer karma whores and more good friends.
Go ahead, friend.
http://tinyurl.com/4ny52
there are 4 petabytes of files being shared, a search through 150 terabytes dosn't seem so big.
I saw a demonstration essentially the same technology at Compaq's CRL about two and a half years ago (formerly DEC CRL, or Cambridge Research Lab, the guys who did research for AltaVista). It did exactly the same thing. It broke sound files down into phonemes, then searched based on the phoneme. It was mostly used for finding a clip on the web rather than a specific place in a long file, but it was the same idea. The nice thing was that it was OK for its application if it missed once or twice. If the audio file was relevant, the word or phrase was probably used multiple times in the clip. It was pretty good at finding NPR stories about certain events. In fact, you can try it out for yourself at an online demo.
I think there should be three tabs instead of one 'Audio' one:
Less is more !
http://audio.google.com/search?hl=en&ie=UTF-8&oe=U TF-8&q=binladen+ahh+pains
Buffer overflow!!
[hmm!]
quote:port 17 udp
Paul
there isnt any linking structure
The wise programmer is told about the Tao and follows it. The average
programmer is told about the Tao and searches for it. The foolish programmer
is told about the Tao and laughs at it. If it were not for laughter, there
would be no Tao.
The highest sounds are the hardest to hear. Going forward is a way to
retreat. Greater talent shows itself late in life. Even a perfect program
still has bugs.
-- Geoffrey James, "The Tao of Programming"
- this post brought to you by the Automated Last Post Generator...