Slashdot Mirror
Google Researchers Create TV Audio Analysis System
segphault writes "Ars Technica reports on a paper (PDF) about ambient audio analysis authored by Google researchers. The system described in the paper can effectively determine what television show a user is watching just by capturing a short audio clip. The paper explains how a regular computer microphone can be used to record an audio clip that is then converted into a statistical data summary and transmitted to a remote server which matches the clip against archived data in order to ascertain which TV show it is associated with. Apparently, the system is fully viable, and other kinds of ambient noise don't negatively impact its accuracy. The paper also describes how web services can provide contextually relevant information based on a consumer's television viewing activities."
Big Brother is listening you!
There's a system in the UK where you can go out clubbing, here a song you like, dial a number and hold the phone out to the music and it'll text you the name of the song. Assuming they don't hire scores of extremely knowledgable music buffs with quick fingers, surely it's a very similar system. TV dialogue may be less distinctive to the human ear but to a computer it just means a larger amount of data to search through.
and this is useful for what exactly?
Is THIS why Google has been returning so many porn sites on my searches lately?
This seems like a not too complicated idea. You create an inexpensive operation that extracts what features you want from the sound data. Most importantly, you avoid features that are prone to randomness and entropy. It would take some research to figure out what the best features are and that's the audio fingerprint.
Since Google has more storage than you can imagine, they can most likely apply this fingerprinting technique to every episodes of every major show. Then they host the fingerprints in Google style and use their patented "Google Technology" to search it much the same way web content is searched.
Why would you want this? Well, there's the obvious marketing ploys. You know that people who watch Darma & Greg like to shop at Trader Joe's and like Odwalla brand food so you offer free episodes of Darma & Greg with only Trader Joe's & Odwalla episodes. You let the sponsors (Trader Joe's and Odwalla) foot the bill for the bandwidth/royalties or whatever.
The second useful implication would be cross suggesting shows to a user based on random sampling of the shows. You could allow users to watch old TV shows on the internet and then build a profile of them and their shows. Much how Amazon works, you could then suggest other shows, other DVDs of shows or perhaps build a site that randomly shows the user episodes that they might like based on prior viewings and statistics of other users.
The take away from this article for me was the fact that Google has vested interest in archiving and now television will be archived Google style.
I can't think of many other uses for this as the system isn't really "inferring" or "thinking" about data samples but is more so matching extracted features against a database. You know, voice recognition software allows for decent voice fingerprinting. You could most likely easily identify characters based on voices (but not actors due to stars like Hank Azaria who do multiple voices). Then you wouldn't need a database of all shows but more so just a database of character voice fingerprints. I would find this sort of approach more interesting but less specific and useful.
Aside from showing this off to your friends, it's not very useful. What I personally would like to see this new Google strategy applied to is all the tapes recorded of famous people like the United States Presidents. If you divided those up into sessions and I was listening to a particular tape of the Nixon set where he talked about the "new right", perhaps a database with references would then point me to some tapes or materials on Joe McCarthey's staunch views on the right.
My work here is dung.
Designed to maximize user privacy while minimizing dependency on unique hardware, the system described in the paper seems interesting and feasible. In order to protect user privacy, the software uses "summary statistics" automatically generated from ambient audio rather than transmitting an actual recording. The actual audio cannot be extrapolated from the summary statistic data, so the system doesn't "overhear" or transmit user conversations.
Still, if the data reveals what show the person is watching, your President or anyone else who gets to see the data might start treating you differently depending on what you are watching latley.
Yet another way to target the marketing. Just what I need.
will help to add meta data to all those mpeg4's you have bittorrented or recorded on your DVR
...I obey the laws of physics....
So Google researchers find a way to find you TV watching habits with only a simple computer microphone, and in the same paper they describe how they could use the microphone to find more about you for your online profile?
This seems to be just asking for privacy concerns.
// MD_Update(&m,buf,j);
TV watches YOU!
This , I think assumes US tv. Does it work with clasical music, Canadian French TV, would it work on The National?
What, thay haven't patented it yet?
Oh... I guess that would have to be a dupe^H^H^H^Hseparate story in YRO.
There could be all kind of films with any number of people, doing all kinds of things, all using the same exact soundtrack! Have they thought of THAT?!
Contextual advertising... for everything you watch. You're watching the news and they're discussing pregnancy... bam, pregnancy tester ad!
i would like to see a graphical qeualizer or something similar so i can remove all the loud music (drama effect) in TV shows, i hate it when the music is drownding out the dialog/conversation
Politics is Treachery, Religion is Brainwashing
Will it finally prove the undeniable fact that commercials are considerably louder than program content? (Which the industry continues to deny...)
shame on google
4 10208
remember cuecat? that funky little free barcode reader from radioshack?
http://yro.slashdot.org/article.pl?sid=01/06/10/0
in one incarnation i beleive they included a jack on the device and the end user was suppose to hook up an audio cable from one's tv to cuecat v.2. the computer would do all the heavy lifting, eventually finding a hidden tone that would magically pull up an advertiser's web page.
it was spam magic that never took off. gee, i wonder why.
Do you remember the MTV show Popup Video? They showed older music videos with popup balloons that gave extra information, like actors in the video that later became famous or mistakes made during production. If Google analyzed the sounds coming into your laptop and gave you a link to a site like the Internet Movie Database then you could have Popup Television. Learn more about the specific episode you are watching, and even have the ability to edit that information yourself.
It'd make an interesting toy. I'm sure that anyone with some imagination could think of even cooler applications.
AlpineR
I agree, sounds like google is creating there own "echelon" system, but with a different purpose.
If I had a sound-removing tool for TV, I'd block out the laugh tracks. I recently realized that the only network shows I watch, My Name Is Earl and The Office, don't have laugh tracks. I just can't bare to watch the majority of sitcoms that do.
I have not owned a TV for 17 years, so Google subcontracting or farming out to their own sub-conglomerate-company 'researching' the use patterns of TV viewers is in no way going to directly affect me getting spam calls to find out what I'm watching, and then sell me more spam calls or directed advertising on based on use. Is Google really that determined to become the next M$, just as invasive and just as annoying? I had hopes they were not.
let me see if I understod well, does this means then if google goes into Tv market (lets say GoogleTv) you would be able to search your tv program just because of it's audio? this looks good.
then would the search query "yes, yes, yes" be filtered because of the highly mature content at results
Actually - it appears they do the same thing Google's researchers talk about already: Reference
No TV.
San Francisco Photographers
In the mean time, I avoid non free software and even have bad thoughts about my cell phone.
Friends don't help friends install M$ junk.
I wish it could get it right, and record my "dateline" and not,
football head baby and big fat cartoon man talking about his ass gas hour...
every day http://en.wikipedia.org/wiki/Special:Random
This reminds me of a past Slashdot story.
Dan East
Better known as 318230.
and while whatchine Fox news, I was pointed here: http://tinyurl.com/z9x2y
Don't fight for your country, if your country does not fight for you.
I don't watch TV much, so I could care less about identifying the TV shows. But what I really would like is an app that would accurately identify mp3 files and apply artist, track #, ect. I've tried a few of the availible programs such as Replay Music and their accuracy is horrid. Maybe Google can do it better. Of course the other use I see for this is identifying music in movies and older TV shows. Newer TV shows do a great job of identifying music, but some older shows (season 1 of The Wire) have great music clips that aren't named in the credits.
And the technology it uses is now owned by BMI.
They been doing this for years with the microphone in my TV?
Posts a screenshot with something like:
You mean like peoplemeter? There is an audio version of this for radio as well.
----
All of whose base are belong to the what-now?
I'm wondering if a similar system can be used to cut out commercials from TV shows? It would be great to have some system that doesn't rely on anything incredibly sophisticated to accomplish this. Perhaps like tripping on commercial type keywords and then using IR remote to mute the TV or switch it to an usused video input to blank the screen.
Please provide specific proof of this claim. Thanks.
Rather than "improving" the content of the commercials I see, how about using the technology to recognize and mute commercials that I've previously flagged as either really, really annoying (eg. that oil company's talking cars, which is like being stuck on the subway listening to really stupid people talk about stupid things), or way too loud relative to the actual TV show, or simply shown too many times?
I, for one, would gladly pay $10 extra per month to have a button on my remote that when pressed kills the audio feed for the currently on-screen commercial now and whenever that commercial comes up again. I wouldn't even mind if a message was sent to the advertiser saying "Hey, somebody is actually paying not to hear your crap". Negative feedback can be a good thing.
Remain calm! All is well!
Not sure about PPM's tech, but Nielsen's A/P meter does exactly what TFA describes. That's the only way Nielsen Media could roll out Time Shifted Viewing at all (disclosure: I work for them). To say that Google "created" it is an insult to the people I work with every day.
I see a patent suit in Google's future. As much as I hate patents and like Google, I'd like to at least see some full disclosure here. To (erroneously) state one one hand that they invented the technology and then admit (on page 4 of the PDF) that they intend to compete with the actual inventors, they're begging to get sued anyway.
This very statement presupposes that other noise is irrelevant, which seems bogus.
Snoring is background noise, and suggests non-watching.
Laughter is background noise, and suggests careful watching.
Of course, the laughter might not be about what's on TV...
It seems to me that watching is an activity involving the eyes and mental processing. It seems to me that audio of what is coming out of the TV is not a statement about either the eyes or about mental processing. This technology of Google's may be an advance in something, but I hope the advertisers paying for this data have their eyes open about the nature of what they are buying because (to re-mix a metaphor) to my eyes this sounds a bit suspect.
Sociologically, it sounds like a foot in the door to get harmless censors in place. Oops, Freudian slip there. That's sensors, I mean. Google would never involve itself with censorship.
Once the sensors are in place, when "we" realize that it's not getting "us" the data "we" want, we'll just do a few "harmless" downloads of "upgrades", perhaps causing a minor tweak to look at the video data rather than the audio, or perhaps doing language processing after all, and ... With user-friendly software like this, who needs spyware?
I also question the claim that because no information is transmitted back to Google that this is the definition of not invading privacy. How is this fundamentally different than the claim that if the police search your house but find nothing, they have not invaded your privacy because they've not placed any record of illegal activity on your permanent record?
It seems to me that once you place a Turing Machine into someone's environment, capable of doing arbitrary processing, and all it sends is a sanitized report, you have all the mechanism in place for abuse. What if the Turing Machine, capable of arbitrary processing, decides that it doesn't want to send a sanitized report. Who is auditing what is sanitized and what is not?
What if it turns out to later be possible to lift information from the supposedly cleansed records? Who will audit the use of that data?
There seem to me to be a lot of slippery slopes here.
Kent M Pitman
Philosopher, Technologist, Writer
Wouldn't grabbing the title from say the tv guide be a lot simplier and only require a few bytes of data? I would try to impliment that then recording audio, seems like to much hassle and massive computer resources.
This exposes an example of how even sanitized information is not sanitized. How can you return information on what people are watching and not return at least some very personal information. Of course, some people might say "well, I don't watch that kind of stuff, so I don't mind", but the issue is subtle. First, that attitude leads quickly to an assumption that there are two classes of people: people who don't mind being tracked, and people who like porn. (One sees similar reasoning often used informally about drug screenings--assuming that the people who refuse drug screenings do so "for a reason". Such reasoning is sometims formally forbidden by courts, but I suspect jurors employ such "common sense" reasoning anyway and just don't say they are doing so.) But second, it's not all black and white, like with porn, where we all "know" that porn is Bad and the only Good people are those who don't watch it. (Yes, that remark was tongue in cheek, in case there was any doubt.) If this technology says what you're watching is benign info that violates no privacy, then implicit in that is an assumption that the information about whether you watch Fox News or CNN is not potentially volatile. Who's to say this, or some other such thing, doesn't identify Democrats or whatever. I have little doubt that Homeland Security will be asking for the data just in case they can mine something from it. So the burden seems to me to be on Google to say what data is being transmitted and why they believe that doing so is safe.
The entire Internet Experiment is mankind's first face-to-face confrontation with the difference between "first-order information" and "the consequences of second-order derived information", and so far I'm not sure that it's at all clear that people understand the implications of information they freely offer, such that I'm not sure it's fair to say that they are, or even can be, giving informed consent. (Of course, that statement itself is subject to the same reasoning, and one doesn't want a paternalistic society that forbids them from engaging in such things either. However, I think on balance one can't say that the Internet has so far erred too far in terms of being conservative about what information is shared and processed, so that I think a few conservative statements of concern are likely to tip the balance away from freedom. At this point, I'm fairly confident in saying that Freedom is more threatened by people giving away information too freely without understanding the consequences than by people being told they cannot or should not. We should, of course, periodically audit the truth of this assumption.)
Kent M Pitman
Philosopher, Technologist, Writer
I'd hate google desktop (or any other google utilitty) spying on my mic to discover my musical preft or anyting else. no tv in my home, but what about the speed at which i type or the general noise in my home or how often my phone goes off or how hard or long my baby cries.. do not listen on my mic, please: 'click' . imagine how many things can be recorded and easily recognized in a home. and many a pc/laptop/headset has a builtin mic, useful to skype, which can thus be used. horror.
I'd like to implement something like this for myself, but with conversational noise instead of TV. I sometimes use my laptop as a visual aid during conversations in my living room. If we're talking about a particular topic, I may pull up a relevant wikipedia article, or something like that. I wouldn't mind if this were more automated.
I can envision running a speech-to-text translator on my laptop mic and then piping that text into my beagle desktop searcher, or maybe even one of those google desktop search tools on windows. I'd rather not send this data to google, for privacy reasons, though.
I could see this being useful at work, or in a conference or class, too. I could stand to have relevant pieces of notes that I took from previous classes pulled up with my professor mentions a particular topic.
Anyone know of a tool or project like this?
Alright, the article says that you need a microphone listening to your TV right? Now, unless Google is sneaking microphones into everyone's homes, the only way you could be spied on is if you agreed to have one placed in your home. If you agree to have one placed in your home, you probably aren't worried about Google spying on you.
Saying that Google will use this to spy on people is like saying that the NSA will spy on people who email them all of their personal information, daily habits, etc.
-1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
So to sum this up: I give up my privacy at home. For...better targetted ads?
I'm very skeptical this wouldn't be abused - if not by Google, then by someone else. And even if this is not abused, I run the risk for what?
I don't like ads now.
Everyone who loves the idea of personalized ads, put up your hand!
----------
From the other side, what will your friends think when that "random" ad for viagra pops up?
-- Life is good. Tastes like chicken.
I was thinking exactly the same thing. I wonder if eventually we'll see google toolbar unnoticebly turning on microphone to listen in.
Figuring out TV ratings is an expensive things for corps. Google can get a huge market by automating the job of figuring out whose watching what at any given time---if they can only convince the users to let them (``hey, install this google audio-ad analyzer and get 10 gigs added to your gmail account---and it lets you see tv listings in real time, as well as clips from tv programs'')
"If anything can go wrong, it will." - Murphy
What's more, from a commercial standpoint, this doesn't have to be directly related to the program at all. Certain inaudable (to the viewer) clips could be inserted into all kinds of programming to trigger a specific function from the computer. Certainly there are privacy ramifications to this, but I think google is doing the right thing by using their creative staff to push the boundary and experiment on projects such as this. YMMV.
Is this something new? At least 3 years ago I was using www.yes.net to get the name of a song on the radio. They've changed their homepage since, probably to commercialize it more, but I used to input my city, the radio station, and the time of day. Their site would display the song that played and offer a link to purchase the disc.
Visit: http://www.411song.com/
You dial the number and play an audio clip. The system recognizes the song and sends you an SMS with information about the song.
I didn't even RTFA, but from the summary I have an idea on how to implement this idea, it's fairly simple, although it's probably not as computationally efficient as what they came up with, no need to be a great engineer, if you have studied digital signal processing for a few monthes it will be enough.
So you take that audio clip, and you simply cross-correlate (reverse in the time-domain and convolve) it with your audio data base. The highest peak in your results denote a correlation between the audio clip and a show. The only problem being if the audio clip recorded some blank part in the show. However with this technique even if there's quite some noise in the audio clip or even someone talking over it it's all good.
You just got troll'd!
nearly all desktops don't have built-in microphones and most notebooks don't have them either. Dell decided to remove the microphone in the Latitude D610, while the D600 has one.
There was a paper at Webmedia 2005 describing a system -- deployed in early 2005 -- used for real-time audio finger printing that does the same, AFAIK.
http://doi.acm.org/10.1145/1114223.1114238
This paper describes a scalable real-time audio fingerprinting system developed by IBOPE Midia for radio and TV broadcast monitoring. A special temporal feature extraction strategy based on the Short-Time Fourier Transform has been designed. When given an input stream to analyse, the system matches it against the database and automatically recognizes instances of the previously registered samples within the input stream. The algorithm exploits the temporal evolution of the signal frequency spectrum in order to identify patterns and produce the final classification. The database is clusterized in order to provide an efficient and scalable search strategy. The system has been assessed using a database containing 393 distinct commercials. A 41-hour audio stream from three different TV channels has been analysed in less than 3 hours, attaining a 95.4% recognition rate.