Slashdot Mirror
Reading Lips In Software
SEWilco writes "The Register points out that Intel has released code for reading lips from a video image, Audio Visual Speech Recognition (AVSR). They do point out that better results would probably be achieved by combining video and audio recognition processing. I don't know if they have any patents, we all know some prior "art" from 2001, er.. 1968. HAL's accomplishment was also mentioned by CNN during 2001 in an article about this group's work."
Men and women, boys and girls. All with really thick, dirty, obscuring mustaches.
What is this world coming to?
Go calculate something
Oh wait, that was a different lip reading session...
If brevity is the soul of wit, then how does one explain Twitter?
Anybody else reminded of the Read My Lips videos that fit clips to songs?
Maybe now with a cluster at our finger tips and this sound visual lip analyser thing, we may be able to (finally) understand what all those foreign heavy accented professors are actually mumbling about...
And well, beats manual note taking if the computer can read the board and his mouth and his voice.
there should be a place near the colleg that sells notes.
I don't know if they have any patents, we all know some prior "art" from 2001, er.. 1968. Did Clarke ever file a patent for the geosynchronous satellites?
a reason to really hunker down and learn an obscure Chinese dialect.
I've been putting it off for far too long.
It breaks my pluginses, my precious!
A couple months ago, a very fine article was posted to /. about work at MIT regarding speech-->video synthesis using pre-recorded syllables. This means in the near future we'll be able to have avatars which an communicate to other people by videophone and/or other computers should we wish to do so. I'm reposting the old link because it got /.'ed for about 2 months (the professor took down the link) before putting the vids back up. So check out the amazing work that's on the flip-side of this article.
l ts/results.html
http://cerboli.mit.edu:8000/research/mary101/resu
-Christopher Wu
http://www.christopherwu.net/
Body language should be even easier than lip reading. I want to know if I'm wasting my time or whether I should invite her back to my place.
Wow, that must have taken a lot of hard work to do. First you'd have to recognize the location of the lips in the images (they might not stand out that much, especially in a crowd scene), then find the region in which the lips are moving, then finally use the positions of the lips to extrapolate for the current shape of the inside of the person's mouth, and make a haphazard guess at the sound being produced. And you'd need to be able to recognize the lips from any angle whatsoever. Sounds near impossible to me... and besides, by the point at which the person is beyond the range of the audio pickup of a security camera (I'm assuming that's what this would be used for), it would also be beyond the point of bad resolution. (unless the target is in a crowd, in which case the lips would be obscured frequently by people moving around in front of the target).
Fry, Leela, and Bender are hiding out in the shower discussing how to turn of Planet Express Delivery Ship. The little red light is on, the screen is scrolling back and forth between the lips as Leela gives orders and Bender objects. Then the ship says, "Oh, if only I could read lips!"
Galium Arsenide is the material of the future, and always will be.
Open the pod bay doors HAL!
Cameras randomly zooming on the lips of the crowd, if somebody says someting from some "list" of words, they keep tracking that person and make some face recognition also.
... but I think it is interesting that Arthur C. Clarke thought HAL reading lips was the only implausible scene in the film. You know, as opposed to the whole aliens thing. :P Just goes to show you the perils of trying to predict the future...
There is no excellent beauty that hath not some strangeness in the proportion. -- Francis Bacon
Sigh... the signal to noise ratio alone is enough to lend you reasonable anonymity. There's just way too much information that would need to be grepped through in order to listen in on your dinner conversation. No one, (or their Big Brother), is going to bother unless they have a really good reason to be investigating you in the first place.
I'm thinking that the 'good' will outweigh the 'evil' here...
"Michael, I did nothing. I did absolutely nothing - and it was everything that I thought it could be."
I don't know if they have any patents, we all know some prior "art" from 2001
Just in case anyone gets the wrong idea here, copyrighted works cannot be used to contravene a patent.
during the judging phase of the competition.
You know in Ace Ventura, when Ace was talking out of his asshole? Can it translate that?
i doubt it would help. The way i see it, the image would have to be clear, and the person can't be moving to much, and they have to be annunciating their english. after all, i hardly move my lips while speaking, so it couldnt read mine
YOU SUCK BALLS!
Call me cynical but has this been released as open source so it will be rapidly improved before being used in an Intel product?
Usage of IRC across the globe suddenly drops as users are dismayed by the number of people asking to sweep with them.
Read my lips, "No New Taxes!" Can the software identify a liar?
I may have done better in my AI class if I was able to read lisp. All those damned parenthesis made life very difficult.
paintball
No... more... taxes.
I know that English is one of the more easy languages to "lip read." It goes into the latin roots, and such. I'm sure that using slang will make it much harder, but I'd be curious how it works with other languages. I think that Japanese (when spoken clearly, and not using dialects) would be incredibly easy where Chinese could be very difficult. If anybody has time and a desire to hack on it, keep me posted if you do multi-lingual work. I'm really curious on how it goes.
Dacels Jewelers can't be trusted.
I only use sign language!
fools...
ummm wait.
The Kruger Dunning explains most post on
Along with the DNS protocol and France.
I've investigated Intel's vision library, OpenCV, before... and it does appear to be available for Linux if you look hard enough... but I couldn't find any Linux applications using it to actually *do* something.
Has anyone had any success with OpenCV/Video4Linux?...
...someone recording to video a person *speaking* the source code of DeCSS and then using this tool in combination with gcc to generate libDVDCSS?
Would this tool then be declared a "circumvention device" under the DMCA, or would the courts finally realize that code can be considered protected speech? The code was, after all, spoken in its original form in this case.
This same question could also be applied to audio-to-text converters as well. Maybe there's hope the DMCA will be declared unconstitutional after all.
Interesting food for thought...
David
ersonally-pay, i-ay(?) erfer-pay o-tay use pig latin.
geeze, that really wasn't worth the effort...
Software and business model patents have evidently effected comprehension of what a patent entails.
"A computer, examining a set of video images, to perform lip reading" is not patentable. HAL would be prior art for this; but it doesn't matter because there isn't any inventive step here anyway.
"A computer, processing a set of video images by locating what appears to be a set of lips, selecting recognizable points, using the movement of those points to track the deformation against a 3D model, comparing against a table of syllables to compute the probability of each particular syllable, and using knowledge about a language to determine which syllables are most likely to follow each other" could be patented. HAL would not be prior art for this, because there is no indication of how HAL performed the lip reading.
Tarsnap: Online backups for the truly paranoid
of course, you wouldn't know about them from experience, nore would most of the slashdot crowd. No, the only experience you have with that set of lips is while masterbating to the illustrations in your high school anatomy class. all though, i am quite familiar the lips of your mom softly groping my cock. peace out and happy trolling.
It probably wouldn't work for Greta "Lips" Van Susteren
This lobster was alive when it hit the frothy, boiling water.
Read my lips: "No new invasions of privacy... hey, wait a minute!"
There's a growing sense that even if The Future comes,
most of us won't be able to afford it.
-- Lemmy
patents are supposed to be on inventions, not ideas. (very) generally speaking, you have to demonstrate you know how to do something for it to count as prior art. actually building something counts, as does a patent application (since the patent application has to explain how the invention works at a reasonable level of detail, for an admittedly arguable legal definition of reasonable).
ianal, but the last i heard, a mention in a science fiction book or movie wouldn't typically be considered prior art. a person skilled in the art can't tell from 2001 how to make a computer read lips.
The evil trolls inside my head keep trying to make a joke about women, scanners and a lack of pants, but it's just not coming together.
Boobies never hurt anyone. - Sherry Glaser.
Oh oh!! L-I-P-S!!
First I thought, Jeeze... I can already read Lisp, emacs style...
Then... ohhhh... they mean Lisps... like a speech impedement... That would be cool, to read lisps.
But reading lips makes much more sense.
in speech recognition if it does no more than allow input from a camera to aid in separating out which sounds came from which speakers. Simply fixing the background noise problem would be a huge advance.
This could solve my fundamental beef with speech as an interface - privacy! Dictating email and documents would be great, if I didn't have to broadcast to everyone around me. Not to mention the annoyance of hearing the guy in the next cube complain to his girlfriend over IM...
Mouthing words silently takes some getting used to, but it has advantages. No more trying to type on a tiny PDA keyboard - etc. Obviously this is a ways off, but it seems doable.
That is only interesting if you are a sweaty, pear-shaped, socialist whiner nerd.
If you started to worry more about personal hygene and less about the DMCA, maybe you could actually convince a female to talk to you.
If you beat a dead horse, will it die some more?
but when can I get this on my desktop? it would be really neat to chat through IRC without making a sound. oh wait...
My potato gun was confiscated by the United Nations. They said I wasn't allowed to have weapons of mash destruction.
The article submitter says:
"I don't know if they have any patents, we all know some prior "art" from 2001, er.. 1968. HAL's accomplishment was also mentioned by CNN during 2001 in an article about this group's work."
Is there not a difference between the idea and the way to implement the technical solution. Meaning thay cannot patent the idea, but they can still patent the code itself for the way thwe code works.
Just curious. What does everyone think?
:-b
HAL, as in "2001" for one thing. You all know about that already.
The REAL THREAT of this is "them" using camera's to look at people from afar (or by whatever means) and eavesdropping on people when they can't get a microphone in..
You can be sure that H.L.S. will jump on this like white on rice...
It's been done at Carnegie Mellon as well.
And if I'm approached to remove it, I'll know that someone is trying to monitor me. Could this be why CDC is concerned with SARS? Can't read lips with SARS masks blocking the 'flow of information". :)
1. stop SARS.
2. collect information.
3. profit.
I can imagine the source video material quality may be quite critical to this. It would be much easier to process a signal from a DVD, for example, than a composite video camera.
:)
But then on a DVD you'd just hit the subtitle button and problem sorted
Please read my Canon EOS tech blog at http://www.everyothershot.com
Lyndsey Nagle: Do I detect a note of sarcasm?
Frink: (With sarcasm detector) Are you kidding? This baby is off the charts mm-hai.
CBG: A sarcasm detector, that's a real useful invention.
(Sarcasm detector explodes)
std::disclaimer<std::legalese> sig=new std::disclaimer; sig->dump(); delete sig;
The human mind parses speech by using both senses of sight and sound. They demonstrated this on the news one time by repeating a word over and over. They instructed the viewers to look at the screen while listening, then at some random time, to close their eyes and then open them again after waiting an interval, all while continuing to listen. Sure enough, when I closed my eyes, the word I heard was a completely different word, even though when I looked at the screen, I wasn't necessarily looking at the person's lips. In other words, your cabeza does this automatically. Obviously, the two words were similar enough in sound that this worked, but it demonstrated that in addition to using context to provide meaning, your brain uses other information as well.
I know what I want this for- I want to read the lips of all the coaches and players during basketball/baseball/whatever broadcasts. Maybe ESPN could offer this as a feature, censoring as needed. :-)
Laugh at stupidity: mod idiots +1 Funny.
While it probably wouldn't work - the prof moving back and forth, not pronouncing words clearly - it'd be a great help if it actually did... Last semester I had an interpreter for one of my classes. (I'm deaf and the teacher had a heavy accent) My interpreter couldn't understand him either!
Twenties Retirement
In Soviet Union .... computer reads you.
XML causes global warming.
I'm trying to transcribe some tapes of lectures right now, and I'm looking for an easy way out. I know speech recognition programs are out there, but from what I know, they need significant training of the user with the program in order to work.
Unfortunately, my voice is not the one giving the lectures, and there are actually two or three different lecturers. Since training is impossible (AFAIK, at least), I'm wondering how far speech-to-text technology has come, especially in the open source community. Can I just pipe the output from the wav file into the input of a speech-to-text program, and if so, what sort of signal-to-noise ratio can I expect on the output without training? (graphical interface would be nice) Right now it's taking me about four hours to transcribe 50 minutes worth of lecture.
This whole, "maybe we could apply 'blah' to rule the DMCA unconstitutional!" thing is turning into the next "wow, imagine a beowulf cluster of [...]".
STOP IT WHILE YOU CAN. PLEASE! I BEG OF YOU.
Very good point... for that matter, how would the courts handle it even without this new technology? Even without programs that can read words from video, it is still theoretically possible (though maybe not practically possible) that someone could read the source code to DeCSS aloud onto a video tape, such that someone else at the receiving end could manually record that code into a source file and compile it.
(And if you wanted to be really ironic about it, you could always store the video on a DVD :-) )
Do you think the almost invulnerable association that we make between the video and audio recording of somebody speaking and the term "free speech" would give this medium any better legal footing than the traditional source-code-on-magnetic-disks? If I remember correctly, at one point the PGP developers were in the business of exporting their strong encryption to non-US territories by means of publishing their source code in a book... correct me if that is wrong.
I have taken many years of ASL classes and am pretty involved with Deaf culture; one of the biggest myths about it is peoples ability to read lips.
;)
The idea most people have of lipreaders, like in the movie See No Evil Hear No Evil (Richard Pryor Gene Wilder comedy) or the Seinfeld lipreader episode just really isn't possible. Many sounds such as "t" and "d" look the exact same, and many such as "k" and "g" are not visible at all. The best lipreaders really can only get 2/3 of what is being said, (if they are entirely Deaf, which many Deaf people are not, if your hearing loss is not total it can be far more efective) and that is with the person speaking slowly, facing them, and human intuition (context). Throw in facial contortions, (like yelling... "they can't hear me so if I yell it will help") low light, bad angle, fast talking, etc. and the accuracy drops dramatically.
Computers lack the ability to figure out what word is being said based on context when the lips don't provide adequate information. They are also historically terribly poor at things like complex image recognition. Registration script busting is based on what? Image recognition with noise in the image (i.e. type the word that appears in the next form box) and no one has even come close to a functional computer ASL interpreter and ASL is far easier to disguish visibly than speech.
I don't see that 40% word error rate it is currently having being able to improve much at all, and I'm guessing the video feed that's off of isn't anything like fullspeed nonexagerated human speech.
Your fears of the video cameras on the streets logging your conversations are pretty unfounded
There's a big difference between the HAL computer and software that can read lips. I am surprised though not shocked to see that the analogy was used. After all, HAL was supposedly a thinking entity that taught itself to read lips (we presume in the movie that 'it' was programmed to do so). Where, in this instance, we have a computer being explicitly programmed to read lips.
My computer hath been able to read lithpth for yearth.
All things in moderation; including moderation
The idea of combining it with speech recognition in an adaptive fashion, using one source to cross-check the other, could open up a whole new area of privacy invasion.
Imagine this stuff running on all the CCTVs in the town where you live...
oh brave new world, that has such people in it!
In french the word "benjamin" and the french translation for "eat shit" have exactly the same lip movement pattern. Just a tought like that
... lavender soap. I've heard those two phrases can be difficult to tell apart by a lip reader.
As for computer lip reading, there's a chapter in Hal's Legacy about this very topic.
i believe the science fiction story 'colossus, the forbin project' makes use of the computer not only reading lips, but also body movement.
I think it'd be about the same as if you were to try to sell an audiobook you recorded of someone else's text. Free speech only covers things you created in the first place. It's not free usage of speech.
StoneCypher is Full of BS