Slashdot Mirror
Cell Phones for the Deaf
nitzan writes "Quoting from the article: 'the software translates the voice on the other side of the line into a three dimensional animated face on the computer, whose lips move in real time synch with the voice allowing the receiver to lip read.' Unfortunately this only works with laptops, but a pda version is in the works." The company website has a demonstration.
Still no?
Ok, can you hear me now? Still no?
Ok....
...so now we'll all have to learn how to sign "Turn off your fucking phone, asshole!"
What was wrong with speech to text?
"With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea...."
RFC 1925
I worked with deaf people for a while and they were (and I am sure still are) disappointed that cell phones are not compatible with TTY devices. How difficult is this to do?
No downloadable ring-tones.
Tuus crepidae innexilis sunt.
This just seems complicated, why can't they just improve the speech to text capability. It seems like drawing a face with life-like facial movements to enable lip reading is a little beyond the scope of power for a PDA.
This is a fantastic idea which will enable communication for the vast numbers of hearing impaired, however if the web-site is any indication, the technology needs improvement. I'm pretty good at reading lips and I was working pretty hard to figure out what was being said with the sound off.
Visit Jonesblog and say hello.
Being a severely hearing impaired person, I do find the virtual person's "O"'s to be highly disturbing if not graphic. Yikes.
I lived with a deaf room-mate last year. It took me about 2 months for me to understand what he was saying, and took him about the same to get used to my lips. Anytime he meets someone new, its very hard for him to read their lips (i.e. every time a new telemarketer tries to prey on the deaf user). Also, its not just the lips, its the tounge also. It'd probably be easier to use speach-> text software than this stuff....and what about background noise? I doubt this thing works well if not at all.
I sure as hell couldn't tell you what they were saying, even when I knew what words were coming out of their mouth. And this is not to mention cell phone static, distractions, contractions, mumbling, and lots of "ummm" and "uhhhh" that occurs during normal speech. I really don't see how this is a viable communication method.
Maybe it's because I'm not experienced with lip reading. Maybe people who are deaf are better at it than I am, but I can usually tell what Football coaches are saying on the sidelines of games (of course, that's limited to "Bull****" and "You've gotta be ****ing kidding me!", but still...)
I still cnat get coverage, or hear the other person clearly, why should the deaf be different? But i can ply 3 different games and send a fucking picture of a duck. Stupid phone companies. Its a fucking phone!! First, fix it so i can hear someone, THEN gimme the damn bowling games.
OK, this might be a troll. Im not sure myself. Its definately a vent. Fucking sprint. Oh well.
All Troll + "offtopic" mods are meta moderated as "Unfair", because you abused the system.
I just can not picture myself on a bus looking at this wildly articulate mout while yelling back: "Can yoo reepeeet dat agaannn???" Yes, I am hearing impaired. I would NEVER touch this thing. I'll stick with 2 way messaging.
look at a lip reader and say:
'I want a fig newton'
IMHO:
too many flaws, the investors will back out
pretzel_logic
Someone can say "Pot" and yet with the same lip movement, can also say "My". Men with bushy mustaches are a lip-reading disaster.
For me, I've adapted in my own way: I rely heavily on my hearing aids. That combination of both lip-reading and hearing the audio stream from your mouth enables me to achieve at least a 70% success rate (under ideal conditions, if it's a party atomosphere, fudgeddaboutit). I've had hearing aids since I was 1 1/2, and only with extensive speech therapy can I speak well. I'm one of the few deaf-from-birth people that can do it this well. So, from that perspective, I can speak on a phone (as long as I can understand that mangled audio coming out the receiver, which is 0%).
Why don't they just focus on speech recognition? A great speech recognition phone would enable deaf people that speak to use phones for near real-time conversations. In addition, such technology can also be (easily?) adapted to foreign language translators for tourists.
However, until such technology is available at the consumer level, I'm stuck with two-way text messaging devices like the T-Mobile SideKick.
-Cyc
/.'s 10 Millionth
I thought it seemed a little weird at first, but then I checked out the other demos. When I knew what the words were ("Thank you" in English, German, French, Spanish, and Japanese), I could easily tell what was being said.
I notice a lot of people complaining about improving text-to-speech, which is far more advanced than this technology. Speech sounds come out in a continuous flow. Getting a computer to recognize the breaks between words, properly spell them reliably, etc. is hard enough on a desktop system, much less a PDA. Especially considering in languages like English, where most vowels in unstressed syllables are rendered vocally as "uh".
This system simply has to hear a sound, and immediately display an associated... well, not "grapheme", since this isn't writing... maybe "pixeme". It is the graphical equivalent of attempting to spell perfectly phonetically.
Also, if you didn't notice it, "invisible" sounds that occur on the back of the tongue are indicated by circles on the cheeks (like hard 'g' and 'k'), and nasal sounds are indicated by a darkening of the nose.
All in all, I think this is an interesting idea. It will be even cooler when they can render different faces so the "avatar" resembles the person to whom you're speaking.
bytesmythe
Hypocrisy is the resin that holds the plywood of society together.
-- Scott Meyer
Partly, because speech to text isn't very good.
Speech to text isn't very good because its very hard to turn phonetics into words. Our ability to understand people is very reliant on context. Knowing what's been said helps you understand what's being said.
Some will say that speech to text is getting fairly good in English, which is somewhat true. Obviously, though, there are bigger markets in other languages.
So how does this thing work, if it doesn't do speech to text? It does speech to phonetics, and phonetics to lips.
For example, its relatively easy to understand when someone has said "h -ee- r", but knowing if that's supposed to be "here" or "hear" is quite difficult.
This is why the same software works across languages. "Th" is "Th" in any language, and your single algorithm doesn't have to care.
-Zipwow
I don't know which is more depressing, that 2/3 didn't care enough to vote, or that 1/2 of those that did are crazy.
Seems like it's not over-engineering. This is less steps than speech-to-text as far as I can see.
You have to record the speech and convert those sounds into phonemes. Now all you do is use the picture(s) that go with that phoneme, which is going to be more or less consistent.
With speech-to-text you have to use probability and word banks to figure out what the heck words those phonemes are supposed to go with, which is the hardest part by far, because spelling and grammar is so inconsistent. That requires a lot more time and computing power, and you are prone to a bunch more mistakes of course.
If you want to make an apple pie from scratch, you must first create the universe. -- Carl Sagan
This is clearly a solution for the large population of completely illiterate deaf people, for whom speech-to-text is not an option.
... if you have this software running on a phone then if you are hearing impared you could get real time conversation with the other party without having to go through a human being.
I've spoken with a hearing impared person on a phone before through a TTY system and it is painfully slow. First you have to say your sentence and then they send it. Then the other end needs to read it, type in a response, and then send it at which point it is read back to you. Imagine having a conversation over an Instant Messenger except you're secretary was reading the screen and typing for you. (IM for the blind for example)
I agree that we need better voice to text and text to voice translation. That technology would give use better access for everyone. You could have "hearing" for the hearing impared (speech to text), "reading" for the vision impaired (text to speech), and you could even have "writing" for those with fine muscle control imparement or who are lacking the necessary limbs for various reasons.
But this is an interesting approach to solve one of the three problems.
42 - So long and thanks for all the fish.
David Stork has a chapter computer lip reading on in the book "Hal's Legacy" on A.I. methods. The combination is much more reliable that either audio or visual.
I'm thinking about taking that new chick from Logistics. If things go right I might be showing her my O-face. You know: Oh! Oh!
Reading this makes me realize that my Lightbulbs for the Blind scheme was not crazy! Bundles of cash, here I come!
C - A language that combines the speed of assembly with the ease of use of assembly.
Posting late, but wtf.
By way of introduction: I developed the core coarticulation and other algorithms for lip synching when I worked at a now-defunct company called...wait for it...LIPSinc. We thought the resulting lip synching was pretty damn convincing, so on my own I tested out our stuff with a hearing-impaired friend, with mixed results. Anyway, I don't know a little about this stuff, I know a *lot* about it.
What these guys have done is map phonemes onto exaggerated visemes (the pictures of the mouth). Not a bad idea at all! Bunch of problems, though. First, there's a data data reduction of about 3x in going from sound to video--there are 40-50 distinguishable phonemes, and 9-16 distinguishable visemes, depending on how you count each. This is because the visible part of the face only makes up the end of the vocal tract, a lot of distinctions between letters occurs without the involvement of the lips, like the difference between F and V, while others, like K, can be pronounced with the face in virtually any position. This is part of what makes lip reading so hard with a real person, and why they need a lot of context to pull it off. They also seem to be slowing down the timing, as if they recognized the phonemes and then synthesized each at the same length. This gives longer to recognize each one, but wrecks the visual prosody (rhthym) of the speech, which is a good cue for where the parts of speech are. Then there's the rest of the face. The eyebrows and head positions help you figure out key words, ends of clauses, tell if something is a question, etc.
Those who say that TTS is superior to lip reading have a point. Good TTS contains *more* accurate information than an uninterpreted stream of phonemes (itself 3x richer than a stream of visemes, as I said above), because the machine can do a Viterbi search to find the most likely sequence of words from a continuous stream of phonemes. Words also open up higher NLP functions, so you can do constraint relaxation to test whether "wreck a nice beach" or "recognize speech" fits better in the context.
Still, I'd like to see an experiment where the raw phonemes are fed, as text, to the recipient. I think with practice, your brain would start to decode the string (it manages with the sound, right?), despite the lack of word boundaries and the errors in phoneme detection (which is not all that high without text-I think seventy-something percent). Seems like an easier pattern recognition problem than lip reading. Who wants to go get funding?
We have tools like Sprint Relay On-Line that will do text-to-speech... and every state provides confidential relay services to begin with. Many states are moving towards making 711 a standard relay number.
If a deaf person wanted a "cell phone", they'll probably have one from Wynd Communications, a two-way pager with text/e-mail and other services built right into the damn thing. They're all the rage here. Screw lip reading over the phone. This technology is pure eye-candy. Nice, but how useful will it really be?
--You will rephrase your request for me to go to hell. Goto statements are not acceptable programming constructs