Slashdot Mirror
Microsoft Research Uses Kinect To Translate Between Spoken and Sign Languages
An anonymous reader writes in with a neat project Microsoft is working on to translate sign language with a Kinect. "Microsoft Research is now using the Kinect to bridge the gap between folks who don't speak the same language, whether they can hear or not. The Kinect Sign Language Translator is a research prototype that can translate sign language into spoken language and vice versa. The best part? It does it all in real time."
They do. They use iPhones (and iPads). iOS has an accessibility feature called VoiceOver which changes the input paradigm from a touch initiating a 'click' action to one where a touch reads out the description of the UI element with text-to-speech. Two taps on the item will send a tap to the UI, and a three-finger swipe will initiate scrolling. So you can basically drag your finger across the screen and find things with your ears instead of your eyes, then navigate and interact as a sighted person would once you get your bearings.
This is one of the stupidest things I've read all day. You think that pupils at a school for the deaf are sending each other text messages as they stand next to each other? Signing is still very popular among the deaf, and is even a part of the distinct culture that has arisen in deaf communities.
No, and neither has she.
Disclaimer: yes I work for Microsoft. No not on these projects.
This was demo'd live in front of 30K MSFT employees at our annual company meeting. It nearly brought me to tears. Yes, I can see through demoware and and yes it's highly imperfect, but honestly it was the single most impressive use of technology I've ever seen. It was both novel and simple. It combined hardware, algorithms, user experience, and cloud scale. I don't know if it will ever go anywhere though I expect that it will. The key point here is that these are off the shelf components. Kinect and gesture APIs combined with machine translation and text to speech. It's important that these are, all or nearly all public production APIs. Such a system 10 years ago even if possible, would never make it to market because of the tiny user base. Today we can build such apps for the 0.01% of the population that need Mandarin Sign Language translated to English. And it can be cost effectively. That is the point. Technology being used to address real problems for under served communities. So yes, maybe people researched automated sign language recognition years ago, but bringing it to market and enabling a scenario for real people is a wholly different beast