Robotic Hand Translates Speech into Sign Language

usermilk writes "Robot educators Keita Matsuo and Hirotsugu Sakai have created a robot hand that translate the spoken word into sign language for the deaf. From the article: 'A microchip in the robot recognizes the 50-character hiragana syllabary and about 10 simple phrases such as "ohayo" (good morning) and sends the information to a central computer, which sends commands to 18 micromotors in the joints of the robotic hand, translating the sound it hears into sign language.'"

Comments

How to sign First Post?

[A Dafa Disciple]

Good lord! I imagine the Japanese language with its 1945+ character alphabet is hard enough to learn; learning Japanese sign language must really suck.

You know what would really spoil those deaf kids is, instead of a robot doing sign language, a robot that shows images or words based on what a speaker says. I know, I know; creating a robot to do this is a feat within itself and impressive in its own right, but perhaps there are better ways of communicating with a robot if it can already perform more than adequate speech recognition.

Re:How to sign First Post?

[Kokuyo]

Uhhh, isn't sign language universal? I thought it didn't depend on the spoken language. I might be wrong, of course :).

Re:How to sign First Post?

[Anonymous Coward]

Why would learning Japanese suck anymore than learning BSL or ASL?

Re:How to sign First Post?

[tpgp]

Good lord! I imagine the Japanese language with its 1945+ character alphabet is hard enough to learn; learning Japanese sign language must really suck.

The relationship between a language & sign language does not work like that.

From the wikipedia sign language page

A common misconception is that sign languages are somehow dependent on oral languages, that is, that they are oral language spelled out in gesture, or that they were invented by hearing people

and

On the whole, deaf sign languages are independent of oral languages and follow their own paths of developmental. For example, British Sign Language and American Sign Language are quite different and mutually unintelligible, even though the hearing people of Britain and America share the same oral language.

You know what would really spoil those deaf kids is, instead of a robot doing sign language, a robot that shows images or words based on what a speaker says.

That doesn't really sound like a robot, but speech recognition software connected to a teleprompter (or monitor)

Re:How to sign First Post?

[Anonymous Coward]

I believe it's a joke... but just to point out, there's ASL and BSL and...

Re:How to sign First Post?

[Jugalator]

Unfortunately, no ;-)

Re:How to sign First Post?

[lewp]

Japanese has a whole bunch of kanji, but the various words in the language can be formed from a much smaller (hiragana, mentioned in TFA) character set that represents the various syllables in the words. These syllables are always pronounced consistently, unlike languages like English where sometimes it seems like nothing is consistent (and I'm a native speaker). Thus, the first thing that came to my mind was that teaching a robot spoken Japanese is probably quite a bit easier than teaching one English (though neither is a trivial task, obviously).

I know nothing about Japanese sign language, and practically nothing about American sign language, but I believe American sign language shares a similarity to written Japanese in that there are signs for common words most any competent signer knows (similar to kanji), and any particularly uncommon words can be signed out with the letter (or in the Japanese case, hiragana syllable) signs. Thus, I doubt teaching a robot enough Japanese sign language to be understandable wouldn't be any harder than teaching a robot American sign language. Which, once you've turned the speech into letters/syllables in the speech recognition part and programmed in the gestures, would be pretty much trivial. Japanese children's TV and manga aimed at kids (I'm told) mimmicks this behavior by mixing the simple kanji school children will have learned at a young age with the hiragana for the words that aren't expected to be known.

I'm shooting from the hip here based on what little experience I have with this stuff, so feel free to correct me, experts.

Re:How to sign First Post?

[magefile]

I know nothing about Japanese sign language, and practically nothing about American sign language, but I believe American sign language shares a similarity to written Japanese in that there are signs for common words most any competent signer knows (similar to kanji), and any particularly uncommon words can be signed out with the letter (or in the Japanese case, hiragana syllable) signs.

Sorta, but not quite. You can fingerspell words you don't know, and some words are derived from their associated letters (i.e., one of the possible signs for "what" looks a lot like a "W" snapped into a "T", and one of the signs for toilet looks like a shaken "T"), but some of these are frowned upon culturally (cultural baggage due to decades of surpression of sign by hearing people). Too, if you depend on fingerspelling too much, you'll find it difficult to communicate; you won't be able to receive well, and while Deaf will put up with receiving it if they know you're learning, it's not sign language, and everyone knows it. Doesn't fit in all that well with the syntax and grammar of ASL, either.

Re:How to sign First Post?

[Andrzej Sawicki]

That would not require a robot, just a screen to display the messages.

Re:How to sign First Post?

[Anonymous Coward]

Not only that, but sign is extremely dependent on context. A single spoken English word (I typically use 'run') will have numerous signs depedent on context. I.e. 'run to the store' is a different sign from 'the car is running', and from 'I have a run in my stocking', or 'my nose is running'.

In short, this is neat, but there's still a long way to go to replace real, human interpreters.

Re:How to sign First Post?

[Zaatxe]

This robot could be used by deaf dogs, which (I'm not sure it's a myth or not) can't see flat images.

Re:How to sign First Post?

[Zaatxe]

You think signing language could be hard to learn? Imagine blind people trying to learn japanese in Braille!!

Re:How to sign First Post?

[smoker2]

You know what would really spoil those deaf kids is, instead of a robot doing sign language, a robot that shows images or words based on what a speaker says.

Which do you think is better ?

a)A robot that translates words into pictures

b)A robot that translates one language into another

Deaf people aren't stupid, at least no worse than the rest of us. You do realise that they probably already know sign language (which is the reason for the robot) so to be shown stupid pictures would be a little demeaning, don't you think ?

Not forgetting that they will have to communicate between themselves as well, should they use pictures there too ? They would need a pretty big file to carry all those picture cards around in.

Have you ever travelled abroad ? How do you think people would react if you tried to communicate with them by showing them pictures, instead of at least trying to make yourself understood in their own language.

Re:How to sign First Post?

[Squalish]

We already have a means of getting Spoken Word into the visual spectrum - it's called Written Word. Are the deaf unable to read? Why not just take a speech recognition program, and put the words on the screen, rather than translating them to sign language? Evem the deaf-blind have mechanized Braille, which I'm sure is more comfortable reading quickly than a robotic hand (at high speed, known as a 'The Mangler').

Re:How to sign First Post?

[davidsyes]

To be pedantic, Japanese doesn't have an "alphabet". It has syllables, representations of sound and utterance. If I may quote a portion of a page (out of some 250-plus pages from Gene Nishi's "Japanese Step by Step (ISBN: 0-658-01490-0):

A syllable is defined as a single uniterrupted sound forming part of a word or, in some cases an entire word. (Example left out...)

We know that there are only 26 letters in the English alphabet. But how many different syllabic combination sare there? According to the late Prof. Minoru Umegaki of Tezukayama Gakuin Junior COllege, as quoted by Prof. Haruhiko Kindaichi in his book "Nihonga" (Iwanami Shinsho, 1985), there are more than 3,000 different syllables in English.

Japanese Syllables:

What about Japanese? You will be surprised to know that there is no alphabet, and there are only 101 syllables in Japanese. It appears to be an easy language to learn doesn't it? But wait. Only 101 syllables to compose hundreds of thousands of Japanese worlds! The consequence is a clear as day: there must be a huge number of homonyms and heteronyms in Japanese."

(It goes on to explain that accent and use of kanji characters to avoid ambiguity. However, my instructor long ago also told us that HAND SIGNALS or writing in the palm and sketching out the expression in the air was sometimes necessary. The Japanese (and other languages' users, I'm sure) have an enormous number of rules and contextual nuances that-- if left out of the equation could-- be embarrassing or fatal (literally, for friendship, job, or mortality...), depending on the context...)

But, what IIII want to know: Will this robot be modded to flip the birdie?

Re:How to sign First Post?

[JoshuaLawrence]

Because sign language is the native language of many deaf people, not Japanese or English - whether written or spoken. Wouldn't you prefer your robots to speak in your native language, instead of some foreign language?

Scutters!

[LiquidCoooled]

They only need to put it on wheels and it can become a scutter.

Additional warning:
Do not let this robot pat you on the back whilst near the top of the stairs.

Re:Scutters!

[IAstudent]

Only a smeghead like you would come up with such thoughts. ;)

Re:Scutters!

[meringuoid]

Do not let this robot pat you on the back whilst near the top of the stairs.

But you must be protected. You must go down the stairs...

Re:Scutters!

[AndroidCat]

Yeah, the scutters were pretty good at sign language, weren't they?

Over Kill?

[xoip]

Call me culturally insensitive but, why not simply translate speech to text?

Re:Over Kill?

[chr1sb]

I agree. This is an example of something that is fun but impractical. I assume they did it because they could, and because it would pique the students' interest in robotics, which is a worthy goal. As a real-world application however, a text-to-speech unit would be cheaper, much more robust, much smaller and more portable, most likely faster, useful to a wider market, etc. etc. But it wouldn't look as cool.

Re:Over Kill?

[tpgp]

Call me culturally insensitive but, why not simply translate speech to text?

Because signing is the native 'tongue' for most deaf people - and it is easier for them to communicate using sign language (over text) - just as its easier for you to understand speech (over text).

Basically - the same reason that some British TV (and undoubtedly many other channels around the world) have a signer translating the news rather then scrolling text.

Re:Over Kill?

[Anonymous Coward]

The idea that Deaf people could just read something and that it's "good enough" can be a rather dangerous assumption to make. In the US at least, the use of sign language varies greatly. One one end you will have people using American Sign Language which is quite different from English -- it has it's own structure/syntax/rules. On the other end you'll have people using a type of sign language that is heavily based on English word order (some of the signs are actually different in order to code the English language). And you'll even have people that float between these two extremes.

Why does this matter? Because from my experience (I worked as a sign language interpreter for 5 years - up until last year), there are a lot of Deaf people out there that don't sign using strict English word order because they don't have a strong grasp of the English language. There's been many times when a Deaf person has come up to me and asked me to interpret a news paper article because they don't understand the English. There's been many times I've had to read a Deaf person's writing and the only way I could understand it was to sign it word for word because they are doing their English is much close to American Sign Langauge than the language that we use everyday.

As for my statement I made earlier about it being dangerous to assume that Deaf people can read and fully understand English...I went to a job at a clinic. The Deaf person had been going there for 20 years and never had an interpreter (the doctor thought it was good enough to write back and forth, because all Deaf people can read English, right?). During the appointment it came out that they had many "discussions" (really writing back and forth) about the high chance that the Deaf person has cancer -- and that a major problem was this person's lack of motivation to do anything about it. After a bit, it came out that the Deaf person had no idea what the word "cancer" meant.

Re:Over Kill?

[Aragorn17]

I thought the same thing initially... but then I thought about deaf *children* who don't know how to read. I have a neice who falls within this category and she can sign quite a bit... but can't read a single word yet (she's only 2).

Re:Over Kill?

[JesseMcDonald]

just as its easier for you to understand speech (over text).

I realize I may be a bit unusual, but I tend to understand text faster and more accurately than speech. It is certainly faster to speak than it is to write out the text, but I can read an entire paragraph in a second or two, whereas communicating the same information as speech would take much longer. Also, I usually visualize the speech as text in my mind and then "read" it, rather than interpreting the words directly.

Re:Over Kill?

[Anonymous Coward]

Actually, that's not quite right. ASL and many other sign languages do not have a written form. Text (in English) is not written ASL. So, it's not really a fair analogy. This is why a signing robot would be more useful than text.

Re:Over Kill?

[Blakey Rat]

Because if you put an LCD screen onto your humanoid android, it looks less human.

Re:Over Kill?

[Anonymous Monkey]

That brings up another interesting point. For this thing to be really useful, it will need to translate English to ASL. The grammar is different and many signs don't have an exact English translation.

For example sign for the English word 'worst' is better described as 'dramatic increase or change.' So, if I say 'Wow, my score on Asteroids has improved' I would use the signs 'wow, game, asteroids score worse' finger-spelling the word asteroids and having my eyebrows up and smiling as I sign worse. Also, the English word 'my' would be inferred by context.

The point being, this will never REALY work until some one develops a universal translator.

Does it also distinguish...

[MadJo]

... Does it also distinguish between different 'dialects' in sign language?
I seem to recall that sign languages differ between countries, same as 'natural' language.

However this is really great for deaf people.

Re:Does it also distinguish...

[davidsyes]

Dialects also differ between, say, Osaka, Hokkaido, and Tokyo. Tokyo dialect is the official, trained one, but with a good ear, despite the frustrations, communication CAN be had.

However, going from Bahston to Hyooton, to Narlenz can be harrowing. Anyone remember the campaign trail with Bush version 1 was traipsing thru Louisiana? I can, a bit. I remember all that twisty, silky, unintelligible stuff that obviously was aired for entertainment value by the networks. (I am a person of less than a lighter color, so maybe it's partly-ok for me to hint at the language stuff, and I have ancestors from Louisiana, so maybe I can take a little liberty with ancestral punning/jabbing...)

Picture of robotic hand

[commodoresloat]

signing "I'll be back"

More Useful As Software

[Afty0r]

Would this not be more useful as software, able to render simple 3d hands with low microprocessor overheads, and preferably available for mobile phones and PDAs?

Deaf people could carry a PDA, and when they need to find out what someone is saying, they can hold the PDA up like a microphone, and watch the screen, assuming the translation is at least reasonable accurate...
Of course they could lipread too but some find that harder than others, and this could also be used eventually to cross language barriers?

I imagine it's extremely hard to lipread a foreign language.

Re:More Useful As Software

[Anonymous Coward]

or just use speech to text on the pda

Re:More Useful As Software

[John Hasler]

> Would this not be more useful as software, able to render simple
> 3d hands with low microprocessor overheads...

There is no need for 3d. I know a woman who makes her living as an ASL translator. She spends most of her time sitting sitting in front of a monitor and camera wearing earphones.

The robot hand is pointless. The computer could just as well generate cartoon images.

Re:More Useful As Software

[whimdot]

With a real hand it could be used by deaf-blind people who need to touch the signing hand to understand.

Re:More Useful As Software

[kidcharles]

You forgot where this was developed: Japan. First step in the Japanese engineering design process is to answer the question: "Can we build a robot to do it?"

Amy Pretty

[Inkieminstrel]

Tickle... Amy... Tickle

http://www.imdb.com/title/tt0112715/

Re:Amy Pretty

[davidsyes]

Might be redundant, but talk about "Talk to da hand"...

Say the wrong thing and you might be "bitch-slapped". Or, with that alloy, maybe beaten to a shallow tallow pulp... (woo... any onomatopoeia coming from that hand, a la "AKA I'M BATMAN!!!"?... squish, khoomp, BAM, POW, THOONG...

But, certainly, the Japanese model might actually USE the middle finger gesture. Over there it is NOT the same thing as here or in some English-speaking countries. And, the Japanese arm/hand most likely WON'T be waving "come here" with the palm facing the waver, or the back of the hand towards the subject. That gesture is used for hailing dogs and other animals, and is quite condescending when used on (umm, toward) a human. But, if the user abuses or "mis-talks" to da hand, I am sure it can be modded to instruct the semantics of "onomatopoeia" in action. FEEL THE POWER OF THINE HAND.

But, it might give a a new meaning to/for the Western Geeku: "Handu-jobbu", hehehe... as in replacing all those photo booths all over Harajuku and Akihabara....

Talk to the hand

[99luftballon]

The possibilities for this are endless - converting 'wanker' into an off the wrist gesture, raising one or two fingers for the US or UK symbol for f*** off, the list goes on... On a practical level however this is surely of limited use. Conversion to text would be far more useful and allow deaf people to talk to non-deaf folk. Cute application mind you can could have some spin offs for better robotic hands in the future.

Re:Talk to the hand

[AllahsAvatar]

I am pretty sure this is a Japanese sign for friend, but I can't find any linkage to back me up on this.

I've gotta hand it to these scientists

[commodoresloat]

The robotic hand was shown at a two-legged robot tournament

So it's not just a hand, but a hand with two legs!

Re:I've gotta hand it to these scientists

[Anonymous Coward]

No - the two-legged robot tournament means that the second leg is the return fixture in a couple of weeks?

foriegn language

[Anonymous Coward]

To them, its not a foriegn language.

Other uses?

[Anonymous Coward]

Anyone else thinking there might be some other 'alternate' use for this device?

Re:Other uses?

[KiroDude]

yes!!! use it and ask repeatedly to translate "wanker" ...

This gives a whole new meaning

[sikandril]

This gives a whole new meaning to the phrase "talk to the hand"

Re:This gives a whole new meaning

[Meester Nice Guy]

And the hand can talk back to YOU!

it's about form factor

[commodoresloat]

Would this not be more useful as software?

Yes but not nearly as intimidating. Who's going to get their lunch money taken -- deaf kid with a PDA, or deaf kid with a giant robot hand?

Re:it's about form factor

[dancallaghan]

Parent is not kidding! From TFA:

An 80-centimeter robotic hand

!!!

Re:it's about form factor

[meringuoid]

Who's going to get their lunch money taken -- deaf kid with a PDA, or deaf kid with a giant robot hand?

Remember, though, that this is Japan. Kid with PDA probably merges with the Wired. Kid with part of giant robot merges with... well... pretty much everything, after a while.

Once that happens, your lunch money is the least of your concerns.

A different dept.

[Anonymous Coward]

This might as well be from the "doing-it-just-because-we-can" dept. As many slashdotters have already pointed out, this is pretty impractical.

Re:A different dept.

[Anonymous Coward]

Oops, thought it was the kind of sign language for blind-deaf people. Never mind, ignore me.

I gotta stop posting on /. before my morning coffee.

Mentifex Artificial Intelligence

[Anonymous Coward]

Mentifex AI might benefit from this technology of sign language as a form of speech output.

Microsoft will probably find a means of censorship for users in China.

Text is not the same

[QMO]

Since sign languages are different all over the world, I don't know if there is the same problem in Japan, but:

American Sign Language is not English (American or other).

Thus, translating speech to ASL would reach people that that understand ASL but don't read Englih.

Re:Text is not the same

[emmadw]

I suspect that the numbers of ASL users who can't read is small. However, as sign languages are linguistically very different from written languages, many profoundly deaf signers aren't good readers. (To start with, we have a lot of pretty meaningless words in text ... a ... the ... etc) So, signing makes it much easier to get the richness of speech. When you textualise speech, you have to cut words out, and loose a lot of nuances. (Have a look at the closed captions on the news sometime).

Re:Text is not the same

[Anonymous Coward]

Thus, translating speech to ASL would reach people that that understand ASL but don't read Englih.

1) There is virtually noone who understands ASL, but doesn't read English.

2) I think it might translate to Japanese sign language hey?

Re:Text is not the same

[magefile]

Literacy is very low among people whose first language is ASL. I believe the commonly quoted statistic is that much of the deaf population reads at a 5th grade level? Thus, part of the job description of an ASL interpreter can be to translate English text into ASL.

Re:Text is not the same

[emmadw]

Yes, I think that the statistic that I read (in the UK) is that a profoundly deaf adult (who's never had hearing), is about age 8-9 - which is I guess about the same.

Having said that, the average reading age of the average adult isn't that great - for example http://news.bbc.co.uk/1/hi/talking_point/3166967.s tm cites a study that say "millions of adults do not have the skills of the average 11 year old" (implying that it's because their skill set is lower, not higher!).

So, certainly in the UK, though profoundly deaf adults aren't always proficient readers, there's a significant proportion of adults who aren't deaf, who also aren't great at reading.

Hence, speech or signing benefits is likely to benefit more people than text.

Re:Text is not the same

[aude_sapere]

Why not just give it the dual ability to write and sign depending on circumstance...signing is a start in the direction of more complex hand motions, the next big step is to construct a hand that can balance well enough to write.

Wonderful

[solanum0]

Finally deaf people can use a computer too.. err.. wait a second..

This is great and all...

[romiir]

Without a camera that translates sign language into spoken language.. This is kinda useless isn't it?

You can talk to the hand, sure but, that doesn't help you read the deaf persons hands..
--
In retrospect: ...and yes.. I know some deaf people can talk sorta.. So I guess it helps there.

Re:This is great and all...

[Anonymous Coward]

What deaf person? I wanted to use it as a wireless router.

Re:This is great and all...

[sobriquet.net]

Work in the "sign language to spoken language" translation is definitely being done. There have been gloves (similar to the one "Amy" wears in Congo) which have had some success.

I've found a paper on Australian research into Auslan recognition with "PowerGloves", for instance. And I know there's plenty of other research out there!

As for ones which recognise sign language from a camera, I haven't heard of it...

The article makes no sense

[MobyDisk]

...that can covert spoken words and simple phrases into sign language...

Ignoring the spelling, this implies that it has speech recognition. It converts SPOKEN words.

...recognizes the 50-character hiragana syllabary and about 10 simple phrases such as "ohayo" (good morning)...

Hiragana is the Japanese phonetic alphabet, so it READS. Huh? Now, does it read only 10 simple phrases, or does it read anything plus it recognizes 10 simple audio phrases. I guess the breakthrough is the hand articulation and the idea, not the rest.

Re:The article makes no sense

[Anti_Climax]

I guess the breakthrough is the hand articulation and the idea, not the rest

Actually, some seniors from the local DeVry made a robotic hand, controlled by an HC11 microcontroller, that could be used for signing. That was around 2000.

Deaf Wife/Son

[SirASCII]

Awesome! This could be very cost saving for when my son starts school. Just train the robot on his teachers voice to sign what they are saying... It looks like the only thing I'll have to figure out how to do is to translate SEE - Signed Exact English to ASL - American Sign Language...
Hope they make a good SDK...

Universal Signs

[digitaldc]

Can you flip off someone in Japanese? Give them the OK sign? Give them a stop with the full palm?

It would be interesting to know how these motions translate, if at all.

Re:Universal Signs

[Boccaccio]

I teach in Japan and I've found some gestures to be the same whilst others are quite different. The OK sign in the west is often misinterpreted as it is similar to the the sign the Japanese use for money. When you call someone over to you or hail a cab you use your hand palm down and make a kind of gesture that looks like a western childs goodbye gesture. These are just a couple of examples.

Re:Universal Signs

[Boccaccio]

Oh yeah, if you give someone the bird in Japan then, well, you're giving them the bird. Might be good to check they don't have a lot of tattoos or are driving a car with black tinted windows first ;-)

Re:Universal Signs

[2obvious4u]

Then there is classic expample of this when Richard Nixon visited Australia and gave the sign for Victory. Well that wasn't exactly what he was doing in their minds... http://en.wikipedia.org/wiki/V_sign

what if

[jaimz22]

if i tell someone to f-off will it flip them off for me?

Visibility?

[SeanDuggan]

A good sign language interpreter can read signs from a fair distance, well across a board room at least. How far away can the PDA be before you stop being able to read the text on the screen?

Universal Sign Language

[SeanDuggan]

Unfortunately, no ;-)
Yup, much the same as how, unfortunately, no one's come up with a universal spoken or written language. Gosh, let alone trying to get a universal programming language...

Re:Universal Sign Language

[cayenne8]

"...no one's come up with a universal spoken or written language..."

Hmm...isn't the middle finger signifying "Fuck You" pretty much universal?

:-)

I wonder if the robot hand translates that properly?

Japanese Sign Language link

[SeanDuggan]

For the sake of being informative, here's a good page on Japanese Sign Language. It's not the same as American Sign Language, which isn't the same as British Sign Language as someone's sure to post eventually. *sigh* Short of Gestuno, there is no universal sign language, no more than there is a universal spoken or written language. *rolls eyes* Except, of course, Esperanto, which everyone speaks by now, right?

Re:Japanese Sign Language link

[nblender]

This has always been something of sheer amazement to me. I've always been personally keen on ASL and have taken an ASL course. It hasn't done me much good because unless you practice and use it; it just goes away... But the deaf community has really lost an opportunity here. If, in retrospect, the community had developed a common sign language across cultures around the world, then I believe this USL (Universal Sign Language) would have become the lowest common denominator for communication among the non-hearing impaired. As a consequence, more people would know sign language and there would be fewer barriers for the hearing impaired as a result. Currently, when I travel to a place like Stockholm, I often see situations where a non-swedish speaking tourist, say German, will walk into a restaurant and will use very broken english to speak to the swedish waiter (who is actually Japanese and was just minutes prior speaking Japanese to his wife in the kitchen). English is the common denominator in these situations. Imagine what the world would be like if that common denominator were USL?

We tought our child sign language when he was 6mos old. By 8 mos, he was asking for 'more milk'. By 1 year, he had about 45 words in his vocabulary; long before he was able to use oral speech; he was able to tell us what he wanted for lunch; or that he was looking for a specific toy... I urge all of you geeks to look into this should you be lucky enough to have children some day.

This book is what got us started. http://www.amazon.com/gp/product/0966836774/sr=1-1 /qid=1137518491/ref=pd_bbs_1/102-4440407-6915304?_ encoding=UTF8

Re:Japanese Sign Language link

[davidsyes]

Out of context, your *rolls eyes* could be:

-- You're on crack, and drooling

-- You snorted Vegemite while hiking at YO-seh-MIGHT (Yosemite), but ended up (butt-up) a sulphur spring, except it had Killer Sulphur like in parts of Japan

-- You got whacked aback of the head with a finely-tuned, perfect-pitch, exquisitely-balanced duratanium rod, and freeze frame caught your eyes before they popped out

-- You're on your back, approaching orgasm, and drooling (just before being hammered by a duratanium rod, totally oblivious as Sil was rockin' so hard your airbed's scrubbing the wall clogged your ears)

-- You're imitating the Uneeda Medical Supply guy in Return of the Living Dead, as you're explaining, "Wanna see'em?" Freddie: See what? "Well, *rolls eyes* you knowwww, Thuuu CORrrpSEZ (tongue hangs out while eyes still roll)

-- You were buried vertically, head down

-- That hand walked over and sphincter-nated you...

Anybody notice that after his license-less encounter with the backing vehicle that Arny looked like he had a bout with a Terminator? I wonder how many pics were "photo-re-re-re-touched" out of deference to him. No more THRILL rides and no more ice cream for YOU. Hopefully, T-2 (if he ever becomes that) obtains a license AND obtains riding lessons from someone with the prescribed license...

Signing, but not reading

[QMO]

After reading a couple of replies to my parent post, I was thinking about people that might understand signing, but not read or hear.

It is my understanding that children can learn to sign before they can learn to read. (In fact hearing children can learn to sign before learning to speak.)

Similarly, developmentally challenged people, such as certain people with Down's Syndrome, never learn to read, but can sign just fine.

Reading takes certain specific brain functions, and it is not inconceivable that there are people who have had head trauma that damaged the reading part of their brain, and the hearing part, but can still understand sign language.

These are just quick thoughts and may have lots of holes, and little sense. Please feel free to expand/correct/flame/whatever.

Re:Signing, but not reading

[zsau]

There's also the fact that as sign languages are typically independent of spoken ones, people who know (say) Auslan will need to learn to read and write English as a second language, and the process is (I'm told) difficult.

Re:Signing, but not reading

[Anonymous Coward]

Just a couple corrections...

It is my understanding that children can learn to sign before they can learn to read.

If your native language is ASL, for example, learning to read written English involves learning a new language. So, that makes it more difficult than hearing children learning to read.

(In fact hearing children can learn to sign before learning to speak.)

It's true that hearing babies can make signs before they start using spoken language, but they are not really using the signs as a language in the way an adult would (they are more like gestures).

Reading takes certain specific brain functions, and it is not inconceivable that there are people who have had head trauma that damaged the reading part of their brain, and the hearing part, but can still understand sign language.

Yup, that's true. For Deaf people however, many of the brain areas that would normally be used to process speech in hearing people get recruited to process sign language. So, damaging them would affet sign language ability.

Recognziing Sign Language

[SWroclawski]

There's a researcher at Gallaudet working on the other side of this equation with a system designed to recognize sign langauge, which seems like a much harder problem.

ASL isn't like English in that there are always specific words- a lot of it has to do with spacial context (where in the signing space the sign was made) and a whole class of signs that don't translate directly into words (they're hand shapes which can translate into an event or a description of an object or set of objects).

And, as the research page shows, facial expressions and even facial movements can be part of a sign.

Of course, this is American Sign Language, Japanese Sign Language may be very different.

Re:Recognziing Sign Language

[cp.tar]

And, as the research page shows, facial expressions and even facial movements can be part of a sign.

Actually, AFAIK facial expressions, body tilt etc. have a para-linguistic meaning, much like the tone of voice and facial expressions in hearing communication.

Therefore, they are not - strictly linguistically (structuralistically) speaking, part of the sign itself, but rather a part of the co-text and context.
Anyway, that's what I remember from that one lecture a year and a half ago...

Re:Recognziing Sign Language

[SWroclawski]

That's *partially* true.

They have para-linguistic meaning but they're an important part of signining. It's not equilivant to the tone of voice as it would appear in a Western language (English, French, German), but it's often part of the sign itself in that certain facial expressions or mouthings should accompany certain signs.

They're also *vital* for things like questions, where eyebrow position are the indicator that the statement is a question. translated to English, it might be more akin to tone: "Going to the store." vs "Going to the store?"

Re:Recognziing Sign Language

[Anonymous Coward]

The basis of the problem from the paper (the only one not downloadable)

"The major challenge that faces American Sign Language (ASL) recognition now is to develop methods that will scale well with increasing vocabulary size. Unlike in spoken languages, phonemes can occur simultaneously in ASL. The number of possible combinations of phonemes is approximately 1.5 x 10^9, which cannot be tackled by conventional hidden Markov model-based methods. Gesture recognition, which is less constrained than ASL recognition, suffers from the same problem."

Re:Recognziing Sign Language

[cp.tar]

They're also *vital* for things like questions, where eyebrow position are the indicator that the statement is a question. translated to English, it might be more akin to tone: "Going to the store." vs "Going to the store?"

Which is exactly what I'd said. That is para-linguistic.

Thing is, hearing people who learn signing later in life more often than not do not learn the expressions, body tilt etc. properly, and even use them rather incorrectly when they try to. It does not make them unintelligible, but - I guess - rather like hearing communication with a speaker with an accent different from yours.

Don't get me wrong; I'm not saying para-linguistic meaning is any less important; it carries an awful lot of meaning. It's just that I'm nitpicking.

Wait...

[void24601]

So now I have to learn Japanese to communicate with the deaf?

this has been done dozen of times

[peter303]

I remember reading about Stanford grad student project doing this ten years ago and a winner of the National Science Fair doing this three or four years ago.

i wonder...

[ladyKae]

... will a version be demonstrated at future Ann Summers parties...

Very large?

[necro81]

The article states that the hand is 80 cm large (doesn't specify, but I'm guessing that's height). 80 cm is almost three feet for non-metric types. My own hand is only about 12 cm long. Is this the largest communicating hand on the planet? Or, as is more likely, the 80 cm takes into account the massive box of micromotors and computing. Pay no attention to the man behind the curtain.

Re:Very large?

[Anonymous Coward]

I kind of doubt the 80 cm is the size of the hand, I suspect it's the length of the arm... In many languages the word for "hand" can mean the entire arm, so it might be just a translation issue.

Been There, Done That...

[Carcass666]

There is an article Evolution of Mechanical Fingerspelling Hands for People who are Deaf-Blind that talks about the development of this technology since 1977.

There are a couple of challenges with this type of technology. Sign language does not depend only on finger movements but gestures and facial expressions to convey emotion and context. Finger-spelling hands, being mechanical, can only accept data so fast before they start "choking" and sezing up/breaking (we tried hooking one up to a teleprompter application, and its middle finger got stuck - go figure).

This technology can be exciting on a small scale, but is not meant (not able) to act as a replacement for sign language or even closed captioning.

Not the first...

[necro81]

There have been other efforts to develop speech-to-sign robots. I recall one being featured on the Discovery channel many years ago that was able to fingerspell a variant of ASL that is used by persons that are both deaf and blind. That was nearly 10 years ago. In that case, the person "listens" by placing their hand over the signer's hand, and feels the different handshapes.

On another note, this sort of translation is actually more difficult than a voice-to-text, text-to-sign translation. As someone who studied sign language for several years, I can categorically state that such a direct translation is not the same as ASL, which has a different grammar and word order. The way things are described and conveyed in ASL is often not just a matter of stringing different signs that represent different words together into a sentence. Oftentimes it uses spatial and directional relations as well. Humans are able to understand these meanings quite naturally, but it is difficult to program that kind of style into a computer. The translation of voice-to-sign is just as difficult as translating between, say, spoken english and chinese. A more difficult task than voice-to-sign would be to go the other way around: a camera or motion-based system that extrapolates what a signer is saying, and then translate that to normal English.

JSL vs ASL

[SeanDuggan]

Interestingly enough, Japanese Sign Language has a trait which makes it less appropriate for this application than American Sign Language. Ambiguous signs are generally distinguished by mouthing the letter in JSL versus the finger-signed letters in ASL.

The next question which I have is the significance of body positioning of signs in JSL. Most ASL signs have migrated to the face and upper-chest region, but I know some sign languages have a great amount of significance in the body positioning and it may range all over the place.

Re:JSL vs ASL

[sobriquet.net]

My studies are mostly based in Auslan (Australian Sign Language), but I believe all sign languages have five elements to their signs:

1. Handshape
2. Orientation
3. Location
4. Movement
5. Expression

So, basically, the body location is just as relevent in JSL as any other sign language - change it, and the whole sign can have a different meaning.

There's plenty of examples in Auslan of changing one of the HOLME elements in a sign to make a completely new sign - change the orientation of "Not Yet", and it becomes "F&^k you", change the facial expression when signing "Rat", and it becomes "Orgasm", and other more boring examples...

There's a bit about this on wikipedia, too.

Can't see?, you can feel the fingers...

[eMpTyBeitler]

In the early 90's I worked with the robotic finger spelling hands called "Dexter" & "Ralph". Those devices were intended for individuals who are both deaf and blind. An individual with this kind of disability must rest their hand on the back of someone's hand (or on the back of the robotic hand) and feel letters as they are signed by the hand/fingers.

Re:Can't see?, you can feel the fingers...

[2obvious4u]

Ding. We have a winner. Then again you could always use a braille reader for this as well. http://www.tiresias.org/equipment/eb7_b.htm But the robot hand would be more fun than a Braille Monitor.

That's Cool

[Greyfox]

Now I can get a freaking ROBOT HAND attached to my HEAD so I can WRITE GOOD MORNING and have aforementioned ROBOT HAND sign it to a deaf guy. That's a LOT EASIER than just writing good morning and SHOWING IT TO THE DEAF GUY!

MST3000

[GigsVT]

Bah, Joel invented this on MST3000 years ago. Where's my edible sneakers?

blind and deaf, teaching, and other uses

[snooo53]

Someone could be blind and deaf. But then why not use braille? The situation I can imagine is maybe a person knew sign language but then became blind later in life. So that would be one of the only ways to communicate. From what I understand a lot of older people have eyesight problems, so for the deaf this is even worse.

The other use could be for teaching sign language. There's a lot of people that know a little sign language, but perhaps not enough to teach someone. Seeing a robotic hand do it in three dimensions might help.

Also if you develop the technology for a sign language hand there's probably other uses for it. Imagine a robotic hand on the end of a stick that you could use to grab fragile things from high shelves. There's thousands of things it could be used for if you use your imagination

Why didn't they just use a monitor with 3D image?

[iamacat]

It's not like that robotic hand actually has to manipulate anything. That way, the program would be actually usable by any deaf person with a notebook that has a microphone.

Meh - then just have video of someone signing...

[JMZero]

In the time it takes to program the robot to do a bad simulacrum of someone doing each sign, they could have just video'd someone doing all the signs. Then it's more visible to a bigger audience, too.

I'm not saying it's not an interesting project, but it's not a practical solution to the problem.

Great!

[bcnu]

Can I get one of these for the back of my car?

interesting but how practical?

[iamhassi]

Wouldn't it just be easier to have the computer type the words to a screen? I mean if you have the equipment to carry around a robotic hand I'd imagine a LCD screen would be much cheaper and it could probably print more words to a screen than the hand could sign and do so faster.

guess this is more for the sake of doing rather than being practical

It's cool, but...

[nicholaides]

Why a robotic hand? Why not simply text on a screen?

Huh...

[Anonymous Coward]

So when someone tells a deaf person 'f--- you,' does the robotic hand give the deaf person the finger?

now if it could just translate morse code

[Anonymous Coward]

then slashdot readers would really get excited.

Yes, render

[Poromenos1]

So, the steps are these: Recognize language, use translator (of the babelfish kind) to translate to sign language, render signing hand.

Why not just type it out to the screen? :(

Re:Yes, render

[Anonymous Coward]

Because written English is in a different language than the sign language.

Why use a robot?

[sbaker]

Why on earth use something as complex as a robot? What's wrong with using ultra-cheap computer graphics instead? Surely the effect must be identical for the viewer. Anything with that many motors has got to be expensive and unreliable.

Re:Why use a robot?

[aude_sapere]

this is about more than just cost benefit. These small achievements are improving the level of complexity in future AI. Besides, the better they can improve the complexity of andriod hand/arm movement, the faster you'll get that better bionic/cyber hand. Furthermore, if they can get this hand to write, think about the benfits for amputees.

ASL users and bad reading

[SeanDuggan]

I heard a similar statistic in my ASL class. A lot of it, particularly with the older generation, is because deaf people were either put into ASL-only schools who often could not attract the better teaching talent or into speaking schools where they were actively discouraged from signing, often by tying their hands to the desks, and therefore could not properly partake in the learning process. I wouldn't be surprised if some of the bad writing processes have propagated in a manner like ebonics; as a tight community that's a definite minority and surrounded by people who largely can't understand the life they live, they likely have a taboo against too much correction of bad grammar and spelling within their community.

The grammar is, of course, different from English, but many children learn multiple languages growing up. So long as you're exposed to fluent speakers and forced to use the languages, anyone can pick up a language. The only reason very little children are seen to pick up languages easily is a) we excuse a lot of grammar and spelling on their part due to their age and b) lacking a language to fall back on, they have to learn quickly. There's a small segment of tonal languages which are easier to pick up as a child, but that has more to do with perfect (or near-perfect) pitch getting fixed at an early age.

Re:ASL users and bad reading

[AhtirTano]

The grammar is, of course, different from English, but many children learn multiple languages growing up. So long as you're exposed to fluent speakers and forced to use the languages, anyone can pick up a language.

But kids don't spontaneously pick up on writing. What you are asking these deaf children to do is learn ASL (which is basically not written), and the written form of a foreign language. That's like only hearing English (no writing) and only seeing written Chinese (no speaking). That's tricky to learn, and almost impossible to do without extensive training.

More than just words

[SeanDuggan]

In the time it takes to program the robot to do a bad simulacrum of someone doing each sign, they could have just video'd someone doing all the signs. Then it's more visible to a bigger audience, too.
Sign languages tend to be more than just words. the positioning and motion of a sign conveys location and tense of nouns and verbs. It would be like speaking English without being able to conjugate any of the nouns or verbs.

They could, perhaps, dynamically generate pictures of the signs to convey more information, but that means you have to get more information out of the original language (often the much tricker part) and even then, you still have an issue of the signing only being visible from one direction and a fixed distance (which I suspect would be even worse for a field unit which would probably have an LCD screen, which tend to be pretty unviewable from any direction but straight on.)

Oh for heaven's sake...

[viksit]

Don't you guys ever consider the fact that some of these breakthroughs are not built for commercial applications?
Instead of trying to analyze these achievements in the rather constricted mould of "Why not 3D graphics" or "Why not text on a screen", consider the use of this technology in the future - when say, the robots to help disabled people finally get off the assembly lines. By then, this process would've been refined to the point of being able to do an excellent job in communications.
As a researcher in the field of robotics, a lot of work which I do or goes on around me, has definite implications - if not now, atleast in the next decade or so. And don't we owe to ourselves to look at developments such as this just for the *sake of the development itself*?

OK, you're just being silly.

[JMZero]

the positioning and motion of a sign conveys location and tense of nouns and verbs

You're just being silly. All this robot does is take words and map them to gestures. It doesn't convey all this crap you're imagining. And if you're going to do signs which require relation to body parts - as many do - you're going to need a big f'ing robot body to make it visible to lots of people - and you're back to viewing from one direction.

In the very worst case scenario, they could have a 3d representation of a hand on the screen (or on a giant screen, or on ten screens - either way still many times cheaper than a robot hand with this kind of articulation). It could then carry out exactly the same motions as the robot hand - only way, way cheaper, easier and more easily replicable. No maintenance, easier viewing, everything.

Again, this thing is cool - but 100% not practical.

Actually that's an interesting point.

[Spy der Mann]

The device in the Congo movie used a sign language => speech converter. The japanese story is about a speech => sign language converter.

If we consider the gestures as a series of movements produced by predetermined actuators (junctions), they can be quantized and stored in a vector, it's just numerical input, and could be classified as a different kind of speech.

Training the a gesture reader would be equivalent to searching inside a soundwave database (find the closest match, reject if there's any significant difference).

However, speech is more than soundwaves, they have to be interpreted thru various phases, i.e. recognizing the phonemes, generating a soundex, then searched against predetermined words to find a match, AND determining the exact word based on the previous textual context.

In other words, the japanese speech recognition (whether it has a robotic arm or not) is technologically superior to the (ficticious?) gesture reader in the Congo movie.

I bet deaf people hate it

[Locke2005]

From my observation, much of the "color" or entertainment value in signed conversations comes not from the movement of the hands but the expressions on the face etc. combined with the movements. Clearly, this robot is still nowhere near being capable of the same range of expression as a human being. As a simple test, I'd like to see the robot tell a joke and get the same laughs as a proficient human signer telling the same joke...

sigh.. incorrect Japanese

[wolf359alpha]

oh come on, in romanji good morning is spelled "Ohaiyo" NOT "ohayo".. :/

Re: Kanji and *SL

[Blink Tag]

Japanese has only about 2,000 "daily use" kanji, those required to read the newspaper, or graduate from high school. Calling it an "alphabet" as one post did is rather inacurate, as "alphabet" denotes characters that make up parts of sounds, while Japanese is based on syllables (Only 40; or 80 is you include minor variations).

While in Japan I was able to learn some JSL at the hands of some of the top translators in the country (my failure to learn more was my fault, not theirs). I was surprised to find JSL actually shares many of the same gestures as ASL, with minor fingering differences depending one the characters in the word. Other signs in Japanses are actually based on their Kanji representations.

Universal Gesture Less than So

[SeanDuggan]

Hmm...isn't the middle finger signifying "Fuck You" pretty much universal?
I know it's a joke you're making, but actually, I don't believe it is universal although it's rapidly spreading coupled with English. I want to say that most cultures have an "up yours" gesture of some sort involving a hand punching up with some kind of finger gesture, but that's probably my ethnocentrism speaking.

Re:Universal Gesture Less than So

[IngramJames]

I want to say that most cultures have an "up yours" gesture of some sort involving a hand punching up with some kind of finger gesture, but that's probably my ethnocentrism speaking.

Not really ethnocentric. The phallic symbolism is pretty much global. Though one or two cultures prefer to make a round shape, indicating a different set of genitalia are in play.

In rural Greece, showing the palm of the hand (link covers many types of "finger") is the rudest gesture you can make. If you go there, don't wave at anyone. Really.

In the USA, making a "V" sign is always a good sign; meaning victory, or success, or maybe just "two". In the UK, if the knuckles face inwards, it means "victory" or "good". If they face outwards, it's the equivalent of the US middle-digit. Not polite. At all. I remember being momentarily stunned when a guy on a market in NYC indicated a price of two dollars by, apparently, gesturing that I should F-Off. It took my UK-based American friend who was next to me to remind me that "honey, we ain't in Kansas any more." And everyone finds those pictures of Sir Winston Churchill doing it the wrong way around very amusing indeed. It's like a photo of JFK flipping the bird at an entire crowd, who cheer him as he does so.

How smart..?

[daddyrief]

If this robot gives the one-finger salute when cussed at, I'll be getting one.

Picture of the robotic hand

[T.Sawamoto]

Original Article of Yomiuri Shinbun(Japanese)
(Babelfish translation)

It is a very large hand. :-)

I was an interpreter for about 7 years

[bigattichouse]

No way in hell a computer can actually interpret in sign language within the next 10 years - at least. I appreciate the difficulty in doing what they have done, but there are intense subtlies to sign that will never be attainable by a machine without a human-like form. Puffing the cheeks, eyebrows, and other expressions are semantic modifiers that can change "I drove" into "I was driving forever!" or "I had to drive those kids ALL day!".
I remember once telling a story, having given place names in space and subjects at the start, that used no further nouns or "pronouns" for the rest of the story. Imagine writing a paragraph without no further nouns/pronouns after the first sentence.
Heck.. "I think therefore I am" doesn't really translate to ASL, as there are no forms of be. Typically, such a quote would be done in SEE, using english terms... but literally it translates to ("I" is implied) "Think then live" or "Here because think" ... I've always been confounded by trying to translate the intent of the statement. It was easy to interpret in general use, but always gave me a sort of stumble when I tried to interpret it accurately. I think My best was "I (honorific) proof because I (hon.) think"

Where are the photos and videos ?

[Anonymous Coward]

I may be breaking a cliché about Japanese photography but ...
where are the pictures, hundreds of photos and the videos ?
Any luck anyone ? I want proofs !

Not only paralinguistic

[Krischi]

Sorry, now I have to nitpick. People who use facial expressions incorrectly *are* unintelligible, depending on what function the expression has, because facial expressions function in roles other than paralinguistic.

For instance:

- negation: in ASL the manual marker for "NOT" is optional. The associated facial expression is mandatory. If the manual sign is omitted, the facial expression must extend over the clause being negated.

- temporal: eye aperture can indicate whether an action took place in the past. There is at least one documented instance of where an interpreter misrepresented what a client said in court, because the interpreter was not aware of this usage.

- adverbial: a facial expression modifies the meaning of the verb

- topic marker: essential to denote the subject of the sentence if the word order is any other than SVO.

And so on ...

Re:Not only paralinguistic

[cp.tar]

Hm. Seems I had concluded some things from our guest speaker's lecture that are less than correct.

One thing that should excuse me is the fact i know rather less about ASL than about its Croatian counterpart.

Fair enough.