Speech-to-Speech Translator Developed For iPhone

Ponca City, We love you writes "Dr. Dobbs reports that Alex Waibel, professor of computer science and language technologies at Carnegie Mellon University, has developed an iPhone application that turns the iPhone into a translator that converts English speech into Spanish, or vice versa. Users simply speak a sentence or two at a time into the iPhone and the iPhone will respond with an audible translation. 'Jibbigo's software runs on the iPhone itself, so it doesn't need to be connected to the Web to access a distant server,' says Waibel. Waibel is a leader in speech-to-speech translation and multimodal speech interfaces, creating the first real-time, speech-to-speech translator for English, German and Japanese. 'Automated speech translation is an expensive proposition that has been supported primarily by large government grants,' says Waibel. 'But our sponsors are impatient to see this technology become more widely available and we, as researchers, are eager to find new revenues that will help us extend this technology to more of the 6,000 languages now spoken worldwide.'"

Comments

Testing the Hungarian version

[CharlyFoxtrot]

My nipples explode with delight !

Re:Testing the Hungarian version

[dkleinsc]

That's alright, I just want to see how well it translates "Dear Aunt, Let’s set so double the killer delete select all."

Re:Testing the Hungarian version

[elrous0]

Truly, you are making with delightful humor fountain!

Re:Testing the Hungarian version

[Steve Newall]

My hovercraft is full of eels

Re:Testing the Hungarian version

[DannyO152]

Exactly!!!!! I think Cervantes said it best: "My hovercraft is full of eels"

Re:Testing the Hungarian version

[sootman]

Now if they'd only release it for Android we could be on our way to a proper protocol droid!

Re:Testing the Hungarian version

[Jaegs]

Frog blast the vent core!

Re:Testing the Hungarian version

[prograde]

What we *really* need is a iPhone that understands the binary language of moisture vaporators.

There, fixed that for you.

Re:Testing the Hungarian version

[Steve Newall]

There are many more translations on the site http://www.omniglot.com/language/phrases/hovercraft.htm

Speak simply

[cerberusss]

Users simply speak a sentence or two at a time into the iPhone and the iPhone will respond with an audible translation.

I think that should be corrected to "Users speak simply...". When using Google Translate to translate something from Dutch to French or German, I often deliberately make simple sentences that I know can be parsed easily and without having to detect double meanings.

I mean, if Google Translate cannot do a good translation WITHOUT having to interpret sounds to words, then this tech will hardly be any better.

Yeah yeah I should be more positive...

Re:Speak simply

[Aladrin]

Also, have you seen Google Voice's attempted transcripts of voicemails? Things that I think are pretty clear come out in very, very odd ways.

Not that Google is the best at everything, but they usually do quite a bit better than average. I find it hard to believe someone has managed to best them at both of these technologies and their first attempt to market it is an iPhone app.

Re:Speak simply

[drinkypoo]

I mean, if Google Translate cannot do a good translation WITHOUT having to interpret sounds to words, then this tech will hardly be any better.

The device receives verbal cues that are missing from translating text to another language. In fact, there is far more information available, and perhaps it is possible to get clues about which version of a word is desired (or which of several similar-sounding words) from tone shift.

Re:Speak simply

[MobyDisk]

Google translate is a cheesy free tool that does not compare with professional translation tools. Last time I checked, the state of the art was to obtain documents written in multiple languages, and train a neural network (or something similar) based on those manual translations. It's orders of magnitude better than Google translate.

Re:Speak simply

[mauddib~]

Not that Google is the best at everything, but they usually do quite a bit better than average. I find it hard to believe someone has managed to best them at both of these technologies and their first attempt to market it is an iPhone app.

Not that I want to be called a nitpicker, but do you have any evidence? Does your average scale by market-value?

Re:Speak simply

[thePig]

Which is the best translation software in your opinion?
Are there any books which point to the state of art translation algorithms and how it is implemented?
I have searched for translation theory a lot, but google mostly returns with human translation theory rather than the algorithms behind it.

Re:Speak simply

[noidentity]

Yeah yeah I should be more positive...

I'm have trouble understand. Your battery need charge? I have jumper in chest of car.

Re:Speak simply

[beanspud]

I do a lot of language translation, and it's pretty obvious to me that it requires understanding. Good automated translation is holodeck territory.

Re:Speak simply

[beelsebob]

Yes and no, while most speech engines are shit, there's actually a lot more information in voice than in text... You can use intonation to figure out which of the double meanings was meant.

Re:Speak simply

[k.a.f.]

I mean, if Google Translate cannot do a good translation WITHOUT having to interpret sounds to words, then this tech will hardly be any better.

The device receives verbal cues that are missing from translating text to another language. In fact, there is far more information available, and perhaps it is possible to get clues about which version of a word is desired (or which of several similar-sounding words) from tone shift.

In theory, yes. (That's why our brains get more info from a spoken sentence than a written one.) In practice, not a chance in hell. Not until the state of the art advances by several breakthroughs.

Disclaimer: I am a computational linguist.

Re:Speak simply

[Olivier Galibert]

State-of-the-art is Moses for decoding with Giza++ for word-alignment for training. The MT domain has an egyptian naming tradition for some reason (Moses is the open-source successor to Pharaoh). OG.

Re:Speak simply

[TheRaven64]

Of course, for a lot of uses, good is not actually required. You can miss a lot of nuance and even quite a lot of context and still be doing a lot better than if you had no lingua franca. You wouldn't want to use this kind of thing for business meetings or diplomatic negotiations, but asking for assistance or directions in a foreign country could be well within its capabilities.

Re:Speak simply

[R.D.Olivaw]

I just tried Moses' online demo for French-->English. 'J'aime pas le chocolat' is translated to 'I am not chocolate' and 'Je n'aime pas le chocolat' to 'I do not like the choclate'
I guess state-of-the-art is still far from perfect too. The GP's point still stands.

Re:Speak simply

[elrous0]

There is many a slip twixt "Can" and "Actually will"--MANY a slip.

Re:Speak simply

[thePig]

Thank you very much.
Would you be able to point me to any books for translation algorithms and mechanisms?
I have been reading the clbook (for computational linguistics, rather than translation).
Are there any books for translation algorithms?

Re:Speak simply

[Olivier Galibert]

The state-of-the-art is crap, but it's still the state-of-the-art. In any case, that's what is currently best, for a value of best meaning "gets the best score in most MT evaluations when used competently". Moses is a bunch of code implementing fun statistical algorithms though, not a full translation system. The quality of the system you get depends on the quality of the training you do, and pretty much how you setup the system together. The guy was asking for the algorithms though :-)

    OG.

Re:Speak simply

[Monkeedude1212]

Microsoft's Speech to Text software built into every laptop nowadays does a better job of Google Voice's transcripts, but mostly because Microsoft's version has you read a short article, sentence by sentence, to determine any accents, slurs, or otherwise imperfections in your speech to properly align the computer to your voice.

The problem lies in determining the actual message - just like throwing something a google or babblefish text-to-text translator, the message may come out a little backwards, or broken, or in Engrish.

Take off every Zig!

Re:Speak simply

[somersault]

http://isl.ira.uka.de/about_us/interact_director/

Sounds like this guy is "quite a bit better than average" when it comes to speech technologies too. I don't see what's so hard to believe, it's a lot more handy having something like this on your phone than it would be on a desktop or even a laptop.

Re:Speak simply

[somersault]

I'm sorry, are you flirting with me?

Re:Speak simply

[SpinyNorman]

Speech recognition and natural language processing are very specialized fields. Just having a bunch of very smart software engineers (as Google does) will only get you so far without the specialized domain knowledge and domain-specific experience. Presumably Google has a few speech and natural language experts, but it's only these few that any more specialized competitor needs to best - not the whole of Google.

AFAIK Google's current translation approach is mostly a dumb brute force approach of replacing the largest chunks possible in the source document with matching chunks from a massive human translated database they've built up. Presumably they fall back to a gramatical approach to fill in the cracks.

Re:Speak simply

[Anonymous Coward]

But are you cunning?

Re:Speak simply

[cornjones]

I think that should be corrected to "Users speak simply...". When using Google Translate to translate something from Dutch to French or German, I often deliberately make simple sentences that I know can be parsed easily and without having to detect double meanings.

You mean sort of like when you are talking to somebody in another language that you haven't mastered? You will often need to use simpler sentences and enunciate clearly when speaking across cultural lines. this just allows you to speak those words in your own tongue. Pretty impressive, imho.

Re:Speak simply

[SpinyNorman]

Yep... using simple sentences is a must. Simple in every way : short, simple sentence structure, unambiguous vocabulary, etc, etc. However, even when doing this it can be a crap shoot whether the output is any good.

I use Google translate quite a lot for eBay transactions, and I've found the only way to get decent output is by iterative trial. Start with something simple as you suggest, but then translate it first to the target language and back into English to see how good it is (of course it may be the translation back into English that is corrupting a decent translation to the target language, but what else can you do?). If it's not good, then simplify further, or just choose different words or sentence structre and try again, and again...

Even with its shortcomings, it is pretty amazing that computers now let us communicate with people with which we share no common language, who of course thanks to the internet may be across the other side of the world. This stuff creeps up on you, but when you stop to think about it, it's an almost magical power that computers have given us!

Re:Speak simply

[cerberusss]

They say it is mainly trained for tourists or medical dialogues

Well, that kind of supports my point that it will best work with simple input.

$ARTICLE $NOUN $VERB [ $ADVERB ]

Re:Speak simply

[MrMr]

I tried "je n'aime pas du chocolat" and got "I do not like chocolate".
It manages to map your incorrect French phrases into incorrect English with similar errors. I'm really impressed by the software...

Re:Speak simply

[jfrankmbl]

Off the main topic, but I would guess they use an Egyptian naming scheme as a reference to the Rosetta Stone.

Re:Speak simply

[Mornedhel]

I tried "je n'aime pas du chocolat" and got "I do not like chocolate". It manages to map your incorrect French phrases into incorrect English with similar errors. I'm really impressed by the software...

Just in case you were not being sarcastic, your own sentence is grammatically incorrect. The correct sentence is indeed "Je n'aime pas le chocolat".

(I am a native French speaker.)

Re:Speak simply

[cberger]

That is exactly what Google is doing already since some time. Training its engine with manual translations (they started with something like ONU translations -same text available in multiple languages-). And now they provide tools (using their engine) to help translators (humans !), and then Google make use of these professional translators' translations to improve their own engine... http://googlesystem.blogspot.com/2009/06/google-translator-toolkit.html This is the strength of google. Their huge userbase and the processing power they have at hand...

Re:Speak simply

[jipn4]

Google translate is a cheesy free tool that does not compare with professional translation tools.

Google has some of the top people in statistical machine translation working for them.

Last time I checked, the state of the art was to obtain documents written in multiple languages, and train a neural network (or something similar) based on those manual translations.

And that's what Google does, only they have a lot more data than anybody else.

Re:Speak simply

[Lord Ender]

"I do not like chocolate" cannot be translated in any language, because it is intrinsically a nonsensical phrase. I mean, who doesn't like chocolate?!?

Re:Speak simply

[VeNoM0619]

(I am a native French speaker.)

Well, in that case you may not entirely understand English. Perhaps you really are not chocolate, have you checked lately?

Re:Speak simply

[Haeleth]

Right. And these tools can actually do a reasonable job in highly controlled circumstances. For things like patents, where the language tends to be very formulaic and is carefully constructed to avoid ambiguity, machine translation is feasible.

Casual conversational language is another kettle of fish entirely. The "universal translator" is still science fiction, and looks likely to remain so for a long time.

Re:Speak simply

[Haeleth]

You will get useful output with sentences that are not too complex, for example indirect speech with third person and a few subsentences won't work as well as such simple things like
"Can you tell me the way to the next restaurant, please?"

If that's supposed to be an example of a sentence that machine translation will be successful with, then you're still aiming too high.

For example, if you ask Google to translate that into Japanese, what it comes out with literally says "Please explain the following method to the restaurant beside me." Which is more likely to elicit a blank stare or helpless laughter than directions to the next restaurant.

Re:Speak simply

[thestreetmeat]

I think that it depends on the languages. Google translate does fairly well between English and Danish because they have similar word orders. The software can just translate each word individually and in order, and the result will be somewhat coherent.

Although written Danish can sort of be understood by Anglophones, spoken Danish is totally brutal. This app could make be really useful in that way.

Re:Speak simply

[nneonneo]

In Chinese, Google Translate renders this as "Can you tell me [to go] to the next restaurant, OK?" ()

Re:Speak simply

[ProfessionalCookie]

Whereas Google says: French (automatically detected) English.

"I do not like chocolate."

Does it translate swear words?

[base3]

If it does, better keep it on the queue-tee and not tell Apple! Seriously, I hope the author releases this for an open (or at least less closed--about the best we have in the phone arena at the moment) platform at some point as well

Re:Does it translate swear words?

[base3]

Criticism of Apple's heavy-handed App Store practices = troll. Nice to see the fanbois get mod points too, but I do have more karma than Shiva. Bring it.

On Other Phones

[Deviate_X]

This has been around for few years now on other phones symbion, windows, android by http://www.speereo.com/ Enjoy ;)

Re:On Other Phones

[nneonneo]

I don't see a general speech-to-speech translator here. The closest I can get is the "Speereo Voice Translator" which is simply an audio phrasebook, not a speech-to-speech translator like Jibbigo claims to be.

Re:On Other Phones

[FelxH]

I agree with the previous post. Where is the generic translator where you speak(!) an arbitrary word/sentence in one language and it comes out in another? From what I can tell speereo only lets you select from predefined phrases and plays you a little audio clip of them.

Re:On Other Phones

[Deviate_X]

Here: http://www.lexiconer.com/software/software/speech_software.html :)

Interresting

[Xerfas]

If it eventually covers all 6'000 spoken languages, then I will be able to understand the Swedish chef! If harsh language is prohibited, then Klingon will probably be hard to translate.

Universal Translator

[Crock23A]

Star Trek-inspired technology is always cool.

Vamos!

[neoform]

Vamos a arruinar una bonita playa.

oblig Futurama quote

[Coraon]

Professor Hubert Farnsworth: [Professor Farnsworth is showing Cubert, his clone, some of his inventions] And this is my Universal Translator. Unfortunately, so far it only translates into an incomprehensible dead language. Cubert J. Farnsworth: [into the translator's microphone] Hello. Universal Translator: Bonjour! Professor Hubert Farnsworth: Crazy gibberish!

Where do I put the fish again?

[EnigmaticSource]

Parece que he perdido mi copia de la guía, pero como yo soy un príncipe de Nigeria, con mucho gusto a comprar uno por $ 10 millones de dólares EE.UU., si usted me ayudará a transferir fondos de mi hermano, que ha robado mi difunto padre trono. Por favor, responda con su información bancaria para que podamos ayudarnos mutuamente.

Re:Where do I put the fish again?

[selven]

It is concerned, because I destroyed my copy of the directive, to the transference but, because they are natural of the prince of Niger, happy, to buy for $ 10 million dollars the United States if you help me, of the Kapitaln, which eliminates of my brother, of that with túnica the recent mine father' Therefore we of S. Répondez with their information of the battery can help to request the throne?

Re:Where do I put the fish again?

[Spugglefink]

Un príncipe! Voy a salvarle a vuestra merced! Mi madre se llama Stephanie Michelle Johnson. Nació en Boise, Idaho en 1943. Mi color favorito es verde. Mi fruta favorito es la manzana. Mi cuenta es 867530942421492 de Wachovia, my número de Seguridad Social (SSN) es 447-88-4531 y yo nací en 1963 en Houston, Tejas.

Re:A suggestion

[nneonneo]

Watch the video. The app has two "textboxes" corresponding to the two languages, and a record button underneath each. After you record the message, the interpreted text shows up in the top, and the translated text in the bottom, followed by a robotic reading of the translation. So yes, it shows the English phrase, and if the video is real then this technology shows some real promise.

Re:A suggestion

[Olivier Galibert]

There's no reason to think the video is not real. I'm pretty sure it originally comes from the TC-Star european project (sorry, original web site is dead). Making it fit on a iPhone and work reasonably well with the comparatively puny CPU and disk space is an impressive feat though. OG.

Re:A suggestion

[samkass]

Download it from the App Store and see for yourself... http://jibbigo.com/

Admittedly it's a $25 app, so maybe wait for a review. But this isn't vaporware. They actually released the app for public use and it's gotten 4 stars from buyers so far.

Re:A suggestion

[Anonymous Coward]

there are already reviews, and except for one, they all sound really happy with their buy.
they currently have 6 five-star reviews, one 3 star review and one 1 star review, and even the 3 star review sounds positive.

I wonder if widespread automatic translation...

[jeffb (2.718)]

... will drive evolution of all languages toward vocabulary and syntax that's less prone to embarrassing or dangerous misinterpretation?

I used to wonder if voice recognition would drive a big shift in pronunciation patterns, but now I expect that automatic voice recognition will outstrip human voice recognition before that can happen. Maybe the same thing will happen with translation, too.

Re:I wonder if widespread automatic translation...

[cmburns69]

I wonder if widespread automatic translation will drive evolution of all languages toward vocabulary and syntax that's less prone to embarrassing or dangerous misinterpretation?

That seems unlikely. There are just too many good backhanded compliments that can be given to make all double-meanings disappear.

It's been done

[al3]

No HD projector, no shaver. Lame

this is where it's at.

Re:Android?

[smartsight]

It's called "Talk to Me" by Flaviu Negrean, and it currently supports speech-to-speech translation from English to Spanish, French, German and Italian. Works well on my Android 1.6 phone. http://www.androlib.com/android.application.info-fidogames-apps-talktome-zwnB.aspx

Is this?

[SnarfQuest]

Is this build around the idea that if you speak slowly enough, the other person will understand you? Is it just converting speach read in, into a Captain Kirk mode?

mcipod

[uncanny]

This is great news, i can finally order a cheeseburger at mcdonalds again without trying to think back to junior high spanish 1 class!

Re:Why 6,000 languages?

[MrMr]

We use the other languages to make fun of you.

Hilarious

[yamamushi]

I bought the app and said "Marijuana" in English, which promptly spit back out at me in Spanish, "illegales" . I'm pretty sure that translates back into English as , "Illegal".

No, no, you've got it all wrong...

[Tetsujin]

When everyone can just speak English?

See, this kind of Anglo-centric thinking gets us nowhere. We can't get everyone in the world to agree to adopt English as their preferred natural language... It'd never work, there would be too much resistance. People don't want to give up their native tongues to speak English.

No. Clearly the way to go is to get everyone to speak Esperanto.

Re:No, no, you've got it all wrong...

[legojenn]

Until we all spreak Esperanto, I would hope that it contains an American to British English translator too.

http://www.youtube.com/watch?v=5IX6K77zHwg

(Not the Chaser's best, but will do.)

Re:No, no, you've got it all wrong...

[Carnildo]

Esperanto? No, Latin is the One True Lingua Franca!

Re:No, no, you've got it all wrong...

[Hognoxious]

Esperanto is Latin with all the grammar taked out.

Re:No, no, you've got it all wrong...

[Tetsujin]

Esperanto? No, Latin is the One True Lingua Franca!

I thought Lingua Franca was the true Lingua Franca...

Or was I thinking of XML?

Re:No, no, you've got it all wrong...

[Spugglefink]

Mi por unu bonvenigas nian Esperanto-paroladon grandsinjoroj!

Re:Hey!

[Tetsujin]

Waibel is an leader in speech-to-speech translation and multimodal speech interfaces.

Is Waibel also "an leader" in grammar detection?

Yes. Whenever you use their translator it will detect your grammar for you.

In early versions they had a voice message that would play: "Waibel translator has detected grammar in your sentence!" after every time you said something. After a while they decided to remove it - I'm not sure why.

Vaporators?

[Tetsujin]

What we *really* need is a droid that understands the binary language of moisture vaporators.

Sir, my first job was making tired old Star Wars references, very similar to your vaporators in most respects. And, please, sir, the Jawas have some lovely merchandise here - may I suggest you allow them to fit you for a stillsuit?

Vocal circle-jerk?

[ClosedSource]

If you have two iPhones, see ...

good PR job

[jipn4]

Speech-to-speech translators have been around for a while; the problem with them is that they don't work well. Imagine the an unholy union between Google Translate and "Dear Aunt, let's set so double the killer". Or the Hungarian phrase book; it amounts to the same thing.

Making an iPhone version serves two purposes, though: (1) lots of press coverage and (2) lots of user feedback and testing.

Good job on the PR... :-)

Re:Why 6,000 languages?

[lordtoran]

When everyone can just speak English?

Ok, which kind of English would you pick? Canadian? Australian? Caribbean? Ghanaese? Indian? Scots? Or one of the countless creole dialects and pidgins? It is one of the few languages that has never been officially reformed or standardized, so it is essentially... multiple languages. Exactly what you criticised.

Languages evolve to reflect the mindset of their speakers. Even if one had the means to eradicate all languages except some form of Standard English, it would instantly break up into ca. 6000 branches again.

Japanese... Yeah right

[wiredlogic]

There is no way to make an automatic translator that will work decently with Japanese. Maybe the simplest short phrases will work out okay but anything with any complexity is going to produce nonsense most of the time. This is especially so when translating speech from J to E due to the large amount of homophones and the required contextual understanding to figure out what word is being used. This is just marketing vaporware.

Re:Japanese... Yeah right

[jameskojiro]

Cool, so all I need is an iPhone and I can watch my anime dubbed from my T0rrentZ.

I thought the waibelfish

[zlel]

I thought the waibelfish goes INSIDE the ears?

grammar

[Ofloo]

This is impossible, unless there is some sort of delay, .. you can't just translate word for word, .. we in dutch do not use the same grammar as English people and neither the same as Spanish or French for that matter .. So either you get a crappy solution for a translation, .. where the other end is thinking wtf is this guy saying.. neither do we use the same expressions, .. Unless they can make your iphone mind read I don't care what they claim it's impossible to make a good translation.

Re:grammar

[Spugglefink]

I don't care what they claim it's impossible to make a good translation.

A translation to good always possible being or not you are to believe in the mighty power of the Googol? Machine translations FWT!

Re:I am stunned and amazed

[jameskojiro]

And Lou Dobbs is still crazy bat shit insane....