Slashdot Mirror
Google's AI Translation Tool Creates Its Own Secret Language (techcrunch.com)
After a little over a month of learning more languages to translate beyond Spanish, Google's recently announced Neural Machine Translation system has used deep learning to develop its own internal language. TechCrunch reports: GNMT's creators were curious about something. If you teach the translation system to translate English to Korean and vice versa, and also English to Japanese and vice versa... could it translate Korean to Japanese, without resorting to English as a bridge between them? They made this helpful gif to illustrate the idea of what they call "zero-shot translation" (it's the orange one). As it turns out -- yes! It produces "reasonable" translations between two languages that it has not explicitly linked in any way. Remember, no English allowed. But this raised a second question. If the computer is able to make connections between concepts and words that have not been formally linked... does that mean that the computer has formed a concept of shared meaning for those words, meaning at a deeper level than simply that one word or phrase is the equivalent of another? In other words, has the computer developed its own internal language to represent the concepts it uses to translate between other languages? Based on how various sentences are related to one another in the memory space of the neural network, Google's language and AI boffins think that it has. The paper describing the researchers' work (primarily on efficient multi-language translation but touching on the mysterious interlingua) can be read at Arxiv.
Tell that to the Korean translators.
It little behooves the best of us to comment on the rest of us.
Learning internal representations are what neural networks are all about.
Conventional wisdom is that each successive layer in a feed-forward network detects higher-level features based on the lower-level features detected by the previous layer. That's why deep networks can do their magic.
Sheesh, evil *and* a jerk. -- Jade
As a translator, these last couple of years have been grim. For things like marketing efforts and full-length books, where a very polished translation is desired from the get-go, there's still work out there for human translators. However, the bread and butter of a lot of translators was things like multinationals' internal documentation, or catalogues that consist of lots of simple listings and not much actual prose, where polish and shine isn't as vital. Companies are increasingly running their material through Google Translate, and then hiring a native speaker of the target language to proofread and correct that clunky output a vastly lower price than human translation.
It has often been said here on Slashdot that the development of self-driving trucks will put 3 million people out of work in the US alone. But translation is a field where, very quietly, automation is hitting the white-collar sector hard.
On the other hand TFS is basically gibberish.
There is no 'secret' language, or even deeper understanding. The notion that they aren't using english as a bridge language just means that they aren't translating Japanese-to-English-to-Korean.
But for example... if I train you that cat = gato in italian, and that cat = chat in french. And then ask you to spit out the french if give you the "gato" that's not exactly magic. It looks up 'gato' in italian and sees a reference to "chat". And it can do this without explicitly looking up the english "cat" and then feeding "cat" back in to look up the french.
English is still the bridge language that was used to train it.
Now this neural network is a lot more complex because lanaguage is a lot more complex than simple word substitutions but the neural network is still basically encoding that chat (french) = cat (english); and cat (english) = gato (italian) and the way this information is mapped into the neural network -- that it can now retrieve equivalencies between french and italian without being EXPLICITLY trained on them.
Its neat... but whoo... the neural network structure inherently models the transitive property of equivalence. That's kind of the whole point of the thing (to effectively build a weighted mapping of language equivalences) so it would almost be more surprising if it couldn't do some reasonable transalations between languages it wasn't explicitly trained on -- because english is the bridge between them in how the knowledge was built even if they aren't explicitly using english now.
I mean... train it on english to french, and train it on japanese to korean, and see if it can go from korean to english. It won't. Because it won't have ANYTHING to bridging those two sets of knowledge.
That would be nice if translating sentences was the same as looking up words in a dictionary. It's not. So pointing out that there are words that have correspondences is meaningless.
Languages have a fuzzy haze of concepts and ways to parse them. I could say "I feel sick" or "I am sick" in English and they're not the same, the latter expresses certainty. But in Icelandic you'd generally say "Ég er lasin(n)" or "Ég er veik(ur)" - aka, "I am sick" - for both of them. Not "I feel sick". You *can* say "I feel as if I'm sick", but that gives a sort of connotation as if you're doubting yourself, more than "I feel sick" does in English. The latter case is "Mér líður eins og ég sé veik(ur)", which is literally "Me (dative, not nominative) feels same and I would-be(pres.) sick (depends on gender)" There's an awful lot going on in there that a word-for-word translation just doesn't catch. Even if you catch phrases, like "eins og" -> "like" rather than literally "same and", you still don't have anything close to a one-to-one mapping.
And here we're talking two Germanic languages.
A neural net that can handle translations in a way where the results aren't terrible must have a concept of the fuzziness, the interplay of how different concepts are presented in different languages. And indeed, that's what the graphic that they show seems to suggest, where you have these branching clusters with varying pathways that dart between them for different languages. Perhaps calling that internal representation a "secret language" is a stretch, but it's most definitely nothing like having "English as a bridge language".
Wingus, Dingus! Listen up!
To follow up a bit further on that, there are some concepts that take whole sentences, paragraphs or more to describe. Back in the day I had a Japanese song, with English lyrics... except that one word in the middle remained untranslated ("Our satori are just floating in the core"). I asked a professor about what it means and it ended up as a whole lecture on Buddhist concepts and Japanese relations between the true self and the self that one presents to others in different contexts.
In Icelandic for me it often comes up in terms of geological terms. For example, someone will ask, "What does Reykjavík" mean, and I usually just give a quick "Smoking Cove" or "Smoking Bay" or something like that. But that's not really right, English doesn't really have a word that describes a "vík". A "vík" is where the coastline "víkur". To víkja is to give way, like if someone's tailgating you on the road and you pull off to the side to let them past. So where the coastline "víkur" - on a certain scale, at least - that's a "vík". It's often where a river empties out, but not all river mouths end in víkur, and not all víkur are river mouths, some are more like coves or small bays. But you wouldn't mistake a "vík" for a "fjörður" or anything like that. We divide "field" up into "akur", "tún", "völlur", maybe even more depending on the concept (melur maybe, if it's rocky? garður even in some contexts? Lots of possibilities). So, I mean, we can just pick a random word, but you'll lose context - and when you translate back you can come up with something that's just wrong.
Even the "smoking" part isn't quite right, as most people in English hear smoke and think of burning things, but "reykur" in Icelandic place names is often used to denote geothermal steam - even though it technically means smoke.
My favorite mismatched concept has to be the verb "nenna", generally used in the negative (e.g. "Ég nenni ekki!"). In the negative it's sort of like "can't be bothered to do X", "not in the mood to do X", "don't waaaanna do X", "it's not worth my time/effort to do X", or just plain "Meh". A lazy translation is often "can't be bothered", but it sounds weird as English speakers don't usually talk like that. I've noticed some people who learn Icelandic end up taking that verb back into English, or even noun-ifying it ("I don't have the nenn to do that right now...")
Wingus, Dingus! Listen up!
Paging Wittgenstein!
brwski
"Because without beer, things do not seem to go as well''
It's quite likely that there is a shared representation. That's what neural nets do: if you feed train them on similar input/output pairs, they will develop common activation patterns. They would do so regardless of the language, since they don't know which language is being presented.
Humans, OTOH, do know that they're being presented with a different language, and demonstrably do something called "code switching": a cognitive effort to use another language resource. Therefore, in the human brain, the shared connection is supposed to lie outside the language faculty (there are other reasons to assume it, too).
On this side of the pond "can't be bothered" is in common usage. A lot of the time the colloquialism "can't be arsed" is used to mean the same thing.
"Wait. Something's happening. It's opening up! My God, it's full of apricots!"