Slashdot Mirror
Google's AI Translation Tool Creates Its Own Secret Language (techcrunch.com)
After a little over a month of learning more languages to translate beyond Spanish, Google's recently announced Neural Machine Translation system has used deep learning to develop its own internal language. TechCrunch reports: GNMT's creators were curious about something. If you teach the translation system to translate English to Korean and vice versa, and also English to Japanese and vice versa... could it translate Korean to Japanese, without resorting to English as a bridge between them? They made this helpful gif to illustrate the idea of what they call "zero-shot translation" (it's the orange one). As it turns out -- yes! It produces "reasonable" translations between two languages that it has not explicitly linked in any way. Remember, no English allowed. But this raised a second question. If the computer is able to make connections between concepts and words that have not been formally linked... does that mean that the computer has formed a concept of shared meaning for those words, meaning at a deeper level than simply that one word or phrase is the equivalent of another? In other words, has the computer developed its own internal language to represent the concepts it uses to translate between other languages? Based on how various sentences are related to one another in the memory space of the neural network, Google's language and AI boffins think that it has. The paper describing the researchers' work (primarily on efficient multi-language translation but touching on the mysterious interlingua) can be read at Arxiv.
if it's so secret, then no comms
The translation system is using English as a "baseline" language. It knows how to translate both Korean and Japanese to/from English. So it's implicitly using English to link the two languages.
It isn't magic. A human would not be able to translate Korean to Japanese without some intermediate language either.
Learning internal representations are what neural networks are all about.
Conventional wisdom is that each successive layer in a feed-forward network detects higher-level features based on the lower-level features detected by the previous layer. That's why deep networks can do their magic.
Sheesh, evil *and* a jerk. -- Jade
Colossus and Guardian had their own secret language too. What could possible go wrong?
As a translator, these last couple of years have been grim. For things like marketing efforts and full-length books, where a very polished translation is desired from the get-go, there's still work out there for human translators. However, the bread and butter of a lot of translators was things like multinationals' internal documentation, or catalogues that consist of lots of simple listings and not much actual prose, where polish and shine isn't as vital. Companies are increasingly running their material through Google Translate, and then hiring a native speaker of the target language to proofread and correct that clunky output a vastly lower price than human translation.
It has often been said here on Slashdot that the development of self-driving trucks will put 3 million people out of work in the US alone. But translation is a field where, very quietly, automation is hitting the white-collar sector hard.
There has been a long-running philosophical debate about this - do we all think in the same language (a language of thought), which goes through translators into other languages for us to read, speak, and think in? The outcome described, the "secret language", would steer us toward affirming that hypothesis.
On the other hand, maybe not: for example, when we catch a ball, out brain is solving several multiple-order polynomial equations to make the hand catch the ball. The question is "does the brain actually do that?" or do it just "do", and make the catch. You could argue both ways.
So what? The internal language IS still an intermediate language. It's just substituting MEANING for English as the central link. Whether MEANINGS are stored as English text or a bunch of bits or a collage of relevant images doesn't matter.
Instead of...
lookupKoreanFromEnglishBaseline( getEnglishBaseline(some_japanese_phrase) ) ...it would be...
lookupKoreanFromMeaning( getMeaning(some_japanese_phrase) )
Who the hell is impressed by that? Furthermore, direct translations would square the necessary data storage, which would be retarded. Even for google's data centers. If this is what they are doing, THEN THEY ARE RETARDED.
That's how Turing tests (duck tests) work. If you can carry on a conversation with it and a human and you can't tell which is which...then you have AI.
Language encodes thought. From 1984's newspeak to fifty words (or whatever) for different kinds of snow, language defines how (if?) the language-user "thinks".
I find this development both exciting and frightening. The singularity will be . Don't know if this is it, but when it gets here it will be.
"Reality is that which, when you stop believing in it, doesn't go away." - Philip K. Dick
Paging Wittgenstein!
brwski
"Because without beer, things do not seem to go as well''
Bring me Forbin!
More like some rapid sorting or look up to give a fast gui flow with modern gpu, cpu, ram designs.
At some point it gets words like dog or glasses hardcoded in and all new languages get filled in when needed for advertising, mil/gov, a product, service or paying client.
Every translation will then feel fast and responsive to the user even with very different teams get tasked to add a new language years later.
Great for a mil or gov or NGO paying for slag, jargon, a very regional dialect very quickly to win hearts and minds.
Domestic spying is now "Benign Information Gathering"
It's quite likely that there is a shared representation. That's what neural nets do: if you feed train them on similar input/output pairs, they will develop common activation patterns. They would do so regardless of the language, since they don't know which language is being presented.
Humans, OTOH, do know that they're being presented with a different language, and demonstrably do something called "code switching": a cognitive effort to use another language resource. Therefore, in the human brain, the shared connection is supposed to lie outside the language faculty (there are other reasons to assume it, too).
I'm old, spent 40 years sweating over a hot computer. That said, this is worrying. As other commentators on this thread have said, this is predictable and useful in many ways. In the 1980s I worked with SYSTRAN: https://en.wikipedia.org/wiki/... which worked (works?) on pairs and the EU Commission, which has a huge translation burden was looking for pivots, even then.
However, consider this, a neural net that takes care of business in an oil refinery (or worse, nuclear installation) 'decides' that it can knock up a much more efficient control language. That's rational and perhaps beneficial, but, at that stage, there's also a creeping loss of control/comprehension in a system that controls actuators: https://en.wikipedia.org/wiki/.... Also from 1983, a much cited paper that is also is debated in the fly-by-wire community (pdf alert!): https://www.ise.ncsu.edu/nsf_i...
So, long story short, I'm not at all sure about surrendering control, somewhat unconsciously as a by-product of optimisation, itself (perhaps) a by-product of economics and 'cost effectiveness'. Also, when we deal with neural nets, we deal with the sub-symbolic, a system that is not going to 'explain', just say I did that because of 42. Don't mistake me, I'm not a Luddite, I love a good computer and have plenty at home, but this 'gives pause'.
On y va, qui mal y pense!
These creators didn't know, but modern Korean language heavily borrowed from Japanese language on every genre you can think of.
In fact, it is possible for a Japanese person to guess what a Korean news article is about by writing it with a mix of Chinese characters and Hangle (Korean alphabets) since so many nouns and verbs are in Japanese!
In the example they used, "stratosphere" and "altitude" are words that Japanese created in late 19th/early 20th century as translations for English (maybe German) words.
These words were then imported by Korean from Japanese language in early 20th century.
It is simply logical that GNMT will get a decent translation because it is matching Japanese words that Korean language imported with original Japanese words.
They should not have used Japanese and Korean to test since it is too easy for GNMT to guess correctly.
> Furthermore, direct translations would square the necessary data storage, which would be retarded. Even for google's data centers. If this is what they are doing, THEN THEY ARE RETARDED.
Don't be so harsh, it might have been justified by better accuracy. It hasn't always been better to use an inter-lingua in multi-language translation.
Google can't even effectively translate Japanese to English. To claim even that produces 'reasonable' results is an absolute act of denial. One can often barely decipher the meaning from the few parts it gets right, but it neither literally translates correctly nor does it provide a more relatable paraphrase.
Bull shit.
Source: Living in Japan and using Google translate daily.
"If link not restored action will be taken."
The Google authors omitted to mention that Pedro Carolino created something far more stylish in 1853.
https://en.wikipedia.org/wiki/...
Carolino's translation of "to wait patiently for someone to open a door" as "to craunch the marmoset" isn't going to be bettered by these young upstarts.
USB, USB, USB!
I wonder a bunch of things. It looks like the internal representation of language the GNMT uses (if there is one) could come in handy, if we could just figure out how to use it without understanding it.
A 2D Fourier transform of anything non-trivial is incomprehensible, but they can be used to reconstruct the original, as-is or with some tweaking. Tweaking of the FT, tweaking of the reconstructing process.
Perhaps something somewhat analogous could be done with these internal language representations. What, I surely don't know.
Maybe humans can reverse-engineer it by treating it as a cryptography problem. Like with known plaintext, and the ability to create new plaintext-cyphertext pairs as needed.
What's the difference in that internal representation between "Spike is a cat" and "Spike is a dog", and how does that differ from the difference between "Mike is a cat" and "Mike is a dog"? Throw in "Fluffy is a wolverine" and "Fluffy is a cat", and see if you can now synthesize "Spike is a wolverine".
Other ideas, anyone?
There's no time like the present. Well, the past used to be.
The "internal language" reminds me of some of the attributes of the "focused" people in Vernor Vinge's A Deepness in the Sky. They were, after all, (spoiler) human automation.