Google Faces Plagiarism Questions Over Chinese Software

yaohua2000 writes "Google's laboratory in China has launched its first product, a Pinyin Input Method Editor. The software allows the romanized characters to be translated to more traditional Chinese symbols , via entering on a QWERTY keyboard. Users soon discovered that the data Google used for the product was unusually similar to the data used by a Chinese rival, Sogou. Google has evaded the question about software similarities, reports PC World. 'The similarities, which included an error involving the name of a celebrity, were noted on a Google Labs discussion board about its Pinyin IME. Users noted that entering the Pinyin pinggong into the Google IME incorrectly produced the name of Feng Gong, an actor and comedian.'"

Comments

Google Should Defend Themselves the OpenBSD Way

[Anonymous Coward]

Blame the Sogou authors, and call them inhuman. Also say it isn't plagarism because it's beta.

I'm a stupid American, so...

[Anonymous Coward]

Let me be the first to say... WHAT?

not saying it's the case

[creativeHavoc]

while i am not insisting that it is the case, it seems like it could easily be the same logic flaw. Different algorithms and code can produce the same mistake if you are using the same mis guided logic behind the problem. Thats why you see the same bugs in students' code in university, even when worked on separatly during a lab.

This is big news in China

[Anonymous Coward]

Unfortunately, since the IME is only used by Chinese speakers, most reports and discussions about this are in Chinese as well. For example, Sina has published an announcement (in Chinese) from Google admitting that they indeed "used data from non-Google sources" during the testing stage.

There were actually much more evidence than the PC World article mentioned, the most convincing being that Google IME included many names of the developers of Sogou IME.

Although according to the other users (I don't use Google Pinyin myself now, or Windows for that matter), the error has been fixed - and those developer names has been removed - in the most recent version of Google IME (1.0.17.0).

Ming

Re:This is big news in China

[epine]

I was involved in a very early effort to develop a pinyin based IME. Think 4.77Mhz. It worked quite well, in fact. Good dictionaries are hard to come by. Back then, not easy at all. In fact, we liberated data quite freely from any resource we could obtain. I made it a policy that each dictionary term had to come from at least two independent sources (sources unlikely to have stolen from each other). The singletons had to be manually reviewed by a qualified linguist. It's like that old saying: stealing from one source is plagiarism, stealing from multiple sources is research.

Eventually I found an extremely effective compression method (the IME portion of our system fit into 128K including dictionary) using a hash table approach. Collisions in the hash table generated spurious terms. The spurious terms that conflicted with legitimate terms were suppressed by a "phantom dictionary". The rest of the phantoms were allowed to remain. These only came up for pinyin bigrams (almost always bigrams) that were non-productive in the stock dictionary. The user supplied dictionary took priority over the system dictionary (and the phantoms it contained) so conflicts didn't arise.

Because of the way the hash table was constructed, our dictionary generated an exponentially increasing number of phantoms with increasing phrase length. By the time you got to four character phrases, the phantoms vastly outnumbered the legitimate vocabulary. Note that our system distinguished 8000 hanzi characters for the input system, so the space of possible four character phrases was up in the trillions, and the phantoms were extremely sparse by that metric, and never seen in the wild.

Any competitor who had decided to enumerate our dictionary (I could have suggested several practical ways to achieve this) would have ended up with barrels of nonsense, unless they also devoted the resources, as we had, to "research" rather than plagiarise.

Nor was it possible to copy our dictionary directly in its compressed format, as the hash function was tied to a hardware dongle. I never heard that the algorithm embedded in the dongle was ever cracked directly, but I do know that the vendor's recommended algorithm for feeding the dongle was awful, and failed most of my statistical tests. We beefed up the routine until many (but far from all) of the statistical tests for randomness were satisified, and then ran the device ten times overspec to get the performance we required. Fun times.

A funny story is that our software was listed as "cracked" on some hacker site because some l33t dude had removed the code to test for the presence of a functioning dongle, and the message we displayed "where's your dongle?" (OK, it wasn't quite like that) without noticing that with the dongle absent, the pinyin input method used white noise as the dictionary hash function, and produced nothing but chicken soup for the hanzi output text. To successfully change the hash function and maintain the dictionary compression ratio, you had to solve a bipartite graph matching problem and then recompute the phantom table, and none of that code shipped with the product.

In this era, with the amount of data you can scrape off the internet on a the barest whim, I'm a bit shocked that anyone still stoops to our tried and true "research" methodologies from the mid eighties. My involvement ended around 1991 as it became apparent that Windows 3.x was going to take over the world. My joy in life at that time was writing bug-free code, and I didn't see any way to achieve that the way the world was turning. If someone tapped me on the shoulder and woke me up after my fifteen year snooze, I could probably suggest many fascinating IME features I had planned back then that still haven't been implemented, though I haven't checked on this in a long while. We already had simplified/classical, Mandarin/Cantonese working from a single dictionary. It wasn't proper dialectic Cantonese though, that was something I wished to do, but never completed. We did all this pre Unicode, so we had to invent our own Unicode, too. Anyone need a first edition Unicode standard? I think I've got three.

Re:This wouldn't be the first time...

[Anonymous Coward]

the dozens of person-years that went into writing the actual dictionaries for aspell were simply co-opted by Google. Get off your high horse - you're just another holy roller.

Thousands of people donate their time, money, and code to GPL-licensed projects. As one of those contributors, I can tell you that I don't believe that Google is doing anything wrong at all with aspell. The terms of the license are clear. Users are no way required to give attribution. In fact, there is not even a suggestion, hint, or implication that attribution would be nice. You suggesting that it should be that way is fine, but to state that aspell was "co-opted" is factually incorrect and falsely implies that Google is doing something against the GPL license.

If you, as a contributor to aspell, don't like aspell's license terms, you are free to start another project with similar goals under different license terms.

Combing

[eMbry00s]

Everybody who says something along the lines of "bah, chinese complaining about stealing" should note that all Chinese are not connected into one single conscious entity, but are different individuals.

The people who own this IP need not have stolen any other IP.

It is as dumb as saying that all Americans are christian, guntouting, fat fuckasses.

Re:This wouldn't be the first time...

[cubic6]

Care to release those words that prove that Google uses Aspell? I don't see any proof in your post, just claims that are impossible to verify because you give very little information. You're an author of some dictionary that's used in Aspell, you put intentionally misspelled words in your dictionary, but you don't tell us which dictionary or which words, so what do we have to go by? Why is your post any more trustworthy than any other AC post? Furthermore, it's pretty suspicious that you claim that you INTENTIONALLY put incorrect words in your dictionary to catch people using it as part of a larger project, when such use is perfectly legal. Things like that undermine Aspell's credibility as a reference tool, which, as a contributor, I would think you'd care about.

Re:Identical typos...

[Plutonite]

Not really. I'm not defending Google here, but you seem to be talking about an essay not an algorithm. If you have algorithms that are similar enough, they do not even need to be "structured the same way" to produce the same output(errors included). Anybody who has been to an ACM contest will tell you this.

As such this story is useless. The internet needs no more speculation as it is, it's hard enough arguing what is wrong or right when concrete evidence is available. Our flamewars should be founded on solid ground.

Re:Identical typos...

[ReallyEvilCanine]

According to TFA, Sohu has patents in several areas related to how popular Internet search terms can be used for predictive text input. Google does, too. And unlike most others, Google constantly tweaks algorithms. Have you noticed how the Google Toolbar now predicts your search terms? And every time you deviate, they do modifications for you personally and tabulate in general to see if other's are also going after such similar versions.

I work in I18N and deal with IMEs all the time, from the basic, non-learning MS Windows versions to the ones which come with the NJ Star and give preference to lesser-used terms previously selected to various other proprietary variants. There are only so many ways to write an IME, and there are only so many ways to do good prediction. If I type "go" in Japanese, my first choice will usually be "5" followed by the symbol for "language" and the game "Go", then various other possibilities. Only when I next type a "z" or a "g" do the symbols for a.m. and p.m. move to the front. Now if I'd written an IME and wanted to protect it I might have it always bring up "Mifune Go" ( as the fifth selection or, more subtly, bring up "Go" as the fifth possibility if you typed a "G" or "Go" after "Mifune". This isn't the case here.

With Google's work and implementation of prediction methods, I find it hard to accuse the company of plagiarism for having the same bug (which comes as a result of predictive methods) as some other company. This is a bug, not some zyzzyx or easter egg which a programmer included to catch thieves. It was unintentional on Sogou's part and likely equally unintentional on Google's.

Then again, there's a lot of pressure to excel at Google and maybe someone gave in to temptation despite working for a company that knows more about data than anyone else out there. Unlikely, but possible... and if Google issue a statement that someone did indeed plagiarise Sohu's work, fine. It could happen anywhere. Doesn't make Google bad, only one programmer. It makes the company culpable, but it hardly looks malicious.

Input method

[DrYak]

Just fucking google it ;)

Chinese is a complex language to write. It doesn't use an alphabet (like most western languages). It doesn't even use syllables (like, for example, 2 of the Japanese writing system), it uses logographs : in an over-simplified way, we can say they use 1 symbol for every different word/idea/etc.
This makes thousands of different symbols (According to wikipedia : a little less than 50k variants in the Kangxi dictionary).

This ISN'T something you can put on a regular occidental 107 keys keyboard.

Therefor you have several solutions :

- Custom keyboards :
Use special keyboards where the most frequently couple of thousand of symbols are present.
Not very practical (symbols harder to find compared to looking for a letter on a 107 keyboard). Wikipedia has a picture.

- By shape of characters :
Either by handwriting recognition, or by decomposing charachters (the different strokes) and putting them on a regular keyboard layout.

- By sound of words :
Either by using something like Zhuyin which is system that was invented to help teaching chinese. It has 31 symbols, 1 for each consonant or vowel in chinese. As such, it can be used for other purposes, like putting it on a keyboard : the person type the sound and the software guess the corresponding word/logogram.
Or an alternative method is the Pinyin : it uses latin letters to write the sound. (And thus is interesting for computers on which latin keyboards are widespread).

The mapping of sound to logographs isn't completely straightforward, for example Chinese is a tonal language, but some system don't require the writer to specify tones using marks. Some software work is required. And this software isn't infallible.

Google released such a software. User can phonetically type Chinese on any occidental keyboard using (tone-less) pinyin, and the software tries to convert it to actual Chinese characters.
This software produce the same correct results as another popular one. (Hopefully. If the google soft didn't give the correct results, there would be problems. I wouldn't be a functional pinyin input system).
Sometime, the software hesitates and give a choice of possibilities. Most of the time, the same as the concurrent (Possibly explained by the fact that both softwares have to process the same user input, using the same pronunciation system that isn't unambiguous).
But, sometime the Google soft is plain wrong, and produces the same errors as the concurrent. And THIS is suspicious, because maybe some part of the software uses piece from the concurrent (part of the algorithm ? statistical data ?)

The company is suing googles on the grounds that if both softwares behave the same down to the bugs, maybe some part could have been illegally copied.

Meanwhile, adepts of Google Seppuku rejoiced world wide a cheap and easy to find software that could also be used to produce random chinese caracter to be subsequently imported into Google as Kanji.

Google evolves

[Lorean]

Google has learned how to do business in China.
Congrats to them.