Baidu Forced To Withdraw Last Month's ImageNet Test Results

elwinc writes: Back in mid-May, Baidu, a computer research and services organization in Mainland China, announced impressive results on the ImageNet "Large Scale Visual Recognition Challenge," besting results posted by Google and Microsoft. Turns out, Baidu gamed the system, creating 30 accounts and running far more than the 2 tests per week allowed in the contest. Having been caught cheating, Baidu has been banned for a year from the challenge. I believe all competitors are using variations on the convolutional neural network, AKA deep network. Running the test dozens of times per week might allow a competitor to pre-tune parameters for the particular problem, thus producing results that might not generalize to other problems. All of which makes it quite ironic that a Baidu scientist crowed "Our company is now leading the race in computer intelligence!"

Comments

Lead from behind....

[bobbied]

That's what I always say.... (/sarcasm)

WHAT!

[war4peace]

Chinese company caught cheating? NO WAY!
Seriously though, raise your hand if you're surprised.

Re:WHAT!

[jandersen]

Chinese company caught cheating? NO WAY!
Seriously though, raise your hand if you're surprised.

I'm not surprised, but not for the reason I suspect you have in mind: "Because Chinese are just so and so...".

However, it does not really surprise me that we see this from some Chinese companies. China is a developing nation, and they are still relatively new to the way companies play it in the West - not that we are in fact more honest in the West, we have just learned how and when to be dishonest in a way that doesn't make as much noise. I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour. There is no shortage of examples.

But there is another thing in it: lack of regulation. It should not be a surprise to anyone that when there is too little regulation, the most ruthless will feel entitled to bully others - the free, unregulated market can never work to the benefit of everybody, because there will always some, that ruthlessly go for maximising their own short-term advantage, and and that behaviour pushes out the competition and creates monopolies. It is perhaps ironical that this argument is exactly analogous with the argument against Communism: "people are selfish, so if they don't have a reason to work harder, most won't".

We see this in all developing countries, but perhaps most tragilcally in Russia, where they tried to go from tightly regulated Communism to a kind of laissez-faire Capitalism overnight and got horribly burned. And they ended up with the same kind of masters as before, because scum always rises to the top of the pond.

Re:WHAT!

[war4peace]

" I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour."

When all else is equal...
All companies have these kind of skeletons in the closet. Chinese companies simply seem to have some more on top of those which everyone else lovingly owns.
Lack of regulation might be a reason when seeing this internally (within the country) - and then again, when all else is equal... But here we're talking about international events, and that's where you see companies A, B, C playing by the rules and company D (Chinese, more often than not) trying to cheat its way in.
No more than a couple weeks ago there was a story here on /. about Chinese students taking exams instead of the ones who should. After many, many, MANY such stories over the years one can't help but develop a stereotype.
From fake $ITEM to cheating in competitions, China seems on top of the world as count of occurrences.

I googled "chinese cheating": got 22.6M results, top results are about exam cheating.
I also googled "americans cheating", got 14.8M results, top results are about marital cheating.
Of course, this might not mean much, but it's a start. Anyone wants to send a research grant my way? :)

Re:WHAT!

[Ralph Wiggam]

China is a developing nation, and they are still relatively new to the way companies play it in the West

Their GDP is twice as big as Japan's, the #3 economy in the world. At what point do they stop getting to play the "we're just a poor developing country, we can't be expected to follow the rules" card?

Re:Chinese Hyper-competitiveness

[Anonymous Coward]

people will just try anything get ahead, completely oblivious to the wider problem or goal they are trying to solve

Their goal is to make money, even if it means screwing the customer. Their culture is such that if you let them screw you it's your own fault and you should be mad at yourself, not the person who cheated you.

I really don't know how us westerners with our 40hr work weeks, healthcare and pensions are going to eventually compete with that until we too are faced with the desperation of trying to escape from abject poverty along with 1 billion other people.

The key word there is "eventually". Long term "us Westerners" are leading and will continue to work together to make real progress instead of pretending to be ahead by faking results and cheating on contests and other things like SAT tests.

Or easier still

[Virtucon]

They'll just go in and steal the research from another competitor and call it their own. Cheating and espionage are familiar bedfellows.

Banned for 1 year?

[Anonymous Coward]

Maybe that's appropriate punishment for children, but these are professional scientists. The only reason nobody has the brass to ban them for life is because their country owns us.

Re:Chinese cheat

[gstoddart]

You know, when your parent society doesn't value honesty, and everybody around you is cheating ... you're a fool to think there's any value in being that one guy who says "gee, I should be honest here".

In situations in which it's a liability to be honest, only suckers are honest.

And in governments who have spend decades saying "there is no higher power than the state", if the state is rampantly corrupt, "integrity" is a relative term.

Give it a few more years, and you'll discover that integrity in America is a much more malleable concept than you realize -- in fact, it's probably already there.

The mentality of "it's OK as long as I don't get caught" isn't a new thing to humanity.

Re:Chinese Hyper-competitiveness

[Anonymous Coward]

"China has no such internal conflict."

You must be kidding, or not know anything about China.

Re:Chinese cheat

[mark-t]

Ah...the "everybody does it" excuse, otherwise known as the Golden Rationalization. As for suggesting that one is somehow foolish to try and live with integrity when others are cheating.... Well thats just a variant on an ad hominem, called Poisoning the Well.

Baidu Team's Apology Appended to Official Notice

[eldavojohn]

From the official announcement found in the NYT article (full of details we mostly already know) there comes an update with the team's response:

Message from the team in question:

Dear ILSVRC community,

Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

Ren Wu – Baidu Heterogeneous Computing Team

So, while they deserve the year ban, the apology is nice. It's a shame we can never know what results a fair competition could have yielded ... and an even bigger shame that the media misreported Baidu as overpowering Google. I suppose the damage is done and the ILSVRC has made the right choice.

Perhaps I'm misunderstanding the classification problem but why isn't this run like most other classification problems (like Netflix and many other data challenges) where you get ~80% for training and the remaining 20% are held back for the final testing and scoring? Is the tagged data set too small to do this? Seems like wikimedia would contain a wealth of ripe public domain images for this purpose ...