Slashdot Mirror

← Back to Stories (view on slashdot.org)

Baidu Forced To Withdraw Last Month's ImageNet Test Results

Posted by timothy on from the please-reconsider dept.

elwinc writes: Back in mid-May, Baidu, a computer research and services organization in Mainland China, announced impressive results on the ImageNet "Large Scale Visual Recognition Challenge," besting results posted by Google and Microsoft. Turns out, Baidu gamed the system, creating 30 accounts and running far more than the 2 tests per week allowed in the contest. Having been caught cheating, Baidu has been banned for a year from the challenge. I believe all competitors are using variations on the convolutional neural network, AKA deep network. Running the test dozens of times per week might allow a competitor to pre-tune parameters for the particular problem, thus producing results that might not generalize to other problems. All of which makes it quite ironic that a Baidu scientist crowed "Our company is now leading the race in computer intelligence!"

31 of 94 comments (clear)

  1. Lead from behind.... by bobbied · · Score: 3, Insightful

    That's what I always say.... (/sarcasm)

    --
    "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
  2. WHAT! by war4peace · · Score: 5, Insightful

    Chinese company caught cheating? NO WAY!
    Seriously though, raise your hand if you're surprised.

    --
    ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    1. Re:WHAT! by retchdog · · Score: 5, Funny

      it's not called cheating there. the ideograms translate roughly to "auspicious cooperation with jade dragon of opportunity."

      --
      "They were pure niggers." – Noam Chomsky
    2. Re:WHAT! by NotDrWho · · Score: 3, Funny

      Chinese company caught cheating? NO WAY!

      I almost broke my string of pearls from clutching them so hard. You just don't expect this kind of corruption out of China.

      --
      SJW's don't eliminate discrimination. They just expropriate it for themselves.
    3. Re:WHAT! by NotDrWho · · Score: 5, Funny

      I think that's from Confucius's 65th Rule of Acquisition.

      --
      SJW's don't eliminate discrimination. They just expropriate it for themselves.
    4. Re:WHAT! by Guy+From+V · · Score: 2

      I just got the vapors.

    5. Re:WHAT! by jandersen · · Score: 5, Insightful

      Chinese company caught cheating? NO WAY!
      Seriously though, raise your hand if you're surprised.

      I'm not surprised, but not for the reason I suspect you have in mind: "Because Chinese are just so and so...".

      However, it does not really surprise me that we see this from some Chinese companies. China is a developing nation, and they are still relatively new to the way companies play it in the West - not that we are in fact more honest in the West, we have just learned how and when to be dishonest in a way that doesn't make as much noise. I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour. There is no shortage of examples.

      But there is another thing in it: lack of regulation. It should not be a surprise to anyone that when there is too little regulation, the most ruthless will feel entitled to bully others - the free, unregulated market can never work to the benefit of everybody, because there will always some, that ruthlessly go for maximising their own short-term advantage, and and that behaviour pushes out the competition and creates monopolies. It is perhaps ironical that this argument is exactly analogous with the argument against Communism: "people are selfish, so if they don't have a reason to work harder, most won't".

      We see this in all developing countries, but perhaps most tragilcally in Russia, where they tried to go from tightly regulated Communism to a kind of laissez-faire Capitalism overnight and got horribly burned. And they ended up with the same kind of masters as before, because scum always rises to the top of the pond.

    6. Re:WHAT! by war4peace · · Score: 5, Insightful

      " I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour."

      When all else is equal...
      All companies have these kind of skeletons in the closet. Chinese companies simply seem to have some more on top of those which everyone else lovingly owns.
      Lack of regulation might be a reason when seeing this internally (within the country) - and then again, when all else is equal... But here we're talking about international events, and that's where you see companies A, B, C playing by the rules and company D (Chinese, more often than not) trying to cheat its way in.
      No more than a couple weeks ago there was a story here on /. about Chinese students taking exams instead of the ones who should. After many, many, MANY such stories over the years one can't help but develop a stereotype.
      From fake $ITEM to cheating in competitions, China seems on top of the world as count of occurrences.

      I googled "chinese cheating": got 22.6M results, top results are about exam cheating.
      I also googled "americans cheating", got 14.8M results, top results are about marital cheating.
      Of course, this might not mean much, but it's a start. Anyone wants to send a research grant my way? :)

      --
      ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    7. Re:WHAT! by ArcadeMan · · Score: 2

      I think you're mixing the Chinese with the Ferengi here...

      --
      Get free satoshi (Bitcoin) and Dogecoins
    8. Re:WHAT! by ShanghaiBill · · Score: 3, Interesting

      it's not called cheating there.

      Except it wasn't "there". It was "here". Baidu's main research lab, where this AI research was done, is in Sunnyvale, California, and many employees there are American citizens.

    9. Re:WHAT! by Ralph+Wiggam · · Score: 2, Insightful

      China is a developing nation, and they are still relatively new to the way companies play it in the West

      Their GDP is twice as big as Japan's, the #3 economy in the world. At what point do they stop getting to play the "we're just a poor developing country, we can't be expected to follow the rules" card?

    10. Re:WHAT! by drinkypoo · · Score: 2

      Congratulations, you killed that joke dead.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    11. Re:WHAT! by barc0001 · · Score: 2

      Corporate culture is corporate culture. Location is largely irrelevant.

    12. Re:WHAT! by Ralph+Wiggam · · Score: 2

      he didn't say they were small or poor. Putting words into someone else's mouth and hitting them with it -- I think that is called a strawman.

      developing nation
      "A nation where the average income is much lower than in industrial nations..."

      http://dictionary.reference.com/browse/developing+nation

  3. Chinese Hyper-competitiveness by Anonymous Coward · · Score: 4, Interesting

    This reminds me of some Chinese ball screws (rotary to linear motion components) my company once ordered from Alibaba. The companies had pictures, drawings and put together quotes for parts but then delivered samples that were just totally totally useless. Some of these 'precision' parts looked like they had been made with a file. It just didn't make any sense that they would waste their and our time on such clearly incompetent products.

    But when you go there you realise the problem. It is basically an economy in a state of hyper competition. There is so much competition that people will just try anything get ahead, completely oblivious to the wider problem or goal they are trying to solve. You can see that in how the government had to rationalise the solar industry because nobody could make any money. They are just really really crazy competitive.

    The trouble though is that there are now many good Chinese engineers who know what they are doing but are still hyper competitive. I really don't know how us westerners with our 40hr work weeks, healthcare and pensions are going to eventually compete with that until we too are faced with the desperation of trying to escape from abject poverty along with 1 billion other people.

    1. Re:Chinese Hyper-competitiveness by Anonymous Coward · · Score: 3, Insightful

      people will just try anything get ahead, completely oblivious to the wider problem or goal they are trying to solve

      Their goal is to make money, even if it means screwing the customer. Their culture is such that if you let them screw you it's your own fault and you should be mad at yourself, not the person who cheated you.

      I really don't know how us westerners with our 40hr work weeks, healthcare and pensions are going to eventually compete with that until we too are faced with the desperation of trying to escape from abject poverty along with 1 billion other people.

      The key word there is "eventually". Long term "us Westerners" are leading and will continue to work together to make real progress instead of pretending to be ahead by faking results and cheating on contests and other things like SAT tests.

    2. Re:Chinese Hyper-competitiveness by JeffOwl · · Score: 4, Interesting

      You need the buyers to stand up to that kind of crap. These guys screwed (sorry) you once, black list them. I had a frustrated guy in procurement say "You could make tons of money in x business by just doing what you say you are going to do." When it came down to it, he was not always allowed to make the smart decision and go with a reliable company's bid if it wasn't close to the bottom.

      Part of the problem we see in China is that they spawn new companies like rabbits. One will eventually go under doing shady stuff only for the same people to come back, selling the exact same crap under a different name. We had this happen and you could look at the product and see that they were using the exact same molds, same defects.

    3. Re:Chinese Hyper-competitiveness by Anonymous Coward · · Score: 2, Insightful

      "China has no such internal conflict."

      You must be kidding, or not know anything about China.

    4. Re:Chinese Hyper-competitiveness by drinkypoo · · Score: 2

      Except they tend not to be major companies with government backing.

      Let's bail out the banks... then bail out the banks again!

      Let's bail out the automakers... then bail out the automakers again!

      Get real.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  4. Chinese cheat by mi · · Score: 5, Interesting

    People growing up under oppressive governments have much fewer problems with cheating — because cheating government is a fair game. It rubs off — and the attitude is quickly extended to non-governmental institutions large and even smaller ones.

    This is not "racism" — ex-Soviets like myself often have the same problem... A cheating Western student fears (or used to fear) the shame of being exposed. A Chinese — or a Soviet — fears merely getting caught. Like a speeding ticket — there is no shame in driving fast, only in being stopped by "the bear".

    China today uses drones to catch cheaters — America had not felt the need for such measures. Perhaps, it was a foolish attitude, because we the immigrants bring all our traits to the "wonderful tapestry of diversity", not just the good ones...

    Anybody dealing with Chinese companies (or Russian ones, if you can find any), ought to be careful and not depend merely on trust.

    --
    In Soviet Washington the swamp drains you.
    1. Re:Chinese cheat by mark-t · · Score: 2

      People growing up under oppressive governments have much fewer problems with cheating â" because cheating government is a fair game.

      Yeah... because two wrongs make a right.

      The rules of life are not "whoever dies with the most toys wins", but it seems to me that people who think cheating is a good idea often seem to live their lives as though that were the case.

      Striving to live one's life with integrity may not be easy, but then nothing that is really worth doing ever is.

      --
      File under 'M' for 'Manic ranting'
    2. Re:Chinese cheat by gstoddart · · Score: 4, Insightful

      You know, when your parent society doesn't value honesty, and everybody around you is cheating ... you're a fool to think there's any value in being that one guy who says "gee, I should be honest here".

      In situations in which it's a liability to be honest, only suckers are honest.

      And in governments who have spend decades saying "there is no higher power than the state", if the state is rampantly corrupt, "integrity" is a relative term.

      Give it a few more years, and you'll discover that integrity in America is a much more malleable concept than you realize -- in fact, it's probably already there.

      The mentality of "it's OK as long as I don't get caught" isn't a new thing to humanity.

      --
      Lost at C:>. Found at C.
    3. Re:Chinese cheat by bobbied · · Score: 2

      Yes, well.. Truth be told, cheating and being dishonest are usually not long term strategies for success under any system. You cheat in school? You will pay later when the next course comes along that uses the material you don't know. Cheat again and you just increase your chances of getting caught. One cheat leads to another, then more, usually bigger and bigger ones until you get caught and pay the piper for cheating.

      In my world view, the reward in doing the right thing over the long haul is in knowing that your success is honestly yours. I may be taken for a sucker at times, but I generally catch on pretty quick, limit my losses and learn to do better next time. I can also sleep at night because I don't have to keep my stories straight and invent new schemes to keep from being exposed.

      Finally, the old adage "Cheaters never prosper" while not always true, is usually true. And in my long term world view we all face judgment in the end where cheaters will be punished and eventually justice will prevail. So I'm adopting the long term view of doing right.

      So doing right is it's own reward NOW, is in my view less risky for tomorrow, and avoids future punishment in the hear after.... Your mileage may vary on the last one, but the first to are good enough reasons to be honest...

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    4. Re:Chinese cheat by ShanghaiBill · · Score: 2, Interesting

      The rules of life are not "whoever dies with the most toys wins"

      But the rules are "whoever doesn't eat loses". Historically, this was not about accumulating toys, but basic survival. During the Great Leap Forward, 30 million Chinese starved to death. There were another few million deaths from starvation and violence during the Cultural Revolution. Both of those were caused by government abuse and incompetence. Even today, China has a hereditary class system, that denies hundreds of millions of people access to housing, healthcare, and even the right to send their children to public schools. When the system treats you like that, you feel no obligation to be honest in return.

      Striving to live one's life with integrity may not be easy, but then nothing that is really worth doing ever is.

      That is a very western attitude. You should not project your another cultural values onto a culture that does not share them. To the Chinese, it is not 'integrity" that is honorable, but loyalty to your family. So if you cheat to help your family, or abuse your authority to give all the plum government jobs to your nephews, that is not only tolerated, but admired.

    5. Re:Chinese cheat by mark-t · · Score: 3, Insightful

      Ah...the "everybody does it" excuse, otherwise known as the Golden Rationalization. As for suggesting that one is somehow foolish to try and live with integrity when others are cheating.... Well thats just a variant on an ad hominem, called Poisoning the Well.

      --
      File under 'M' for 'Manic ranting'
  5. Or easier still by Virtucon · · Score: 4, Insightful

    They'll just go in and steal the research from another competitor and call it their own. Cheating and espionage are familiar bedfellows.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
  6. Nothing new by lorinc · · Score: 2

    Such cheat have been used for years in the field. Before ImageNet, there was the Pascal VOC challenge with about the same rules, and I'm pretty sure all winners were optimizing the hyperparameters their submission on the test dataset.

    Seriously, as long as computer vision benchmark are based on a single train/test split, there will be such abuses. If there were several splits with meaningful statistics computed on it, I would be less worried by the overfitting you get by optimizing the hyperparameters.

    But hey, you're never gonna make it to CVPR without tunning your method so as to fool reviewers that it performs much better than the state of the art. 0.1% for a good idea, 99.9% for engineering tricks.

    --
    Video of some good progressive thrash music
  7. Banned for 1 year? by Anonymous Coward · · Score: 2, Insightful

    Maybe that's appropriate punishment for children, but these are professional scientists. The only reason nobody has the brass to ban them for life is because their country owns us.

  8. We know what Baidu is, thankyouverymuch. by captaindomon · · Score: 4, Informative

    Baidu isn't just "a computer research and services organization", they're the Chinese version of Google. They're a massive company with eight billion USD in revenue last year. The headline is either misleading or completely clueless.

    --
    Just because I can hook a shark from a boat, I do no offer to wrestle it in the water.
  9. China cheating?? by gatkinso · · Score: 2

    Surely you jest!

    --
    I am very small, utmostly microscopic.
  10. Baidu Team's Apology Appended to Official Notice by eldavojohn · · Score: 3, Insightful
    From the official announcement found in the NYT article (full of details we mostly already know) there comes an update with the team's response:

    Message from the team in question:

    Dear ILSVRC community,

    Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

    We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

    We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

    Ren Wu – Baidu Heterogeneous Computing Team

    So, while they deserve the year ban, the apology is nice. It's a shame we can never know what results a fair competition could have yielded ... and an even bigger shame that the media misreported Baidu as overpowering Google. I suppose the damage is done and the ILSVRC has made the right choice.

    Perhaps I'm misunderstanding the classification problem but why isn't this run like most other classification problems (like Netflix and many other data challenges) where you get ~80% for training and the remaining 20% are held back for the final testing and scoring? Is the tagged data set too small to do this? Seems like wikimedia would contain a wealth of ripe public domain images for this purpose ...

    --
    My work here is dung.