Slashdot Mirror

← Back to Stories (view on slashdot.org)

Huawei Caught Cheating Performance Test For New Phones (techcrunch.com)

Posted by BeauHD on from the can't-say-I'm-surprised dept.

An anonymous reader quotes a report from TechCrunch: UL, the company behind the tablet and phone performance benchmark app 3DMark, has delisted new Huawei phones from its "Best Smartphone" leaderboard after AnandTech discovered the phone maker was boosting its performance to ace the app's test. The phones delisted were the P20, P20 Pro, Nova 3 and the Honor Play. "After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings," the company said in a statement.

For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances. In other words, the phones aren't so smart after all.

9 of 74 comments (clear)

  1. Re:Stop using foreign products by Anonymous Coward · · Score: 5, Informative

    You DO know Apple phones are made in China, right?

  2. AI by 110010001000 · · Score: 5, Insightful

    "Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI"

    Whenever someone makes an AI claim you know they are lying.

  3. Hardly seems worth it by Anonymous Coward · · Score: 4, Insightful

    If you know what chipset is in a phone, then you know as much as you usefully need to know about the real world performance of that phone relative to other phones with the same chipset, i.e. pretty much the same. If you see a benchmark saying one such phone is outperforming its peers, then you assume that, if it's not down to actual cheating, then it's a quirk of the benchmark, because you know the same cpu/gpu at the same speed gives the same performance. Does anybody actually look at benchmarks when buying a phone?

    1. Re:Hardly seems worth it by Moof123 · · Score: 2

      "For a portable device, that means more battery drain. So even if you can run apps a few % faster, you'll only be able to do (roughly) the same amount of work before battery runs empty"
      Actually no. In the olden days the supply voltage was fixed, leakage current was low resulting in a constant energy per cycle for a given operation. In modern times the supply voltage is carefully adjusted with clock speed. Run the part down at 1V and it might only run at 1 GHz, crank it up to 1.5V and it goes at 2 GHz, but consumes about 4x the power and also has much increased leakage power.
      Cell phone SOC's go even further these days, containing both low power cores and high performance cores to further scale power savings across a wide range of operating scenarios. The low power core might be half the performance at a given clock rate, but a quarter or less the power.
      Firmware choices as to which cores to activate and what clock rate to run at can make huge difference net power consumption and perceived speed.

    2. Re:Hardly seems worth it by Desler · · Score: 2

      Not true at all. The same SoC can have a wide difference in performance due to differing thermal headroom depending on the device.

  4. Re:Stop using foreign products by Anonymous Coward · · Score: 2, Insightful

    There is no such thing as an American-made phone. In fact, there is almost no American-made anything when it comes to electronics. Prior administrations have allowed the manufacturing sector to move almost everything to enemy states.

    Make no mistake. If there were ever a war, the US would lose, and quickly, because we cannot maintain, repair, or build anything without China's help. We have nearly zero electronics manufacturing capability compared to what would be needed to prosecute a War.

  5. Re:Stop using foreign products by alvinrod · · Score: 2

    The U.S. still does a lot of chip design and fabrication. It's just that the chips get shipped overseas for packaging and final product assembly.

    Also, U.S. manufacturing output has been increasing year over year even though we were moving large chunks of it overseas. I think now would be a good time to start reinvesting in local manufacturing, but that would be done with machines. A lot of the jobs aren't coming back, but that's okay because it means that labor is free to do something that's more productive instead.

  6. Heavens by guygo · · Score: 2

    Who wouldve thought the company that gives approval rights for their designs to the Chinese government would lie to us? But for sure there's no possibilty of any backdoors. Trust us!

  7. Re:But is this what GPU manufacturers have been do by Littleman_TAMU · · Score: 2

    Yes, it was wrong for them and it's wrong for Huawei now. Anandtech, HardOCP, Tom's, and others caught GPU manufacturers doing this and called them out just like now. Part of the reason for Anandtech's development of their own test suite was because they didn't completely trust that vendors weren't cheating the known industry benchmarks.

    So far as I know, the above sites are still looking for cheating and GPU manufacturers have stopped. They've at least stopped obviously cheating by looking for when a known benchmark is running like Huawei got caught doing. They may still be tuning for benchmarks, and not real-world performance, but that's why real-world tests like Anandtech and HardOCP do are still useful for me.