Slashdot Mirror
Huawei Caught Cheating Performance Test For New Phones (techcrunch.com)
An anonymous reader quotes a report from TechCrunch: UL, the company behind the tablet and phone performance benchmark app 3DMark, has delisted new Huawei phones from its "Best Smartphone" leaderboard after AnandTech discovered the phone maker was boosting its performance to ace the app's test. The phones delisted were the P20, P20 Pro, Nova 3 and the Honor Play. "After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings," the company said in a statement.
For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances. In other words, the phones aren't so smart after all.
For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances. In other words, the phones aren't so smart after all.
You DO know Apple phones are made in China, right?
"Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI"
Whenever someone makes an AI claim you know they are lying.
If you know what chipset is in a phone, then you know as much as you usefully need to know about the real world performance of that phone relative to other phones with the same chipset, i.e. pretty much the same. If you see a benchmark saying one such phone is outperforming its peers, then you assume that, if it's not down to actual cheating, then it's a quirk of the benchmark, because you know the same cpu/gpu at the same speed gives the same performance. Does anybody actually look at benchmarks when buying a phone?