Casting a Jaundiced Eye On AnTuTu Benchmark Claims Favoring Intel
MojoKid writes "Recently, industry analysts came forward with the dubious claim that Intel's Clover Trail+ low power processor for mobile devices had somehow seized a massive lead over ARM's products, though there were suspicious discrepancies in the popular AnTuTu benchmark that was utilized to showcase performance. It turns out that the situation is far shadier than initially thought. The version used in testing with the benchmark isn't just tilted to favor Intel — it seems to flat-out cheat to accomplish it. The new 3.3 version of AnTuTu was compiled using Intel's C++ Compiler, while GCC was used for the ARM variants. The Intel code was auto-vectorized, the ARM code wasn't — there are no NEON instructions in the ARM version of the application. Granted, GCC isn't currently very good at auto-vectorization, but NEON is now standard on every Cortex-A9 and Cortex-A15 SoC — and these are the parts people will be benchmarking. But compiler optimizations are just the beginning. Apparently the Intel code deliberately breaks the benchmark's function. At a certain point, it runs a loop that's meant to be performed 32x just once, then reports to the benchmark that the task completed successfully. Now, the optimization in question is part of ICC (the Intel C++ compiler), but was only added recently. It's not the kind of procedure you'd call by accident. AnTuTu has released an updated "new" version of the benchmark in which Intel performance drops back down 20-50%. Systems based on high-end ARM devices again win the benchmark overall, as they did previously."
Of course not.
Make them do real work loads. With Monkeys.
It is the suite of tools, not just the processor. If intel offers a better processor/compiler package than is available for arm why shouldn't they tout it? I'm not saying they are presenting it in the correct way, but I do think they have a valid point they want to make. That with Intel you get more than a CPU, you get a heck of a lot of tool expertise. And for some people that is worth something.
http://hardware.slashdot.org/story/13/07/12/1558209/new-analysis-casts-doubt-on-intels-smartphone-performance-vs-arm-devices?sdsrc=rel
Compiler was one of many skews in this "honest" benchmark. Aside of deliberately "fixing" benchmark code for intel and deliberately breaking ARM benchmark by disabling NEON. In my opinion they should run identical code, trying to maximize its performance on both platforme and in case of Intel use both compilers and post both results. This would lead potential customer to correct conclusions - as opposed to a bunch of lies and misinterpretations AnTuTu actually posted.
In fairness to AnTuTu they released a new version which tries to rectify the problem:
http://www.eetimes.com/author.asp?section_id=36&doc_id=1318894&
At least Linpack performs actual linear algebra, so coding to that particular test will help some people with real workloads (i.e. scientific software that uses Linpack). It's definitely not representative of everyone's workload, though.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
...where companies used to rig benchmarks?
Oh right, we're still not past them.
AND WE'LL NEVER BE!
Always use real world applications, in actual, real usage. Never benchmarks.
Looking for people to chat about multicopters, coding, music. skype: gtsiros
I know some ignorant people that will take these benchmarks as gospel in their righteous views.
ARM looks like a sore loser here.
>GCC isn't currently very good at auto-vectorization, but NEON is now standard on every Cortex-A9 and Cortex-A15 SoC
So the conclusion is to remove intel optimizations instead of improving ARM ones?
Who logs in to gdm? Not I, said the duck.
>high-performance computing
winner, no contest: Intel's best CPU, plus the best GPU money can buy. Why hobble a kick-ass GPU with a second-rate CPU?
> excitement
I don't know about YOU, but I get more excited by maximum performance than by power consumption, cost, or marketing. Winner: Intel
I don't wish AMD ill. I'd *much* rather see AMD pulling ahead of Intel & forcing both into a deathmatch for better performance. Let's not forget that AMD happily showed us that they can be as expensive & mean as Intel given half a chance. I *want* to see AMD & Intel trading the top spot every 4-8 months, like in the old days :-)
The last time I checked, neither Windows NOR Linux truly has realtime 2560x1600 120hz alpha-blended raytracing or consequence-free desktop eyecandy yet. There's still plenty of room for improvement, as long as we can slow down Microsoft & Ubuntu's mad quest to roll back performance to the nasty level of tablets and netbooks. I don't want a 1mm pseudo-notebook with virtual keyboard & cpu that spends most of its time at 700MHz... I want a 6" thick lunchbox with 3 screens (1920x1200 main, flanked by a pair of hinged 960x1200 panels), mechanical-switch keyboard (Cherry Green, or ideally buckling spring), and the fastest cpu available. With optional 24,000MAh battery & Thunderbolt-connected pseudo-netbook remote LCD+keyboard+Trackpoint, so that when I fly I can put the lunchbox under the seat & still use it ;-)
>high-performance computing
winner, no contest: Intel's best CPU, plus the best GPU money can buy. Why hobble a kick-ass GPU with a second-rate CPU?
Because power consumption is very important for HPC...
It might not matter so much for a single user's desktop, but when you scale to thousands of processors the extra power consumption can cost serious amounts of money to keep running both in the power it consumes, and the extra power consumed keeping it cooled.
If your HPC workload primarily uses the GPU, then the CPU may even be sitting idle most of the time, your CPU only needs to be fast enough to keep the GPU fed with data.
Also for HPC, throughput is important... Current CPUs are much faster than the memory to which they're connected, so while some workloads can fit in the cache and run very fast those which result in lots of memory access can be considerably slower. That's why several of the IBM top500 supercomputers use relatively slow embedded PPC cpus, the CPU won't be wasting lots of its time and energy waiting for memory to catch up.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Instead of pointing a finger and tell people about it, why not fix it and SHOW people the actual numbers after optimisations for both platforms are set into place, and without those optimisations on both platforms.... Meanwhile you can also say that Intel's C++ compiler is just better than GCC's as appearantly Intel compiler already has all optimisations ON by default and GCC doesn't....