Speed Test: Comparing Intel C++, GNU C++, and LLVM Clang Compilers

Nerval's Lobster writes "Benchmarking is a tricky business: a valid benchmarking tries to remove all extraneous variables in order to get an accurate measurement, a process that's often problematic: sometimes it's nearly impossible to remove all outside influences, and often the process of taking the measurement can skew the results. In deciding to compare three compilers (the Intel C++ compiler, the GNU C++ compiler (g++), and the LLVM clang compiler), developer and editor Jeff Cogswell takes a number of 'real world' factors into account, such as how each compiler deals with templates, and comes to certain conclusions. 'It's interesting that the code built with the g++ compiler performed the best in most cases, although the clang compiler proved to be the fastest in terms of compilation time,' he writes. 'But I wasn't able to test much regarding the parallel processing with clang, since its Cilk Plus extension aren't quite ready, and the Threading Building Blocks team hasn't ported it yet.' Follow his work and see if you agree, and suggest where he can go from here."

Comments

Funny benchmarks

[Anonymous Coward]

The benchmarks in TFA are a little funny. Why is system time so large while user time so small? The only time I've seen this in real applications is when there is major core contention for resources.

Re:Funny benchmarks

[robthebloke]

I agree. He's not testing compiled code performance, he's just created a set of tests which will all be memory bandwidth limited. FTA:

I’m testing these with an array size of two billion.

That's all I needed to read to ignore him completely. Completely and utterly pointless. If g++ won, it is likely because it utilised stream intrinsics to avoid writing data back to the CPU cache, which would have freed up more cache, and minimised the number of page faults. This will not in anyway test the performance of the CPU code, it will just prove that your 1333Mhz memory is slower than your 3Ghz processor . This is why you don't profile code (wrapped up in a stupid for loop), but profile whole applications instead. From my own tests (measuring the performance of large scale applications using real world data sets), intel > clang > g++ (although the difference between them is shrinking). The author of the article hasn't got a clue what he's doing. FTA:

Notice the system time is higher than the elapsed time. That’s because we’re dealing with multiple cores.

No it isn't. It's because your CPU is sat idle whilst it waits for something to do.

Re:Funny benchmarks

[Anonymous Coward]

Here's another tipoff that the guy is clueless about benchmarking, talking about a test which does FP math:

I’m not initializing the arrays, and that’s okay

Actually, it's not. This is a bad mistake which totally invalidates the data. Many FPUs have variable execution time depending on input data. There is often a large penalty for computations involving denormalized numbers. If uninitialized data arrays happen to be different across different compilers (and they might well be), execution time can vary quite a lot for reasons completely unrelated to compiled code quality.

It's not limited to FP, either. I remember at least one PowerPC CPU which had variable execution time for integer multiplies -- the multiplier could detect "early out" conditions when one of the operands was a small number, allowing it to shave a cycle or two off the execution time.

The moral of the story: making sure that input data for benchmarks is always the same is very important, even when it's trivially obvious that the code will execute the exact same instruction count for any data set.

Compile time is irrelevant.

[Anonymous Coward]

Which one produced the fastest code?

Re:Compile time is irrelevant.

[0123456]

Which one produced the fastest code?

My current project takes two hours to compile from scratch, and uses around 20% CPU when it runs. So yes, compile time can be more important than how fast the code runs.

Re:Compile time is irrelevant.

[ledow]

But over the lifetime of any average, runtime should outweigh compile time by orders of magnitude.

Otherwise, honestly, why bother to write the program efficiently at all?

And if you want to decrease compile times, it's easy - throw more machines and more power at the job. If you want to decrease runtime, then ALL of your users have to do that.

Honestly, if your compile times are that much, and that much of a burden, you need to upgrade, and you also need to modularise your code more. The fact is that most of that compile time isn't actually needed for 90% of compiles unless your code is very crap.

Re:Compile time is irrelevant.

[ShanghaiBill]

Which one produced the fastest code?

It doesn't matter. It may matter which one compiles your code faster. Depending on your use of things like templates, classes, etc. that may be a different compiler than the best for the benchmarks. But even that is unlikely to matter much. I doubt if their is much more than a few percentage difference. More important are issues like standard compliance, good warning messages, tool-chain/IDE integration, etc.

Measuring pebbles

[T.E.D.]

Interesting info, but I have a couple of issues:

First off, why wasn't Microsoft's C++ compiler included in this? That's the one we use at work, so that's the one I'd really like compared to all those others. Are we the only ones still using it or something?

More importantly, why on earth was compilation speed the only thing compared? I mean, I suppose its nice for g++ users to know that their 10 minute compiles would have been 2 minutes longer if they used the Intel compiler, but Intel users might not really care if they believe their resulting code is going to run faster. Speed of compilation of optimized code is a particularly useless metric, because different compilers have different definitions of "unoptimized", so its guaranteed you aren't comparing apples to apples.

I suppose compilation speed is a nice metric to brag about between compiler writers. But for compiler users, the most important things are roughly these, in order: Toolchain support, language feature support (eg: C++2012/14 features), clarity of error/warning messages, speed of generated code (optimization), and lastly speed of compilation. I'm not really sure why you took it upon yourself to measure the least important factor, and only that one.

Intel C++ produced fastest code for us

[pauljlucas]

This information is perhaps 2 years out of date, but back for one of my projects, when we switched from g++ to Intel C++, our software got about twice as fast with no other changes. It got even faster when we took advantage of SSE3 instructions.

Re:DIe Buisness Intelligence DIE

[MrNemesis]

Oxymoronic.

Re:This benchmark is pointless

[Anonymous Coward]

I am a scholar and study parallel computing.

aka I'm a second year computer science student.

No, give the guy a break... English is not his first language. You can tell from the "what your compiler is doing in your back", instead of "behind your back", that sort of thing. From timezone, European seems most likely... from the sentence structure... French?

Re:Four-hour compile times means a 1 day turnaroun

[namgge]

six release cycles a day is probably why you have bugs in the first place...