Speed Test: Comparing Intel C++, GNU C++, and LLVM Clang Compilers

Nerval's Lobster writes "Benchmarking is a tricky business: a valid benchmarking tries to remove all extraneous variables in order to get an accurate measurement, a process that's often problematic: sometimes it's nearly impossible to remove all outside influences, and often the process of taking the measurement can skew the results. In deciding to compare three compilers (the Intel C++ compiler, the GNU C++ compiler (g++), and the LLVM clang compiler), developer and editor Jeff Cogswell takes a number of 'real world' factors into account, such as how each compiler deals with templates, and comes to certain conclusions. 'It's interesting that the code built with the g++ compiler performed the best in most cases, although the clang compiler proved to be the fastest in terms of compilation time,' he writes. 'But I wasn't able to test much regarding the parallel processing with clang, since its Cilk Plus extension aren't quite ready, and the Threading Building Blocks team hasn't ported it yet.' Follow his work and see if you agree, and suggest where he can go from here."

Comments

first post

[Anonymous Coward]

first post

Re:first post

[Anonymous Coward]

compiled with clang

Re:first post

[Anonymous Coward]

Mod this up please.

Re:first post

[sce7mjm]

first post++

Re:first post

[Anonymous Coward]

fuck you, and fuck the GP

Re:first post

[Anonymous Coward]

man, it took a long time to read it.

Re:first post

[Mitchell314]

Why, so we can have more first posters?

Re:first post

[neokushan]

first ++pre

Re:first post

[Anonymous Coward]

You're that retarded?

Re:first post

[TheNastyInThePasty]

int FirstPost(int a, int b)
{
   if(a < b)
       printf("I got first post!");
   else
      printf("No, I got first post!");
}

int main(int argc, const char** argv)
{
   int i = 0;

   // What prints out here?
   FirstPost(i++, i++);
}

Re:first post

[mark-t]

Assuming typical C calling convention.... "No, I got first post" will be printed, where a will be 1 and b will be 0 in the call to FirstPost. This is because generally, final arguments are evaluated and pushed onto the stack before earlier ones.

Although typically, the standard may say this behavior is undefined, in practice, almost all modern C compilers will produce the output I've described here.

Re:first post

[Anonymous Coward]

The result may be nasal demons.

Re:first post

[Anonymous Coward]

Actually, it's not undefined behavior. It's unspecified behavior.

Re:first post

[Anonymous Coward]

It will not print anything. I compile with -Wall -Werror and get:

x.C: In function ‘int main(int, const char**)’:
x.C:17:23: error: operation on ‘i’ may be undefined [-Werror=sequence-point]
        FirstPost(i++, i++);
                                              ^
cc1plus: all warnings being treated as errors

Re:first post

[EvanED]

If it were just up to the order of evaluation of the function arguments, then it would be unspecified. However, the program also modifies the same object twice without an intervening sequence point, and that puts it into undefined behavior territory (6.5/2, C99 draft standard).

Re:first post

[Anonymous Coward]

hello world

Re:first post

[elbonia]

Clang will just issue a warning that you are making multiple unsequenced modifications. This is undefined in the C spec and the compiler just increments i sequently printing "I got first post!." Sequence points like this are hard to clarify for all cases which is why the C99 spec leaves it undefined. In C11 a detailed memory model has been created which should define most cases. http://en.wikipedia.org/wiki/C11_(C_standard_revision)

Confirmed with:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

Re:first post

[Anonymous Coward]

Why, so we can have more first posters?

Based on my analysis of 1,000 random /. stories I have found that the average number of first posts per story is exactly 1. There's no reason to spread this FUD about more first posters.

Re:first post

[seebs]

Pretty sure there is no intention whatsoever of turning that into defined behavior.

Re:first post

[seebs]

I am not at all convinced about this "almost all modern C compilers", given how many will do fairly awesome things once they determine that the behavior is undefined.

Re:first post

[smash]

Turn warnings off...

Re:first post

[Anonymous Coward]

Both happen to be the same person.

Re:first post

[Anonymous Coward]

I am no expert at C; I would have guessed the i++ occurs twice at the location being the semicolon after FirstPost(i++, i++).
Isn't the ++ operator supposed to occur "after" the statement has completed?

Re: first post

[Anonymous Coward]

nethack?

Re:first post

[Anonymous Coward]

No. The evaluation of each argument is a separate expression/sequence point, thus there is no undefined behavior. It is unspecified.

Re:first post

[ZorroXXX]

No, the ++ operation will take place before the next sequence point (super important concept! If you do not fully grok sequence points, you are not really programming C). The end of a statement is one sequence point, a function call is another sequence point.

Here you have two modifications to i before that, and that is what is invoking undefined behaviour (in the same way i = array[i++]; is also undefined behaviour since i is modified twice before the end of the statement).

Re:first post

[EvanED]

Check Annex C: there is no sequence point between evaluation of arguments, only after evaluation of all of the arguments is complete. (Note that the comma separating arguments aren't a comma operator.)

In addition, the standard explicitly states that it is not necessary to completely evaluate one argument before moving onto the next: "The order of evaluation of the function designator, the actual arguments, and subexpressions within the actual arguments is unspeciïed, but there is a sequence point before the actual call" (6.5.2.2/10 of C99 draft). This means, for example, that f(g1() + g2(), g3() + g4()) could be evaluated by calling the g# functions in any order (as each is a subexpression within the actual arguments), and if those functions produced side effects then that would be a counterexample to your claim that there is a sequence point between arguments.

Re:first post

[mark-t]

Actually, you don't strictly need sequence points to determine the order of events, sometimes they can be guaranteed simply by the defined order of operations, even if they result in side effects.

Consider the statement x = a[i++] + b[i++].. This should be equivalent to temp = a[i++], temp = temp + b[i++], x = temp, because the the order in which the + operator evaluates its operands is determined by the standard. Even without sequencing points.

But the initial example, where one passes in an argument to a function that is also being modified more than once technically isn't defined by the standard at all since the standard makes no guarantee about the order of evaluation of function arguments.

Nonetheless, one will still find that in practice, most C compilers will always evaluate the last argument first to a function first (including its side effects), as long as the compiler is configured to utilize the C calling convention for construction of stack frames (cdecl, in many compilers). If configured differently, it can, of course, have different behaviour.

Re:first post

[EvanED]

Actually I don't think my "counterexample" argument holds, because there would actually still be a sequence point at each call. You could change it to:

volitile int x1, x2, x3, xy;
f( (x1=1) + (x2 = 1), (x3 = 1) + (x4 = 1) );

and the order in which the writes to the xs occur could be any, including, for example, x1, x3, x2, x4.

Re:first post

[Anonymous Coward]

It prints "EXTERMINATE"

Re:first post

[ignavus]

Just tried this with TCC (Fabrice Bellard's Tiny C Compiler) and GCC:

tcc: I got first post!

gcc: No, I got first post!

Funny benchmarks

[Anonymous Coward]

The benchmarks in TFA are a little funny. Why is system time so large while user time so small? The only time I've seen this in real applications is when there is major core contention for resources.

Re:Funny benchmarks

[Trepidity]

This looks like it's testing the compile-time, in which case a large % of time being system time isn't that uncommon. Lots of opening and closing small files, generating temp files, general banging on the filesystem. Can heavily depend on your storage speed: compiling the Linux kernel on an SSD is much faster than on spinning rust.

Re: Funny benchmarks

[Anonymous Coward]

No compiler version numbers provided ? G++ versions differ wildly even in small increments. One needs to know the version of the benchmarked software for a fair comparison.

Re:Funny benchmarks

[godrik]

that's normal. There is hyperthreading on that machine, it screws up that kind of measurement. You should always use wall-clock time when dealing with parallel codes. You should also repeat the test multiple times and discard the first few results which the author did not do. It is very standard in parallel programming benchmark. And since the author did not do that, I assume he does not know much about benchmarking. Lots of parallel middleware have high initialization overhead. This tends to be particularly true for intel tools.

Re:Funny benchmarks

[robthebloke]

I agree. He's not testing compiled code performance, he's just created a set of tests which will all be memory bandwidth limited. FTA:

I’m testing these with an array size of two billion.

That's all I needed to read to ignore him completely. Completely and utterly pointless. If g++ won, it is likely because it utilised stream intrinsics to avoid writing data back to the CPU cache, which would have freed up more cache, and minimised the number of page faults. This will not in anyway test the performance of the CPU code, it will just prove that your 1333Mhz memory is slower than your 3Ghz processor . This is why you don't profile code (wrapped up in a stupid for loop), but profile whole applications instead. From my own tests (measuring the performance of large scale applications using real world data sets), intel > clang > g++ (although the difference between them is shrinking). The author of the article hasn't got a clue what he's doing. FTA:

Notice the system time is higher than the elapsed time. That’s because we’re dealing with multiple cores.

No it isn't. It's because your CPU is sat idle whilst it waits for something to do.

Re: Funny benchmarks

[coats]

Nor what flags he means by "full optimization". Nor what the hardware really is - for some problems, with the right flags Haswell performance is very different from Core-2.

Re:Funny benchmarks

[Anonymous Coward]

Here's another tipoff that the guy is clueless about benchmarking, talking about a test which does FP math:

I’m not initializing the arrays, and that’s okay

Actually, it's not. This is a bad mistake which totally invalidates the data. Many FPUs have variable execution time depending on input data. There is often a large penalty for computations involving denormalized numbers. If uninitialized data arrays happen to be different across different compilers (and they might well be), execution time can vary quite a lot for reasons completely unrelated to compiled code quality.

It's not limited to FP, either. I remember at least one PowerPC CPU which had variable execution time for integer multiplies -- the multiplier could detect "early out" conditions when one of the operands was a small number, allowing it to shave a cycle or two off the execution time.

The moral of the story: making sure that input data for benchmarks is always the same is very important, even when it's trivially obvious that the code will execute the exact same instruction count for any data set.

Re:Funny benchmarks

[Anonymous Coward]

uh... if the CPU sat idle whilst it waits for something to do, wouldn't the elapse time be higher than the system time?

you're an idiot.

Re:Funny benchmarks

[robthebloke]

Last time I checked, 11 minutes was greater than 250 seconds. ('%e' is the default format for user/system time [seconds], '%E' is the format time for elapsed time [minutes]. The different formatting should have been a big hint).

Re:Funny benchmarks

[robthebloke]

Although the fact that elapsed time is in minutes whilst the other times are in seconds, might have something to do with it....

Re:Funny benchmarks

[Anonymous Coward]

Here's another tipoff that the guy is clueless about benchmarking

I hear Mikey over at Moronix is hiring. Seems he's making out big on the click bait lately.

Re:Funny benchmarks

[Anonymous Coward]

There is another rather profound issue here. The author is using Intel processors, this greatly helps Intel's compiler. Intel's compiler tends to check the cpuid information for the string "GenuineIntel", rather than the feature flags for whether particular processor features are available. The consequence is both Intel's compiler and code generated by Intel's compiler runs much faster on Intel hardware, but has terrible performance on other processors. The numbers I recall from the article by Tom's Hardware where they had a Cyrix processor where the BIOS could tell the processor to report fake cpuid information is: cpuid reporting "GenuineIntel", performance increased by 33%; cpuid reporting "AuthenticAMD", performance increased by 20%.

Re:Funny benchmarks

[godrik]

From my own tests (measuring the performance of large scale applications using real world data sets), intel > clang > g++ (although the difference between them is shrinking).

I made lots of expeirments in this area as well. And my overall conclusions was that the intel compiler, the pgi compiler and GCC are all good compilers. But their performance vary significantly depending on what you are compiling. For some other applications you would get different results. Or that within your application some set of functions are more efficiently compiled by the intel compiler while other are better compiled by GCC. It is really difficult to say which is "the best" compiler in term of performance.

Re:Funny benchmarks

[godrik]

Let alone the fact that on linux an memory page is allocated the first time (ans so placed in memory) it is touched. So if you do not intialize it, the allocation and placement happens during the traversal, which is probably not things you want to time.

Re:Funny benchmarks

[Raenex]

Phoronix actually benchmarks applications. They also keep track of stuff like GPU support on Linux. Is there anybody that comes close to filling this Linux niche? I don't understand all the hate.

Compile time is irrelevant.

[Anonymous Coward]

Which one produced the fastest code?

Re:Compile time is irrelevant.

[0123456]

Which one produced the fastest code?

My current project takes two hours to compile from scratch, and uses around 20% CPU when it runs. So yes, compile time can be more important than how fast the code runs.

Re:Compile time is irrelevant.

[Anonymous Coward]

g++.
It's right in TFS.

Re:Compile time is irrelevant.

[Anonymous Coward]

'It's interesting that the code built with the g++ compiler performed the best in most cases,

are you too stupid to read?

Re:Compile time is irrelevant.

[Anonymous Coward]

g++.
It's right in TFS.

Without knowing the margin over the others, it's meaningless.

Re:Compile time is irrelevant.

[ledow]

But over the lifetime of any average, runtime should outweigh compile time by orders of magnitude.

Otherwise, honestly, why bother to write the program efficiently at all?

And if you want to decrease compile times, it's easy - throw more machines and more power at the job. If you want to decrease runtime, then ALL of your users have to do that.

Honestly, if your compile times are that much, and that much of a burden, you need to upgrade, and you also need to modularise your code more. The fact is that most of that compile time isn't actually needed for 90% of compiles unless your code is very crap.

Re:Compile time is irrelevant.

[Anonymous Coward]

Not really. Once you use LLVM in a JIT environment, you care.

Re: Compile time is irrelevant.

[Anonymous Coward]

Visual C++ 2013, but they were too scared to include it in their tests.

Re:Compile time is irrelevant.

[ShanghaiBill]

Which one produced the fastest code?

It doesn't matter. It may matter which one compiles your code faster. Depending on your use of things like templates, classes, etc. that may be a different compiler than the best for the benchmarks. But even that is unlikely to matter much. I doubt if their is much more than a few percentage difference. More important are issues like standard compliance, good warning messages, tool-chain/IDE integration, etc.

Re:Compile time is irrelevant.

[Anonymous Coward]

But over the lifetime of any average, runtime should outweigh compile time by orders of magnitude.

Which is irrelevant for a application that is sitting idle for 80% of the time.

Honestly, if your compile times are that much, and that much of a burden, you need to upgrade, and you also need to modularise your code more. The fact is that most of that compile time isn't actually needed for 90% of compiles unless your code is very crap.

And if he does all that, the faster compiler still wins.

Re:Compile time is irrelevant.

[marcello_dl]

The time spent on running code vs compiling code, for me, is like 10000:1, to be optimistic. Compilation time is pretty irrelevant for me and I daresay most users.

Re: Compile time is irrelevant.

[Anonymous Coward]

They only tested on a Linux box... they even say the reason they don't test TBB with clang is because they couldn't find a Mac.

Re:Compile time is irrelevant.

[rfolkker]

I have worked on projects that have taken upwards of 8 hours for a full compile. There is a lot of validity behind the business impact of different compilers.

The current mentality of throw more horse power at a problem is not always the practical, or the logical conclusion. If you can improve your overall compile time, it can improve your productive time.

From a Build Engineering perspective, analyzing why it takes time for a project to compile is one of the most important metrics.

Not only do I monitor how long a project takes to compile, but I also keep an active average, and try to maintain highs and lows to identify compile spikes.

We monitor processor(s), disk access speeds, memory loads, build warnings, change size, concurred builds, etc.

We look at all possible solutions. With the current build tools we have, we can either provision another build system for the queue, or if necessary increase memory, or disk space, or faster drives, more processors, or even upgraded software. We have gone as far as home-grown fixes to get around issues until better solutions become available.

All of this needs to be accounted for, so, not only is compile time relevant, but what is CAUSING compile times is relevant.

Re:Compile time is irrelevant.

[0123456]

Honestly, if your compile times are that much, and that much of a burden, you need to upgrade, and you also need to modularise your code more. The fact is that most of that compile time isn't actually needed for 90% of compiles unless your code is very crap.

Hint: I said 'two hours to compile from scratch'. You can't avoid compiling all your source if you just did a clean checkout from SVN into an empty source tree; as you would, for example, before building a release or release candidate.

Re:Compile time is irrelevant.

[BasilBrush]

Compilation time is pretty irrelevant for me and I daresay most users.

Compilers are selected by developers, not users.

Re:Compile time is irrelevant.

[non0score]

Or maybe people should just stop using so much templates, and use code generation instead. Oh, and also supporting the Apple/LLVM initiative to modularize C++ would be double plus good too (think #import in C++).

Re:Compile time is irrelevant.

[Anonymous Coward]

I have plenty of code that takes GCC 4.7 over 8 hours for a single C-file, where GCC 4.4 or clang can compile it in just over 2 minutes. And what's worse: GCC 4.4 produces faster code than GCC 4.7 in this case. The C++ rewrite of GCC is pure crap.

I also have plenty of generated code that is typically compiled and executed only once or twice. Here compile-time is king.

Re:Compile time is irrelevant.

[Anonymous Coward]

Oh yeah, forgot to mention. That's gcc -O0 taking 8 hours for GCC 4.7. It is the front-end that got slower, not the optimizer... So the developers don't care.

Re:Compile time is irrelevant.

[TheGavster]

While any user-facing application is going to spend most of its time waiting for the user to do something, the latency to finish that task is still something the user will want to see optimized. Further, if a long-running task tops out at 20% CPU, apparently optimization was weighted too much towards CPU and you need to look into optimizing your IO or memory usage.

Re:Compile time is irrelevant.

[Gothmolly]

You're IO bound. Get a real disk subsystem.

Re:Compile time is irrelevant.

[Anonymous Coward]

Eight hours? Try a week.
Grab this book and fix your build: Large-Scale C++ Software Design by John Lakos

Re:Compile time is irrelevant.

[Anonymous Coward]

Who is your customer?

If you're working on a worthwhile project you will have %BIGNUMs of customers (or at least %BIGNUMs of customer hours per day). You have to compile each build once. Your customers will see the benefit of faster code all day, every day. So unless you're working in a very low-competition niche, improving the efficiency of your customers should be a plus to your marketability. See how hard and fast the droid manufacturers play with benchmark rules just to snatch a percent or so. So compile time is important (and yes, the workarounds, architectural solutions etc. you mention are valid but ultimately speed of compiled code should take priority over speed to compile the code.

Re:Compile time is irrelevant.

[Anonymous Coward]

Developers might scratch their own itch, but >=99% of them want others to use their software and thereby feel good about themselves. Faster execution time will impact on potential users. Therefore developers should pay more attention to that than to compile time. Few users (gentoo exempt!) care about compile times.

Re:Compile time is irrelevant.

[ebno-10db]

Lipstick on a pig.

Re:Compile time is irrelevant.

[F.Ultra]

A week? Try Gentoo.

Re:Compile time is irrelevant.

[adisakp]

Faster compile times make for faster iteration... which lets you test global changes for examples - which optimizations actually work - more easily. Not to mention that having better iteration on a program usually produces a superior product.

Also, as a developer, faster compile times make my life a little less frustrating so I'll be less likely to pull out all my hair while waiting on the computer.

Re:Compile time is irrelevant.

[Anonymous Coward]

But over the lifetime of any average, runtime should outweigh compile time by orders of magnitude.

This is not +5 insightful thinking. Counterintuitively, for large projects, a very fast compiler with middling optimization will often produce superior runtimes compared to a slower compiler with high quality optimization. A programmer able to do a lot more compile-test cycles in the same amount of time is one who effectively enjoys more time to prototype, test, and debug large-scale algorithmic optimizations. These tend to absolutely kill the kind of micro-optimizations compilers are good at.

Consider a reductio ad absurdum: would anybody use a compiler which compiles 1000x slower to deliver 10x better runtimes than the next best compiler? Such a compiler actually would find a niche, IMO, but it would be just a niche: large projects simply wouldn't be possible to develop if they relied exclusively on that one compiler. Provided that there is object file / linker compatibility, many projects would use it for a handful of performance-critical functions which seldom need recompilation, relying on a faster compiler to compile the rest of the project.

Real world projects have to deal with finite time to write and debug code. Ignoring this and insisting on a single figure of merit (optimization quality) may produce poor decisions.

Re:Compile time is irrelevant.

[wiredlogic]

An excessively long build time can inflate development costs if the delay in testing new code becomes prohibitively long. A large codebase that takes 4 hours to build on a slow compiler will force developers to frequently wait over night for test results to come back. If a different compiler can build that code 4x faster you have many more opportunities to observe test results during a work day. Upgrading the build system isn't always an option when you have to support legacy platforms with inherently slow hardware and cross compiling isn't an option.

Re:Compile time is irrelevant.

[flargleblarg]

A single C source code file? 8 hours? How many lines? Even 8 minutes seems off by several order of magnitude.

Re:Compile time is irrelevant.

[Aighearach]

No, in most cases execution time will vary much less than compile time.

Re:Compile time is irrelevant.

[Zero__Kelvin]

Yes, because whenever I'm running an application that eats 80% of my CPU time doing something trivial I think to myself" "It's no big deal really. I bet it compiled like a bat out of hell!"

Re:Compile time is irrelevant.

[Zero__Kelvin]

"Which is irrelevant for a application that is sitting idle for 80% of the time."

When you stop using DOS it will become quite relevant actually.

Re:Compile time is irrelevant.

[Zero__Kelvin]

Lets not bring your Mother into the conversation.

Re:Compile time is irrelevant.

[VortexCortex]

Which one produced the fastest code?

My current project takes two hours to compile from scratch, and uses around 20% CPU when it runs. So yes, compile time can be more important than how fast the code runs.

I had a C++ project like that once... It was a tightly coupled scripting language that could be compiled down to the host language if parts needed to be sped up. I noticed that I was mostly avoiding C++ features since they didn't exist ( eg: using multiple inheritance with non-pure virtual base classes -- Which the scripting lang allowed by allowing variables to be virtual ) and implementing them in C instead. So, I ditched C++ and coded to C99 instead. When I got all the C++ out of the codebase (thus making it compilable in either) the compile time dropped from an hour and a half in C++ to 15 minutes in C. Since I absolutely must have scripting and the VM lang optionally allows GC transparently across C or script (by replacing malloc and friends), and it has more flexible OOP (entities can change "state" and thus remap method behaviors (function pointers) making large improvements over jump-tables (switch statements) for my typically highly stateful code: I avoid C++ like the plague.

In fact, since the scripting language can translate itself into C, I don't touch C much either for my projects unless I'm extending the language itself. Over the years I've ported the "compiled" output to support Java and Perl, and Javascript (and am working on targeting ASM.js). It's grueling work just for my games and OS hobby projects, but I really can't bring myself to use a compilable language that doesn't double as its own scripting language -- That's asinine, especially if it compiles an order of magnitude slower.

Don't get me wrong, I get the utility of a general purpose OOP language built around the most general purpose use cases possible; However when you design something for everyone, you've actually designed it for no one at all. I'll take a language with full language features applicable to its templating (code generation), like my scripting language (or Lisp) over C++ or Java any day. (Note: Rhino or Lua JIT + Java is a close contender as far as usability goes, but nothing beats native compiled code for my applications' use case.)

WRT to the "insightful" commenter above: Deadlines are far more important to code being able to run than the distributed minute performance gains on end user systems which are influenced by moore's law. Release date is far more significant: The code has 0% usability if I can't produce it in time. Unfortunately, some project depend on emergent behaviors and thus require fast revisions to tweak (this goes doubly for me, hence the scripting component requirement).

Re:Compile time is irrelevant.

[Anonymous Coward]

If it spent more time running the code than compiling then you'd use an interpreted language.

Re:Compile time is irrelevant.

[smash]

However, faster compile time means faster development and debugging.

Re:Compile time is irrelevant.

[jandrese]

My current project takes two hours to compile from scratch,

Ah, another satisifed Boost user I see.

Re:Compile time is irrelevant.

[Anonymous Coward]

I have worked on projects that have taken upwards of 8 hours for a full compile. There is a lot of validity behind the business impact of different compilers.

I have done that too, then we moved all the dependencies to a company wide cache and replaced most hard drives with SSDs, the compile now takes max 10 minutes - the first time, everything after that is a few seconds at most. If possible make your code modular, if that is not possible something is likely wrong with the overall design.

Re:Compile time is irrelevant.

[TheManInTheMoon]

You obviously haven't programmed using Visual Studio 2005 or later! ;)

Re:Compile time is irrelevant.

[semi-extrinsic]

I have to say, if we ignore Firefox/Linux/other ridiculously huge codes, the people having hours of compile time must be doing something wrong. The 30 000 line Fortran code I'm working on now takes

$ time make clean optim_ifort
...
make clean optim_ifort 50.16s user 0.86s system 98% cpu 51.731 total

in serial, and it's roughly 3x faster for the parallell cmake build. This is with -O3 and inter-procedural optimizations turned on, generating AVX-tuned code. If I have only edited a file or two, the compilation takes less time than a sip of coffee.

Re:Compile time is irrelevant.

[TheRaven64]

Any chance of posting the code somewhere? I can well believe 8 hours for a C++ source file (technically, parsing C++ is a non-computable problem in the general case and there are some nice examples of fairly small C++ files where parsing and template instantiation will never terminate in a fully correct compiler, although in practice you'll hit stack limits), but for C it sounds quite improbable because parsing C is just not that complicated. The only thing I can think of is that you have a lot of nested macros (for accurate error reporting, you keep track of all macro instantiations so you can give helpful diagnostics. GCC 4.4 didn't do this), or lots of deeply nested scopes so that symbol resolutions are very slow. Or possibly a mixture of the two.

Re:Compile time is irrelevant.

[TheRaven64]

30,000 lines of code is a tiny project. I have codebases I wrote myself that are larger. Anything developed by a team is likely to be at least an order of magnitude larger. You're also comparing Fortran to C++, so you get a much faster compile because Fortran doesn't encourage large compile-time code generation in the way C++ templates do, which make parsing very slow, and makes alias analysis trivial, which makes a lot of optimisations easier.

Re:Compile time is irrelevant.

[TheRaven64]

You're assuming that you'll use the same environment for debug and release builds. It's fairly common to use -O0 or -O1 for debug builds so that you get the fast turn-around, but -O2 or -O3 for release builds so you get the faster execution. It doesn't matter if the release build takes a lot longer to compile, because it's not going to be part of an iterative compile-test-debug cycle (at least, not for many iterations). There are some exceptions. For example, Apple strongly discourages developers (kernel developers, at least, not sure about the rest) from running code with debugging instrumentation enabled because it makes it easy for them to blame their build for slowdowns and not fix them, and by the time you get close to release everyone's introduced a few small performance regressions that all add up.

Re:Compile time is irrelevant.

[Bert64]

As a gentoo user i don't particularly care about compile times either, i have both ccache and distcc set up to speed up compilation, i build binary packages where i have a large number of identical machines. I can always let compiles run over night and have them ready by the morning, and even that isn't usually necessary with modern hardware.

On the other hand, having the resulting binaries run faster and use less memory either because of better compiler options, or more efficient compile time flags (eg disabling unwanted features) is a fairly significant benefit. When you have a large number of vm images running, small gains in each one soon add up.

Re:Compile time is irrelevant.

[Rockoon]

sigh...

His 20% is 1 of 6 cores (16.666%) plus a little bit.

Doesnt seem like anyone here even has a cursory understanding...

Re:Compile time is irrelevant.

[Anonymous Coward]

Which one produced the fastest code?

It doesn't matter. It may matter which one compiles your code faster. Depending on your use of things like templates, classes, etc. that may be a different compiler than the best for the benchmarks. But even that is unlikely to matter much. I doubt if their is much more than a few percentage difference. More important are issues like standard compliance, good warning messages, tool-chain/IDE integration, etc.

Clang wins as far as detailed varnings.
Clang wins as far as speed is concerned (and by way more than a few percent)
tool-chain/IDE integration gcc wins, but clang has wrappers to emulate most functionality...

Re:Compile time is irrelevant.

[AmiMoJo]

How often is the compiler the bottleneck? It seems more likely that optimizing the build process and hardware (particularly by adding more RAM and an SSD) would have a much, much greater effect than switching compiler and at no expense to your run-time performance.

Re:Compile time is irrelevant.

[Anonymous Coward]

That's how long it takes to compile the full system from kernel to libreoffice and firefox, including glibc twice (before and after GCC). On 5 year old HW.

Re:Compile time is irrelevant.

[Anonymous Coward]

A 'Gentoo' is a unit of measurement somewhat counter-intuitively for measuring processor load, as Gentoo is always (re)compiling.

Re:Compile time is irrelevant.

[Anonymous Coward]

Where did the GGP say anything about the number of cores in his compile?

Re:Compile time is irrelevant.

[adisakp]

30K lines is pretty small. It's smaller than a single library in our code base. For example, both our memory system and our network layers are significantly larger than this and they are just support libraries.

Re:Compile time is irrelevant.

[Anonymous Coward]

But over the lifetime of any average, runtime should outweigh compile time by orders of magnitude.

That depends. In the most extreme case, you write a program that is run exactly once: To get a specific result. A second run would give the exact same result and thus be pointless.

Now I don't think I've written a program (apart from very simple tests) that I've run just once (all my programs have parameters). However I did write programs where I can count the number of runs per compile on my hands.

Re:Compile time is irrelevant.

[Anonymous Coward]

30K lines is pretty small. It's smaller than a single library in our code base. For example, both our memory system and our network layers are significantly larger than this and they are just support libraries.

time to refactor

Re:Compile time is irrelevant.

[david_thornley]

Sometimes the debug and release builds will do different things, typically due to undetected undefined behavior. Usually, once you can localize the problem, you can fix it, but you can't do that with a debug build.

Re:Compile time is irrelevant.

[Rockoon]

Where did the GGP say anything about the number of cores in his compile?

He didn't, because he doesnt know what information to include, because he doesnt know why his cpu usage was so low, because he doesnt know how to enable multi core compiling, because he doesn't know much about what he is talking about.

Re:Compile time is irrelevant.

[jgrahn]

Honestly, if your compile times are that much, and that much of a burden, you need to upgrade, and you also need to modularise your code more. The fact is that most of that compile time isn't actually needed for 90% of compiles unless your code is very crap.

Hint: I said 'two hours to compile from scratch'. You can't avoid compiling all your source if you just did a clean checkout from SVN into an empty source tree; as you would, for example, before building a release or release candidate.

That, to me, sounds like "I don't trust my build system to do an incremental build". A lot of people don't, and end up spending $$$$ on building provably the same thing over and over again, dozens of times a day ... Stop doing that, and you can use a compiler which is ten times slower and still have shorter build times.

BTW, Git does the right thing WRT timestamps and Makefiles: when it changes a real file, it updates the timestamp to the current time, not the time the version was created. Now that I've used it, I don't understand why tools made the wrong choice for thirty years.

How about benchmarking the binary?

[Anonymous Coward]

How about benchmarking the binary? Isn't what it matters anyway? Let's us see compiler benchmarks and the benchmarks of the compiled binary itself.

Re:How about benchmarking the binary?

[adisakp]

You obviously don't work on large projects where build times can be 30 minutes and link times can be 5-10 minutes on top of that. In the past we have tried just about everything possible to make our compiles faster because it allows more iteration and less time waiting on code building. This include minimize include dependencies and looking at dependency graphs, benchmarking distributed build systems (incredibuild), working with pre-compiled headers, examining unity-builds / unified builds (think one CPP that includes many other CPP's in the same system), etc. We also buy fast hardware (8 core CPU's with 16 threads), 32 GB Memory, and fast SSD's. All because minimizing build time is means more productive time for developers.

Re:How about benchmarking the binary?

[petteyg359]

Can you read?

It’s interesting that the code built with the g++ compiler performed the best in most cases, although the clang compiler proved to be the fastest in terms of compilation time.

Just in case you didn't get that: They did benchmark the resulting binaries, and g++ made the best ones.

Re:How about benchmarking the binary?

[Anonymous Coward]

They did benchmark the resulting binaries, and g++ made the best ones.

Not always. In some tests it made the slowest.

Re:How about benchmarking the binary?

[TapeCutter]

Compiling SOAP on Windows or Linux takes about 20min on a well managed VM with respectable grunt. The couple of dozen other binaries that go with our application take about half that time to build in total. SOAP is not even the largest of the component source trees we have, but from the compiler's POV it's certainly takes the most effort.

Re:How about benchmarking the binary?

[flargleblarg]

What's the turnaround time if you change, say, one tiny part of a C function having no ramifications to other modules? Do you have a 1-second recompile time (just for that module) followed by 5-10 minutes of link time before you can re-test? Is there no incremental linking? No dynamic libraries? I'm curious what type of program you have. That seems excessively slow to me.

Re:How about benchmarking the binary?

[cheesybagel]

Don't use templates. Period.

Re:How about benchmarking the binary?

[adisakp]

Moderate template use is fine. FWIW, Unity/Unified Builds seem to help a lot with compile time on template heavy code.

Re:How about benchmarking the binary?

[petteyg359]

So, if it is fails to be best in one case, it is therefore suboptimal in all other cases? Guess we should un-launch all the satellites since a few of them were damaged on the ground, and tell the Mars rover to power down, since other Mars missions have had problems.

Re:How about benchmarking the binary?

[aaaaaaargh!]

I'm skeptical about that. Wasn't Intel's compiler supposed to produce the fastest binaries - on Intel machines, at least?

Re:How about benchmarking the binary?

[Anonymous Coward]

wow, read all that from nine words?

-another ac

Re:How about benchmarking the binary?

[adisakp]

We are targetting custom / closed-wall systems. Single binary EXE for user systems - No user level DLL's or external binary loading allowed (for security purposes). Incremental linking not allowed on retail or profile builds (it's a compiler level hack that potentially adds a thunk per function and at the very least adds one per function moved and also significantly changes the memory footprint for code memory layout from build to build on the functions you are modifying). Additionally, we use link pass optimizations (i.e. link-time inlining) on those builds on certain platforms which on retail and profile builds which require incremental linking to be disabled.

In the best case in the above mentioned builds, we do have about a 15 second compile time followed by the full link time.

But even without these limits, it very easy to find programs where a single change in a common header causes a full recompile and non-cremental link times on all large projects (especially ones with a lot of redundant template functions per translation unit) can grow significantly.

DIe Buisness Intelligence DIE

[Bill, Shooter of Bul]

What on earth does compiler benchmarking have to do with the BI section of slashdot?

Furthermore, why on earth are you idiots creating a blurb on the main screen that just links to a different slashdot article? Its such terrible self promotion. Just freaking write the main article as the main article. No need to make it seem as if the Buisness Intellegence section is actually worth reading, its not.

Re:DIe Buisness Intelligence DIE

[Anonymous Coward]

What's the Business Intelligence section of /.?

Re:DIe Buisness Intelligence DIE

[MrNemesis]

Oxymoronic.

Re:DIe Buisness Intelligence DIE

[Anonymous Coward]

The sad part is, I can easily see managers asking why compilation is taking so long and then suggest moving an entire project to a different compiler to save a few seconds each time. Of course, they have no idea what exactly that would take to implement, but then again such analysis is harder than barking orders based on sound bites.

YOU Are The Problem!

[Anonymous Coward]

Which one produced the fastest code?

My current project takes two hours to compile from scratch, and uses around 20% CPU when it runs. So yes, compile time can be more important than how fast the code runs.

You are the problem with most software today. Your utter selfishness or plain ignorance results in slow software that wastes countless time and offloads your temporary pain to possibly millions of users for years to come.

Great! You saved 20 to 40 hours on the project. But, because you put your needs ahead of the end user's needs your compiled code may be slow, or even extremely slow, because you chose the wrong compiler. Now, your software will run for thousands or millions of man hours, wasting FAR more that the 20 or 40 hours that you save for yourself.

This is the same sort of asshattery that seems to go on everywhere. 'RAM/Disk/CPU is cheap so I'm justified in making it bloated.' 'Why would any other application be running at the same time as my own? I can consume all resources for my bubble sort.'

Re:YOU Are The Problem!

[Anonymous Coward]

Speed up from algorithmic changes will far out perform anything an optimizing compiler can produce. A compiler that compiles faster lets you focus on algorithmic optimizations more than a slow compiling compiler.

Besides if it's spending 80% of the time idle, then the program is waiting for the user not the other way around.

Re:YOU Are The Problem!

[0123456]

Besides if it's spending 80% of the time idle, then the program is waiting for the user not the other way around.

Bingo. When the software is waiting for something to do 80% of the time, and nothing else of any importance is running on that machine, optimization is pretty much irrelevant; at best it would save a tiny amount of power by slightly reducing CPU usage.

Re:YOU Are The Problem!

[Bert64]

And how often will nothing else be running on the machine?
These days a large proportion of servers run under hypervisors, and that 80% of idle time will be used by other virtual machines running on the same physical hardware. If you make your code more efficient, then you can consolidate more functions onto the same hardware which could result in significant cost savings.

And even on single standalone machines, modern powersaving functions will mean that far less power is used during the 80% idle periods, and more efficient code could potentially result in 90% idle or more... This translates to lower power usage, and subsequently longer battery life.

Re:YOU Are The Problem!

[Urza9814]

Besides if it's spending 80% of the time idle, then the program is waiting for the user not the other way around.

Bingo. When the software is waiting for something to do 80% of the time, and nothing else of any importance is running on that machine, optimization is pretty much irrelevant; at best it would save a tiny amount of power by slightly reducing CPU usage.

Yeah, and my work laptop backup software is idle 80% of the time...except for those 4 hours every Friday when the disk utilization pegs to 100% and it starts taking several minutes just to switch to a different folder in Outlook, with no other programs open...and if I need to use SQL Developer or Access or something, it's gonna have to wait for Monday!

Nobody cares how much time your program sits there waiting for someone to push the button. They care about how quickly it reacts once you push that button. Just because it's sitting idle for most of the time doesn't mean your customers wouldn't greatly appreciate some optimization.

Measuring pebbles

[T.E.D.]

Interesting info, but I have a couple of issues:

First off, why wasn't Microsoft's C++ compiler included in this? That's the one we use at work, so that's the one I'd really like compared to all those others. Are we the only ones still using it or something?

More importantly, why on earth was compilation speed the only thing compared? I mean, I suppose its nice for g++ users to know that their 10 minute compiles would have been 2 minutes longer if they used the Intel compiler, but Intel users might not really care if they believe their resulting code is going to run faster. Speed of compilation of optimized code is a particularly useless metric, because different compilers have different definitions of "unoptimized", so its guaranteed you aren't comparing apples to apples.

I suppose compilation speed is a nice metric to brag about between compiler writers. But for compiler users, the most important things are roughly these, in order: Toolchain support, language feature support (eg: C++2012/14 features), clarity of error/warning messages, speed of generated code (optimization), and lastly speed of compilation. I'm not really sure why you took it upon yourself to measure the least important factor, and only that one.

Re:Measuring pebbles

[Anonymous Coward]

Your pre-elementary reading and comprehension skills leave much to be desired.

It’s interesting that the code built with the g++ compiler performed the best in most cases, although the clang compiler proved to be the fastest in terms of compilation time.

Just in case you didn't get that: They did benchmark the resulting binaries, and g++ made the best ones.

Re:Measuring pebbles

[Anonymous Coward]

Because nobody cares about windoze losers. Jeez!

Re:Measuring pebbles

[Gothmolly]

Mod parent up. GP is an asshat.

Re:Measuring pebbles

[T.E.D.]

OK. Much abashed, I went back through the article.

It turns out that there are numbers for an actual code benchmark. Its found about 2/3rds of the way through the report, in the third graph (untitled), after the balance of the text had already been devoted to compilation speed comparisons. Also, it only listed 2 of the 3 compilers, half the data was for unoptimized code (and thus useless), and it was hidden behind a sign that said "Beware of leopard".

OK, perhaps I made that last part up.

For the curious, the difference in at least that one table was never more than 5%. In my mind, hardly a differentiator unless you are doing heavy number crunching or stock trading programs.

Perhaps the remaining 1/3rd is all about more important things? I've lost interest. You're right. I'm weak.

Re:Measuring pebbles

[Anonymous Coward]

While I agree with the priorities you enumerate, I believe those priorities change at different development stages.

>>> Speed of compilation of optimized code is a particularly useless metric, because different compilers have different definitions of "unoptimized", so its guaranteed you aren't comparing apples to apples.

This could be the difference between an hour and 10 minutes for builds of some projects. Think of all the productivity lost waiting to debug a build.

Re:Measuring pebbles

[T.E.D.]

This could be the difference between an hour and 10 minutes for builds of some projects.

If that's really the delta (one is 600% slower), then something is likely seriously pathologically wrong with one of those two compilers. Submit a bug report (not that it helps you, but it will help someone else).

But yes, different users in different phases will have different priorities. I'm not laying down an immutable law here, just trying to restore the proper proportion to a situation that we both agree is way out of wack.

Re:Measuring pebbles

[CastrTroy]

Fast compilation can have it's advantages. It's one of the reasons some developers like working in scripting languages (PHP, Ruby, etc...). If simple mistakes in coding don't cost you 20 minutes of compile time, it can speed up development a lot. I use .Net, which i think has a nice balance between the two. Reasonable compile times, while still having compile time type checking and other advantages of a compiled language.

Re:Measuring pebbles

[Anonymous Coward]

>If simple mistakes in coding don't cost you 20 minutes of compile time, it can speed up development a lot.
Are incremental compile no longer used for development anymore?

Re:Measuring pebbles

[Anonymous Coward]

First off RTFA - resulting binaries were also bench marked.

Secondly, Microsoft's compiler doesn't run on any of the important development platforms so no one really cares how it runs.

Re:Measuring pebbles

[Anonymous Coward]

No, because Makefiles are too ugly. And by "ugly" I mean, people don't get declarative languages and fallback to brain-dead solutions.

Re:Measuring pebbles

[flargleblarg]

First off, why wasn't Microsoft's C++ compiler included in this?

Maybe because most people want Microsoft out of the picture and conveniently choose to ignore it rather than lend credence to it?

Re:Measuring pebbles

[Anonymous Coward]

You're as stupid as the Ms fanboy that dismisses free software for being free software.

Re:Measuring pebbles

[Zero__Kelvin]

"First off, why wasn't Microsoft's C++ compiler included in this?"

Because he was trying to keep things as similar as possible and you can't compare a proprietary compiler chock full of deviations from standard running on a platform written by a company with a history of dishonesty who makes both the OS and the compiler to compilers designed to run on a real Operating System and then trust the results. If you program for Windows M$C++ will always be the best, even if Microsoft has to put code in the OS distributed in an update that does the equivalent of if (exec_name == g++) { wait(1); } in an update.

Re:Measuring pebbles

[Anonymous Coward]

... (eg: C++2012/14 features) ...

It's C++11 -- saying C++2012 makes you sound silly

Re:Measuring pebbles

[Anonymous Coward]

History of dishonesty? Indeed. On the part of liars such as yourselves, supporters of the so-called "real Operating System". Please cite a single instance proof regarding your claim that Microsoft sabotages their operating system to prevent competing products from working.

Yes, I've heard "DOS isn't done until Lotus won't run". And having supported both DOS and Lotus 123 in the field, I know how that really worked. In short, it was bullshit, the same as "history of dishonesty" is now.

People like yourselves have been lying consistently for years. You won't stop. But all your lies will not make a good product bad. So keep the lies coming. They will matter as much as they always have.

Re:Measuring pebbles

[Rockoon]

There are types of optimizations that require every single source file (or at least a lower level proxy of them) loaded at the same time, and as such its quite common for large compiles to run out of memory even on systems with enormous amounts of memory. One implementation may handle the not-out-of-memory case better while another may handle the out-of-memory case better. This does not mean that either are pathologically wrong, but rather than the wrong tool was used for the job.

In GCC's case, there are two ways to enable this type of optimization and one of them is tailored to memory constraints while the other is not. I'm not sure about ICC, but given that TFA doesnt tell us any compiler flags at all we cannot presume that he isn't an even bigger idiot than he already appears to be (ex: using the options tailored for swapping out on one compiler, while not using that option on the others.)

And then the real kicker... Anyone seriously concerned about compilation speed wouldn't be using anything resembling the statement "full optimizations." -- The number of release builds should be very infrequent relative to the number of debug builds. A debug building benchmark might be an article worth reading even if it was done by this same practically worthless clown.

Re:Measuring pebbles

[Zero__Kelvin]

The Netscape fiasco. "But your honor, the web browser is an integral part of the OS, not an application!" Off you go now little troll.

Re:Measuring pebbles

[TangoMargarine]

http://en.wikipedia.org/wiki/Windows_3.1x#DR-DOS_compatibility

See also post below: "Internet Explorer is a crucial part of the OS! It just happens to be really convenient for smothering our competitors, too."

Oh, and while we're at it: http://redmondmag.com/articles/2013/08/22/windows-8-security-issues.aspx
So there are arguments for this, I'm sure, but it rankles me. A computer should be first and foremost under the control of its owner--in the case of PCs, the end user.

QED.

Re:Measuring pebbles

[toddestan]

What, as opposed to the Intel compiler?

Re:Measuring pebbles

[Zero__Kelvin]

What OS does Intel sell? I must have missed the announcement.

Re:Measuring pebbles

[TemporalBeing]

First off, why wasn't Microsoft's C++ compiler included in this? That's the one we use at work, so that's the one I'd really like compared to all those others. Are we the only ones still using it or something?

Probably because it doesn't run on Linux, and one of the few compilers that is not multi-platform.

future compiler trends

[sander]

The main claim for g++ for a very long time was "while it does not optimize much or support all of the language, it is FREE". With clang on the scene and offering a comparable feature set and speed of compiled code, it will be interesting to see how g++ and the gnu compiler collection in general will fare over time. Especially as a part of the canonical GNU core.

Re:future compiler trends

[ShanghaiBill]

The main claim for g++ for a very long time was "while it does not optimize much or support all of the language, it is FREE".

I have never heard that claim, maybe because it isn't true. g++ has always been one of the best at language support. It has not always been the best at low-level processor specific optimizations, but it has made up for that by being really good at higher level optimizations, like recognizing unused code, inlining, and code hoisting. I haven't seen a better compiler at any price.

Re:future compiler trends

[ebno-10db]

it has made up for that by being really good at higher level optimizations

Heh, heh, heh, don't remember the great EGCS split of '97, do you sonny? Yep, us old timers knew that gcc was a dog of an optimizer, but them EGCS whippersnappers fixed it, and even got the fork accepted as the official gcc. Remember, you probably got to where you are today by running over the body of some crusty old-timer.

Re:future compiler trends

[cheesybagel]

The main problem back then was X86 optimizations. Not the high level optimizations although it was lacking in those too. Eventually they started porting the code to use GIMPLE and moved most of the optimizations away from the language dependent trees to the GIMPLE language independent code. This was done before LLVM was even popular.

Re:future compiler trends

[ebno-10db]

No doubt that gcc is a damned good compiler these days, at least in terms of the quality of code produced, if not the speed at which the compiler runs. My point was just that it wasn't always so. Back then gcc was considered a toy compared to some of the commercial compilers, and it was. Thankfully the EGCS people did a lot to change that, and got the ball rolling for future improvements.

Re:future compiler trends

[Rockoon]

Back then GCC was competing with the likes of VC5/VC6, so its not as much of a knock on how bad GCC was back then.

Most C and C++ compilers were completely atrocious on x86 even as little as 15 years ago. "Optimization" meant a little bit of non-extensive peep-hole.

Re:future compiler trends

[Anonymous Coward]

"Old timer." You're funny. I'd sure I've got at least a decade on you (especially given your awfully big uid). And who cares of GCC was like in the past?

clang?

[Anonymous Coward]

I take it they never looked here? TBB works with clang... sort of.

Re:clang?

[Anonymous Coward]

Oh, I see... "Also, Threading Building Blocks hasn’t yet been ported to clang on Linux, and as such, clang wasn’t included in that part of the test. It has on the Mac, but I didn’t have access to a Mac for these tests."

Yes, your benchmarking test is vast and impressive and newsworthy... you're such an amazing developer... that you cannot find anyone from whom you can borrow a Mac for an hour.

Crappy benchmark

[raxx7]

The code in the benchmark runs a parallel for over a 10 billion element array but in steps of 100 elements.
It's going to be limited by the creation and destruction of threads.

Also, by not initializing the input array, the floating point arithmetic is vulnerable to eventual denormal values.

Re:Crappy benchmark

[Anonymous Coward]

By not initializing the input array, the code's behaviour is undefined. Which makes this a test of what these compilers do with complete garbage source that isn't even a valid C++ program.

other compilers

[Anonymous Coward]

don't forget the Microsoft Visual Express and Watcom C++ compilers. :D I must be the only person who uses Watcom C++ compiler.

Re:other compilers

[cheesybagel]

I thought that was something people used back when MS-DOS was a popular OS was not even aware the product still existed.

Re:other compilers

[cheesybagel]

I am talking about Watcom C++ of course.

Re:other compilers

[aiht]

I thought that was something people used back when MS-DOS was a popular OS was not even aware the product still existed.

I am talking about Watcom C++ of course.

It was open sourced some time ago. Now it supports Linux (to some extent) and some other CPU architectures.
It can still make DOS/4GW exes, though. Ahh, nostalgia.

Re:other compilers

[TemporalBeing]

I thought that was something people used back when MS-DOS was a popular OS was not even aware the product still existed.

I am talking about Watcom C++ of course.

It was open sourced some time ago. Now it supports Linux (to some extent) and some other CPU architectures. It can still make DOS/4GW exes, though. Ahh, nostalgia.

As someone that has maintained Watcom C/C++ code, the Watcom and OpenWatcom are slightly different and code needs porting from Watcom to OpenWatcom. How much I don't know...I just know that our code needed quite a bit of work to do that. Would have been nice if we did...but no one wanted to.

Link does not work in some browsers

[Anonymous Coward]

If you're using something like a privacy enhanced chromium, and you have referrer headers turned off, the link to the BI site endlessly reloads and never displays the article.

Speed is irrelevant

[Anonymous Coward]

My company (10,000+ developers, billions of lines of code across hundreds of products some of which everyone uses every day) dumped gcc as soon as LLVM was able to compile our code. We do not care for organizations like the FSF that try to dictate what we can and cannot do with open source so we choose to support projects that are MORE FREE than any bullshit GPL covered crap ever can be.

Re:Speed is irrelevant

[Anonymous Coward]

Why is a company with 10,000+ developers, with billions of lines of code across hundreds of products, some of which everyone uses every day, using free compilers in the first place, particularly if they're so willing to dump gcc as soon as LLVM appeared? Surely such a behemoth could have afforded to buy a different compiler that didn't come with such a prohibitive license as the GPL? Surely you're not just another of the bullshitters riddling Slashdot today!

Re:Speed is irrelevant

[Anonymous Coward]

You are an idiot.

Re:Speed is irrelevant

[Anonymous Coward]

NO U

Re:Speed is irrelevant

[cheesybagel]

More than likely your main gcc use was for Mac or iOS applications and you changed compiler because you can't even figure out how to change the defaults in XCode. Having tried both with my applications I can perfectly tell that clang is not up to snuff. Sure it compiles quickly and the syntax errors have color highlighting but the quality of the code, in terms of execution speed or size, it produces is vastly inferior.

Re:Speed is irrelevant

[aiht]

You are an idiot.

NO U

No, _I_ am Idiotus!

Forgot multiple platforms too.

[Anonymous Coward]

Yah, besides missing compiler flags, how does it perform on different intel processors, how about different AMDs?

Plus, the huge system times seems to indicate this more a kernel test than a compiler one.

Re:Forgot multiple platforms too.

[Curupira]

Yah, besides missing compiler flags, how does it perform on different intel processors, how about different AMDs?
Plus, the huge system times seems to indicate this more a kernel test than a compiler one.

Sorry, AC, I will have to let go my positive mod point to you so I can reinforce what you've said. Next time, please consider making an account so you can escape the Score: 0 limbo when you post on Slashdot :(

Since Intel has been caught red-handed crippling AMD processors on code produced by Intel C++ Compiler, I think that testing on Intel and AMD processors should the duty of every single compiler benchmark -- that is posted in Slashdot, at least.

I see why you care about compile speed

[Anonymous Coward]

Cogswell takes a number of 'real world' factors into account, such as how each compiler deals with templates, and comes to certain conclusions

Being that this was the only thing said to be explicitly tested, I'm guessing that is the focus. If your code is so overwhelmed with template code that compiling speed is an issue, perhaps that is a good indicator that you are abusing templates instead of using proper inheritance/overloading

This benchmark is pointless

[godrik]

I am a scholar and study parallel computing. These benchmarks are pretty much pointless. You can not make any conclusions out of these results. Here the author take the time whole time of the execution for the creation of the process to its destruction. That means that are included lots of overhead which would be included in startup time in a real application.

There is also apparently no thread pinning to computational cores. This is known to make a HUGE difference.

Then the authors compared cilk result. cilk is known to be slow for simple codes that do not require workstealing and have complex dependencies. For the record, I know they are also comparing TBB. But TBB is implemented on top of the cilk engine in the intel compiler (I don't know about gcc).

In these results hyperthreading is enabled. The proper use of hyperthreading is complicated. There are some problems where it helps, other where it harms, and I would not be surprise that this behavior be compiler dependent.

Finally, it is almost impossible to compare compilers. On different platforms, with the same compilers you will get different results. Some functions are better compiled by one compiler and some functions are better compiled by the other compiler. This has been reported over and over and over again.

If you care about performance, you should not rely on what your compiler is doing in your back. You need to know what it is doing. Depending on memory alignment (and what the compiler knows about it), depending how the vectorization happen, depending on potential memory aliasing you will get different results.

If you care about performance, you need to benchmark and you need to optimize and you need to know what the compiler does.

Re:This benchmark is pointless

[Anonymous Coward]

I am a scholar...

But are you a gentleman as well?

Re:This benchmark is pointless

[Anonymous Coward]

I am a scholar and study parallel computing.

aka I'm a second year computer science student.

Re:This benchmark is pointless

[Anonymous Coward]

I am a scholar and study parallel computing.

aka I'm a second year computer science student.

No, give the guy a break... English is not his first language. You can tell from the "what your compiler is doing in your back", instead of "behind your back", that sort of thing. From timezone, European seems most likely... from the sentence structure... French?

Re:This benchmark is pointless

[aiht]

I am a scholar and study parallel computing.

aka I'm a second year computer science student.

No, give the guy a break... English is not his first language. You can tell from the "what your compiler is doing in your back", instead of "behind your back", that sort of thing. From timezone, European seems most likely... from the sentence structure... French?

Indeed, godrik's English is quite fluent. It's just those few subtle points that give away that they're probably not a native speaker.
And well picked, AC: godrik has mentioned living in France in the past, so may very well be a French speaker.

Of course, we could just ask... but where's the fun in that?

Re:This benchmark is pointless

[Anonymous Coward]

A few good points, but also a few misleading ones.

TBB is *not* implemented using the Cilk Plus runtime, either in the Intel compiler or in the Cilk Plus branch of GCC. TBB is implemented using a completely separate runtime from Cilk Plus. You can take my word for it that I know what I'm talking about, or you can confirm it by studying the sources online, since they are both publicly available. :)

Pinning threads to cores can help on some benchmarks, but it is less useful for others. In particular, for codes implemented in TBB or Cilk Plus, which use work-stealing schedulers, the performance benefits of pinning can be modest, almost negligible, or sometimes even hurt performance.

The authors never compared any parallel Cilk Plus code with other platforms, so there is no evidence presented in the article that Cilk Plus is slow or fast. More generally, the claim that Cilk is "known to be slow" for codes that do not require work-stealing is a vague statement that has no meaning. There are situations when using Cilk is appropriate, and others when it is not. Optimization for programs for Cilk can be different than optimizing programs using pthreads or OpenMP, so it can be tricky to do truly meaningful performance comparisons.
In this case though, the discussion of speed of Cilk Plus for parallelization is mostly uninteresting. As long as the compiler-generated code is comparable (i.e., the loop is vectorized), the performance of the original parallel loop the author tested is likely to be bound by the memory bandwidth of the system anyway, and most platforms for multicore parallelization should get comparable performance.

Re:This benchmark is pointless

[godrik]

You are indeed correct, I am french. And for the first AC that replied to me, I was a 2nd computer science student something like 10 years ago.

I am currently a CS professor in a US university and I have been doing low level performance study on various architecture (intel xeon, nvidia GPU, recently Xeon Phi, distributed memory machine) for the last 4 years. So I might not know everything about performance benchmarks, but clearly the methodology of the original article is flawed. I would whip (figuratively of course) students that turn in something like that to me.

Re:This benchmark is pointless

[Anonymous Coward]

No, give the guy a break... English is not his first language. You can tell from the "what your compiler is doing in your back", instead of "behind your back",

I think his English is correct, it's just that he's used to using MS Visual Studio.

Re:This benchmark is pointless

[godrik]

TBB is *not* implemented using the Cilk Plus runtime, either in the Intel compiler or in the Cilk Plus branch of GCC. TBB is implemented using a completely separate runtime from Cilk Plus. You can take my word for it that I know what I'm talking about, or you can confirm it by studying the sources online, since they are both publicly available. :)

Interesting. I never looked at how it is implemented by ICC. But my understanding of it is that (some parts of) TBB used a workstealing engine for execution and that it was reusing a significant portion of the cilk runtime. I might have understood wrong.

Pinning threads to cores can help on some benchmarks, but it is less useful for others. In particular, for codes implemented in TBB or Cilk Plus, which use work-stealing schedulers, the performance benefits of pinning can be modest, almost negligible, or sometimes even hurt performance.

Well, the point of pinning is to increase memory locality. Workstealing engine typically try to keep things local to avoid that problem. So you would rather have little migration. Now that I think about it more. Cilk Plus tends to create more threads than cores/hardware thread. So pinning might actually be a catastrophe in that case. I guess what I meant was that ignoring pinning in a parallel benchmark is not the way to do it.

About the speed of Cilk Plus. It is my personnal experience that Cilk Plus is slow. I forwarded a couple of performance related problem to Intel. I remember excluding Cilk Plus result from some charts because they were embarassing (although I might have done something wrong). Whenever you see academic publication using that kind of technology you realize that people emphasize things like parallelism, speedup, load balance, but rarely performance. And when you dig what actually happened in the performance area, you realize that the numbers are not that good. (Though I agree with you that in this case it should not matter)

Bingo Fail

[Anonymous Coward]

Besides if it's spending 80% of the time idle, then the program is waiting for the user not the other way around.

Bingo. When the software is waiting for something to do 80% of the time, and nothing else of any importance is running on that machine, optimization is pretty much irrelevant; at best it would save a tiny amount of power by slightly reducing CPU usage.

Even if it sits idle for 80% of the time, that's still 4.8 hours per day suffering at the hand of a selfish programmer and his crap software.

Don't try to justify the issue. OWN IT!

Intel C++ produced fastest code for us

[pauljlucas]

This information is perhaps 2 years out of date, but back for one of my projects, when we switched from g++ to Intel C++, our software got about twice as fast with no other changes. It got even faster when we took advantage of SSE3 instructions.

Re:Intel C++ produced fastest code for us

[Anonymous Coward]

lol, no.

Re:Intel C++ produced fastest code for us

[Anonymous Coward]

Intel's compiler suite has always (and continues) to be faster than GCC. But that isn't super unexpected, they have a better working knowledge of their CPUs quirks, 10x more comp sci PHDs on staff than the gcc team can muster, and their product isn't portable to dozens of archs like gcc.

Re:Intel C++ produced fastest code for us

[Anonymous Coward]

I'd suspect your project mostly had machines with Intel processors, while the benchmarking here was likely done on AMD (or Cyrix) processors. Intel's compiler produces code that has this nasty misfeature of checking the cpuid instruction for "GenuineIntel", rather than checking the feature flags. There was a benchmark run by Tom's Hardware where they took a machine with a Cyrix processor (which allows you to change the information cpuid gives), with one benchmark program when they told it to report "AuthenticAMD" performance increased by 20%% when they had it report "GenuineIntel" performance increased by 33%% (this is from memory). This is why avoiding Intel's compiler is often a Good Thing(tm).

for complilers

[mjwalshe]

It is speed that is important which is why a lot of HPC people still prefer the intel compilers.

Re:for complilers

[Anonymous Coward]

Because they generate faster binaries contrary to what some ill defined benchmarks may tell you (even faster if you use fortran 2008).

Compilation time

[amightywind]

Compilation time is irrelevant if you use ccache or distcc, nitwits. GCC is and has always been the logical choice.

Four-hour compile times means a 1 day turnaround

[Anonymous Coward]

Four-hour compile times means a 1 day turnaround for any bugfix for production.

A one-hour compile time means four to six bugfix/test cycles per day.

Re:Four-hour compile times means a 1 day turnaroun

[namgge]

six release cycles a day is probably why you have bugs in the first place...

Cilk Extensions and Clang

[tyrione]

Why in the hell are you testing Clang with either Cilk or OpenMP when neither have moved into mainline trunk of LLVM/Clang? This test is as worthless as Phoronix's test suite on benchmarking apps that require OpenMP and they note that Clang takes it in the shorts because it presently doesn't have OpenMP implemented. Complete waste of time.

Re:Four-hour compile times means a 1 day turnaroun

[smash]

he said bugfix/test, not release.

Re:Four-hour compile times means a 1 day turnaroun

[euroq]

Although you have a good point, his point is still valid.

Re:Four-hour compile times means a 1 day turnaroun

[Bert64]

Who's to say you have to recompile everything? Surely if your making a small bugfix you just recompile the files which have changed...
Plus you have tools like distcc, ccache etc

GCC post

[DrYak]

I could have beaten him with my highly optimized GCC-devel compile of "FirstPost.cxx",
but I didn't quite understand the error message regarding the templates.

In our times

[ThatsNotPudding]

In the times we live in - and the knowledge Ed S. has given us - do you really still trust a black-box compiler from a huge US corporation with intimate government ties?