Slashdot Mirror
Speed Test 2: Comparing C++ Compilers On WIndows
Nerval's Lobster writes "In a previous posting, developer and programmer Jeff Cogswell compared a few C++ compilers on Linux. Now he's going to perform a similar set of tests for Windows. "Like all things Windows, it can get costly doing C++ development in this environment," he writes. "However, there are a couple notable exceptions" such as free and open-source cygwin, mingW, Express Versions of Visual Studio, and Embacadero. He also matched up the Intel C++ Compiler, Microsoft C++ Compiler, and the Embarcadero C++ 6.70 Compiler. He found some interesting things — for example, Intel's compiler is pretty fast, but its annoying habit of occasionally "calling home" to check licensing information kept throwing off the rests. Read on to see how the compilers matched up in his testing."
Oh, they meant the tests.
Did calling home really throw off the results? Since that is something that ordinary users would have to put up with, I would think it should be part of the test. It might be difficult to get an average, but testing Intel's compiler only when it is at its fastest doesn't seem fair.
does the Intel one still slow down on AMD systems and or trun out code with AMD slow down blocks?
Based on his description, he is using a very synthetic benchmark:
The code I’m testing contains no #include directives, and makes use of only standard C++ code. It starts with one class, and then is followed by 6084 small classes derived from various instantiations of the template classes. (So these 6084 classes are technically not templates themselves.) Then I create 6084 instantiations of the original template class, using each of the 6084 classes. The end result is 6084 different template instantiations. Now, obviously in real life we wouldn’t write like that (at least I hope you don’t).
So in his own words, the code does not reflect realistic compiles. There is no reason to assume that the result generalise to any programs that anyone actually cares about.
Also, there are no error bars of any kind listed.
In other words, I have no reason to assign any meaning to these numbers.
Depends on the paranoia of the DRM implementation.
A simple "We sell to corporations, so we don't expect cracks but we do expect optimists trying to install a few extra seats" licensing system really just needs to (ideally over SSL, hear that, LG?) phone home the serial number at intervals and get a 'yes/no' in response.
The more paranoid systems, designed on the assumption that the client will be under active attack, may be considerably murkier. Something like 'BD+', for instance, for Blu-ray DRM, mandates a proprietary, blackbox virtual machine (with access to the client hardware's memory and the ability to run native code, including applying persistent patches to the client system, in addition to what it does internally) that executes whatever blob of BD+ code burned into the disk you try to play.
A system like that, or an AV-like 'signature' system that is designed to have one or more parts of the client DRM continually interrogating one another in novel ways to detect compromises in the authentication mechanism would be more or less indistinguishable from a binary, because it would be one.
For something like a compiler, rather than a game, or some precious 'premium content', I'd expect the former; but I've underestimated human stupidity before.
You might think so, but no. If the licensing check is done in the clear, one could spoof the reply and thus bypass the licensing effortlessly. More likely it's encoded/encrypted in some way that you can't (easily) fake a green light. So you may be able to identify the transmission of "very large binary" or equivalent, but differentiating between a "license check" and "tiny binary" would be tricky since you won't be reading the stream.
I took a quick took at their website. It looks quite scammy, they only talk about how much you will save, not about how much it will cost.
After clicking through the buy-now buttons twice, I found the C++ version was $4000.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
My understanding is that they never explicitly 'slowed down' AMD systems; but that the binaries produced by their compiler refused to honor the capabilities flags of non-intel processors (eg. even if an AMD CPU lists 'SSE2, SSE3' among supported instructions, it would get the fallback to non-SSE instructions, while Intel CPUs would get whatever their supported instructions lists specified). No actual 'here be lots of NOPs for no reason'; but x87 on a machine that can do recent SSE is probably enough to achieve the same effect...)
This doesn't test the speed of generated code. I like to know which compiler produces faster code when looking at benchmarks.
According to the fine article, "The Intel compiler occasionally âoecalls homeâ to an Intel-owned Website to check licensing information. When it does so, it prints out a message about when the current license expires. I didnâ(TM)t use the results when that happens, since it would add time and skew the timing results. " WRONG. The tester should not have excluded these results where time was wasted with this nonsense: If WE the users have to put up with it, it SHOULD be included in the benchmarks.
Sometimes the "writing on the wall" is blood spatter...
My understanding is that they never explicitly 'slowed down' AMD systems; but that the binaries produced by their compiler refused to honor the capabilities flags of non-intel processors
Oh, my. Just how many major non-Intel x86-64 CPU vendors are there? AMD, and...? It's suspiciously similar to the ACPI and SecureBoot affairs, don't you think?
Ezekiel 23:20
It would help if he actually measured something worthwhile. In the 20+ years I've been coding, I've never once picked a compiler on the basis of how long it takes to spit out a binary. There are just so many other more interesting features and characteristics to consider.
I want this account deleted.
I'd just like to see a C++11 compiler for windows.
File under 'M' for 'Manic ranting'
Benchmarking compilers on how long it takes to compile would be like benchmarking cars based on how long it takes to fill the gas tank.
There are so many things that can affect compile time more than the compiler - and the end customer really doesn't care anyway. Frankly, if you want a 3-5x speedup, just put the whole thing on an SSD and let it fly.
I have mod points and I am not afraid to use them
VIA was also one that was affected by Intels compiler behaviour.
In the 20+ years I've been coding, I've never once picked a compiler on the basis of how long it takes to spit out a binary.
Then you don't compile big projects. Neither do I, but there are projects where a full build can take hours. Sure build farms and stuff help, but so does a 2x faster compiler.
If memory serves, their argument varies (depending on whether the FTC appears interested or not) between 'fuck you, it's the Intel compiler collection, and it'll do what's best for Intel. Go suck an Opteron if you like AMD so much.' and 'Gosh, we sure know about the capabilities flags; but we can't be sure of the details of other vendors'(*cough*shoddy, probably reverse engineered illegally*cough*) implementations of certain complex features, and our customers expect our compiler suite to provide stable, correct output, so reverting to the x87 codepath is our only real option..."
The Intel compilers do NOT "phone home" for licensing. What they do "phone home" for is to send anonymous usage data. When you install, you're asked if you want to opt in to this - it is not enabled by default. Licensing is done entirely locally for single-user licenses. See http://software.intel.com/en-us/articles/software-improvement-program for more information.
Long, long ago some review site ran a Via CPU based system while spoofing the CPU ID to appear as an Intel CPU of similar capabilities.
They expected a few percent gain in the FP and INT benches, but oddly got an 8-fold increase in reported memory bandwidth. The other benchmarks appeared to reflect a real increase in memory performance.
Don't wipe your arse with Intel, they're so dirty you'll end up shittier.
I do work on large projects in the multi-hour range for a full rebuild, and the compile time is still pretty much the lowest priority in selecting a compiler. All things being equal, of course I'd like to have the fastest compiles possible. But more important than that is that I can write the code I need to write without dodging compiler bugs / shortcomings all day, and deliver a binary which is optimized well for the target.
You can adapt to slow compiles. Breaking the project up into libraries, for example. You can't readily adjust to other compiler problems.
Visual C++ has this handy /MP option which tells the compiler to do multi-threaded compiles. On some of our build machines (with 16 cores) this gives an almost linear increase in build speeds. It's obvious from the author's discussion of multi-core that he is not aware of this option and did not use it.
A performance benchmark which doesn't turn on the go-fast option is not going to produce meaningful results.
The author also doesn't discuss debug symbols. VC++ generates debug symbols by default, whereas the other compilers do not. Generating builds without symbols is not a reasonable scenario for most builds, so this makes the file size comparisons rather meaningless.
Iterate twice as fast? Only if all you do is compile. Some of us compile, test some, think, make changes, and then recompile. I heartily recommend the "test" and "think" ideas if you're not used to them.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes