Slashdot Mirror
Speed Test 2: Comparing C++ Compilers On WIndows
Nerval's Lobster writes "In a previous posting, developer and programmer Jeff Cogswell compared a few C++ compilers on Linux. Now he's going to perform a similar set of tests for Windows. "Like all things Windows, it can get costly doing C++ development in this environment," he writes. "However, there are a couple notable exceptions" such as free and open-source cygwin, mingW, Express Versions of Visual Studio, and Embacadero. He also matched up the Intel C++ Compiler, Microsoft C++ Compiler, and the Embarcadero C++ 6.70 Compiler. He found some interesting things — for example, Intel's compiler is pretty fast, but its annoying habit of occasionally "calling home" to check licensing information kept throwing off the rests. Read on to see how the compilers matched up in his testing."
Did calling home really throw off the results? Since that is something that ordinary users would have to put up with, I would think it should be part of the test. It might be difficult to get an average, but testing Intel's compiler only when it is at its fastest doesn't seem fair.
does the Intel one still slow down on AMD systems and or trun out code with AMD slow down blocks?
Based on his description, he is using a very synthetic benchmark:
The code I’m testing contains no #include directives, and makes use of only standard C++ code. It starts with one class, and then is followed by 6084 small classes derived from various instantiations of the template classes. (So these 6084 classes are technically not templates themselves.) Then I create 6084 instantiations of the original template class, using each of the 6084 classes. The end result is 6084 different template instantiations. Now, obviously in real life we wouldn’t write like that (at least I hope you don’t).
So in his own words, the code does not reflect realistic compiles. There is no reason to assume that the result generalise to any programs that anyone actually cares about.
Also, there are no error bars of any kind listed.
In other words, I have no reason to assign any meaning to these numbers.
Depends on the paranoia of the DRM implementation.
A simple "We sell to corporations, so we don't expect cracks but we do expect optimists trying to install a few extra seats" licensing system really just needs to (ideally over SSL, hear that, LG?) phone home the serial number at intervals and get a 'yes/no' in response.
The more paranoid systems, designed on the assumption that the client will be under active attack, may be considerably murkier. Something like 'BD+', for instance, for Blu-ray DRM, mandates a proprietary, blackbox virtual machine (with access to the client hardware's memory and the ability to run native code, including applying persistent patches to the client system, in addition to what it does internally) that executes whatever blob of BD+ code burned into the disk you try to play.
A system like that, or an AV-like 'signature' system that is designed to have one or more parts of the client DRM continually interrogating one another in novel ways to detect compromises in the authentication mechanism would be more or less indistinguishable from a binary, because it would be one.
For something like a compiler, rather than a game, or some precious 'premium content', I'd expect the former; but I've underestimated human stupidity before.
You might think so, but no. If the licensing check is done in the clear, one could spoof the reply and thus bypass the licensing effortlessly. More likely it's encoded/encrypted in some way that you can't (easily) fake a green light. So you may be able to identify the transmission of "very large binary" or equivalent, but differentiating between a "license check" and "tiny binary" would be tricky since you won't be reading the stream.
My understanding is that they never explicitly 'slowed down' AMD systems; but that the binaries produced by their compiler refused to honor the capabilities flags of non-intel processors (eg. even if an AMD CPU lists 'SSE2, SSE3' among supported instructions, it would get the fallback to non-SSE instructions, while Intel CPUs would get whatever their supported instructions lists specified). No actual 'here be lots of NOPs for no reason'; but x87 on a machine that can do recent SSE is probably enough to achieve the same effect...)
This doesn't test the speed of generated code. I like to know which compiler produces faster code when looking at benchmarks.
According to the fine article, "The Intel compiler occasionally âoecalls homeâ to an Intel-owned Website to check licensing information. When it does so, it prints out a message about when the current license expires. I didnâ(TM)t use the results when that happens, since it would add time and skew the timing results. " WRONG. The tester should not have excluded these results where time was wasted with this nonsense: If WE the users have to put up with it, it SHOULD be included in the benchmarks.
Sometimes the "writing on the wall" is blood spatter...
It would help if he actually measured something worthwhile. In the 20+ years I've been coding, I've never once picked a compiler on the basis of how long it takes to spit out a binary. There are just so many other more interesting features and characteristics to consider.
I want this account deleted.
Benchmarking compilers on how long it takes to compile would be like benchmarking cars based on how long it takes to fill the gas tank.
There are so many things that can affect compile time more than the compiler - and the end customer really doesn't care anyway. Frankly, if you want a 3-5x speedup, just put the whole thing on an SSD and let it fly.
I have mod points and I am not afraid to use them
If memory serves, their argument varies (depending on whether the FTC appears interested or not) between 'fuck you, it's the Intel compiler collection, and it'll do what's best for Intel. Go suck an Opteron if you like AMD so much.' and 'Gosh, we sure know about the capabilities flags; but we can't be sure of the details of other vendors'(*cough*shoddy, probably reverse engineered illegally*cough*) implementations of certain complex features, and our customers expect our compiler suite to provide stable, correct output, so reverting to the x87 codepath is our only real option..."
The Intel compilers do NOT "phone home" for licensing. What they do "phone home" for is to send anonymous usage data. When you install, you're asked if you want to opt in to this - it is not enabled by default. Licensing is done entirely locally for single-user licenses. See http://software.intel.com/en-us/articles/software-improvement-program for more information.