Distributed Compilation, a Programmer's Delight

cyberpead writes in with a Developerworks article on the open source tool options that can help speed up your build process by distributing the process across multiple machines in a local area network.

Comments

What about Excuse #1?

[dsginter]

Sorry - compiling

Re:What about Excuse #1?

[Thiez]

That would allow for people to inject malware, wouldn't it?

To compile:

void printhello() {
  printf("Hello world!\n");
}

evil bastard changes to:

void printhello() {
  {
   
  }
  printf("Hello world\n");
}

Since the most practical way to spot the evil binary would be to compile the code yourself and compare, that sort of defeats the purpose of having someone else compile it. I guess you could have many random people compile the same piece of source-code and then compare all produced code, but that makes the whole thing rather complicated.
Also, the p2p thing would only be useful for open source, as I doubt it would be smart for people trying to produce some closed source product to send their source to a p2p network that may or may not store everything.
And this is all assuming the delays introduced by sending all this stuff over the internet are not so large that compiling locally is faster or almost as fast.

It's probably best to compile your stuff on your lan, on machines that are close, and that can be trusted.

Bulk building is more effective

[TheThiefMaster]

Due to a strange quirk in the way compilers are designed, it's (MUCH) faster to build a dozen files that include every file in your project than to build thousands of files.

Once build times are down to 5 - 15 minutes you don't need distributed compiling. The link step is typically the most expensive anyway, so distributed compiling doesn't get you much.

Minor error

[pipatron]

There's a minor error in the article, which claims that your servers need access to the source. distcc was designed to not need this.

In other news

[adonoman]

Slashdot readership plummets to an all-time low as programmers actually have to work.

Re:distcc has one fatal flaw

[TeknoHog]

From man distcc:

In pump mode, distcc runs the preprocessor remotely too. To do so, the preprocessor must have access to all the files that it would have accessed if had been running locally. In pump mode, therefore, distcc gathers all of the recursively included headers, except the ones that are default system headers, and sends them along with the source file to the compilation server.

Preprocessing in C

[Frans Faase]

I guess you are refering to the preprocessing step of C and C++ compilers, which was really a lame hack, I think. If you have a lot of include files, preprocessing produces large intermediate files, which contain a lot of overlapping code, that has to be compiled over and over again.

Preprocessing should have been removed a long time ago, but nasty backwards compatability issue, it was never done. Other languages, such as Java and D, solve this problem in a much better way. Just as did TurboPascal with its TPU files in the late 1980's.