Distributed Compilation, a Programmer's Delight

cyberpead writes in with a Developerworks article on the open source tool options that can help speed up your build process by distributing the process across multiple machines in a local area network.

Comments

Is this new?

[daveewart]

Article summary: use 'make -j', 'distcc' and 'ccache' or something combination of these. These utilities are well known and widely used already, no?

Re:Minor error

[SleptThroughClass]

There's a minor error in the article, which claims that your servers need access to the source. distcc was designed to not need this.

That implies you read the article, but that can't be the case.

Re:What about Excuse #1?

[Thiez]

That would allow for people to inject malware, wouldn't it?

To compile:

void printhello() {
  printf("Hello world!\n");
}

evil bastard changes to:

void printhello() {
  {
   
  }
  printf("Hello world\n");
}

Since the most practical way to spot the evil binary would be to compile the code yourself and compare, that sort of defeats the purpose of having someone else compile it. I guess you could have many random people compile the same piece of source-code and then compare all produced code, but that makes the whole thing rather complicated.
Also, the p2p thing would only be useful for open source, as I doubt it would be smart for people trying to produce some closed source product to send their source to a p2p network that may or may not store everything.
And this is all assuming the delays introduced by sending all this stuff over the internet are not so large that compiling locally is faster or almost as fast.

It's probably best to compile your stuff on your lan, on machines that are close, and that can be trusted.

Re:What about Excuse #1?

[ORBAT]

If by powerful system you mean steam-powered Analytical Engine, yes, it'll take days.

The longest OpenOffice compile I've ever done was something around 5 hours, and that was with the system doing other stuff on the side. Distcc et al reduce the compile time to around 2h.

Re:Minor error

[cbreaker]

"False - distcc does require the same version of the OS. I've tried using distcc to spread compilation of the kernel across a ubuntu & redhat machine that I have (both x64), and it refused."

You didn't read my post or you have low comprehension skills. I said "Depending on the source you're compiling."

A Kernel build might require specific libraries to be the same version. Building Firefox might not. Some apps you can build on Linux and use a cygwin box running distcc to help. Others you cannot.

It's not a RULE. It depends the the source. That's what I said. Then again you might have actually realized that and that's why you posted AC.