Distributed Compilation, a Programmer's Delight

cyberpead writes in with a Developerworks article on the open source tool options that can help speed up your build process by distributing the process across multiple machines in a local area network.

Comments

What about Excuse #1?

[dsginter]

Sorry - compiling

Re:What about Excuse #1?

[khellendros1984]

That was my problem. Broken ebuilds. Conflicting requirement lists that the updater script wasn't any good at working out. Gentoo made me run back to Slackware for a while, and eventually to Ubuntu (about 2-3 years ago, to see what the buzz was about).

Re:What about Excuse #1?

[Thiez]

That would allow for people to inject malware, wouldn't it?

To compile:

void printhello() {
  printf("Hello world!\n");
}

evil bastard changes to:

void printhello() {
  {
   
  }
  printf("Hello world\n");
}

Since the most practical way to spot the evil binary would be to compile the code yourself and compare, that sort of defeats the purpose of having someone else compile it. I guess you could have many random people compile the same piece of source-code and then compare all produced code, but that makes the whole thing rather complicated.
Also, the p2p thing would only be useful for open source, as I doubt it would be smart for people trying to produce some closed source product to send their source to a p2p network that may or may not store everything.
And this is all assuming the delays introduced by sending all this stuff over the internet are not so large that compiling locally is faster or almost as fast.

It's probably best to compile your stuff on your lan, on machines that are close, and that can be trusted.

Re:What about Excuse #1?

[ORBAT]

If by powerful system you mean steam-powered Analytical Engine, yes, it'll take days.

The longest OpenOffice compile I've ever done was something around 5 hours, and that was with the system doing other stuff on the side. Distcc et al reduce the compile time to around 2h.

Bulk building is more effective

[TheThiefMaster]

Due to a strange quirk in the way compilers are designed, it's (MUCH) faster to build a dozen files that include every file in your project than to build thousands of files.

Once build times are down to 5 - 15 minutes you don't need distributed compiling. The link step is typically the most expensive anyway, so distributed compiling doesn't get you much.

Is this new?

[daveewart]

Article summary: use 'make -j', 'distcc' and 'ccache' or something combination of these. These utilities are well known and widely used already, no?

Minor error

[pipatron]

There's a minor error in the article, which claims that your servers need access to the source. distcc was designed to not need this.

Re:Minor error

[SleptThroughClass]

There's a minor error in the article, which claims that your servers need access to the source. distcc was designed to not need this.

That implies you read the article, but that can't be the case.

Re:Minor error

[cbreaker]

I read it too, and it's true - they DO say all of the machines need access to the source which they do not.

Maybe there's some special cases, but I've never had to have a shared source repository in order to use distcc.

They also say the machines need to be exactly the same configuration, and they do elaborate on that a little bit, but it's not strictly true. Depending on the source you're compiling, you might only need to just have the same major version of GCC.

Re:Minor error

[pipatron]

False - I regularly use windows as a host for example, running distcc in cygwin. The only thing you have to make sure is that the compiler called by distcc will create object files for the client system. You can have a renderfarm of sparcs generating code for your ARM router if you like.

Re:Minor error

[cbreaker]

"False - distcc does require the same version of the OS. I've tried using distcc to spread compilation of the kernel across a ubuntu & redhat machine that I have (both x64), and it refused."

You didn't read my post or you have low comprehension skills. I said "Depending on the source you're compiling."

A Kernel build might require specific libraries to be the same version. Building Firefox might not. Some apps you can build on Linux and use a cygwin box running distcc to help. Others you cannot.

It's not a RULE. It depends the the source. That's what I said. Then again you might have actually realized that and that's why you posted AC.

In other news

[adonoman]

Slashdot readership plummets to an all-time low as programmers actually have to work.

raggle fraggle

[TheRealMindChild]

Sky rockets in flight... distcc delight......
distcc deliiiiiiiight.

Re:distcc has one fatal flaw

[TeknoHog]

From man distcc:

In pump mode, distcc runs the preprocessor remotely too. To do so, the preprocessor must have access to all the files that it would have accessed if had been running locally. In pump mode, therefore, distcc gathers all of the recursively included headers, except the ones that are default system headers, and sends them along with the source file to the compilation server.

Preprocessing in C

[Frans Faase]

I guess you are refering to the preprocessing step of C and C++ compilers, which was really a lame hack, I think. If you have a lot of include files, preprocessing produces large intermediate files, which contain a lot of overlapping code, that has to be compiled over and over again.

Preprocessing should have been removed a long time ago, but nasty backwards compatability issue, it was never done. Other languages, such as Java and D, solve this problem in a much better way. Just as did TurboPascal with its TPU files in the late 1980's.

Re:Preprocessing in C

[Anonymous Coward]

It's not sufficient for large projects; disk I/O is still a very large overhead when compiling. Switching to a 'Unity' build scheme reduced compile times significantly (more-so than the distributed compile solution we used since it still had to read the files off disk multiple times in addition to sending them over the wire to multiple machines). .CPPs and .Hs make up about 110mb on our project.

SMEs

[hachete]

The reason for a lot of build machines in the rack may not be horsepower but rather you need x different machine versions, or a certain build only builds on a certain machine because of licence restrictions or you may only have one windows box with the Japanese character set installed because it causes so many problems that multiplying the problems just isn't worth it and so on and so forth. Building across n number of the same machine version just isn't worth the work IMO. Just get a bigger machine and save on the machine maintenance.

So the real benefit of distcc might be parallel compilation; I see a big future for this, particularly with the chipsets becoming commonplace. Once upon a time, I would not countenance a dual-chip machine in the rack because of the indeterminate mayhem it would sometimes cause to a random piece of code deep in the bowels. Those problems are well gone.

Umm. I wonder how this plays out how with VMWARE? A distributed compiler smart enough to use the (correct) local compiler across a varied build set would be worth having ...

Icecream.

[Sir_Lewk]

If you are interested in distributed compiling, you may want to check out icecream. http://en.opensuse.org/Icecream

It's similar to distcc, but with some notable benefits.