Slashdot Mirror
Optimizing distcc
IceFox writes "Having fallen in love with distcc and its ability to speed up compiling (insert anyone who compiles like Gentoo users or Linux developers). I recently got the chance to dive deeper into distcc. By itself distcc will decrease your build times, but did you know that if you tweak a few things you can get a whole lot better compile times? Through a lot of trial and error, tips from others, profiling, testing and just playing around with distcc, I have put together a nice big article. It shows how developers can get a bigger bang for their buck out of their old computers and distcc with just a few changes."
My life changed the day i found out i could get my super fast P4 Windows XP box to compile for my slow linux box. Distcc for cygwin is a miracle. check out the thread at Gentoo forums
Martin Pool, the brains behind distcc, was interviewed by ZDNet yesterday. How timely.
o ry /0,2000034960,20283318-1,00.htm
http://web.zdnet.com.au/builder/program/work/st
I feel like burning my new site in a bit =)
http://hackish.org/~rufus/distcc.php.html
Developer Spotlight: Martin Pool
Install Sun Grid Engine[1] since it's free and now open source and then not only do you get qmake for distributed builds but you also get a general purpose distributed processing system. And hey! It even has the current buzzword "grid" in the title so your PHB will love you.
[1] http://gridengine.sunsource.net/
Government of the people, by corporate executives, for corporate profits.
Sigh, another experiment that could have benefitted greatly from factorial experimentation. If your unfamiliar with DOE, here is a basic introduction courtesy of NIST:
t io n1/pri11.htm
http://www.itl.nist.gov/div898/handbook/pri/sec
It appears in this case we have a variety of factors and trying to, in this case, have a response of "elapsed time" for compilation and it is a minimization problem. Instead of looking at factors individually, a factorial DOE would have allowed interactions to be analyzed and to look for a global optima rather than just optimizing individual factors and then tossing them all together, it doesn't work that way a lot/most of the time.
If the author of this article is present: Why wasn't a factorial experiment used?
Yes, distcc is nice, but anyone with a really big build (say like hours long) must take a look at John Ousterhout's company Electric Cloud (yeah, John Ousterhout as in Tcl) here. They've built this replacement for gmake that runs the jobs in parallel but is smarter than distcc because it can break open all the recursive makes and run _everything_ in parallel and it works cross platform too. It's $$$ and not OSS :-) but designed to be ultrareliable.
official gentoo distcc guide:
http://www.gentoo.org/doc/en/distcc.xml
You must be new so I will explain it to you. There is a class of users that regularly Karma Whore and get their Karma maxed out and then proceed to burn Karma by posting things like goatse.cx at +1. They do this not only to annoy people but to prove the flaws with the moderation system. While this guy may not be one of those people it is important to not reward somebody for posting an article. Users can easily post the article anonymously and avoid this issue altogether.
Yup, look at the X code preferences for distributed builds. The cool part is they use Rendezvous to automatically find machines to send work. You can set your box to use these others and/or offer service to others. Also on dual processor boxes is will treat them as two machines and do two compiles at once.
Anyway, you can see distcc running when you have X code enabled for distributed builds and running.
--jim
There are some problems though - which do you do first ccache or distcc (answer on my benchmarks is ccache - if it isn't in the cache send it on the network) how fast is your "build" machine - this is critical. The build machine is resonsible for preprocessing the file, checking if it is in the cache and then sending it out to be turned into an object. Especially when you interact the results of ccache (which most of your builds are just the same file over and over - very few "changed" files) and distcc - most of your time is spent in the first pass compiler.
In our environment we had boatloads of dual XEON machines around - they made wonderful build machines, and it didn't hurt that we connected them with Gig Ethernet either. Did wonders for our build times.
Over all distcc and ccache are wonderful tools that should be in every large compile environment - making compiles that used to take days take simple minutes. But you want to make sure that the dependancy between ccache and distcc work optimally in your environment.
I have mod points and I am not afraid to use them
Unfortunately, the makefile creator most people use, automake, creates only recursive makefiles. Maybe a replacement like unsermake will get automake developers thinking about radical changes. I wouldn't mind seeing M4 go away, for one.
Well.. here goes a couple of mod points that I spent.. but I'd thought I'd chime in..
/etc/make.globals? or make.conf).
My friend recently had the same thing happen, and the conclusion we came to was that the compiler versions were different on the distcc servers (3.2.2) versus the client (3.2.3).. and the preprocessed code being sent off had syntax erorrs or something of the like when it was sent off (something to do with one of the new options in the latest gcc). I don't recall exactly what option it was or what package(s) were failing... but I do know that somewhere there was an 'if gcc-version 3.2.3 then add some options to CFLAGS' (maybe
This is one of the biggest things I have found with distcc.. compiler versions have to be pretty similar.. usually even the incremental version changes affect the compiles..
I've not had any problems with using distcc, both with compiling Gentoo packages, along with my own projects..
Kenny