Slashdot Mirror
Distributed Compiling?
Stijn Buys writes in with this interesting question:
"We are a bunch of students conected to the campus network and now we are looking for a way to distribute applications like compiler jobs to several Linux boxes. Each PC should retain it's autonomy, since these are our personal machines, but we'd really like to make them work together. I heard about things like distributed make but I haven't found something about it yet. Are there people who have experience with these kind of things and/or can give us some suggestions?"
With just a quick search, I found this patch to GNU make and something called PGMAKE (dated 1994). What other options for distributed compiling are there?
MOSIX (search on freshmeat..its GPLised) can allow you to migrate processes....PVM and MPI libraries can allow you to use multiple processors (altho you need to write code that calls PVM & MPI). Linux/HA might have some(?) code that asllows you to migrate stuff but thats all i know of...of course you could do a round robin rsh style make.....
Check out pmake in your favourite distribution. If it has been set up you should be able to just replace
> make
with
>pmake
Seems this is the same as what we were using in '92-94 to compile the Sather compiler. It would distribute the compiles to idle workstations on the network and gather up the object files for linking. All completely transparent to the user - except for the significant speedup.
What kills it is screensavers that generate fractals and the like. The machine is never so busy as when it is idle!
dmake and ppmake will do it. They are two PVM applications (or one MPI, one PVM, I can't remember) that will do exactly what you need. Please note that installing PVM and managing it may not be worth your time. Check alternatives first.
A good test of this can be done before distributing by building with make -jX (where X is some number >1), this works whether or not you have SMP, though in the case that you do, it will actually take advantage of your multiprocessor capability. Once you can safely do parallel makes on a single machine, you should be able to extend this to a fully distributed make. (Obviously, such considerations require some thought and effort, so trying to do this for something you just need to make once is silly. But for large apps under active development, which need to be built from the base up periodically, it can be a real win.)
Peace and love, y'all
I was on a project several years ago where the sysadmins wrote some wrapper shell scripts around 'at' to accomplish distributed compiling. (Each complete make task was given to the machine with the lowest load, it didn't distribute an individual run of make where obj's are created in parallel on different machines.) It was all pretty low-tech, but worked fine. Sorry, I don't have any code to share...
There is an add-on to Berkeley make that does this called 'customs'. It's in the latest Red Hat 6.0 in case you are using Linux. 'rpm -ql pmake-customs' only disadvantage is that it probably uses Berkeley RPC, which means the the connections over the network won't be secure by default. You shouldn't use it in an unsafe environment unless you hack it to use ssh or something.
damn html formatted text. sorry.
There is an add-on to Berkeley make that does this called 'customs'.
It's in the latest Red Hat 6.0 in case you are using Linux. 'rpm -ql pmake-customs'
only disadvantage is that it probably uses Berkeley RPC, which means the the connections over the network won't be secure by default. You shouldn't use it in an unsafe environment unless you hack it to use ssh or something.
This doesn't answer your question directly,
but solaris sparcworks has a parallel make
utility called 'dmake' that does exactly what
you want. You can even set permissions on each
system and the number of jobs allowed.
In my experience, I've found that you can't
just use parallel make on anything you find
on the net. The Makefiles have to be double
checked to make sure you have the dependencies
set up right. (using stuff like NO_PARALLEL)
For instance, I've had lots of makes break
because a link references a library member
that hasn't been built. That's kind of scary,
since if you're doing development and doing
a re-make, you might unwittingly be compiling
against an older library member. (and it's
timing-dependent with this kind of hosage -
it might work one time, but not the next)
So double check your makefiles...
good luck