Slashdot Mirror

← Back to Stories (view on slashdot.org)

Distributed Compilation, a Programmer's Delight

Posted by kdawson on from the less-time-for-wacraft dept.

cyberpead writes in with a Developerworks article on the open source tool options that can help speed up your build process by distributing the process across multiple machines in a local area network.

8 of 60 comments (clear)

  1. Bulk building is more effective by TheThiefMaster · · Score: 3, Informative

    Due to a strange quirk in the way compilers are designed, it's (MUCH) faster to build a dozen files that include every file in your project than to build thousands of files.

    Once build times are down to 5 - 15 minutes you don't need distributed compiling. The link step is typically the most expensive anyway, so distributed compiling doesn't get you much.

  2. Minor error by pipatron · · Score: 4, Informative

    There's a minor error in the article, which claims that your servers need access to the source. distcc was designed to not need this.

    --
    c++; /* this makes c bigger but returns the old value */
    1. Re:Minor error by cbreaker · · Score: 2, Informative

      I read it too, and it's true - they DO say all of the machines need access to the source which they do not.

      Maybe there's some special cases, but I've never had to have a shared source repository in order to use distcc.

      They also say the machines need to be exactly the same configuration, and they do elaborate on that a little bit, but it's not strictly true. Depending on the source you're compiling, you might only need to just have the same major version of GCC.

      --
      - It's not the Macs I hate. It's Digg users. -
    2. Re:Minor error by pipatron · · Score: 2, Informative

      False - I regularly use windows as a host for example, running distcc in cygwin. The only thing you have to make sure is that the compiler called by distcc will create object files for the client system. You can have a renderfarm of sparcs generating code for your ARM router if you like.

      --
      c++; /* this makes c bigger but returns the old value */
  3. Re:What about Excuse #1? by khellendros1984 · · Score: 2, Informative

    That was my problem. Broken ebuilds. Conflicting requirement lists that the updater script wasn't any good at working out. Gentoo made me run back to Slackware for a while, and eventually to Ubuntu (about 2-3 years ago, to see what the buzz was about).

    --
    It is pitch black. You are likely to be eaten by a grue.
  4. Re:distcc has one fatal flaw by TeknoHog · · Score: 3, Informative
    From man distcc:

    In pump mode, distcc runs the preprocessor remotely too. To do so, the preprocessor must have access to all the files that it would have accessed if had been running locally. In pump mode, therefore, distcc gathers all of the recursively included headers, except the ones that are default system headers, and sends them along with the source file to the compilation server.

    --
    Escher was the first MC and Giger invented the HR department.
  5. Preprocessing in C by Frans+Faase · · Score: 4, Informative

    I guess you are refering to the preprocessing step of C and C++ compilers, which was really a lame hack, I think. If you have a lot of include files, preprocessing produces large intermediate files, which contain a lot of overlapping code, that has to be compiled over and over again.

    Preprocessing should have been removed a long time ago, but nasty backwards compatability issue, it was never done. Other languages, such as Java and D, solve this problem in a much better way. Just as did TurboPascal with its TPU files in the late 1980's.

    1. Re:Preprocessing in C by Anonymous Coward · · Score: 2, Informative

      It's not sufficient for large projects; disk I/O is still a very large overhead when compiling. Switching to a 'Unity' build scheme reduced compile times significantly (more-so than the distributed compile solution we used since it still had to read the files off disk multiple times in addition to sending them over the wire to multiple machines). .CPPs and .Hs make up about 110mb on our project.