Slashdot Mirror

← Back to Stories (view on slashdot.org)

Distributed Compilation

Posted by michael on from the they-weren't-using-their-processors-anyway dept.

BagOBones writes "Tired of waiting for your source to compile? Dreaming of having your own cluster and having something useful to do with it? Well Trolltech might have the answer. Trolltech Teambuilder lets you turn your network into a clustered C/C++ compiler."

29 comments

  1. 1st compile. by Anonymous Coward · · Score: -1, Redundant

    Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86
    Copyright (C) Microsoft Corp 1984-1998. All rights reserved.

    a.c
    Microsoft (R) Incremental Linker Version 6.00.8447
    Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

    /out:a.exe
    a.obj

  2. hmm.... by Anonymous Coward · · Score: 0

    I can see it coming ....

    "wow! imagine a compiler cluster of those!"

  3. I have nothing against the product by ObviousGuy · · Score: 4, Insightful

    But isn't the point of having subscriptions to avoid advertisements?

    That said, I've certainly worked on products that needed at least 3 hours to build an entire tree. These builds were done regularly around 3 in the morning so that the daily drop was available to QA first thing in the morning. It wasn't really necessary to farm out the compilation across machines because it wasn't a big deal to maximize speed.

    As for developers' machines, it wasn't like every change was accompanied by a full build. You recompile the files that changed and link the object files together. Any smart build system should be able to handle this type of logic. Such a local build would take about 2 or 3 minutes (if that. This time could be made even shorter by using dlls instead of a single binary). I guess this Trolltech system could reduce this wait even further, but I'm not sure I see the point.

    When else are you going to refill your coffee?

    --
    I have been pwned because my /. password was too easy to guess.
    1. Re:I have nothing against the product by Anonymous+Brave+Guy · · Score: 3, Interesting
      As for developers' machines, it wasn't like every change was accompanied by a full build. You recompile the files that changed and link the object files together.

      Sure. OTOH, the project I work on has well over a million lines of C++ code in it, about five developers full time, and a pretty tidy OO design is used to model much of our stuff. If you're playing with a new feature, this quite often goes into a base class to add it across a whole range of areas of the code, and changing the wrong header file can cause a good 10-15 minutes of rebuild (the whole project on a fast machine takes ~1hr). It's the sort of thing that's not normally a problem, but if you do it 10 times in a day while trying to fix that nagging bug, and you're shipping within the week so time is of the essence, then this sort of functionality is priceless. I guess the kind of team where I work is (aside from developing under Windoze) exactly the sort of place they're targetting. To our clients paying a significant sum of money per hour for our time, the $750 to save several hours at crunch time is nothing.

      (And before anyone mentions it, yes, I'm aware of the Cheshire Cat idiom and such, thanks. They aren't in common use on the project, which dates from several years ago, and retrofitting them would be difficult to do without a lot of risk. It's cheaper and safer to just speed up build resources, by using faster PCs or something like the product in question here.)

      --
      If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
  4. Ack... by drgnvale · · Score: 2, Interesting

    Maybe its just me, but $750 for a farm size of 5 seems kinda pricey for a few moments saved here and there. At my job, we have pretty speedy machines, and almost nothing we have built has taken more than 10 min for a complete recompile. And since you rarely recompile everything, its usually a couple seconds to a minute to compile your changes. I can't imagine that this would speed that up enough to justify the price.

    1. Re:Ack... by BagOBones · · Score: 2

      I agree the price seems high.. It was the whole clustering thing that got me interested.

      Hay if you want to compile KDE from source it could come in handy.

      --
      EA David Gardner -"... but the consumers have proven that actually what they want is fun."
    2. Re:Ack... by RangerBob · · Score: 1

      Yeah, this is the problem I see with the Trolltech guys. I mean, I love Qt and wish it would get a heck of a lot more market penetration, but until they finally make their prices reasonable it'll never happen. Look at Qt itself as an example. Even though it's extremely well designed, allows cross platform, and would make Windows developers' lives easier by replacing MFC, it's not worth $1,550. I mean, as much as I'm biased towards it, it's a toolkit and a sorta IDE. I think they'd sell a lot more licenses for stuff in the end if they'd look around and make their prices more competitive.

    3. Re:Ack... by Anonymous Coward · · Score: 2, Insightful

      I doubt dropping the price would make any sense. It doesn't seem likely that halving the price would double the volume: $775 is still not something a hobbiest would spend, and most companies who feel they need something like Qt won't be affected by saving $775 anyway.

    4. Re:Ack... by drgnvale · · Score: 2, Interesting

      The biggest problem with Qt that we have at my job is that the cost is per developer, it cost more to get licenses for multiple platforms, which kinda was the point of it anyway, and the faq used to say you couldn't develop with the free edition and then get licenses before you release. Which stinks if you want to see if your product is doable in Qt before you shell out the cash for your development team to start working.

  5. Trolltech? by Anonymous Coward · · Score: 0

    Get those negative moderation points ready! The trolls are going to have a field day with this...

  6. The Hypocrisy of /. Regarding MS by returnofthe_spork · · Score: -1
    Listen, this has been bothering me for a while. Now, as we all know, most of the people who frequent this site are Libertarians (except for a few socialists - Go back to France!,) which is a fine and noble lifestyle. The problem is that they throw their ideals out the window (bad pun) with any issue concerning Microsoft.

    What exactly do I mean by this? It's easy to see. Any time there is a story about Microsoft, there are 300 posts crying for the government to step in and break MS into two companies, force them to open their code, etc. Excuse me folks, but having the government interfere with the economy is not capitalism. It's communism. I don't want this great nation to end up like Russia, ok?

    Another thing that pisses me off: anytime there is a circle-jerk discussion about Mozilla, people whine and bitch about how Internet Explorer doesn't conform to W3C standards, and how unfair it is. Well guess what? We don't need the W3C (government-wannabe of the Internet) forcing companies to do things. The market has decided - Internet Explorer is the standard. That's how the "Invisible Hand of John Smith" works. If you want to gain market share, follow IE's lead. Otherwise, go back to Cuba or China you commie bastards.

    BTW, if this gets modded down, you can bet the Liberals are out in force modding down all group-think. Fuck You!

    --

    Visit the new Troll site!

    1. Re:The Hypocrisy of /. Regarding MS by Anonymous Coward · · Score: 0

      The Invisible Hand was coined by Adam Smith, not John Smith. Nice point otherwise. Maybe there should be an Anti-Slashdot section inside of Slashdot.

    2. Re:The Hypocrisy of /. Regarding MS by returnofthe_spork · · Score: -1

      Yes I know it's Adam Smith. I did that on purpose. I was wondering when someone would finally point that out though. Cheers.

      --

      Visit the new Troll site!

    3. Re:The Hypocrisy of /. Regarding MS by kermit6306 · · Score: 1

      Wow. Where do I begin? First of all, start referencing specific posts. Next, the situation with Microsoft requires government intervention. Why? Because capitalism must be returned to the industry. One company dictating everything is not capitalism. The government is hoping to correct the situation. The government is also seeking punitive damages against Microsoft. You see, first Microsft broke the law, then they told the government, "OK, from now on, we'll try really hard not to break the law". Our government bought that and now look at the mess we're in.
      As for web standards, you're right, who needs 'em? I'm happy with one company telling what to do. It'll work out in the end. They're rich, right?

  7. Why not dmake? by wickidpisa · · Score: 5, Insightful

    What does this package have that makes it any better than dmake(distributed make)? Convincing anyone that uses linux to go with a $750 product over an open source one seems silly, if they were going to do that they wouldn't be using linux in the first place.

  8. There is a shiney new solution to this by Tablizer · · Score: 3, Interesting

    use an interpreter. (Okay, it is not new. I lied.)

    BTW, is there "lint"-like utilities for any scripting/interpreted languages that detects "suspicious" stuff before running?

    This may help one get the best of both worlds.

    I personally prefer interpreted languages. They are usually easier to read IMO because there is less formality in the way.

    Interpreted languages attempt to reduce problems by making the view clearer, while compiled languages attempt reduce problems by using a fatter air-bag.

    It is sort of like chosing between a fighter plane that is highly menueverable by being light, or one that is heavily armored. They are both different strategies to the same goal. The choice often depends on the pilot and training.

    --
    Table-ized A.I.
    1. Re:There is a shiney new solution to this by costas · · Score: 2

      Well, there are obviously fewer things that untrusted input can break in an interpreted language (no buffer overflows for one).

      I don't know if that's what you have in mind, but at least Python's exec() and eval() allow for namespace definition (i.e. only let the exec/eval'ed code to have access to a portion of the variable namespace.

  9. ccache by ryants · · Score: 4, Interesting
    A similar idea that I've been playing around with and seems to work is to use ccache and put the cache on a network drive.

    That way, if someone has already built the file you're about to build, it's just a copy. Even better is if you have a continous build script caching results ahead of time for you.

    It has worked pretty well for me so far.

    --

    Ryan T. Sammartino
    "Ancora imparo"

    1. Re:ccache by elmartinos · · Score: 3, Informative

      ccache is pretty good, but not especially made for distributed compilation, but there is another project at samba.org:
      distcc.

      /elmartinos

      --
      Open Source Alternatives
    2. Re:ccache by ryants · · Score: 1
      Sweet... thanks for the pointer.

      Now I have yet another thing to play around and waste time with... :)

      --

      Ryan T. Sammartino
      "Ancora imparo"

  10. Mosix? by intu · · Score: 2, Interesting

    What about {open}Mosix cluster?

    http://www.mosix.org/
    http://openmosix.sourcefo rge.net/

  11. free with PVMGmake by bohnsack · · Score: 5, Informative

    You can do this for free with PVMGmake. I have a HOWTO

  12. distcc by Anonymous Coward · · Score: 2, Informative

    check out distcc while you're at it. from Martin Pool.

  13. I don't know about this by Henry+V+.009 · · Score: 3, Interesting

    Program compilation is not an obvious candidate for parallelization because it is an extremely memory intensive process. So to do this right you'd need very fast communication between your CPUs.

    However, what Trolltech is probably doing, is farming out the independant bits of your program that you would link together in the end anyway. That might really help on rebuilding the entire project at once. However, if your program is set up well, you usually only need to recompile one file at a time.

    So the times when you could see the biggest speed up in compilation, you will need this least.

    1. Re:I don't know about this by MrBoring · · Score: 1

      In my case, compilation is a big issue. I may be unique in porting code from an obsolete C++ compiler to another, but I have to go through 4000+ C/C++ files. There is lots of independent libraries that could be built concurrently.
      A build for one platform takes 2-3 hours, which I unfortunately have to test.

    2. Re:I don't know about this by Henry+V+.009 · · Score: 2

      Whoa. It does sound like this sort of thing could help you. How often do you need to do that sort of stuff?

    3. Re:I don't know about this by MrBoring · · Score: 1

      I'm on a porting assignment right now, so I do large builds continually. Fortunately, this assignment is coming to a close, though. Nevertheless, getting $1500 for software is a tough sell for the lowly trenchman.

    4. Re:I don't know about this by Chalky · · Score: 2

      Sure, if you modify one .cc (or .cpp) file at a time, you will only compile one file. And if you keep your sources short and your headers organised it wont take long anyway. But what if you modify a header file? In a library used by the whole application?

      While I was working over summer, we regularly made changes to a core IDL file (hey, I wasn't the guy who put all the system interfaces in one file!) resulting in a 4-hour coffee-break on a decent PC And this was just the system layer and GUI. To rebuild everything (including embedded code for various modules) would take 12 hours or more.

      Sure you may not recompile everything very often - but imagine what the 12 hours (or 4 hours) would be like if you used the 100 developers PCs on that floor alone. Of course, at some point - probably well before 100 - the cost of organising the distributed compile and transferring files around starts to negate the benefits.

      How many full builds do *you* make before each release, when it can take months from initial feature freeze to release? How much will it affect your shipping date? (and how many $$ in wages are you paying people for those hour long coffee breaks all the time?:)

      --
      --Chalky
  14. Clearcase by Steve+Cox · · Score: 1

    You can do distributed builds with ClearCase (I remember doing this a few years back on Digital UNIX). Quite easy to do really. Quite expensive for cleacase licenses (oh yeah, and DEC compiler licenses...)

    Steve.