Make Out with SCons

Nice2Cats writes "I stumbled across this and can't find a mention of it here yet: SCons, a stable alternative for the vilified make. Built-in support, they claim, for C, C++, and Java, built-in support for use with SCCS, CVS, and BitKeeper, supports .NET, and works with Linux, *BSD systems, Windows NT, and Mac OS X. I'd love something to would finally get rid of the awkward (no pun intended) make, and having a system based on Python would make it double good. Maybe one of the serious C/C++ programers here could give a verdict on if this is the dragon slayer we've been waiting for."

Comments

Be careful...

[Anonymous Coward]

SCOns may have SCO's intellectual property!

SCOns?

[puckhead]

They need to change the name.

Re:SCOns?

[Directrix1]

Jeez, just use Ant and be done with it, already.

Re:SCOns?

[wizs]

I think SCS is better.

Re:SCons?

[jpu8086]

I think so. I would never miss out on a chance to make out with MyBride.

Questions

[Zach Garner]

Integrated Autoconf-like support for finding #include files, libraries, functions and typedefs.

Is SCons viewed as a replacement for Autoconf and Automake, or just a replacement for Make?

Built in support for...

What does this mean? Will SCons somehow only work with languages that SCons has built in support for?

How does SCons compare to Apache's Ant?

Re:Questions

[Teancom]

What does this mean? Will SCons somehow only work with languages that SCons has built in support for?

Well, I haven't actually looked at scons, but in Make, there is only built-in support for a limited amount of languages. For instance, (maybe this has changed recently) with ObjC you have to define how to take care of .m file, as it doesn't know to compile them. .c and .cc files it knows to use CC on, etc.

How does SCons compare to Apache's Ant?

Well, the fact that it isn't based on Java or XML gives it a good head start :-)

Re:Questions

[sporty]

Well, the fact that it isn't based on Java or XML gives it a good head start :-)

The good thing about doing ant conf files in XML, si if they change the format, you can use xsl to transform ALL of your xml to a new format.

As for the java thing, run's most everywhere.

Re:Questions

[Elwood P Dowd]

You misread his post. He actually said, "I don't like Ant for religious reasons."

Re:Questions

[pmz]

Well, I haven't actually looked at scons, but in Make, there is only built-in support for a limited amount of languages.

Adding implicit rules in make is not difficult. It's in the man page (even Solaris' crappy man page for make describes it well enough, and the system default rules file has lots of examples).

Well, the fact that it isn't based on Java or XML gives it a good head start :-)

Agreed.

Re:Questions

[pmz]

The good thing about doing ant conf files in XML, si if they change the format, you can use xsl to transform ALL of your xml to a new format.

Layers upon layers of complexity. XML, XSL, Java, Ant Schemas...all for compiling a program. My sig is appropriate, here, too. A build system for software should be easy, intuitive, transparent, etc. etc. There has yet to be a replacement for make that has these attributes (makes tab syntax really isn't that big of a deal, either, even though that's what most people latch onto when complaining about it).

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Re:Questions

[Teancom]

Adding implicit rules in make is not difficult. It's in the man page (even Solaris' crappy man page for make describes it well enough, and the system default rules file has lots of examples).

Sorry, didn't mean to imply that it was difficult. My point was that having "built-in" support for a subset of languages in scons is no different than make. And I don't doubt that adding ObjC support to scons would be much harder...

Re:Questions

[sporty]

Layers upon layers of complexity. XML, XSL, Java, Ant Schemas...all for compiling a program. My sig is appropriate, here, too. A build system for software should be easy, intuitive, transparent, etc. etc. There has yet to be a replacement for make that has these attributes (makes tab syntax really isn't that big of a deal, either, even though that's what most people latch onto when complaining about it).

Whoahoahoa.. no. Blatantly wrong. If ant 1.0 used one particular XML format, and you upgraded to ant 2.0, to go from an ant 1.0 file to 2.0 file, you'd apply an xsl style sheet ONCE to upgrade it all. Any future XML you write would be in the 2.0 format. Simple as that.

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Bash? csh? sh? tcsh? bash 2.0? Which shell are you suggesting is usable across them all?

ant is just a program, just like make is. You run it against a config file of a certain format, just like Makefiles have their own format. Ant is no more compicated than make. hell.. you can use the core stuff from ant to run any program across many files, just like make. The targeted tasks are just to make those easier.

Re:Questions

[jonadab]

As long as the format is regular, it's pretty trivial to write a
simple script to do the transformation. XML helps enforce a certain
amount of regularity, but it's by no means the only regular format.

Personally, I like XML pretty well in general, and I really like
what OO.o has done with it in particular.

I'm less enamored with Java, but that's another thread for another day.

Re:Questions

[Jerf]

A build system for software should be easy, intuitive, transparent, etc. etc.

No. A build system for software must reflect the complexities what it is attempting to build. No build system for C++ programs can ever be fully "easy, intuitive, transparent, etc. etc." because there are C++ programs that aren't any of the above. Same for C.

Ever tried to compile XFree86 by hand from source? It must be possible, because Gentoo can do it, but I've never managed myself. While it can probably be made easier, the simple fact is that it's a complicated set of programs with a complicated build order and no matter how hard you try, it would take a full-fledged AI to make it "easy".

On the other hand, a pure Python project, even a large one, needs no special build system at all, because Python automatically compiles with few or no hassles (and what hassles exist can not be solved by a build system, they tend to be user error). Of course large projects are rarely all Python (building C extensions, compiling human-readable data representations into faster data or code, etc.), but since Python is more simple, the build system can be equally simple (or in this case, non-existant).

Make can be improved on, it's a decades old design with multiple layers of hacks jostling each other. A new, clean design can at least simplify away the parts of the build process that involve "fighting with make". But a build system for C can only become finitely simple, or you'll start to lose capabilities that C has because your build system can't handle them.

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Proof by counterexample: autoconf exists, therefore it is clearly necessary, because nobody would trouble to build it if it didn't address a problem. (Whether it solves it is another question, but given its vast popularity in the open source meritocracy I'd say the onus is on you to show that it truly causes more problems then it solves; every project that uses it is a vote against that statement.) You've clearly never written massively cross-platform code; grab a decent sized package, look at the output autoconf generates, and look at what the program does with it. Odds are every single line you see during configure, with the possible exception of some really standard ones that seem to be a standard part of the package ("CC works?"), is used somewhere in the program, especially lines like "Is your pthreads implementation broken? ... yes, using workaround". The workaround may itself not work if pthreads isn't broken!

Re:Questions

[jonadab]

Shell scripts aren't terribly portable. makefiles aren't either.

Just write everything in Perl, and you don't have these problems.

Re:Questions

[sporty]

As long as the format is regular, it's pretty trivial to write a
simple script to do the transformation. XML helps enforce a certain
amount of regularity, but it's by no means the only regular format.

True, but writing parsers for things like, perl, is a pain in the butt. At least with XML, writing XSL to transform is realtively easy. Even for ant.

Thank god make isnt' written to use perl as its language. *shudder*

Re:Questions

[diewarzau]

Built-in support means you generally don't have to write rules (or "Builders" as they are called on SCons parlance) for these tools. SCons allows you to write your own Builders, which can range from very simple to very complex depending on the needs of the tool, to work with just about any tool chain you need. So, "built-in" simply means you don't have to extend SCons at all to use it, and also migth have other implications such as being able to automatically generate dependencies for certain file types, like figuring out what header files a C file depends on. As far as a comparison to Ant, in my experience Ant is great for building Java projects, but a little bit of a pain to get to work with C++. Also, it must be extended in Java, which is a little less weildy than a scripting language. The one-line comparison is SCons's Java support lags behind Ant currently, but I prefer SCons for just about everything else.

Re:Questions

[p2sam]

To be fair, XFree86 source compilation has improved a lot in recent versions, I downloaded the source, and "make World", "sudo make install", and then I am done.

Re:Questions

[4of12]

look at the output autoconf generates

Worse, as you begin to try to standardize coding of your configure.in and Makefile.in files, it begins to dawn on you just how much more boilerplate could be automatically generated.

Ergo, automake.

However, by the time I started learning the syntax for automake and tried to figure out just exactly how libtool can magically make shared library construction and maintenance a cross-platform reality, I get tired of the whole damn business and think: there has got to be a better way.

I don't know that SCons is that better way, because the problem it has to solve is complicated.

Personally, I had high hopes that the software carpentry project would have chosen some of the XML features of Tan by David Ascher - specifically, having a way for the build to query the system for some information about the best place to find previously installed software.

Fancy autoconf m4 macros (clever, but ugly) can indeed check for existing installations of other packages and accepting user-specified overrides, but somehow it seems to me that having some databases (XML, but simple enough to edit by hand) to describe how you want the build to look for packages is better than specifying 8 options like --with-ssl-include-dir=/some/long/path, etc.

Finally, despite being a Python fan, I've always been skeptical about whether the SCons approach to replacing Bourne shell with the admittedly more powerful language was really a good idea, mainly because of the broader availability of sh.

But time has made less relevant those systems on which Python cannot be built, so perhaps my concerns are overstated.

Nevertheless, if you want to build Python, I notice that it uses autoconf:)

Maybe if Python had a module that could crank out Bourne shell like m4 does, but without the ugly m4 syntax?

Re:Questions

[PD]

I agree with you completely. I just looked at some scons makefiles and one thing was apparent: Python makes a terrible language to specify dependencies in.

I've looked at many many different supposedly better replacements for make. None of them do the job as simply as make.

Re:Questions

[pmz]

A build system for software must reflect the complexities what it is attempting to build.

Compiling a program and linking it into executables, libraries, jar files, etc. simply isn't that hard to do. A programmer that doesn't understand how to do it has some learning to do. Tools that claim to make the learning unnecessary are misleading--what does the programmer do those tools break and leave them stranded?

Ever tried to compile XFree86 by hand from source? ...it's a complicated set of programs with a complicated build order and no matter how hard you try, it would take a full-fledged AI to make it "easy".

There would certainly be many makefiles, but they don't have to be complex. Just the bare minimum rules and environment variables should do. A separate file containing all volatile variables make customizing the build for platforms not difficult. Build systems do not need to be clever nor convoluted. All they need to do is compile the program properly and consistently.

Make can be improved on, it's a decades old design with multiple layers of hacks jostling each other. A new, clean design can at least simplify away the parts of the build process that involve "fighting with make".

Make can certainly be improved, but a new clean design has yet to be presented. Build systems that depend on Java or Python are not clean, because they introduce unneeded and unwanted dependencies into non-Java and non-Python programs, and they make for a build system that can be more complex than the program being built.

Proof by counterexample: autoconf exists, therefore it is clearly necessary, because nobody would trouble to build it if it didn't address a problem.

I'm not exactly sure what the original problem was that motivated autoconf, but autoconf and libtool cause me many many many problems on non-Linux platforms. I'd much rather have a single configuration file that tells gcc and ld where all the needed libraries are. All the tricks and things built into the 15000 (yes, fifteen thousand) lines of shell script that are libtool and configure simply break builds. Often.

given its vast popularity in the open source meritocracy I'd say the onus is on you to show that it truly causes more problems then it solves

The reason it's popular is that they are GNU, and people lap it up. Searches on newsgroup archives, especially for BSD systems, indicate that I am not alone in critizing these tools.

every project that uses it is a vote against that statement

And every project that won't build on Solaris because of libtool is a vote for it.

Odds are every single line you see during configure, with the possible exception of some really standard ones that seem to be a standard part of the package ("CC works?"), is used somewhere in the program

Perhaps the fault is in the overly-complex program? It seems the programmers write what they please and, then, invent these kludges of tools to deal with those programs. It is completely backwards. If a program is requiring tons of smoke and mirrors and special allowances and rules and exceptions, then the program is due for an overhaul.

Re:Questions

[Mr. Piddle]

Simple as that.

What do you do when XSL can't adequately map the files? What do you do when the transformation gets you only 90% of the way there requiring time-consuming debugging and finish work? Murphy's law applied to nifty ideas like XML means that there will absolutely be unanticipated shortcomings that aren't well understood due to the complexity and immaturity of the tools being used.

Bash? csh? sh? tcsh? bash 2.0? Which shell are you suggesting is usable across them all? /bin/sh. That's it.

Ant is no more compicated than make.

Layering the J2SE runtime onto an application for the sake of compiling is "no more complicated"? This is only true for Java applications.

Re:Questions

[Mr. Piddle]

Shell scripts aren't terribly portable. makefiles aren't either. /bin/sh (Bourne shell) and POSIX makefiles should be more portable than Perl. At worst, the portablility is the same, because the platform-dependent paths need variables to configure, for example.

Re:Questions

[IamTheRealMike]

I'm not exactly sure what the original problem was that motivated autoconf, but autoconf and libtool cause me many many many problems on non-Linux platforms.

Nothing is perfect. Autoconf was built, essentially, to abstract imperfection from the developer.

Read up on its history, it's in the info pages. Back when Linux was a nothing, people wanted free software to compile on pretty much any machine they could. They all varied in obscure ways, some had features others did not and so on. Before autoconf, people spent a long time writing their own configuration scripts. Autoconf, which automated the process and provided libraries of high quality scripts (relative to peoples home made ones), was a godsend. The reason it uses m4 by the way is that unfortunately, shell functions did not fall into the lowest common denominator of features that all UNIXen were guaranteed to have.

Now, we have scons. Scons, from what I've seen, is good. I'm still not entirely convinced that Python is the best syntax for describing a build system, but I'll withhold true judgement until I've tried it.

It probably addresses different needs. Today, being able to build on obscure deceased form of UNIX A is arguably less important than having an easy to use build system for 99.99% of the users who are on Linux or one of the BSDs. Windows users often put their own build system in when porting, so that isn't a big deal.

I know that scanline of PicoGUI fame has been playing with a sparkly new build system based on scons and some other stuff, that looks very nifty indeed. It'll be interesting to see if scons, pgbuild and the rest eventually take over from autotools.

Re:Questions

[Jerf]

Compiling a program and linking it into executables, libraries, jar files, etc. simply isn't that hard to do.

I'm not exactly sure what the original problem was that motivated autoconf,

It seems the programmers write what they please and, then, invent these kludges of tools to deal with those programs.

Between these two statements, combined with the fact you didn't deny it despite obviously going over my message with a fine-tooth comb, you're only reinforcing my original guess that you've never written a large program or worked on one.

The first is just flat wrong in static languages; compile orders are frequently hairy and circular dependencies happen with distressing frequency, and tools can only help so much, especially when the large project start including multiple languages or compiled data sources.

The third makes it even more obvious you've never done this stuff; I've had to insert workarounds for critical applications between two different point releases of Netscape 4. Given that the code is out of my control, and given that even in the open source world it is no longer feasible to fix every bug in every library you may have to use (especially since such fixes often would involve major architectural changes in a project you don't really care to get involved in and may themselves break umpteen other projects for whom it's literally not a bug, but a feature), the it's not possible to remove the workarounds. Characterizing them as "smoke and mirrors" just pushes the point home even further; any given project ought to be clean within its own boundaries, yes, but there's no conceivable way, even in theory, for the boundaries of disparate projects to be clean.

You clearly are not a serious programmer and I do not find you competent to have an opinion on project building tools that is of any value to me. Barring pointing to a major product with "pmw" listed as the lead developer, I don't expect you can change my call on that. (And of course everyone else is free to hold their opinions; make sure you actually read the opening sentence of this paragraph.)

Re:Questions

[pmz]

You clearly are not a serious programmer and I do not find you competent to have an opinion...

I'm glad you are such a good judge of character.

I actually don't disagree with what you wrote about your experiences as an uber-elite programmer, because that's the reality of working with a large corporate bureaucratic mess of software that was created over years of short-sighted decision-making. My original argument was simply against the kludges being presented in the open-source world as suitable build tools, when they create as many problems as they solve and make Open Source look no better than Windows (the worst kludge of them all). It doesn't take a programmer to see that progress is not being served either by these tools nor by stubbornness that clings onto what should be embarrasing historical baggage.

Re:Questions

[jonadab]

> Thank god make isnt' written to use perl as its language. *shudder*

If the software were written in pure Perl, you wouldn't need make
to install it.

Re:Questions

[jonadab]

> Shell scripts aren't terribly portable. makefiles aren't either.
> /bin/sh (Bourne shell) and POSIX makefiles should be more portable
> than Perl.

Only in theory.

In practice, some systems use csh as the default sh. In practice,
getting any sh at all on some systems means installing an entire
POSIX layer that doesn't come standard, but Perl can be installed
without that. In practice, some POSIX systems don't implement the
full POSIX standard, so you end up tracking down niggling bugs
where one platform's version of some command doesn't have the
option you were using. Also in practice, Perl is more than a
replacement for sh and whatnot; it is in many cases a replacement
for C and C++, and it is distinctly more portable than those.

Of course, the ultimate holy grail in portability is to just write
all your software to target the z-machine, compile once, and then
it can be run on anything from IBM mainframes to the TRS80 to the
gameboy, using standard z-machine emulators available for every
platform known to man (and several that are generally considered
unknown). The problem with this approach, of course, is that you
can't do things outside the scope of the z-machine's abilities.
It's turing-complete in theory, but it suffers in terms of I/O
because of certain assumptions it doesn't make. (For example,
the z-machine does not assume it's running on a platform that
has a filesystem or can allocate memory at runtime.)

No Screenshots?

[Randolpho]

How am I supposed to know if I like the look and feel if there aren't any screenshots? :D

Re:No Screenshots?

[Stele]

And does it have shadowed menus? I really don't think I could use such a program if it didn't have shadowed menus.

Already there...

[KDan]

Though i guess for now it only supports java - but there's nothing preventing you from writing custom tasks for it.

It's called Ant.

Daniel

SCons?

[shadowbearer]

Am I the only one who saw the title to this and thought is was another article about that sexy Darl McBride and his burgeoning company?

and thought "No thanks"

SB

stable alternative to make?

[Anonymous Coward]

I've been using make for like 10+ years. What exactly is the instability?

One thing I never liked about make was the dependence on whitespace in the makefile. Hopefully this Python-based solution will fi...uh never mind.

I predict Make will be around for a while still. I've used Ant but Make is fine for 90% of what I do.

Re:stable alternative to make?

[metamatic]

I thought you were joking, but then I went and looked at the web page and discovered you were serious, unfortunately.

Yay, let's build a replacement for make, using a language that repeats the biggest design flaw of make.

Re:stable alternative to make?

[digulla]

When I had to do FORTRAN, I hated to have to put one tab or eight spaces in front of every line (that's where you put labels in FORTRAN), so my first contact with languages which depend on a certain input format was a bad one.

Then I found Python. I didn't like it in the beginning but I found that Python code is very easy to read because it enforces a certain format.

Of course, Python doesn't repeat the biggest mistake of Make: It doesn't bother if you use spaces or tabs as long as your tabsize is 8.

And the amount of screen space which can be saved if you don't need block parentheses anymore, is amazing...

I have written code in C, C++, Java, various shell dialects, Tcl/Tk and Python.

Python has a better OO system than any other language, it has a better syntax than anything else and it is easier to understand than any programming language I've seen so far.

It is a bit slow, though, and when you must have a grip on yourself or you can easily write very compact code that does amazing things ... that you don't understand anymore. Oh, and the GUI is still a weak point (it uses Tk and that means that there are thousands of extensions which don't quite fit what you need so you have to write it yourself, again).

But I've yet to find a language which even comes close to the power Python gives me and to the speed one can write solid code in this language.

Re:stable alternative to make?

[metamatic]

The problem is, my tabsize isn't 8, and it's never going to be. So I'm not going to touch Python.

Re:stable alternative to make?

[Anonymous Coward]

I don't know what the parent was thinking. Tab size doesn't have to be 8. Mine is 2 and works fine.

Re:stable alternative to make?

[axxackall]

You don't have to keep your tabsize as 8 - it can be any other value in Python. Moreover, you don't have to use tabs at all - all your program can be just a single line:

print [(x,y) for x in (1,2,3,4) for y in (10,15,3,22) if x*y > 25]

Does not solve the problem, but makes it even wors

[tjansen]

Unfortunately it does not solve the problem. (Auto)Make's main problems are the lack of a robust syntax which would make it easier to give good error messages / feedback, the lack of a syntax that can easily be parsed and modified in IDEs and that it is far too complex for the casual developer / contributor. Most people want to program, not fiddle with the Make system. Scons may have some evolutionary improvements, but does not solve those issues. Actually replacing the make files by python scripts makes it only worse.

Ant solves the problem, but only for Java (admittedly building Java is much easier than C/C++).

YAMR (Yet another make replacement)

[pmz]

I'd love something to would finally get rid of the awkward (no pun intended) make, and having a system based on Python would make it double good.

So now my C, C++, or Java programs can be dependent on yet another language runtime environment? Awesome!

Yes, there are limitations to Make, but why hasn't a genuinely better replacement materialized in the last twenty years and become truly widely adopted (in a manner like Perl, for example)?

Truth is, if you avoid fancy GNU extensions and stick to POSIX, make really isn't all the stuff people cry about.

Re:YAMR (Yet another make replacement)

[egomaniac]

why hasn't a genuinely better replacement materialized in the last twenty years and become truly widely adopted (in a manner like Perl, for example)?

The Java community uses Ant almost exclusively, and Ant kicks the living crap out of make as far as, well, everything.

True, the C++ community has had trouble coming up with anything better than make, but come on -- we're talking about people that consider C++ to be a reasonable 21st-century programming language, which already leaves me with serious doubts about their sanity. Given that, I'm not too surprised that most of them also consider make to be a reasonable 21st-century build tool.

SCONS is awesome

[alyosha1]

After spending several hours trying to get make to do something complicated with shared libraries and multiple configurations, I started looking for a replacement, and stumbled across SCONS. It took me a few moments to download and install it, and a very short time to go from no knowledge of the system to having solved my problem.

Since then I've shifted several of my projects to using SCONS as the build system. I'm sure it's theoretically possible to achieve what I was trying using make, but I wasted many hours wading through man pages and howtos without success, and SCONS did it with almost zero effort.

I haven't tried using it on Windows yet, but my understanding is that you can use the same script under both platforms.

So, just one data point, but my experience with it has been 100% positive.

-alyosha

Re:SCONS is awesome

[Punto]

I haven't tried using it on Windows yet, but my understanding is that you can use the same script under both platforms.

I work on a project of about 600+ cpp files, we build on linux (gcc2 and gcc3) and windows (gcc2 and gcc3 on cygwin and mingw, msvc). Scons worked almost unchanged with the cygwin and mingw versions of gcc. VC++ required some minor changes, mainly because of the specific command line switches, and because the guy who 'ported' it was very anal about integrating it perfectly with his miscrosoft interface (wich he did; he runs scons from within 'microsoft visual c++ .net', or whatever it's called) instead of just using the shell, like a real man ;)

I never thought I'd see the day

[lightspawn]

A /. story about making out!

Re:I never thought I'd see the day

[JDWTopGuy]

Yes, but should it be "Quick & Dirty" or "Correct and Proper"?

the make replacement is (drumroll please): sh

[ArmorFiend]

[rant-on]
make's syntax is so incredibly baroque. Its basically only to do one thing, but over the years its gotten so much crap stapled onto it that its a time trap. Add to that automake, which has to be layered on top just to keep make doing its job. Stop the madness!
[rant-off]

Nowadays I just write my build scripts in sh. Its on every platform. It takes less time in the long run because its a real programming language.

For example, in make when you want to do arcane-thing-X, you have to dig through the manual to figure out what feature is there to help you, then fret about whether you're only supporting GNU make, blah blah blah.

With sh you already know how to do anything since its a general purpose computing language with good support for groping files.

Re:the make replacement is (drumroll please): sh

[SeanAhern]

I'm a bit surprised you aren't more of a make fan. Since make is, by default, based on sh, you should be able to use the power of a general purpose computing language inside the timestamp and rule framework of make.

Not that I'm a big fan of either, but the combination of two really makes a powerful system.

Re:the make replacement is (drumroll please): sh

[ArmorFiend]

Hm, I guess I shrug and say I like my eggs inside my chickens, instead of my chickens inside my eggs. Its perfectly easy to get timestamps from within sh. And there's no tab-versus-space confusion or millions of compatibility headaches.

(well, OK, time for complete honesty. I do use really bash, and I do occassionally have a bug on windows from windows' non-unixy filenames. There, that's off my chest. :)

Re:Does not solve the problem, but makes it even w

[Circuit Breaker]

Spend some time to look at Scons. It's not a silver bullet, but it's a great step forward.

It can, among other things, generate Makefiles or Visual Studio project files, or run the build itself. It does have "autoconf" style checks, which you can embed in the sconstruct/makefile. It's built for parallel make from the start. It does not suffer from network clock skew.

And it has much better multiplatform support than any tool I've seen so far (automake/autoconf, make, qmake, tmake, cmake).

It will take time to mature, and it doesn't solve the problem, but it does makes things much better.

My SCons experience

[vslashg]

My SCons experience is that I like it a lot!

We use SCons at work to build a huge, highly componentized project with upwards of 5000 source files. It's hands down the best experience I've ever had for building large projects. The other two big projects I've been on used GNU make and Microsoft nmake, and neither was satisfactory.

The nmake-based system required running "nmake depends" before a build, and it simply wasn't reliable. Paranoia was high; when we hit a bug while developing, the first thing we did was a clean build, just to make sure we weren't looking at a build problem. (It was justified paranoia -- half of the time, it WAS a build problem :-) The recursive GNU make system was developed in-house and was much more solid, but painfully slow and absolutely horrible to use. It required Cygwin (which is evil, IMO) tools and bizarre mappings from source control, so it was hard to convince other people in the company to build our project.

SCons was loads faster than our (admittedly bad) GNU make recursive build system, though noticably slower than our nmake build. With that cost, though, you get a correct build every time, something we never got in our "make depends" system. The paranoia is gone. SCons caches dependencies, so the cost for this accuracy isn't as bad as you might think. There were hiccups in the past, but the C++ dependency generation has been rock solid for at least nine months now.

I have a little toy project at home with maybe 25 source files which also uses SCons. I use MinGW's gcc at home, and SCons worked well with it "out of the box". Right now all I do is maintain a list of source files to build just like I would with any other system, but the setup was painless, and it's awesome to know that I can write Python to solve any problems I might hit later.

My big complaint with SCons is that a no-op build (running SCons when no source has changed) takes way too long on our big project, making iterative development painful. There's a command (-U) to build only components in a subtree of your file system, but that doesn't really speed things up much. Hopefully they can improve on that. That's about my only gripe, though.

Re:My SCons experience

[diewarzau]

(SCons developer here)

Yes, no-op build time is one thing we continually optimize for. Believe it or not, it's about 50x faster than it was at our initial release :)

Also, there are several things you can do to sacrifice potential build correctness in order to speed up no-op build time, which are documented in the SCons Wiki. Unfortunately the Wiki appears to be on the fritz right now.

Our philosophy is that, by default, we provide an absolutely correct build, which means recalculating/rescanning a lot of stuff every time -- the equivalent of doing a make depends;make every time (I venture that our no-op build time is faster than that!). However, if you know that you have not changed certain things, you can keep SCons from rescanning them, which is great for when you are in the compile/fix/recompile mode. I usually have an alias for "fast, not-so-accurate SCons" that I use for development and "accurate SCons" that I use for testing and release.

Finally, there is a separate effort toi "daemonize" SCons. SCons has an as-yet-unpublished API that allows Python developers to put other wrappers around it other than the SConscript (i.e., makefiles-in-Python) interface that ships with it. The daemon effort keeps SCons around and rescans dependencies in the background a'la MS DevStudio, saving lots of time during the build. I believe the people on this project have seen as much as a 100x performance increase for no-op build times for large projects. Amazingly, it looks like we don't have a link to this on our page. I'll try to dig it up.

Re:My SCons experience

[mughi]

SCons was loads faster than our (admittedly bad) GNU make recursive build system, though noticably slower than our nmake build. With that cost, though, you get a correct build every time, something we never got in our "make depends" system.

Why, of course it was. Remember, Recursive Make Considered Harmful . By just applying that insight to a medium library after first reading the paper, I saw accuracy go way up and scan time go down by an order of magnitude. Oh, and no-op builds went to maybe 10 seconds instead of upwards of two minutes.

A-A-P

[LarryRiedel]

Something similar/related is A-A-P from Bram Moolenaar of vim. A 1.0 version was just released last week. It is a make+Python kind of thing. There was short article about it at osdir a few months ago.

Larry

Re:A-A-P

[DancingSword]

Yeah: I'd have thought that The A-A-P Project or JAM
Perforce.com, Jam
would be the ticket...

Has anybody compared 'em?

Re:A-A-P

[be-fan]

I tried to figure out Jam once. Couldn't do it. It's hideously complex, and the documentation is poorly written. Went with scons instead :)

Re:A-A-P

[DancingSword]

Thanks for the info
: )

So now anyone compare The A-A-P Project with SCons?

Slashdotted?

[metalmaniac1759]

Has the scons site been slashdotted?
And I just checked out - Gentoo has an ebuild for it too. Go Gentoo!!

Nandz.

jam

[Horny Smurf]

look at jam sometime. It's a free make replacer by the same folks who wrote perforce (the commercial cvs killer and bitkeeper competitor).

The jamfiles are easy to set up, and much faster to calculate dependencies -- I built freetype via gnu make and via jam. Jam was done building before make had even started compiling!

Also, props to perforce. It's proprietary (just like bitkeeper), kicks cvs's ass (just like bitkeeper), and is free for open source projects (just like bitkeeper). MySQL uses p4, IIRC. Unlike bitkeeper, you can get a single-seat version free (I use it for my internal projects, though CVS would probably suit my needs).

Re:jam

[egomaniac]

I'm going to pipe up here with some more Perforce lovin'. My team uses it for our sccs here at work, and I honestly can't imagine going back to CVS after having used Perforce. It's a hell of a good tool.

Re:jam

[Anonymous Coward]

correction: bitkeeper is free for open source projects that DON'T COMPETE WITH BITKEEPER

also McVoy is an asswipe.

also, Mac OS X dev tools use Jam, check it out.

Make vs. make

[V. Mole]

One thing I see is a lot of people referring to how slow and unreliable recursive makes are, and they're completely correct. However, you can make your life a lot easier by using make but getting rid of the recursion, using a make with an 'include' feature. See Recursive Make Considered Harmful for nice write up on why recursive makes don't work, and how to do it correctly.

I converted our build system, and no-op builds went from a few minutes to about 5 seconds. Really. If I change one file, all the necessary stuff gets rebuilt, and none of the unnecessary. Every time. With a little gmake if-then-else (all in one place, not scattered around in a bunch of files), the same Makefile and segments work on Linux, AIX, and Solaris. And while the organization of the files is a little unusual for those who are used to the recursive style, they are just makefiles, no new syntax.

The downside? It pretty much requires gmake, but since this particular product is proprietary, we control the build machines. I don't think you can use automake...but I consider that a positive :-). You could use autoconf -- just put all the config stuff in one makefile fragment, and include it into the master.

Re:Make vs. make

[Jason Pollock]

Recursive make works fine for us here. You only need a couple of rules:

1) Get rid of cyclical dependencies!
2) Separate global vs local interfaces.
3) Have a centralised make system that works 90% of the time by filling in a list of source files.
4) Differentiate between libraries and executables.

How do you do this? We use a simple build order.

1) Copy out the changed global header files.
2) Build your libraries.
3) Build your shared libraries.
4) Build your executables.

Easy enough to do. We have a subsystem/programs separation. Libraries go in subsystem, anything that needs a library goes into programs.

So, the order of library build doesn't matter, the only dependency is on the interfaces (header files). The build order of executables doesn't matter, they only depend on the libraries... No fancy directed graphs, simple to explain. We even have header and library dependencies!

Too easy. :)
Jason Pollock

Re:Make vs. make

[V. Mole]

In other words, you've worked around the flaws in the recursive make system by forcing an artificial structure on yourself. Which is not to say that some of your structure isn't a good idea by itself, but the first time you forget to build something in the right order, you have a broken build. Or, you spend a lot of time building stuff that isn't immediately needed.

I can't over-emphasize how nice it is to be able to be able to type 'make foo' from anywhere in the source tree and be confident that what I'll get is exactly what I need and correct. You simply can't do this with a recursive make system - the global dependency relationships are not available. The fact that it's also faster is just gravy.

Is it worth re-doing a big build system completely from scratch? Maybe not. But if you're starting something new, or already know that you're going to be doing significant work on an existing system, then I strongly suggest you read the article previously referenced. If nothing else, it will tell *why* your recursive makes aren't working reliably, and you'll be able to figure out the required work-arounds.

Re:Make vs. make

[Jason Pollock]

You can type "make" from the main directory, and everything builds correctly. You even get told which downstream changes are needed (side-effect detection! Cool!). You don't get that from "make foo". To top it off, it doesn't build anything that doesn't need to be built either.

As with any large scale software engineering, you will need to put some sort of structure in or you will fail. Not might, not possibly, but will.

You will need some things like code style guides, documentation guides, testing guides, etc. So, a little "artificial structure" makes it so that anyone, regardless of how long they've been with the company and team, is able to pick up the software, find the relevent piece of code and get it to build - the first time.

As for performance, there are many, many tools out there to help with that - try ccache and distcc. We got complete builds from scratch down from 3 hours to 20 minutes. :)

I do intend to read the paper though. :)

Jason Pollock

Re:Make vs. make

[axxackall]

Your workarounds remnd me Portage from Gentoo. Have you ever tried it?

SCons: so far so good

[SilentTristero]

We switched away from Make a few years ago. I looked at jam and other make replacements before settling on Cons, the perl-based precursor of SCons. We are now in the process of switching over from Cons to SCons (slowly), and so far it is working very well.

We are using it on Windows (MSVC and Intel compilers), Linux, IRIX, and Mac OS X (gcc and two versions of CodeWarrior). Handles all of those with ease. It can do things like properly handle dependencies on auto-generated source and header files, which would be a nightmare in make.

Some things like error handling and documentation need some work, but overall it is very solid. Automatic dependency checking, parallel build support, autoconf-like stuff, switchable tools, and easy to extend.

Yes, it requires python, but will run on just about any version back to 1.5.2, and doesn't require any python add-ons, so it's not too onerous a requirement really.

So far I am very happy with it; I'd suggest checking it out if make is making you crazy.

-- Tristero

Blah

[photon317]

Everyone tries to replace make and freinds, and everybody fails. (Well, at least fails to be a true replacement that's actually better in every way). A few small tweaks aside, you can't really make serious design improvements to the standard unix make and related tools without taking some large tradeoffs that make it unacceptable for at least some uses. It's a very flexible tool as it stands, and it's hard to make it better without losing that flexibility.

SCon vs ant for java?

[herrlich_98]

I've used make/gnumake a lot for C/C++ and Java. I've used ant a lot for Java.

For a large primarily Java project ant is the hands down winner. It has two advantages over make. Out of the box, ant "knows" how to build java and java technologies. The same way that the make "knows" how to build C/C++ and UNIX style stuff. Sure you can do this in make but not without a lot of additional rules and setup... and it never feels as seamless as in ant. The second thing ant does is only use one java JVM to do the build. Each java compile happens in the same JVM that ant is running in. With make you are constantly reinvoking the JVM and it really slows things down.

parallel builds on Win32?

[blacksqr]

gnumake's -j option (for executing jobs in parallel) doesn't work on windows... does SCons'? That might be the killer feature.

Re:parallel builds on Win32?

[aroach]

Yes SCons's -j option works on all platforms that SCons works on. I should know, I wrote the -j support.

BB Scons config file

[adamshelley]

mmmmmmmmmmmm

Plan 9's solution

[yandros]

Plan 9's new take on `make' is `mk'. Unfortunately, I wasn't able to find a good on-line source for the original paper on mk:

A. G. Hume, Mk: a successor to make , USENIX Phoenix 1987 Summer Conference Proceedings, 445-457 (1987).

I seem to have no trouble finding things that reference this paper. The following might also be helpful:

• mk man page
• Maintaining Files on Plan 9 with Mk
• Plan 9 Mkfiles

Yet Another Build Tool

[oren]

There are so many of these... And this isn't going to change.

The nice thing about SCons is that you code the "makefile" in Python instead of some build-tool specific language. Of course, that's also its main disadvantage. Build tool languages are typically nicer for this specific problem domain. Not to mention you need to install the Python run-time environment first. Well, not everybody wants to install it just for the sake of a build script.

As for its other claims to fame, there's nothing new about them. All modern build tools (and there are many) do away with "recursive make", "make depend" etc. This has been the case for more than a decade now. Comparing a new build tool with this crude method is the equivalent of a car maker coming out with a new model whose claim to fame is having airbags and ABS. Yawn.

The only reason that (reasonable) people use make is portability; like /bin/sh, you *know* it will exist on your target platform. I wouldn't hold my brath waiting for either makefiles or shell scripts to disappear any time soon.

I doubt SCons is better than cake, jam or cook (to name a few example). There was also the original Cons which had the same idea as SCons but was based on Perl instead of Python; Ant that does the same thing with XML and Java. None have even made a dent in plain old make.

We are stuck with makefiles for almost anything that needs to be delivered as an open source project, regardless of size. I'd probably use GNU make for the larger projects. While GNU make isn't quite up to the task, it is common enough to be acceptable and advanced enough that the pain is tolerable. That is, as long as you do *NOT* use it recursively. Recursive make is an abomination.

For serious in-house projects I personally prefer to use cook, since I'd want to use Aegis as well and cook is the best match for it. Cook is really a great build tool independently from Aegis. It is just that their combination is such an amazingly useful development framework that if you hit on one you usually end up sucked into using the other as well :-)

Of course, most people couldn't care less; in fact, most people use the Visual Studio build system. It makes recursive make look advanced!

Re:Yet Another Build Tool

[IamTheRealMike]

Actually, I think scons could make a difference.

a) It is 100% free software, unlike, say, Jam

b) It can work with C and C++, which are the most common languages (on Linux at any rate) today, whereas for instance Ant is tied to Java.

c) Python is a widely known and easy to learn language.

d) They have great documentation.

I've already encountered several projects pondering moving up the "next level" in tools, ie Subversion with some new build system (usually scons). At some point, one of the new systems has to stick, and I think this one might just be it.

Re:Yet Another Build Tool

[axxackall]

Have you ever consider to use Portage from Gentoo for big in-house projects? IMHO a good way to have a fine-grained dependency control. Also it's Python-based. The downside is that Portage is in its alpha stage of being ported to Cygwin.

By the way, do you use Cook and Aegis in Cygwin? I've tried to build them in Cygwin once - both seem like still not ported completely yet.

Awesome

[be-fan]

I've used SCons in two projects so far, an OS kernel and something for work. Its absolutely awesome. The kernel bit was extremely telling. You can't compile a kernel just like any old userspace program. You have to use special linker scripts and compiler options. Further, the compiler I'm using (Intel C++) allows a special optimization mode that requires a few changes to the usual compile/link process. Implementing support for all that would have been very complex with makefiles. With SCons, the Python code that implements configuration management (deciding what files to build based on features included in the kernel) and software installation (installing headers and binaries to appropriate places in the filesystem) is only 200 lines of code in all. Autoconf-generated Makefiles for a single directory can be larger than that! Having a full programming language on hand when you need it (in my setup, even config files are regular old Python scripts) is very powerful.