Make Out with SCons

Nice2Cats writes "I stumbled across this and can't find a mention of it here yet: SCons, a stable alternative for the vilified make. Built-in support, they claim, for C, C++, and Java, built-in support for use with SCCS, CVS, and BitKeeper, supports .NET, and works with Linux, *BSD systems, Windows NT, and Mac OS X. I'd love something to would finally get rid of the awkward (no pun intended) make, and having a system based on Python would make it double good. Maybe one of the serious C/C++ programers here could give a verdict on if this is the dragon slayer we've been waiting for."

Comments

Questions

[Zach Garner]

Integrated Autoconf-like support for finding #include files, libraries, functions and typedefs.

Is SCons viewed as a replacement for Autoconf and Automake, or just a replacement for Make?

Built in support for...

What does this mean? Will SCons somehow only work with languages that SCons has built in support for?

How does SCons compare to Apache's Ant?

Re:Questions

[Teancom]

What does this mean? Will SCons somehow only work with languages that SCons has built in support for?

Well, I haven't actually looked at scons, but in Make, there is only built-in support for a limited amount of languages. For instance, (maybe this has changed recently) with ObjC you have to define how to take care of .m file, as it doesn't know to compile them. .c and .cc files it knows to use CC on, etc.

How does SCons compare to Apache's Ant?

Well, the fact that it isn't based on Java or XML gives it a good head start :-)

Re:Questions

[pmz]

Well, I haven't actually looked at scons, but in Make, there is only built-in support for a limited amount of languages.

Adding implicit rules in make is not difficult. It's in the man page (even Solaris' crappy man page for make describes it well enough, and the system default rules file has lots of examples).

Well, the fact that it isn't based on Java or XML gives it a good head start :-)

Agreed.

Re:Questions

[pmz]

The good thing about doing ant conf files in XML, si if they change the format, you can use xsl to transform ALL of your xml to a new format.

Layers upon layers of complexity. XML, XSL, Java, Ant Schemas...all for compiling a program. My sig is appropriate, here, too. A build system for software should be easy, intuitive, transparent, etc. etc. There has yet to be a replacement for make that has these attributes (makes tab syntax really isn't that big of a deal, either, even though that's what most people latch onto when complaining about it).

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Re:Questions

[sporty]

Layers upon layers of complexity. XML, XSL, Java, Ant Schemas...all for compiling a program. My sig is appropriate, here, too. A build system for software should be easy, intuitive, transparent, etc. etc. There has yet to be a replacement for make that has these attributes (makes tab syntax really isn't that big of a deal, either, even though that's what most people latch onto when complaining about it).

Whoahoahoa.. no. Blatantly wrong. If ant 1.0 used one particular XML format, and you upgraded to ant 2.0, to go from an ant 1.0 file to 2.0 file, you'd apply an xsl style sheet ONCE to upgrade it all. Any future XML you write would be in the 2.0 format. Simple as that.

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Bash? csh? sh? tcsh? bash 2.0? Which shell are you suggesting is usable across them all?

ant is just a program, just like make is. You run it against a config file of a certain format, just like Makefiles have their own format. Ant is no more compicated than make. hell.. you can use the core stuff from ant to run any program across many files, just like make. The targeted tasks are just to make those easier.

Re:Questions

[jonadab]

As long as the format is regular, it's pretty trivial to write a
simple script to do the transformation. XML helps enforce a certain
amount of regularity, but it's by no means the only regular format.

Personally, I like XML pretty well in general, and I really like
what OO.o has done with it in particular.

I'm less enamored with Java, but that's another thread for another day.

Re:Questions

[Jerf]

A build system for software should be easy, intuitive, transparent, etc. etc.

No. A build system for software must reflect the complexities what it is attempting to build. No build system for C++ programs can ever be fully "easy, intuitive, transparent, etc. etc." because there are C++ programs that aren't any of the above. Same for C.

Ever tried to compile XFree86 by hand from source? It must be possible, because Gentoo can do it, but I've never managed myself. While it can probably be made easier, the simple fact is that it's a complicated set of programs with a complicated build order and no matter how hard you try, it would take a full-fledged AI to make it "easy".

On the other hand, a pure Python project, even a large one, needs no special build system at all, because Python automatically compiles with few or no hassles (and what hassles exist can not be solved by a build system, they tend to be user error). Of course large projects are rarely all Python (building C extensions, compiling human-readable data representations into faster data or code, etc.), but since Python is more simple, the build system can be equally simple (or in this case, non-existant).

Make can be improved on, it's a decades old design with multiple layers of hacks jostling each other. A new, clean design can at least simplify away the parts of the build process that involve "fighting with make". But a build system for C can only become finitely simple, or you'll start to lose capabilities that C has because your build system can't handle them.

Small shell scripts and makefiles. Is more really necessary to call cc and ld or javac or whatever for each file in a directory tree? The differences among platforms can't be so dire as to require gigantic build automation tools, which introduce more problems than they solve!

Proof by counterexample: autoconf exists, therefore it is clearly necessary, because nobody would trouble to build it if it didn't address a problem. (Whether it solves it is another question, but given its vast popularity in the open source meritocracy I'd say the onus is on you to show that it truly causes more problems then it solves; every project that uses it is a vote against that statement.) You've clearly never written massively cross-platform code; grab a decent sized package, look at the output autoconf generates, and look at what the program does with it. Odds are every single line you see during configure, with the possible exception of some really standard ones that seem to be a standard part of the package ("CC works?"), is used somewhere in the program, especially lines like "Is your pthreads implementation broken? ... yes, using workaround". The workaround may itself not work if pthreads isn't broken!

No Screenshots?

[Randolpho]

How am I supposed to know if I like the look and feel if there aren't any screenshots? :D

Already there...

[KDan]

Though i guess for now it only supports java - but there's nothing preventing you from writing custom tasks for it.

It's called Ant.

Daniel

stable alternative to make?

[Anonymous Coward]

I've been using make for like 10+ years. What exactly is the instability?

One thing I never liked about make was the dependence on whitespace in the makefile. Hopefully this Python-based solution will fi...uh never mind.

I predict Make will be around for a while still. I've used Ant but Make is fine for 90% of what I do.

SCONS is awesome

[alyosha1]

After spending several hours trying to get make to do something complicated with shared libraries and multiple configurations, I started looking for a replacement, and stumbled across SCONS. It took me a few moments to download and install it, and a very short time to go from no knowledge of the system to having solved my problem.

Since then I've shifted several of my projects to using SCONS as the build system. I'm sure it's theoretically possible to achieve what I was trying using make, but I wasted many hours wading through man pages and howtos without success, and SCONS did it with almost zero effort.

I haven't tried using it on Windows yet, but my understanding is that you can use the same script under both platforms.

So, just one data point, but my experience with it has been 100% positive.

-alyosha

Re:Does not solve the problem, but makes it even w

[Circuit Breaker]

Spend some time to look at Scons. It's not a silver bullet, but it's a great step forward.

It can, among other things, generate Makefiles or Visual Studio project files, or run the build itself. It does have "autoconf" style checks, which you can embed in the sconstruct/makefile. It's built for parallel make from the start. It does not suffer from network clock skew.

And it has much better multiplatform support than any tool I've seen so far (automake/autoconf, make, qmake, tmake, cmake).

It will take time to mature, and it doesn't solve the problem, but it does makes things much better.

My SCons experience

[vslashg]

My SCons experience is that I like it a lot!

We use SCons at work to build a huge, highly componentized project with upwards of 5000 source files. It's hands down the best experience I've ever had for building large projects. The other two big projects I've been on used GNU make and Microsoft nmake, and neither was satisfactory.

The nmake-based system required running "nmake depends" before a build, and it simply wasn't reliable. Paranoia was high; when we hit a bug while developing, the first thing we did was a clean build, just to make sure we weren't looking at a build problem. (It was justified paranoia -- half of the time, it WAS a build problem :-) The recursive GNU make system was developed in-house and was much more solid, but painfully slow and absolutely horrible to use. It required Cygwin (which is evil, IMO) tools and bizarre mappings from source control, so it was hard to convince other people in the company to build our project.

SCons was loads faster than our (admittedly bad) GNU make recursive build system, though noticably slower than our nmake build. With that cost, though, you get a correct build every time, something we never got in our "make depends" system. The paranoia is gone. SCons caches dependencies, so the cost for this accuracy isn't as bad as you might think. There were hiccups in the past, but the C++ dependency generation has been rock solid for at least nine months now.

I have a little toy project at home with maybe 25 source files which also uses SCons. I use MinGW's gcc at home, and SCons worked well with it "out of the box". Right now all I do is maintain a list of source files to build just like I would with any other system, but the setup was painless, and it's awesome to know that I can write Python to solve any problems I might hit later.

My big complaint with SCons is that a no-op build (running SCons when no source has changed) takes way too long on our big project, making iterative development painful. There's a command (-U) to build only components in a subtree of your file system, but that doesn't really speed things up much. Hopefully they can improve on that. That's about my only gripe, though.

Re:My SCons experience

[diewarzau]

(SCons developer here)

Yes, no-op build time is one thing we continually optimize for. Believe it or not, it's about 50x faster than it was at our initial release :)

Also, there are several things you can do to sacrifice potential build correctness in order to speed up no-op build time, which are documented in the SCons Wiki. Unfortunately the Wiki appears to be on the fritz right now.

Our philosophy is that, by default, we provide an absolutely correct build, which means recalculating/rescanning a lot of stuff every time -- the equivalent of doing a make depends;make every time (I venture that our no-op build time is faster than that!). However, if you know that you have not changed certain things, you can keep SCons from rescanning them, which is great for when you are in the compile/fix/recompile mode. I usually have an alias for "fast, not-so-accurate SCons" that I use for development and "accurate SCons" that I use for testing and release.

Finally, there is a separate effort toi "daemonize" SCons. SCons has an as-yet-unpublished API that allows Python developers to put other wrappers around it other than the SConscript (i.e., makefiles-in-Python) interface that ships with it. The daemon effort keeps SCons around and rescans dependencies in the background a'la MS DevStudio, saving lots of time during the build. I believe the people on this project have seen as much as a 100x performance increase for no-op build times for large projects. Amazingly, it looks like we don't have a link to this on our page. I'll try to dig it up.

A-A-P

[LarryRiedel]

Something similar/related is A-A-P from Bram Moolenaar of vim. A 1.0 version was just released last week. It is a make+Python kind of thing. There was short article about it at osdir a few months ago.

Larry

jam

[Horny Smurf]

look at jam sometime. It's a free make replacer by the same folks who wrote perforce (the commercial cvs killer and bitkeeper competitor).

The jamfiles are easy to set up, and much faster to calculate dependencies -- I built freetype via gnu make and via jam. Jam was done building before make had even started compiling!

Also, props to perforce. It's proprietary (just like bitkeeper), kicks cvs's ass (just like bitkeeper), and is free for open source projects (just like bitkeeper). MySQL uses p4, IIRC. Unlike bitkeeper, you can get a single-seat version free (I use it for my internal projects, though CVS would probably suit my needs).

Make vs. make

[V. Mole]

One thing I see is a lot of people referring to how slow and unreliable recursive makes are, and they're completely correct. However, you can make your life a lot easier by using make but getting rid of the recursion, using a make with an 'include' feature. See Recursive Make Considered Harmful for nice write up on why recursive makes don't work, and how to do it correctly.

I converted our build system, and no-op builds went from a few minutes to about 5 seconds. Really. If I change one file, all the necessary stuff gets rebuilt, and none of the unnecessary. Every time. With a little gmake if-then-else (all in one place, not scattered around in a bunch of files), the same Makefile and segments work on Linux, AIX, and Solaris. And while the organization of the files is a little unusual for those who are used to the recursive style, they are just makefiles, no new syntax.

The downside? It pretty much requires gmake, but since this particular product is proprietary, we control the build machines. I don't think you can use automake...but I consider that a positive :-). You could use autoconf -- just put all the config stuff in one makefile fragment, and include it into the master.

SCons: so far so good

[SilentTristero]

We switched away from Make a few years ago. I looked at jam and other make replacements before settling on Cons, the perl-based precursor of SCons. We are now in the process of switching over from Cons to SCons (slowly), and so far it is working very well.

We are using it on Windows (MSVC and Intel compilers), Linux, IRIX, and Mac OS X (gcc and two versions of CodeWarrior). Handles all of those with ease. It can do things like properly handle dependencies on auto-generated source and header files, which would be a nightmare in make.

Some things like error handling and documentation need some work, but overall it is very solid. Automatic dependency checking, parallel build support, autoconf-like stuff, switchable tools, and easy to extend.

Yes, it requires python, but will run on just about any version back to 1.5.2, and doesn't require any python add-ons, so it's not too onerous a requirement really.

So far I am very happy with it; I'd suggest checking it out if make is making you crazy.

-- Tristero

BB Scons config file

[adamshelley]

mmmmmmmmmmmm