Slashdot Mirror
Why Switch a Big Software Project to autoconf?
woggo queries: "I'm a CS grad student working on a large research project (over 1 million lines of code, supported on many platforms). The project has been under development for several years, and the build system is nontrivial for end-users. We'd like to make it easier to build our software, and I'm investigating the feasibility of migrating to GNU autoconf. I need to demonstrate that the benefits of autoconf outweigh the costs of migrating a large system of makefiles with a lot of ad-hoc kludge-enabling logic. Has anyone made a similar case to their advisor/manager? Does anyone have some good 'autoconfiscation' war stories to share? (I've already seen the Berkeley amd story and the obvious links from a google search....)" Depending on the intricacies of the build process, such a conversion might take an awful lot of work. It might be easier to put a nicer face on the "nontrivial build process", although there is something to be said for the ease of "./configure; make; make install"
Why cant we have an equivilent of setup.exe that untars the package and runs ./configure && make && make install? The process the user sees would be comparable to an InstallShield installer. Open, ask a few questions, pass the answers onto configure, such as --prefix for example.
Why would that be so difficult?
Syllable : It's an Operating System
I have found that autoconf has problems when you are cross-compiling. If you are or plan to cross-compile your project, stay far away from autoconf.
/usr/share/autoconf/acspecific.m4 or not do the test!
A few macros (particularly AC_TRY_RUN, and anything that calls it) compile and run a program on the host to test for functionality. Unfortunately, there is no way to tell autoconf that you aren't going to be able to run the compiled program, because it's for your target platform.
Many of the AC_FUNC_ macros simply DIE when running in a cross-compiling environment. For example, let's cross-compile the ncftp package. The configure.in uses the AC_FUNC_SETPGRP macro. Running autoconf results in:
configure.in:146: warning: AC_TRY_RUN called without default to allow cross compiling
That's right. This macro (defined elsewhere in the autoconf package) uses AC_TRY_RUN without a default clause. So instead of recovering nicely, the configure script dies when it tries to do this test. The only way to get around this is to modify
When I started to work for my last employer, they where using RCS and static Makefiles plus a hand-crufted make and version control system that a prior contractor had introduced. The code base for the large integrated system was about 250.000 LoC c/C++ code that ran on a number of Solaris 2.5, Linux-Boxen and Windows maschines. It used GNU-Tools and libraries quite liberally and thus was dependant on the regular introduction of external code.
But since a number of developers didn't really grasp what make was doing, they would simply copy the makefile of one of the larger subprojects and tweak until they got it working. This led to each of the about 20 sub-projects having Makefiles of about 2000 lines each. Quite unmaintainiable.
I started to use auto* for my own library that, in the beginning, was fairly separate from the other projects. More from curiosity, than actual need (I know how to use make B-) ), I put first autoconf, then automake and libtool into the code that was at that time about 20.000 LoC. I also got everybody to switch to CVS, since it supports multiple developers on the same project much better.
It worked quite well I must say. For the following reasons:
- not everybody is on a intimate level with make. If at least one person knows what they are doing the automake Macros allow encoding of that knowledge for others to use with new Makefiles.
- Specific problems with one of the plattforms may be detected, together with the activation of workarounds, in a single place, which is configure.in. One earlier poster was saying that he was having problems with changing Makefiles. I allways found that if you take the log of what happend and debug from there, changing the behavior on a given plattform as needed becomes trivial.
- You have a single place where to configure the inclusion of features, glues to other packages and the paths/names of libraries, config-files, etc. If you write your Makefiles and helper utils to do the same, it will just be duplication of effort with a smaller user-base. YMMV
- with the config.status file you can save what you have configured in a separate place and reproduce the configuration just by copying it.
The war story ends with a system that has about 2 MLoc now, and whith the help of a script that one of my fellow developers wrote, will automaticaly pull itself out of CVS (using the supplied branch) and configure and build each of the sub-packages in the proper order and install the packages on the proper machines (or at least schedule them for installation. Due to security concerns only 2 humans working together may actually install a component). My sub-project has grown to 140KLoC and the public part is on sourceforge now.Could this be done with only some make magic? Yes, if everybod knew make well. But in the real world people that really grok make are few and far between.
would your own solution handle shared libs on multiple systems? Maybe you are that good. I am not. I use libtool.
like any software there are problems with these. For instance the handling of languages like java is suboptimal in automake. But this is opensource. If you have a problem solve you itch and trust that others will solve other itches.
This is why so much development effort has been put into creating nice, user friendly graphical binary installers, and why so little effort has been put into more general Tk configure frontends. Development energy can be better spent elsewhere.
Besides, the problem of build failure is a serious one. The other day I compiled an (autoconf-based) application that was supposed to build out of the box on my system. The developer, however, had not tested the application with gcc 3, and I had to make half a dozen changes to the source to get it to compile. What do you propose your graphical newbie-friendly builder does at this point? You're pretty much screwed, because there's no way to recover from "error X on line Y" aside from telling the user that there was an insurmountable failure. Binary installers work much better for newbies, because your installer can detect and report (and possibly fetch and install) missing libraries on which the new application is dependent. Then, with very high probability, the installation will succeed. (At that point, the only stumbling block should be things like the locations of configuration files.)
i may well be in the minority among this crowd, but i think you should avoid autoconf like the plauge. people's most common reason for using autoconf/configure is portability, but that's a cop-out.
basically, the autoconf/configure school of portability says "forget about actually writing portable code, just write it for each variant and let the build process pick". that's whacked. you'll continually be running accross new variants, new ways in which systems are different, or just new systems. for example, i use Plan 9, which has a very good ANSI/POSIX compatability system (better than many *nix systems). despite being near-fully ANSI compliant, pretty much every autoconf fails because it doesn't recognize the output of uname or something stupid like that (of course, that says noting about when programs that claim to be ANSI or POSIX arn't, like including non-ANSI/POSIX headers, typically BSD stuff).
this school of portability also typically makes your code far less readable - littering it with ifdef's every third line - and much larger. it ends up taking much more time than just slowing down and disiplining yourself to write real portable code.
the argument 'configure ; make ; make install' is easy is stupid, as well. y'know why? 'cause 'make install' is easier. build your makefiles well. the 'install' target should have prerequisits, obviously. make will build them for install. and 'configure' is slower and more prone to failure than 'cp makefile.linux.386 makefile'. and light years more reliable. and editing an example makefile is way easier than putzing with autoconf/configure if something doesn't work. not to mention easier to debug (uh, 'echo' anyone?).
on a personal note, having done portability work a good bunch, i'd offer just a little extra advice: do not require GNU-specific stuff. don't use any of the gmake-specific features or gcc language extentions. GNU propaganda, that'll just kill your portability. and if you need something both simpler and more powerful than make, check out mk from the Plan 9 and Inferno distrubutions. Vita Nuova's got a version that runs on various unixes and win32 systems. it's very similar to make, but a bit more general and flexible. but, given that you've already got all the makefiles, i'd suggest your best bet is just sticking with plain makefiles and cleaning them up.
i speak for myself and those who like what i say.
...is what breaks autoconf and makes it often unusable.
Look at the PHP project. They use autoconf. Yes, it (autoconf) works most of the time pretty good for PHP.
However, you can use only certain versins of it because the older versions do not have the necessary features, and the new ones break BC or just plain have bugs that prevent PHP from building with that new version, so you are often locked in to a certain couple versions of autoconf, which is one of the main problems autoconf is supposed to fix in the first place.
It does contribute a bit to developer frustration, but, in the long run, it is much better than most other things I have seen so far, so it's probably not that bad of a choice after all...
:)
Jobs? Which jobs?
From trolltech; The version that comes with qt-2 is Perl and thus as cross platform as you can probably hope for. tmakes are significantly easier to maintain directly than makefiles, configure scripts or any other beast I've looked at.
In my experience with software projects, a straightforward build system is essential. It permits consistent builds, which is essential for debugging complex problems. It permits easy builds, which is essential for developer testing. The hardest part of programming is debugging, and good debugging requires speeding up the edit/compile/debug cycle, while ensuring that everybody still gets the same result for each build.
So I would say that no matter what, you should improve your build system.
Given that, should you use the GNU build system, which is autoconf/automake/libtool? Well, it depends. There is a definite learning curve to these tools. They are written in several different languages (sh, m4, perl) and dealing with certain problems can require understanding each of those languages and how they interact. Using these tools will not automatically make your code portable or easy to build; you have to use them correctly, and you have to understand what you are doing.
On the other hand, every other system which supports a complex build process also has a steep learning curve. There's no easy solution--if there were one, everybody would be using it.
The GNU build system is widely used, so there are a lot of people who can help you with it. The mailing lists are active and helpful. There is also a book, but I'm one of the co-authors so you'd better take that with a grain of salt.
I've converted large projects from ad hoc build systems to using the GNU build system. It's usually pretty straightforward--a matter of a week or so--but then I have a lot of experience with the tools.
I've never used most of the other build systems (e.g., Odin, Jam, ANT) for a serious project, so I can't honestly evaluate how they stack up. I can recommend against imake.