Build From Source vs. Packages?

mod_critical asks: "I am a student at the University of Minnesota and I work with a professor performing research and managing more than ten Linux based servers. When it comes to installing services on these machines I am a die-hard build-from-source fanatic, while the professor I work with prefers to install and maintain everything from packages. I want to know what Slashdot readers tend to think is the best way to do things. How you feel about the ease and simplicity of installing and maintaining packaged programs versus the optimization and control that can be achieved by building from source? What are your experiences?"

Comments

Support

[ackthpt]

After you've gone it will be easier for the prof to get support on a package than something custom. From experience, the less something you have resembles what tech support is expecting the more finger pointing and the less gets done.

As often as I've lamented how much employers spend on PC's, vs build them themselves from parts, they would rather not have to rely on someone in-house to support hardware.

Re:Support

[vrTeach]

This is very much the case. I have managed 15-20 linux machines for the past seven years, and have moved from largely building from source to largely depending on packages. The porting of apt to rpm systems has completely changed my work for the better, so if at all posible I use the packages and a small subset of apt repositories. My next step is probably develop our own apt repository.

In some cases, the packaged version won't play well with something that I need, or I particularly don't want upgrades to disturb something. In that case I put together a pseudo-script that gets and builds the source and dependencies, and mark the packages as "Ignore" in my apt configuration.

eks

Gentoo is something of a middle ground.

[Novanix]

Gentoo is a great OS as instead of having binary packaged systems, it builds everything from source but can build it effeciently and automatically. In addition it can allow you to just use it to manage the source and you compile it yourself. If you were dealing with many systems you could setup your own gentoo sync server and distribute custom copies of various packages exactly to your specs and compiling details. In addition it can easily determine dependencies, and even install them for you if needed. Gentoo is kind of like a bare bones OS that simply makes it easy to install whatever you want and rather helps shortcut the process of dealing with installing things by compiling things for you.

Who are these people?

[bperkins]

While building from source can be fun, and necessary sometimes, I don't think it makes sense. You spend far too much time tweaking minor issues, and lose sight of major problems.

One problem that I've noticed is the fact the build from source people tend to install things in a way that's completely different than anyone else. This means that anyone who tried to maintain the machine is hopelessly lost trying to figure out what the previous person did. OTOH, When (e.g.) RedHat does something weird, the explanation and fix is usually just a few google queries away.

Most (all?) package formats have source packages that can be modified and rebuild in case you need some really special feature.

Re:Who are these people?

[bwy]

You spend far too much time tweaking minor issues, and lose sight of major problems.

Good point. There are probably very few cases where spending the extra hours of tweak time ever ends up being something that adds a significant amount of value to anybody, except yourself of course. I can think of a couple exceptions, but they are exactly that- exceptions to the rule. IMHO the ability to standardize installation packages is an important aspect of modern computing.

If time didn't matter, I suppose we'd could all go so far as writing all our own software that would do exactly what we wanted.

Simply

[AchilleTalon]

build packages from source!

Many sources include the SPEC file required to build the package.

--No-Deps

[Doesn't_Comment_Code]

My biggest grievance against packages is the dependacy fiasco. For instance, I have Red Hat at work. And the majority of the programs are .rpm's. Well there was a certain program that I could only get as source, so I compiled and installed it. It turns out that it was required as a basis for other packages I wanted to install. But when I tried to install those, it didn't recognize the prerequisite programs because they weren't installed via rpm.

I don't care for the dependancy model of packages, and I'd much rather install programs myself. That way I know I'm getting the program compiled most efficiently for my computer, and I don't have to worry about dependancy databases.

Re:--No-Deps

[idontgno]

I don't care for the dependancy model of packages, and I'd much rather install programs myself. That way I know I'm getting the program compiled most efficiently for my computer, and I don't have to worry about dependancy databases

That just means that you'll have to store the dependancy databases in your head. A release of a particular software package, whether it's a package or a tarball of source, depends on other software. Always. "config" goes a long ways towards working that out, but if it doesn't work automagically you're going to have to take it by the hand and lead it to wherever your copy of libfoobar.so.17 might happen to be.

I've just started using yum for RPM management and I'm already liking it a lot. At least dependency management seems a bit cleaner and more automatic.

Re:--No-Deps

[Zathrus]

I need to write a compressed bitmap? Okay, then I include code to do so. I need to read a wav file ripped from a CD? Yup, my own code. Calculate an MD5? Inspired heavily on the RFC reference code, but essentially my own.

Man, I'm glad other industries aren't as stupid as the software engineering industry. Otherwise car manufacturers would have to have steel foundaries, cloth weaving, a slaughterhouse and tannery (for leather), and innumerable other ancillary businesses on site just to build a car. And, of course, everyone would have to know how to do absolutely everything.

What you're preaching is directly contrary to the practice of reusing code -- and not just your own. It's insane to reinvent the wheel every time you need to drive to the store -- but that's exactly what you're doing. It's one thing to understand the physics behind the wheel, or the foundary, or the paint shop. It's another to rebuild them from scratch.

I hope there's never a bug in your code... because if there is you're going to have to patch every single code base, and re-issue every single binary (since you prefer to link statically). All because you felt it was better to not trust others and do it yourself. Not to mention the vast amount of time burnt re-implementing that which already works, and works extremely well.

The code I'm working on uses a multitude of libraries -- STL, Boost (primarily for its shared_ptr's; we'd use more but much of it doesn't compile on our platform), OTL, libcurl, libxml, pcre, openssl, and others. In some cases we've ditched libraries and implemented our own solution (in particular, MQSeries, which sucked deeply). But to re-implement all of those libraries would literally add years to development. And to what purpose? To have a less feature complete, more buggy, less supportable code base?

And, yes, we've even used libraries sometimes when the library pretty much sucks. Case in point is cgicc, which we used because it's one of the few C/C++ libraries that interfaces "properly" with fastcgi. It's full of bugs, full of really idiotic #define's, and doesn't implement things quite right... but fixing it took much less time than rewriting it from scratch. Because it doesn't do everything wrong, and there's no reason to toss the baby out with the bath water.

No thanks. I'll happily replicate what's been done in every other scientific and engineering discipline -- to stand on the shoulders of giants while adding my own knowledge to the repository.

But when a package links against it for the sake of using a single function that the programmer could have reproduced in under ten lines of code... Well, that just screams "laziness" to me.

Sure. But that situation is pretty rare, at least among competent developers. If you're seeing that commonly, then you're using crap packages (and god knows there's a ton out there... I've ditched many packages because they had too many esoteric dependancies).

depends on the system

[Evanrude]

I used to be a die-hard build from source person myself back when I ran slackware.
Since that time I have gained more experience with production Linux systems.
When it comes to managing production servers, I use Debian and typically only install programs that are in the stable tree.
Every once in a while I have to build a deb from source, but only in rare circumstances.

Now, when it comes to my development systems I am more likely to compile from source rather than rely on the packages to supply me with the latest and greatest.

It really all just depends on what kind of stability vs. "new" features you need as well as ease of managment. Installing a package takes 30 seconds vs. compiling/installing from source can take longer and requires more hands on.

Depends

[Richard_at_work]

I use OpenBSD, which like most of the BSDs has the ports tree, and also has packages. Most of the ports tree are built as packages and are available on the FTP sites, allowing you to either install 3rd party applications from source preprepared for the job, or install the package that has already been preproduced from that port. Best of both worlds, and indeed if you are after customisation and have a number of systems, you can make the changes on one system, and bingo - you have the package ready to roll out to the other systems.

As for what I use? I used to use solely ports, but now I usually grab all the packages when I do a fresh install, and only use ports for what isnt available as a package, as the packages give me no disadvantage.

From source, definitely.

[mod_gurl]

If you're responsible for the machines you run how can you abdicate that responsibility by using whatever some package maintainer decides to give you? At the University of Michigan we use Linux from Scratch to manage hundreds of machines that provide everything from web servers to IMAP servers to user Desktops & Laptops. The trick is leveraging the work used to administer one machine well out to hundreds of machines. The tool for this is radmind. Radmind doesn't require that you build your software from source, but it leverages the work you put into one machine to manage all of your machines. It also integrates a tripwire with your management software which means you can detect unwanted filesystem changes in addition to managing software.

Re:From source, definitely.

[Kourino]

If you're responsible for the machines you run how can you abdicate that responsibility by using whatever some package maintainer decides to give you?

While in principle I can agree with what you're saying, this is a pretty insulting view to take of all the people who work on GNU/Linux distributions. (Or put another way, how am I better than every Debian developer combined? (Substituting Debian for your distribution of choice, of course.))

Source and un-install

[KenSeymour]

I would have to agree about using packages. One gripe I have about building from source is
that most packages do not have "make uninstall".

With packages, you have a much better chance of removing all the files that were installed with the packages when you need to.

Beyond personally - professionally

[Roadkills-R-Us]

I agree. What the professor wants is a readily supportable, production environment, and tat's what you should supply. That means packages wherever possible. IFF there is a clear need, build from source- a 5% speed optimization may not be worth it (that's the prof's call). A 50% speed improvement (unlikely, but possible) would probably be worth it (prof's call). Otherwise, I'd only build from source when there was not a trustworthy package available, or to add features, fix bugs, etc.

I've been in both your and the prof's position, and this is generally the best bet. It'll make the prof's life a lot easier when you're gone, too.

Re:Beyond personally - professionally

[Shakrai]

Otherwise, I'd only build from source when there was not a trustworthy package available, or to add features, fix bugs, etc.

If you can't find a site with a trustworthy package what makes you think you can find a site with trustworthy source code? Or are you going to review every line of code to make sure it wasn't tampered with?

The paranoia works both ways :(

Re:Beyond personally - professionally

[JAgostoni]

In the parent post's defense, you can almost always get the source code from the "source" or author. However, sometimes you rely on some other guy to produce a .deb or .rpm or whatever which you might not trust as much as the author.

I almost always trust packages from the vendor and the distro and only trust "3rd party" packages when there's been tons of anecdotal evidence that they work.

Source Builds = Administration Nightmares

[JM]

I used to run an ISP, built everything from source, but eventually it got to the point where it was un-manageable.

You end up with different versions, different compile options, upgrades are a mess, and it's hard to support.

Another problem is filesystem pollution. When you do your "make install", it's hard to track what files are installed, and when you upgrade to a new version, you can't be sure it's clean, since you might have configuration files or binaries anywhere on your system.

So, one day, I started to make RPM packages of stuff I needed, and modified existing RPMS, and sent all the patches to the community.

What happened is that Mandrake accepted all my packages, so all I had to do was to install the standard distro, and all I needed was there.

And eventually, I made so many packages that they hired me ;-)

But even if I wouldn't work for Mandrake, I'm still sold on RPMs. You have a clean SPEC file that contains the pristine source code, plus the patches, and basically all the instructions to build the stuff. You can specify the requirements, you can easily rebuild on another machine, uninstall the old stuff, or upgrade, with a single rpm command.

make your own "packages"

[drdanny_orig]

I use fedora, and most often I get the *.src.rpm versions, then tweak the SPEC files as required, build my own binary rpms, and use those. Best of both worlds, IMO.

Re:make your own "packages"

[tjwhaynes]

I use fedora, and most often I get the *.src.rpm versions, then tweak the SPEC files as required, build my own binary rpms, and use those. Best of both worlds, IMO.

And the tweaking need not be that tricky or time consuming either. Decent defaults for building RPMS can be placed in your ~/.rpmrc file (or /etc/rpmrc, etc.). Once you have set your optimising settings, architectural preferences and packager name and cryptographic signature (if you want to submit them to other people), that's done for all future packages.

I used to run a mix of RPM packages and tarballs (./configure --prefix=/usr/local && make && su -c "make install") so I could tell what was under RPM control and what was not, but it became annoying when I wanted to build a Source RPM with dependencies on a package I had built from tarballs. These days I usually try and wrap any install up in an RPM - it's not difficult once you get hold of a skeleton spec file for your distro and it saves much hair pulling later on. Also the dependency requirements of RPMs actually save time in the long run because you know when removing a package will hose your system (or part of it) .

Cheers,
Toby Haynes

Building from source is often just a bloody waste

[multipartmixed]

..of time.

It's like the programmer who spends six hours hand-optimizing the inside of a loop that gets called once a day and already executes in 10ms... but ignores the fact that the program takes 20 times longer to run than it should because of an inefficient algorithm. This programmer doesn't know *why* his program is slow, he's guessing, and he will almost always guess badly. This is why profiling was invented.

Look at it this way. Installing from the packages you get the following benefits:
- You save time compiling (multiply this by the number of patches you have to add over the box's life time)
- You save time tracking down dependencies
- You have a standard platform you can re-deploy at will
- You have something that another administrator can work on without asking where you shoved shit.
- You have a package database you can query for version information, dependencies, etc.
- You have an easily available source of "known good" binaries if you have a suspected intrusion problem.
- Depending on the package system you use, you might be able to stay on top of security vulnerabilities with very little (or no) work.

Now, installing from source, you get the following benefits:
- You can pick where the files go (whoopie)
- You tune the performance for your platform
- You can select specific features
- You can de-select specific features to save disk space

The only one which gains you a lot 99% of the time is where you can select specific features which are turned off in the standard package. If you need those options, you build it from source. If you're doing ten machines, though, you build it from source on *one* machine, package it up, burn it, and install it from YOUR package on all ten machines.

Saving a few CPU cycles is never worth saving a man-hour. You can use the man hour more productively on the macro-optimization level. Similarly, you can take the dollars that you would be pay the man and buy a new CPU with it.

The same argument goes for saving a kilobyte of disk space. If found out that any of my guys spent *any* significant time trying to cut less than a gigabyte out of our application footprint, I would give him a footprint of my own, right in the middle of his colon. Disk is cheap. People are not.

If you have an application is which is CPU-bound and running too slow, find out why (profile the system or binary), and build from sources only what you need to make your application conform to the target specification. Or, if that will take too long, just buy more CPU.

Long story short -- tuning of ANY kind should not be done at the micro-level across the board, that's just a waste of time. Tuning should be done by profiling the system as a whole, identifying the constrants, and relieving them. If that requires micro-tuning of a few things, that's fine... but squeezing every last little bit of performance out of absolutely everything is either impossible or incredibly time-prohibitive. And, of course, if you were going to spend that kind of time, you could either buy new hardware with the money (remember Moore's law), OR you example the system more closely at the macro level and come up with a better way to do things.

Re:Whatever get the job done

[ajs]

the best solution is to do whatever is most efficient at performing those tasks

And if you've ever had to pick up and maintain a system from someone who left you will know that this is just about 100% wrong.

The best solution is one that works and is maintainable. If you are willing to put in the extra work involved in making your from-source installations clearly maintainable and upgradable so that the next guy isn't going to have to spend 6 hours learning how everything works when he needs to upgrade foobnitz to version 2.0, then great. If not, think about letting someome else do that work for you.

Use RPM's or DEB's if at all possible.

[BenRussoUSA]

I've been a UNIX sys-admin for about a decade.
My advice is that for a workstation that is managed by an individual you can let the admin do whatever they want, but for any server that has to be stable and maintainable you want to stick with a well maintained package repository and try to avoid 3rd party packages and tarballs if possible.

You have to understand that there is a software stack in most services.
With the kernel and core libs (like glibc) and such at the bottom of the stack, and applications like Evolution at the top of the stack. In between you can have gdb and openssl and various perl modules (in AMAVIS for example) and you have sasl stuff which may be related to pam and openldap and cyrus or wu.... etc..

The thing is that even though all of those various pieces of the software stack may be linked against different libraries on the box, the maintainer of the library code may not have a QA group to co-ordinate regression testing and compatability testing before the latest CVS commit is enacted to fix a bug referenced in a CERT alert.

RedHat and Debian and SUSE and all the others have package repositories, the repository maintainers do an amazingly fantastic job of QA and testing to make sure that new patches don't break your software stack. As an individual you simply can't keep up with that.

For example the Development team that takes care of OpenSSL doesn't backport their bug fixes and security patches to old versions of the code. They just maintain the latest release version and the current CVS version. If you have an old server running IMAPs and HTTPs and SSH and SMTP/TLS and such, and CERT announces a bug in openssl vX.Y, then the OpenSSL development team will certainly release a patch for the latest version which may be version Z!

That might cause you to have to upgrade APACHE or wu-IMAP or OpenSSH or Postfix etc... Those things might then have divergent dependencies that would cause you to go and rebuild half a dozen other packages, and so on and so on. Also, do you remember all the magic flags you used for configure and make? Do you have the same environment variables set today that you did the last time you built PostFix? The possibilities for problems are endless. And if you do have a problem you are kind of on your own since your system will be a unique box. Whereas if there is a problem with a standard RedHat or Debian package, then you can always go to the general newsgroups and chances are there are a dozen other "me too" posts with answers already.

It is much easier to use apt or up2date.

So, unless you have a very good reason for using a tarball on a production server that requires reliability and security and high availability, then you should stick with packages.

If you want to build the packages from source, feel free! RedHat and Debian and SuSE make the SOURCE packages available so that you can dig in and read all about'em. I'm sure the Debian team could use a new package maintainer, if you are addicted to compiling and testing things, check them out.

Have your cake AND eat it, too!

[tmoertel]

Packages and package managers solve a real problem: Keeping track of software installations, their files, and their interdependencies is hard, hard work. By packaging software and using good, "higher-level", package managers (like yum and apt-get) you can delegate most of this problem to the computer. That's a smart move.

It's still a smart move if you're building from source. Just package your source. Then you can build the sources under the control of a package manager (like RPM), and install the resulting packages. You get the full benefits of build-from-scratch and the full benefits of using packages.

This is exactly the approach I use. In fact, I'm a bit more strict about it: My policy is that I don't install any software that isn't packaged. If I need to install something that isn't packaged, I'll package it first. If I don't like the way a packager built an already existing package, I'll repackage it.

The bottom line is that creating your own packages (or fixing packages you don't like) is much easier than maintaining a from-scratch, unpackaged installation. Or ten of them.

To get you started, here a couple of RPM-building references:

• The fight, my first attempt to make a readable rpm package building introduction.
• Fedora's RPM Building Guidelines

Don't give up the benefits of source. Don't give up the benefits of packaging. Have them both.

Re:Personally

[Frymaster]

In my opinion if Gentoo wants to gain a larger user base it needs one.

and why does gentoo need or want a larger user base? gentoo is geared towards a niche market and those people will be attracted to the distro whizzy installer or no.

porsche has a tiny market share - but nobody suggests they should make a k-car version to get a bigger slice of the pie!

Amen.

[Sevn]

What some people don't seem to understand about Gentoo or the BSD's is that not everyone is hell bent on world domination and market share. Some people want something specific, and Gentoo and the BSD's are there for them. It's not like they are ever going anywhere. BSD "despite the rumors" has never done anything but grow in usership with the steady, yet slow trickle of new users and the fiercly dedicated long time users. Gentoo is growing rather fast, but will no doubt plateau off and settle in the same way the BSD's have. But by all means, continue to have your OS flame wars and make your comparisons and talk about market share or other things that aren't important or even remotely interesting to the majority of most Gentoo and BSD users. It's very humorous. :) HAVE FUN STORMING THE CASTLE!!!

Different stages

[matt-fu]

As far as as I can tell, there are four stages of sysadmin as it relates to installed software:

1) I am a newbie and have to use packages for *.
2) I know my way around. I like the level of control I get with compiling/know how to code/read far too much Slashdot. I compile by default.
3) I manage more than three boxes in my basement now. Having the ability to back out of system changes without a full OS reinstall is a necessity. I build my own packages from source that I've compiled.
4) I manage more than just three boxes in a department now. Now I have to deal with politics, ordering hardware, the freakin' network, and I generally have time for sysadmin. On top of all that I now have a family so spending two or three extra hours per day on my Unix hobby is no longer feasable. Precompiled packages work just fine.

The point?

[Anthony Boyd]

Wow. There sure are a lot of posts about which is better, but I don't see any comments that deal with the underlying problem. And that is this: don't get into a pissing match with your professor. Seriously, what are you hoping to accomplish here?

If you were thinking that you'd get tons of pro-compiling comments, and then put that in front of the professor, stop right there. Coming to Slashdot for validation of your side of the argument is about as helpful as those wives who write to Dear Abby about their husbands. Because no husband on Earth is going to appreciate getting chastised by Dear Abby, and if Abby sides with him, he's going to gloat. It's lose-lose for the wife, just like it's lose-lose for you if you try to use Slashdot as leverage. Screw with the computers that the professor relies on, and he'll find a way to "thank" you for it. Don't sabotage yourself.