Gentoo On Server Considered Harmful

Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."

Comments

This article makes good points.

[suso]

At the same time, the "your system is always approaching the bleeding edge" way of doing things solves one problem that I've always been bothered by with running user servers for suso.org. Eventually, the OS on the server reaches the age where it is no longer supported and updates are no longer coming out for it. This isn't always X years where X is the number of years that a distribution claims to provide package updates for. Its usually X-1. This is because you'd be foolish to use the very latest hasn't been available for more than a day version of Linux. Usually you wait for 6-12 months for it to be mature and have special packages of whatever available for it. Then you spend another month or two setting up the machine and getting it ready for production. By that time, you've already burned over a year of support time. Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left. Upgrading such a system to the latest version of whatever distro means taking the server down for several hours to upgrade, hope to hell that special packages you've built and configurations aren't broken and in nightmare situations, roll back because something is broken and can't be fixed.

The promise of Gentoo for me is being able to continually upgrade and never get outside of that window of support.

I actually have a new shared user system that is running Gentoo that is kinda in beta right now. This article was very useful for me because it brings up those points about stability that concern me. Its kinda an experiment.

I think I may try Debian next.

Re:This article makes good points.

[ePhil_One]

Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left.

Which is why IT Pros prefer Red Hat Linux or its unencumbered variants link CentOS, White Box, and Scientific. Better testing up front thanks to the Red Hat gang, and longer shelf life. Which is why most commercial software chooses to support it first, it provides a stable base.

Re:This article makes good points.

[Anonymous Coward]

Then one day we had to upgrade some of the services.. which in turn required lots of libraries to be upgraded.

In the end, we had to upgrade kernel.. cause libraries didnt support 2.4 kernel.
Stuff change too much in gentoo

How is it Gentoo's fault that the services you run require updated libraries? How is it Gentoo's fault that the libraries you use require a 2.6 kernel?

Seems to me the blame lies with the services and the libraries respectively, and performing the same upgrade would require the same kernel update on other distros too.

Re:This article makes good points.

[Anonymous Coward]

In the case of Red Hat, they'll backport changes for you so that you don't need to upgrade 50 other packages in order to get a security patch for Apache to work.

So in a way, yes, it is Gentoo's fault. It's just the way the distro is designed. Everything at the latest revisions possible. Great for a home system, not good for a server you have to maintain.

Re:This article makes good points.

[zokum]

So, you upgraded from the old 1.x branch to a radically different 2.x branch, known to be a substantial partial rewrite, and expect everything to work out ok all by magic? You also seem to failed the "sentient sys-admin test" by not using 'google' to do some research. Things like say "http://www.gentoo.org/doc/en/apache-upgrading.xml " perhaps?

I run Gentoo on my own machine, and most of my users WANT bleeding edge versions, a lot of custom options here and there. The system is using a hardened kernel, stack protection and everything is compiled for 64bit (k8). I don't know of any distros that can do that for every package. So far I have had 1 package problem, and that was resolved by 'uncaching' some stuff and redo the emerge of that package. In general, gentoo is easy to maintain, provided you update regularly. As for the people whining about compile times, this is a server, using it at 100% cpu now and then, provided the compilation has a low priority impacts noone. Compiler time is a non-issue, i'm not running X, soundcards, usb, video drivers, gui-browsers etc, there's not all that much to upgrade.

It should be noted that I sync the portage tree from a euro-mirror to a local mirror 6 times a day, and having 3-4 meg a sec to the files-repository makes downloads take an average of 2-3 seconds. Coupled with two beefy processors and lots of ram, Gentoo is brilliant for me. And yes, I have permission from the rsync-maintainer to synch that often.

Re:This article makes good points.

[Goeland86]

It's not. The issue here is not which distro is better than the other in some very personal sense, it's whether or not it makes sense to update all the time. I personally feel that, yes, gentoo does require lots of time to update constantly, but it's meant for a park of desktops, not specifically servers, or else you'd better have a number of machines you have a servers + 1 to run updates and then just use packages compiled on your external machine.
Yes new patches come out all the time, but the real question is whether you trust developers to improve their code over time, or to destroy it. We've seen one end of the spectrum with what MS did between 98 and ME, and I believe that gentoo shows us the other end. While you theoretically always ARE at the bleeding edge with Gentoo, it does have a "safe window" built in, the way it handles portage with the keyword system. New packages are usually in CVS within 48 hours of release. If they compile and run, they get thrown into the ~arch (testing) rapidly. Then, depending on what kind of update has been done on it, you'll have to wait anywhere from 2 days to 5 months to see it come down into the actual arch repository, which is deemed the "stable" gentoo. I personally run ~arch, yet I can't seem to recall a problem that portage couldn't solve with minimum input on my part.
Yes, I'm a gentoo fanboy, but I'm not so glued down into distro patriotism to refuse to see flaws where they are.
Some people seem to want to spend time in maintenance to keep a system up to date and continually tinker and let their knowledge grow by frequent maintenance, and other people seem more interested in setting something up and being lazy about having to deal with updates/upgrades. I personally trust that most open source coders, and especially the ones for the big projects like apache, ssh, mysql and others of that caliber, usually improve the code from release to release, not damage it. Security fixes, bug fixes, and plain new features are usually the goal of coders, and I trust that they do that.

Re:This article makes good points.

[neersign]

on the contrary, this article makes NO good points. First of all, he is using old hardware and then he complains about the time it takes to compile packages. Duh. Slow computer + large packages like Apache and MySQL = a lot of time spent compiling. The writer talks about the inital install taking a long time. Yes, my first time installing Gentoo using the CLI took a long time, too, because I was spending more time reading the documentation than performing the steps. The documentation is stupendous, btw. Now that I have gone through a few installs, I can pretty much do it all on my own, but I still refer to the Quick Install Guide for reference. He then complains about the Stage 3 install losing the "compile everything" mentality. Wrong again, because the nature of Gentoo is that as you change USE flags and packages get updated, then the base system will naturally update itself. So, the Stage 3 install simply removes a lot of initial compiling to get you in to a base system quicker. The author then complains about Gentoo wanting you to upgrade everything all of the time and not being stable. Wrong again. You perform an initial `emerge -vauDN world` to update the base system to your new, custom USE flags. Once you install packages and you only want to keep specfic packages up to date, you only need to update that package using `emerge -vau package` (v is verbose, a is ask, both are good so you know exactly what is going on, but not essential). And, as a lot of other people have pointed out, if you do not use the unstable keywords, you will be on a very stable system. Also, no config files will be overwritten unless you tell them to be. The `etc-update` program shows you what needs to be updated, and allows you to merge, overwrite, or ignore the updates. He then says "Gentoo wants you to change a lot of stuff. It wants to be bleeding edge." This is also false. Gentoo wants you to do whatever you want to on your own system. It wants to be as current as you want it to be. Gentoo is all about control and knowing your system.

Obviously, the author has used Gentoo once and now believes he is an expert on the subject. If he had used it more than once, he would see that all of his arguments hold no weight. Every distribution has their own way of doing things, and when you change from one to the other, it takes time to learn the new procedures.

and for the record, I tried to install Debian Woody on my pentium 133 laptop. It took an entire 7 days to download and update the base system. And that was a full 7 days of straight work. Thus, I decided to pick a distribution that was more appropriate for the system and settled on DSL. Also, I switched to Gentoo from Slackware on my server because I always found my self reading the Gentoo documentation for help in setting things up on Slackware, and I like the FreeBSD-esque package manager (Portage is based on Ports). Personally, I would always choose a BSD in a production environment. "BSD on the server; Linux on the desktop," as the old addage goes.

The Problem With Gentoo...

[mattdev121]

The problem with Gentoo Linux is not the system itself, it's the stereotypes that people put against it.

Gentoo is only good for ricers, Gentoo is bleeding edge and unstable, Gentoo is only good for X deployment

The truth about Gentoo is that it is not really a distribution. Gentoo Linux does not make "releases" and it does not aim to cover one area of the market alone.

In Gentoo's packaging system, called portage, the aim is not only to provide up-to-the-minute packages (which it does) but also to provide a wide variety of both tested and verified "stable" packages as well as more bleeding-edge, testing packages.

This, along with a properly configured make.conf and /etc/portage file system, allows you to pull down the packages you want that have been verified as stable (and are also under watch by the Gentoo security project) and keep track of their libraries with revdep-rebuild.

Stop branding Gentoo with stereotypes that label it as X distribution, the project even calls itself a "metadistribution" capable of dropping into multiple roles.

Re:The Problem With Gentoo...

[wolf31o2]

The problem with Gentoo is that Gentoo users assume that most people care about configuration options. They assume that people want the most up-to-date packages. They assume that there's no reason to have stable, long-term supported releases.

Huh? We assume no such thing. In fact, we really don't care what "most people" want, at all. We make no assumptions about support. It is Gentoo detractors who tend to claim that we do. We don't. What we care about is making Gentoo. If Gentoo doesn't fit your needs, don't friggin' use it! Trust me, you won't hurt our feelings. If you think Debian is better, use it. If you think Windows is better, use it. You aren't harming us in any way by using what you feel is the best tool for the job. In fact, that is exactly what we try to give to our users. We give them a set of tools to allow them to build what they want.

I think the biggest issue is that people seem to have this closed-minded view of software and Gentoo. They're stuck in this way of thinking that lends towards doing what the vendor tells you to do. They run Red Hat. They run Debian. They don't think that you can build what you want. Gentoo provides the tools to do just that. For many of my clients, I have built custom Gentoo-based distributions. What they get themselves is slightly different than Gentoo. They get pre-compiled packages. They get a very nice Internet-based update system for these packages. They don't jump into make.conf, at all. They don't need to make these kind of changes. Instead, I have built a custom distribution with the software that the customer wants on it. They install it from CD, and it has exactly what they want on it and nothing else. Gentoo is the tool that builds this system. I am using Gentoo as it was intended, to build exactly what I want. People tend to forget that it is impossible to make something that fits every need. Rather than try to do so, like other distributions do, we instead provide the tools to allow you to build it on your own. It's a completely different philosophy, which is why I understand that so many people simply don't get it.

*sigh*

[Ant P.]

The article makes it sound as if gentoo installs the ~unstable profile by default. The stable one's no more bleeding-edge than Ubuntu.

Re:Some serious crack smoking...

[Anonymous Coward]

You are essentially describing a Slackware system after 20 minutes of install.

You've got to be kidding me...

[God of Lemmings]

There is NOTHING forcing you to "emerge world", "emerge system", and "emerge --sync" every single time Gentoo
updates portage... Emerge flags include "--pretend", "--ask" and "--fetchonly" among several others, learn to
use them.

to each their own

[siddesu]

i didn't read TF blog post, but since i saw a radical view and the word "server" in the same summary, i'll add my 2 yen here. Since we see the word "server", we assume we're talking competent system administrators here. A competent system administrator usually reads and understands the documentation of a software package before making a decision. Having read the documentation of gentoo, I can suggest at least the following ways to ensure a stable distribution:

- one can create a copy of the source files repository
- one can create a repository for self-compiled binary packages and install from there
- one can use the global repositories, and still get a stable version by restricting available packages by version
- finally, as others say, one can use the stable version.

Since the blogger seems to have missed these obvious ways, he hasn't read the documentation, and hence is not a competent administrator, hence his opinion is not very valuable.

Re:It's a dirty job

[VGPowerlord]

Servers are not the place for bleeding tech. Servers are the place for stability.

That is, unless you really dislike your customers that much, be they actual customers or other divisions in your business.

Nonsense

[loxosceles]

You say Gentoo wants to change a lot of stuff?

Any binary distribution has two modes of updates. One is an updated package within the same release; the other is a mass-update from one release to another. Gentoo combines the two, since the distinction is artificial. What you call "changing a lot of stuff" is merely keeping packages reasonably current so that you never have to do a mass-update or complete reinstall.

Anyone who considers the Gentoo update process too difficult either hasn't used Gentoo (upgrades are easy, and there aren't that many of them if you stick to stable x86) or has never dealt with package conflicts in binary distributions. That is the real horror I want to avoid, and I avoid it nicely by running Gentoo.

Updating

[MrEcho.net]

Gentoo gives you 100% control over your system and how things are built.
It does NOT force you to do anything.

"You will need to fix things that break in the upgrade process..." Like what?
This past year there have been some major changes in the Linux world like:
glibc, gcc, xorg, apache(Gentoo went to the standard) and mysql are some the things I can think off of the top of my head.
Because of how Gentoo updates, big updates like these might break things if your not watching what your doing.
And if your blindly updating your system and overwriting confings when you do etc-update, its your own damm fault.

There comes a point in where a package is marked 'stable' for some distros, but if you look on the project site, its old and outdated.

http://gentoo-install.com/

Not anymore.

[a9db0]

Gentoo on a server? No longer.

I used Gentoo for several years. I learned an awful lot about Linux from it. And I appreciate the work that goes into it. But my servers run Debian now, for one reason - quick, reliable updates. I support several small businesses, I don't have the resources to maintain test environnments to check the impact of upgrades. And not having multiple powerful systems at many sites means distcc is not an option. And the recompiles occasionally necessary for apache or samba or postfix or mysql put an unreasonable strain on servers that are typically not high powered and are supporting multiple users. So for quick, reliable system updating apt-get beats emerge every time.

I'm not knocking gentoo. It's a great system for testing stuff, and evaluating software. But in the 3 minutes it took me to type this post, I could update 5 servers that hadn't been updated in a week.

So?

[Enahs]

Call me a jerk, but I found a lot of what was said to be totally accurate. I tried to love Gentoo, off and on, for three years. While it's true that you can start on a fairly complete base system, and while it's true that there are tools available such as glsa-check now and revdep-rebuild (to say nothing of the joys of being able to unmask only what you want to have as totally bleeding-edge) it's true that it's it's a major time sink.

I'll be more than happy to let the folks at Canoical, Red Hat, Novell, or wherever be the ones to put in several hours of work; I simply can't, at home, put in the hours required to maintain a "stable" system. When I quit using Gentoo a couple of years ago, it was to the point where I'd search the forums before I'd ever install a piece of software. And you know what? That gets old. Real old. Especially if you're sitting in front of what should be a desktop machine and you're waiting for revdep-rebuild to rebuild a couple dozen packages because libpng applied a non-backwards-compatible patch that fixed a major security flaw.

Sorry, kids, but although I can deal with running a Gentoo system, I choose to run Kubuntu 6.10. Not because I'm too much of a wuss to run Gentoo, or because I'm too stupid to run anything other than Ubuntu, but because I'd rather spend the hour or so of computer time I have at home some days getting pix and video of my adorable girl (now at toddler age) ready for the grandparents. Not glamorous, and doesn't help push the state of the art, but it's much more gratifying than, say (I'm making this one up), trying to chase down the ruby package maintainer to get him to apply a patch so that you can use Getopt::Long without having to edit files by hand. ;-)

I stuck my head in the sand and I got run over...

[bnomis]

To summarize:

Quote: "If you don't need new features, and things are working, why change anything?"
Translation: "Never change a working system."

Quote: "...I ran the dreaded but most needed "emerge world"..."
Translation: "My system worked but I updated everything"

Quote: "I had nearly no idea of what I was updating..."
Translation: "I didn't bother to check what was going to change"

Quote: "I tried to read the enormous emerge log file..."
Translation: "I didn't bother to read the log file about what had changed"

Quote: "...the machine had to be resuscitated..."
Translation: "I changed it, it doesn't work anymore and I can't be bother to read the documentation"

Basically, he made a bad choice for his environment. Horses for courses.

Not at all

[vandan]

I've been using Gentoo on our database / web / email / many-other-goodies server since August 2003 ( I keep emerge --sync logs ). I'm running the stable branch on our server, and the unstable ( ~x86 ) branch on desktops. I certainly agree that updates on the unstable branch have to be done thoughtfully, but building binary packages when emerging helps a great deal with disaster recovery. It's nothing that can't be fixed with a little searching.

But on the stable branch, I've actually been very surprised with how ... stable ... it is ( coming from the ~x86 branch ). I keep a separate binary packages repository for the server ... just in case ... but haven't actually had to back-track to anything yet. I do updates outside of work hours, and revdep-rebuild when upgrading major parts. I haven't had any catastrophes yet. Actually I haven't even had any mishaps yet. What can I say? If you are confident enough to run Linux on a server, I say you can handle the stable branch of Gentoo.

As for the points the author raised against Gentoo:

1) Too long to do initial install.

This one gives it away from the start. You only install once. But this is at the top of the list. I can't remember how long it took me to install Gentoo on this server, but it was probably 2 days or something. Who cares? That's what time I take installing *any* server. You don't just whack it together and put it into production. You install, you read, you test, you frig around some more. What's wrong with that? The author is no server administrator.

2) Same as point one, just repeated

WTF? Seriously, this author has his head up his arse. On the one hand, he later says that you shouldn't update willy-nilly on servers, and yet then says that it takes ages to update everything. So what, exactly, is he trying to achieve? It takes me about 10 - 15 minutes to update MySQL, which is the most common package I update. What's wrong with that? I back things up, shut down MySQL, emerge the new MySQL package, test, and import form backups if required. No problem? Where is this guy's problem, seriously?

3) Don't like updates, even if they are to more stable packages

Nothing forces you to update packages. Also, no-one claims that packages updates *won't* break things ( though my experience is that in the stable branch, updates *don't* break things ). But if you don't want to update, don't. No problem. If you do want to update, the tools are there to update easily. Sure you should pay attention to what you're doing. It goes without saying.

4) Same as point 3, but with the update impetus being security instead of stablity

Doesn't deserve a response really.

I challenge this author to prove that he's actually used Gentoo Linux for more than 7 days without running crying back to Linspire.

I had an OpenBSD/postfix box

[toadlife]

That would have had around 900 days uptime if my reboot-happy Windows-only-admin coworkers wouldn't have reset it in a panic on multiple occasions to "troubleshoot" (no it was never a problem with my OpenBSD box) mail problems.

I don't know what the hell it is with Windows-only admins and rebooting. The kind of instability that required reboots all the time was reduced drastically with Win2k and win2k3, yet that insatiable urge to reboot first and ask questions later still plauges my Windows-only counterparts.

Re:Bukd your own binaries

[caitriona81]

It still begs the question... Should you really have to build software for a production server environment? (Yes, I know that Gentoo has the ability for binary packages, but their use seems to be actively discouraged by the culture, if not the documentation and the support channels as well.)
Real production environments, at least at the enterprise level, are built around stable, well tested binary packages that just work, change control processes, updates that can be applied safely with minimal technical skill and minimal configuration work, environments which may have one sysadmin for every 500 servers. Server builds should be able to be left to operations staff instead of sysadmins without fear of things going wrong.
Gentoo's strategy of bleeding edge continuous upgrades also doesn't fly in the enterprise world, which insists, for good reason, that functionality shouldn't change at all in production environments except as dictated by the organization's needs. This means that patches have to be back ported, release cycles are a requirement, and product support lifecycles are expected to be measured at least in months, preferably years. Functionality changes require a different kind of risk assessment and planning mentality in the business datacenter world than they do in the workstation world, and this is even more so with laws on the books regulating IT controls such as section 404 of the Sarbanes-Oxley act.and HIPPA in the US alone, and extensive data protection laws in other countries.
Not to mention, the difficulty of one person rolling back changes to 500 servers in an environment built on everything compiled from source is a huge drawback.
While I think that Gentoo is a tremendous technical achievement, it's design places it firmly in the hobbyist and developer workstation realm, and I strongly agree with the article that this leaves it no place in the datacenter for anything other than development work.

Re:Bukd your own binaries

[ajs318]

Yeah, but you have to admit ..... that's getting on for as much effort as it takes to be a Distributor!

If you want something that you know isn't going to change much, and certainly never in a way that breaks anything, use Debian Stable -- and be prepared to build the odd package from source {it really isn't as bad as it's made out to be} if you have to have a massively up-to-date version of something. They have a more-than-King-size package repository.

Re:Not if you're using Debian

[arivanov]

Which is exactly the way I like my infrastructure. 3-6 months freeze with all bugs known, worked around or fixed in the meantime. Once I have gotten it to this point I build on top of that for the actual services which can run something very bleeding edge if necessary, but this is as I pointed out "your daily bread". For the stuff that is not, you need to be sure that it works and if you are a manager to be severely anal about it. So debian stable + 2-3 unavoidable backports and local builds is about right. This is also the reason corporations buy RedHat ES/AS/WS like hot bread. They finally see a model where the base has been frozen long enough to be relied on for building your own services.

Many itadmins and most developers have a problem with understanding of the "establish a platform and build on it" and "platform freeze before development" ideas. They think that everything is a fair game and the results (in man hours wasted on piecing everything together for release) are usually quite obvious.