Gentoo On Server Considered Harmful

Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."

Comments

Re:This article makes good points.

[lordsilence]

Gentoo has proven troublesome in a production environment.
The problem isnt updating often... it's when you DONT update often.

We had one system which we didnt bother to update. (Dont fix what isnt broken)
Then one day we had to upgrade some of the services.. which in turn required lots of libraries to be upgraded.

In the end, we had to upgrade kernel.. cause libraries didnt support 2.4 kernel.
Stuff change too much in gentoo to put it simple.. It's easier to keep updating often

emerge sync && emerge -u world
Then iron out all config-changes. Find out which undocumented features were changed, which keys to add to startup script etc.

Lesson learnt: Dont use gentoo on production systems. Run it on your desktop computer you play around with...

Agreed.

[MrNaz]

I have been a server admin for web/database for about 3 years now. I agree that bleeding edge is *not* where server admins want to be. There's a reason that Debian is widely considered the best server OS despite being rather far behind the bleeding edge. Tried and tested is better than the latest and greatest when you rely on the machine being up. It's also worth noting that the military doesn't use any COTS technology within 5 years of it being released.

The Problem With Gentoo...

[mattdev121]

The problem with Gentoo Linux is not the system itself, it's the stereotypes that people put against it.

Gentoo is only good for ricers, Gentoo is bleeding edge and unstable, Gentoo is only good for X deployment

The truth about Gentoo is that it is not really a distribution. Gentoo Linux does not make "releases" and it does not aim to cover one area of the market alone.

In Gentoo's packaging system, called portage, the aim is not only to provide up-to-the-minute packages (which it does) but also to provide a wide variety of both tested and verified "stable" packages as well as more bleeding-edge, testing packages.

This, along with a properly configured make.conf and /etc/portage file system, allows you to pull down the packages you want that have been verified as stable (and are also under watch by the Gentoo security project) and keep track of their libraries with revdep-rebuild.

Stop branding Gentoo with stereotypes that label it as X distribution, the project even calls itself a "metadistribution" capable of dropping into multiple roles.

Redhat 6.2

[flyingfsck]

Don't fix it if it ain't broke: up 292 days, 22:26 The reason for the short uptime, is PSU upgrades...

some truth, but for many Gentoo is appropriate

[Anonymous Coward]

First of all, I find it interesting that FreeBSD never seems to get these complaints and hate about having to recompile packages with portupgrade all the time, and being able to tweak the flags, etc. In this respect, it's just like gentoo!!! Except without a lot of the fancy features like etc-update and slots and masking and multiple supported versions. Yes, the "base system" is more stable on FreeBSD (which is both a blessing and curse), but what is it about Gentoo that attracts so many haters/inexperienced admins, hmm??

Anyway, I run Gentoo on servers. (Also FreeBSD). I think it's great. I can't stand stuff like Red Hat, which makes it difficult to customize anything, so I'd always resort to installing stuff "by hand", which was a huge pain. Or creating a custom RPM, which was an even bigger pain (RPM is basically a huge clusterfuck in general).

Being able to set up ebuild "overlays" is great. Being able to set up custom profiles that contain all the software needed for a particular app is great. Writing ebuilds is a piece of cake. Turning on/off various features system-wide is very helpful. The mechanism for merging configs (etc-update or dispatch-conf) is nice. Being able to pin down specific versions with masking is good. Etc. For the record, I've never tweaked the CFLAGS in my life.. that's just not why I use Gentoo.

The author writes this:

A profile update will touch a very large number of configuration files, and it may even alter your startup process. Obviously this is not something you want to do to any server. ................. The end result: the machine had to be resuscitated on-site with associated downtime.

I have no idea what happened to him. Updating your profile is basically moving a symlink, which changes some lists of base packages and other high-level build configuration. It doesn't "touch" anything in your system. Sure, you have to some upgrades afterwards, but you have to do that regularly anyway on Gentoo. Compare it to upgrading FreeBSD from 5.x to 6.x, which is much more involved.

As you might be aware, FreeBSD has a nice little program called portaudit........... Now, Gentoo also has something like portupgrade. What it doesn?t have is portaudit. ............ In all fairness, Gentoo has an experimental command called "glsa-check".

I've been using glsa-check for a while now, it works great. It tells me what's got known holes and I just update those packages, and their dependencies. What problem did he have with it, besides the "experimental" status? Yeah it can "do stuff", but I don't use those options, I just use it to get a list of packages with known holes. Heck I could probably write a script to do the very same thing.

Suppose you need to patch one of your installed packages by the way.. it's very easy to create custom ebuilds on Gentoo. Sometimes I plug security holes that I've found on my own for instance.

I have a simple strategy with Gentoo servers: keep an identical test/staging server nearby and do your updates on that machine first. Run your application tests and then upgrade the production machine. If you want, build binary packages on the staging machine. I would do this even with Red Hat, Debian, etc.

Another point: I've NEVER run "emerge -u world". I always do the packages in small groups or chunks and then updated configs, restarted daemons, and run tests after each one. This seems like a much better strategy than what some people do.

Also, I gotta say, it's probably not a good idea to run Gentoo on a production server unless you've got at least 5 years of Linux admin under your built. You also need to FOLLOW the Gentoo newsletter, AT LEAST, so you can get a heads-up when config files change or files are moved around. It happens from time to time.

Really, the only valid point he makes that generalizes to servers other than his own is the following: Gentoo takes more time to keep running. But you have to weigh that against the flexibility you get, just like any "build vs. buy" decision.

My post to the gentoo forums

[RyogaHibiki]

(I posted this on the gentoo forums)

If someone is running a server room with many live production systems where downtime must be in seconds per year, they should ALWAYS have a test environment and a production environment. Gentoo makes it extremely easy to produce this setup. Imagine if you will, this setup:

1) Master rsync system (contains the portage sync used by all the systems)

2) Test boxes for each role needed (perhaps you have 3 different kinds of servers, WWW, Mail, DB)

3) Many production boxes

What you would end up doing is creating a fairly generic gentoo install (by generic, I mean hardware independent - like i686 or whatever you feel comfortable that will be supported for the lifecycle of the servers). All production servers are identical to the test boxes at the beginning of this example and have a simple backup of the whole test environments (perhaps a large tarball saved on a separate drive). A new update is necessary for apache so you do an emerge --sync on the master rsync system. Then you rsync all the test boxes so they have the same portage tree. You then run the necessary installs on the test systems to make sure that it works, if it doesn't, then you research why and figure out if its easier to fix after the update, or if the update needs to be done differently, if you need to, you can restore the test system from the backup and start over. After you have all the test boxes running well, you can then rsync the production boxes and reproduce the steps necessary to get them updated.

Once all this is said and done, the production boxes will all be updated successfully (and the updates were tested on the test boxes) and the test boxes will at this point have the same configuration as the production boxes. You would make a new backup of the test boxes and wait for the next time you have to do this cycle. As long as the boxes really are identical, you could even run konsole (or another xterm that allows you to send your input to multiple console windows) and perform the identical steps on all the same type of boxes (sending your update commands to 20 or even 50 servers at once).

I'm sorry, but in any real production environment, I see NO issues with this setup. It may be a bit time consuming if you have a lot of etc-updates to do, but still, the basic update should be painless to that point.

-Jason Pf.

Re:It's a dirty job

[VGPowerlord]

Servers are not the place for bleeding tech. Servers are the place for stability.

That is, unless you really dislike your customers that much, be they actual customers or other divisions in your business.

Nonsense

[loxosceles]

You say Gentoo wants to change a lot of stuff?

Any binary distribution has two modes of updates. One is an updated package within the same release; the other is a mass-update from one release to another. Gentoo combines the two, since the distinction is artificial. What you call "changing a lot of stuff" is merely keeping packages reasonably current so that you never have to do a mass-update or complete reinstall.

Anyone who considers the Gentoo update process too difficult either hasn't used Gentoo (upgrades are easy, and there aren't that many of them if you stick to stable x86) or has never dealt with package conflicts in binary distributions. That is the real horror I want to avoid, and I avoid it nicely by running Gentoo.

Re:This article makes good points.

[Anonymous Coward]

In the case of Red Hat, they'll backport changes for you so that you don't need to upgrade 50 other packages in order to get a security patch for Apache to work.

So in a way, yes, it is Gentoo's fault. It's just the way the distro is designed. Everything at the latest revisions possible. Great for a home system, not good for a server you have to maintain.

I had an OpenBSD/postfix box

[toadlife]

That would have had around 900 days uptime if my reboot-happy Windows-only-admin coworkers wouldn't have reset it in a panic on multiple occasions to "troubleshoot" (no it was never a problem with my OpenBSD box) mail problems.

I don't know what the hell it is with Windows-only admins and rebooting. The kind of instability that required reboots all the time was reduced drastically with Win2k and win2k3, yet that insatiable urge to reboot first and ask questions later still plauges my Windows-only counterparts.

Re:This article makes good points.

[zokum]

So, you upgraded from the old 1.x branch to a radically different 2.x branch, known to be a substantial partial rewrite, and expect everything to work out ok all by magic? You also seem to failed the "sentient sys-admin test" by not using 'google' to do some research. Things like say "http://www.gentoo.org/doc/en/apache-upgrading.xml " perhaps?

I run Gentoo on my own machine, and most of my users WANT bleeding edge versions, a lot of custom options here and there. The system is using a hardened kernel, stack protection and everything is compiled for 64bit (k8). I don't know of any distros that can do that for every package. So far I have had 1 package problem, and that was resolved by 'uncaching' some stuff and redo the emerge of that package. In general, gentoo is easy to maintain, provided you update regularly. As for the people whining about compile times, this is a server, using it at 100% cpu now and then, provided the compilation has a low priority impacts noone. Compiler time is a non-issue, i'm not running X, soundcards, usb, video drivers, gui-browsers etc, there's not all that much to upgrade.

It should be noted that I sync the portage tree from a euro-mirror to a local mirror 6 times a day, and having 3-4 meg a sec to the files-repository makes downloads take an average of 2-3 seconds. Coupled with two beefy processors and lots of ram, Gentoo is brilliant for me. And yes, I have permission from the rsync-maintainer to synch that often.

Re:This article makes good points.

[lRem]

Gentoo has proven all right in my production enviroment - and that is an ISP.
First reason, is that you don't have to upgrade those production machines all that often. I sit down and read any security advisory that seems to affect me. And, not surprisngly, there are actually very few remote vulnerabilities that hit Gentoo-hardened. Furthermore, those tend to be in software right in a leaf of the dependency tree, or software I might consider disabling (or limiting to trusted hosts) to the next maintainance cycle.
And there comes it - once in 6 months a massive emerge -uDB world && emerge -uDk world && revdep-rebuild && perl-cleaner (better don't omit the latter two). The system is nicely trimmed down and the build runs on a few machines I have available, so it doesn't take any epic amounts of time. In fact, I even seen it done within half an hour. Still, back when it did take a better part of the day, I simply run the first command a day earlier and then used the packages, what of course is a breeze.
Finally comes the configuration updating. I haven't seen it easier anywhere. The first nice thing is that Gentoo developers don't toy around them - they usualy come as the original software developers intended. But what really makes a difference is the toolchain. By far, I have seen no other distro that automagicaly within the standard package system uses revision control for configs. And then, it gets the trivial updates done for me, and puts me into vimdiff anytime any decision is required.
At most times, this means no downtime at all, as everything runs smoothly. In case of a kernel upgrade, or anything going wrong (once till now), we still have redundancy. So there are no visible drawbacks of using Gentoo on those servers... Unless I, and my boss, am missing something.