Slashdot Mirror
Gentoo On Server Considered Harmful
Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."
At the same time, the "your system is always approaching the bleeding edge" way of doing things solves one problem that I've always been bothered by with running user servers for suso.org. Eventually, the OS on the server reaches the age where it is no longer supported and updates are no longer coming out for it. This isn't always X years where X is the number of years that a distribution claims to provide package updates for. Its usually X-1. This is because you'd be foolish to use the very latest hasn't been available for more than a day version of Linux. Usually you wait for 6-12 months for it to be mature and have special packages of whatever available for it. Then you spend another month or two setting up the machine and getting it ready for production. By that time, you've already burned over a year of support time. Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left. Upgrading such a system to the latest version of whatever distro means taking the server down for several hours to upgrade, hope to hell that special packages you've built and configurations aren't broken and in nightmare situations, roll back because something is broken and can't be fixed.
The promise of Gentoo for me is being able to continually upgrade and never get outside of that window of support.
I actually have a new shared user system that is running Gentoo that is kinda in beta right now. This article was very useful for me because it brings up those points about stability that concern me. Its kinda an experiment.
I think I may try Debian next.
Of course some distributions like Debian/Ubuntu/Mandriva let you build the packages if you really want to.
With Debian/Ubuntu it's easily done with apt-build. Mandriva on the other hand, things get a bit more complicated.
Change is certain; progress is not obligatory.
Gentoo is only good for ricers, Gentoo is bleeding edge and unstable, Gentoo is only good for X deployment
The truth about Gentoo is that it is not really a distribution. Gentoo Linux does not make "releases" and it does not aim to cover one area of the market alone.
In Gentoo's packaging system, called portage, the aim is not only to provide up-to-the-minute packages (which it does) but also to provide a wide variety of both tested and verified "stable" packages as well as more bleeding-edge, testing packages.
This, along with a properly configured make.conf and /etc/portage file system, allows you to pull down the packages you want that have been verified as stable (and are also under watch by the Gentoo security project) and keep track of their libraries with revdep-rebuild.
Stop branding Gentoo with stereotypes that label it as X distribution, the project even calls itself a "metadistribution" capable of dropping into multiple roles.
mattdev@server$ touch
cannot touch `/dev/genitals': Permission denied
The article makes it sound as if gentoo installs the ~unstable profile by default. The stable one's no more bleeding-edge than Ubuntu.
You are essentially describing a Slackware system after 20 minutes of install.
There is NOTHING forcing you to "emerge world", "emerge system", and "emerge --sync" every single time Gentoo
updates portage... Emerge flags include "--pretend", "--ask" and "--fetchonly" among several others, learn to
use them.
Non sequitur: Your facts are uncoordinated.
I ran RedHat 6.2....... 8 years ago. Dude, that's not too cool. IMHO, that was back when there were some more serious remote exploits.
i didn't read TF blog post, but since i saw a radical view and the word "server" in the same summary, i'll add my 2 yen here. Since we see the word "server", we assume we're talking competent system administrators here. A competent system administrator usually reads and understands the documentation of a software package before making a decision. Having read the documentation of gentoo, I can suggest at least the following ways to ensure a stable distribution:
- one can create a copy of the source files repository
- one can create a repository for self-compiled binary packages and install from there
- one can use the global repositories, and still get a stable version by restricting available packages by version
- finally, as others say, one can use the stable version.
Since the blogger seems to have missed these obvious ways, he hasn't read the documentation, and hence is not a competent administrator, hence his opinion is not very valuable.
Servers are not the place for bleeding tech. Servers are the place for stability.
That is, unless you really dislike your customers that much, be they actual customers or other divisions in your business.
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
You say Gentoo wants to change a lot of stuff?
Any binary distribution has two modes of updates. One is an updated package within the same release; the other is a mass-update from one release to another. Gentoo combines the two, since the distinction is artificial. What you call "changing a lot of stuff" is merely keeping packages reasonably current so that you never have to do a mass-update or complete reinstall.
Anyone who considers the Gentoo update process too difficult either hasn't used Gentoo (upgrades are easy, and there aren't that many of them if you stick to stable x86) or has never dealt with package conflicts in binary distributions. That is the real horror I want to avoid, and I avoid it nicely by running Gentoo.
Gentoo gives you 100% control over your system and how things are built.
It does NOT force you to do anything.
"You will need to fix things that break in the upgrade process..." Like what?
This past year there have been some major changes in the Linux world like:
glibc, gcc, xorg, apache(Gentoo went to the standard) and mysql are some the things I can think off of the top of my head.
Because of how Gentoo updates, big updates like these might break things if your not watching what your doing.
And if your blindly updating your system and overwriting confings when you do etc-update, its your own damm fault.
There comes a point in where a package is marked 'stable' for some distros, but if you look on the project site, its old and outdated.
http://gentoo-install.com/
In a production eviornment you don't have time for little things like that. RTFM on every little upgrade does'nt really matter when the mysql server suddenly goes down for no reason and thousands of users get pissed off. I reccomend fedora for beginners and slackware for seasoned veterans and people wanting to know what a true linux experience is (no flames please :).. They are very solid for production enviornments. Like previous posts have said, Gentoo is a good distro, but not suitable for production. development boxes, sure.
Gentoo on a server? No longer.
I used Gentoo for several years. I learned an awful lot about Linux from it. And I appreciate the work that goes into it. But my servers run Debian now, for one reason - quick, reliable updates. I support several small businesses, I don't have the resources to maintain test environnments to check the impact of upgrades. And not having multiple powerful systems at many sites means distcc is not an option. And the recompiles occasionally necessary for apache or samba or postfix or mysql put an unreasonable strain on servers that are typically not high powered and are supporting multiple users. So for quick, reliable system updating apt-get beats emerge every time.
I'm not knocking gentoo. It's a great system for testing stuff, and evaluating software. But in the 3 minutes it took me to type this post, I could update 5 servers that hadn't been updated in a week.
-- "Never underestimate the power of human stupidity." - R.A.H.
I'm a long-time Debian user, and I also think it's an ugly legacy UNIX thing. It's much better to have some sort of process supervisor that will restart crashed servers, and that will deal with dependencies in some sort of sane manner. The problem is that Debian is huge, and the amount of work required to switch to a new system would be almost equally as huge, but the benefits are comparatively small, so there's never been a push to change to something different.
The bright side of it is, like most of the advances Debian has made, when it finally does get replaced, it'll probably be replaced with something substantially better, because anything less would be unlikely to win the support of Debian's army of volunteers.
http://outcampaign.org/
I know that apt-get update && apt-get dist-upgrade (on Debian Stable) is unlikely to break anything. Testing is still prudent, but you know that nothing so insanely stupid as an incompatible database upgrade is going to occur. PHP4 and PHP5, for example, are separate packages in Debian. So are MySQL 4 and MySQL 5, Apache and Apache2, and any other package with significantly different versions. Running updates doesn't drastically change your environment. That's a good thing.
Hello? Security anyone? Or maybe someone remember kernel 2.4.11? Don't wanna update that one either should you happen to have it installed back when it was considered stable?
I do agree that there are certain things you needn't update. A local server without a connection to any user you do not trust your data with (i.e. nobody but you, if you're smart) running on rock stable software that gets feature adds rather than bugfixes in new versions is a candidate for this. And for this server (singular, probably worldwide), the setup is ok.
Not updating a server connected to the internet is an invitation for hackers. No matter how "stable" or "solid" or "secure" a system is deemed to be at the moment of its compilation. Time and again there are bugs found in software that has been considered stable and safe for years. OpenSSH is hardly the most insecure application out there, and I would NOT want to see what happens to a server that does not update it.
And, last but not least, when you don't want to update Gentoo, you don't have to. It's fine and satisfied if you don't do an update sync. Actually, you reduce the workload of the servers if you don't.
So what the hell is this fuss about?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Call me a jerk, but I found a lot of what was said to be totally accurate. I tried to love Gentoo, off and on, for three years. While it's true that you can start on a fairly complete base system, and while it's true that there are tools available such as glsa-check now and revdep-rebuild (to say nothing of the joys of being able to unmask only what you want to have as totally bleeding-edge) it's true that it's it's a major time sink.
;-)
I'll be more than happy to let the folks at Canoical, Red Hat, Novell, or wherever be the ones to put in several hours of work; I simply can't, at home, put in the hours required to maintain a "stable" system. When I quit using Gentoo a couple of years ago, it was to the point where I'd search the forums before I'd ever install a piece of software. And you know what? That gets old. Real old. Especially if you're sitting in front of what should be a desktop machine and you're waiting for revdep-rebuild to rebuild a couple dozen packages because libpng applied a non-backwards-compatible patch that fixed a major security flaw.
Sorry, kids, but although I can deal with running a Gentoo system, I choose to run Kubuntu 6.10. Not because I'm too much of a wuss to run Gentoo, or because I'm too stupid to run anything other than Ubuntu, but because I'd rather spend the hour or so of computer time I have at home some days getting pix and video of my adorable girl (now at toddler age) ready for the grandparents. Not glamorous, and doesn't help push the state of the art, but it's much more gratifying than, say (I'm making this one up), trying to chase down the ruby package maintainer to get him to apply a patch so that you can use Getopt::Long without having to edit files by hand.
Stating on Slashdot that I like cheese since 1997.
So how many of the bleeding edge proponents have to support more than 50 systems?
Package management, rpm, dpkg, all came out in response to the shortcomings of compile-yourself approach we can dearly remember from the days Slackware was about the only Linux distro.
I was there. I was the young sysadmin who had to support 2 Linux servers and who was excited by the performance gain I was supposed to get from compiling stuff yourself. In truth, I never noticed it - and I bet 90% of others don't notice it either and 9% see the gain there because they believe in it.
That was around 11 years ago. By the time Gentoo came out I was dealing with RPMs and blessing them.
Nowadays package management software on SuSE, Ubuntu or others even lets you upgrade running system to next release while running.
Do I want to spend hours of my time tweaking compile parameters and wondering why some of them don't work? Do I need "bleeding edge" or stability? For production systems my answer is clear. Yes, there will cases when you want to squeeze the top speed out of the system, so it is good that something like Gentoo is there, too. But I am fairly certain those cases are rare, and in majority of them an upgraded piece of hardware is usually required in the end.
Where I come from, deployments to production are first validated in a QA environment. OS stuff, application updates belong there too.
What happened to backups anyway?
http://stephan.sugarmotor.org
To summarize:
Quote: "If you don't need new features, and things are working, why change anything?"
Translation: "Never change a working system."
Quote: "...I ran the dreaded but most needed "emerge world"..."
Translation: "My system worked but I updated everything"
Quote: "I had nearly no idea of what I was updating..."
Translation: "I didn't bother to check what was going to change"
Quote: "I tried to read the enormous emerge log file..."
Translation: "I didn't bother to read the log file about what had changed"
Quote: "...the machine had to be resuscitated..."
Translation: "I changed it, it doesn't work anymore and I can't be bother to read the documentation"
Basically, he made a bad choice for his environment. Horses for courses.
I've been using Gentoo on our database / web / email / many-other-goodies server since August 2003 ( I keep emerge --sync logs ). I'm running the stable branch on our server, and the unstable ( ~x86 ) branch on desktops. I certainly agree that updates on the unstable branch have to be done thoughtfully, but building binary packages when emerging helps a great deal with disaster recovery. It's nothing that can't be fixed with a little searching.
... stable ... it is ( coming from the ~x86 branch ). I keep a separate binary packages repository for the server ... just in case ... but haven't actually had to back-track to anything yet. I do updates outside of work hours, and revdep-rebuild when upgrading major parts. I haven't had any catastrophes yet. Actually I haven't even had any mishaps yet. What can I say? If you are confident enough to run Linux on a server, I say you can handle the stable branch of Gentoo.
But on the stable branch, I've actually been very surprised with how
As for the points the author raised against Gentoo:
1) Too long to do initial install.
This one gives it away from the start. You only install once. But this is at the top of the list. I can't remember how long it took me to install Gentoo on this server, but it was probably 2 days or something. Who cares? That's what time I take installing *any* server. You don't just whack it together and put it into production. You install, you read, you test, you frig around some more. What's wrong with that? The author is no server administrator.
2) Same as point one, just repeated
WTF? Seriously, this author has his head up his arse. On the one hand, he later says that you shouldn't update willy-nilly on servers, and yet then says that it takes ages to update everything. So what, exactly, is he trying to achieve? It takes me about 10 - 15 minutes to update MySQL, which is the most common package I update. What's wrong with that? I back things up, shut down MySQL, emerge the new MySQL package, test, and import form backups if required. No problem? Where is this guy's problem, seriously?
3) Don't like updates, even if they are to more stable packages
Nothing forces you to update packages. Also, no-one claims that packages updates *won't* break things ( though my experience is that in the stable branch, updates *don't* break things ). But if you don't want to update, don't. No problem. If you do want to update, the tools are there to update easily. Sure you should pay attention to what you're doing. It goes without saying.
4) Same as point 3, but with the update impetus being security instead of stablity
Doesn't deserve a response really.
I challenge this author to prove that he's actually used Gentoo Linux for more than 7 days without running crying back to Linspire.
That would have had around 900 days uptime if my reboot-happy Windows-only-admin coworkers wouldn't have reset it in a panic on multiple occasions to "troubleshoot" (no it was never a problem with my OpenBSD box) mail problems.
I don't know what the hell it is with Windows-only admins and rebooting. The kind of instability that required reboots all the time was reduced drastically with Win2k and win2k3, yet that insatiable urge to reboot first and ask questions later still plauges my Windows-only counterparts.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
A major mysql update should certainly not be done without planning. However, the OP's point was that a standard stabe update included a major mysql upgrade which broke his functionality. If that's indeed the case, there is something very wrong with the way Gentoo manages its packages, and exactly why you wouldn't want it running on any kind of production system.
See, this is where things are slightly different with Gentoo versus any other Linux distribution.
There's no such thing as "Gentoo Version 3" or whatever. A package is marked stable after it is deemed to be... well, stable. Gentoo does have a mechanism whereby you can ask it to tell you what it proposes to update before it actually goes away and does it (emerge -p), and on any system this is exactly what you should use to make sure you're not trying to do a major update on your database.
The one issue here (and it's open to debate whether or not it's an issue) is that portage is designed with a more traditional Unix mindset - a mindset which says "the operator knows best, if they're telling the computer to do something then the computer should do it unless it's physically impossible, regardless of how silly it may seem".
Gentoo is great.
Gentoo is wonderful.
*IF* you're only administrating a small handful of servers.
When you have to look out for a few HUNDRED machines at a time, you **reaaaally** start to appreciate things like calendar based release cycles, binary packages, uniformity, hardware compatibility lists, repository mirroring, etc.
Gentoo is far too schizophrenic to be a reliable environment for n servers, especially in a "real" scenario.
Academically, Gentoo is a wonderful system.... but its one of those things that works "great on paper" but sucks a lot of ass in Real Life. Trust me, you have better things to do than worry about than whether or not upgrading one package for a minor security fix will drag along your system libs and userland utils with it. If this is the sort of thing you concern yourself with on a day to day basis, you're doing something WRONG.
Large environment management is a constant battle with entropy.
Hard drives die, switches fail, nics go bad, boards burn out, storage space fills up, and all this has to be dealt with. Using predictable, understandable, documented, tested and supported systems creates One Less Thing to worry about.
An entire IT staff should not have to be briefed on a daily basis about what the Gentoo Administrator decided to include in his(her?) build flags.
-s
mysqldump, stop mysql, unmerge it, emerge new version, and restore?
Anyway, it sounds like you're blaming Gentoo for something that is MySQL's fault. (Assuming that the format was changed, and not just the db dir location). It's probably because you went from 3.x to 4.x or similar.
Get your own free personal location tracker
It still begs the question... Should you really have to build software for a production server environment? (Yes, I know that Gentoo has the ability for binary packages, but their use seems to be actively discouraged by the culture, if not the documentation and the support channels as well.)
Real production environments, at least at the enterprise level, are built around stable, well tested binary packages that just work, change control processes, updates that can be applied safely with minimal technical skill and minimal configuration work, environments which may have one sysadmin for every 500 servers. Server builds should be able to be left to operations staff instead of sysadmins without fear of things going wrong.
Gentoo's strategy of bleeding edge continuous upgrades also doesn't fly in the enterprise world, which insists, for good reason, that functionality shouldn't change at all in production environments except as dictated by the organization's needs. This means that patches have to be back ported, release cycles are a requirement, and product support lifecycles are expected to be measured at least in months, preferably years. Functionality changes require a different kind of risk assessment and planning mentality in the business datacenter world than they do in the workstation world, and this is even more so with laws on the books regulating IT controls such as section 404 of the Sarbanes-Oxley act.and HIPPA in the US alone, and extensive data protection laws in other countries.
Not to mention, the difficulty of one person rolling back changes to 500 servers in an environment built on everything compiled from source is a huge drawback.
While I think that Gentoo is a tremendous technical achievement, it's design places it firmly in the hobbyist and developer workstation realm, and I strongly agree with the article that this leaves it no place in the datacenter for anything other than development work.
Yeah, but you have to admit ..... that's getting on for as much effort as it takes to be a Distributor!
If you want something that you know isn't going to change much, and certainly never in a way that breaks anything, use Debian Stable -- and be prepared to build the odd package from source {it really isn't as bad as it's made out to be} if you have to have a massively up-to-date version of something. They have a more-than-King-size package repository.
Je fume. Tu fumes. Nous fûmes!
The whole argument of "Gentoo 'wants' you to update a lot of things" is trivially debunked. Gentoo isn't a distro per se, it is a meta-distribution. I have worked in environments where Gentoo was used on servers, desktops, and what have you. The "solution" to Gentoo's frequent changes is simple: maintain your own portage tree mirror, which you keep frozen until you are good and ready to roll out the next major update (which of course you only do after extensive testing, like any Suse, Red Hat, or debian update). You define your own in-house releases, not Gentoo (and you graft security updates to your own tree as they come out--this isn't difficult, as each security update is announced by package).
This is trivial to do, and leads me to suspect the person putting forward the argument against using Gentoo (or any other well-engineered distribution) on servers either has an agenda, hasn't taken much time to ponder the issue, or doesn't understand the technology.
The Future of Human Evolution: Autonomy
Seriously the "if it ain't broke don't fix mentality" is what pays my bills.
... e.g. migrating from Apache 1.x to 2.x... and then there are bugs that haven't affected you yet but are still in the code base. Just because you haven't experienced any problems yet does not mean there aren't any underlying problems in the packages you're using.
There are two kinds of "broke", there are gaps in functionality
Case in point. The company I work for is in a mad dash to upgrade for the DST time change. And for those of you thinking "duh, you just upgrade your timezone files"... no it's not that easy. Some Sun systems require firmware upgrades, almost all of the systems prior to 2005 require binary updates because they can't handle a timezone that has two rulesets (e.g. they would apply the new 2007 rules to timestamps from 2005), most JVM's have to be patched or upgraded and some applications inexplicably do their own calculations and have to be update as well.
The majority of the company has the "if it ain't broke" mentality and were running everything from NT 4.0 on DEC Alpha's and Sun 2.4 to Windows 2003 64-bit and Solaris 10. Upgrading the older machines is an absolute nightmare because the vendor patches are built one, two even three years worth of patches that we haven't applied. What should be a relatively simple upgrade task has broken applications all over the place and has our QA and Engineering staff bleary eyed and ready for it all to just end.
The answer is controlled refresh. Twice a year you sync up your servers with a certain patchset. You don't go crazy... you just get vendor required patches and include them in your dev and qa cycles. And you DO NOT USE EOL OS' in an enterprise environment. Ever. This includes commercial and FOSS packages.
Full Disclosure : I run two gentoo boxes at my house my workstation and my mythtv box. I patch them about once a week because I like to tinker. My web/file/mysql server is running on a stripped down Debian system that only gets patched every few months or if there is an advisory that comes out.
Which is exactly the way I like my infrastructure. 3-6 months freeze with all bugs known, worked around or fixed in the meantime. Once I have gotten it to this point I build on top of that for the actual services which can run something very bleeding edge if necessary, but this is as I pointed out "your daily bread". For the stuff that is not, you need to be sure that it works and if you are a manager to be severely anal about it. So debian stable + 2-3 unavoidable backports and local builds is about right. This is also the reason corporations buy RedHat ES/AS/WS like hot bread. They finally see a model where the base has been frozen long enough to be relied on for building your own services.
Many itadmins and most developers have a problem with understanding of the "establish a platform and build on it" and "platform freeze before development" ideas. They think that everything is a fair game and the results (in man hours wasted on piecing everything together for release) are usually quite obvious.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Having run both FreeBSD and Gentoo systems, I think you're missing the obvious reason for why people are more satisified with FreeBSD: it is precisely because it doesn't have: "slots and masking and multiple supported versions". Okay, wait. It does have multiple supported versions, although rarely.
The problem with USE flags is that every Gentoo build environment is __too__ unique. With FreeBSD, everyone is running, debugging, and fixing the same stuff. Consequently, most of the ports build & work together out of the box.
I've come to the conclusion that Gentoo is solving a problem that plagued the OSS community in the '90s.
Major 'server' software is usually known stable within a month of its release and is usually incorporated into FreeBSD ports with a corresponding lag--except when driven by security issues. The mentality is that everything in ports should 'work'. Instability and brokenness is frowned upon and not excused by allowing somethings to be classified as unstable. When differences of opinion exist about versions (e.g., firefox15 and firefox2) the name space is split, but this is only tolerated when people genuinely disagree as to which is better.
In the end: FreeBSD => less choice. Less choice => more consistency. More consistency => easier administration.
Obviously there can be too little choice at some extreme, but generally FreeBSD saves admin time by adopting some degree of simplicity and uniformity.
It's just not trendy to knock on Slackware, so everyone targets Gentoo.
Some people also love to ignore advances that are made. The article mentions how long it took to install Gentoo. He claims that there was not an installer when he performed his installation more than a year ago. This is false. There was an installer, but it was considered experimental. Since then, the installer has become the de facto installation method on x86/amd64 and will be the default method on other architectures as support is added for them.
As for updates, who in the world out there has a ton of servers, then compiles on all of them? What is this guy smoking and where can I get some? Does he not realize that you can upgrade only what you want? There's nothing forcing you to upgrade the entire system. Hell, there's nothing forcing you to ever update the portage tree on your servers.
Any place where I'm ending up with more than one or two Gentoo boxes, I setup a local "master" server. This server will host my portage tree, which I don't update. It will also host my overlay. I update packages and add my own packages in this overlay. I build binary packages on this "master" server and distribute them to my servers. This really isn't a hard concept. It also isn't much different than the sort of thing that any sane administrator would do with any distribution. You don't just blindly run Red Hat updates, do you?
I think the biggest problem here is one of perception. People seem to ignore that when you buy Red Hat/SuSE, you are buying a product. You're buying support and service. With Gentoo, you're "buying" a toolset and nothing more. It's the difference between buying a car, and getting a bunch of parts and tools. They simply aren't comparable. If you want a nice Gentoo server infrastructure, you have to build it!
Gentoo has projects, such as the Scire project, which are designed to create a more enterprise-ready management set for Gentoo. However, this will not be a default for Gentoo, since few of our users would need it. Like everything else with Gentoo, you pick what you want from the tools provided, and you customize and tailor it to fit your needs. This entire article reads like a rant from someone who chose a tool because of its properties, then complained about those exact properties. You don't get to bitch when the "do-it-yourself" distribution asks you to do it yourself. ;]