Cross-Distro Remote Package Administration?

tobiasly writes "I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches. I currently have to log in to each system, run the appropriate apt-get or yum command to list available updates, determine which ones I need, then run the appropriate install commands. I'd love to have a distro-independent equivalent of the Red Hat Network where I could do all of this remotely using a web-based interface. PackageKit seems to have solved some of the issues regarding cross-distro package maintenance, but their FAQ explicitly states that remote administration is not a goal of their project. Has anyone put together such a system?"

Comments

Tools exist

[PeterBrett]

1. Create a local package repository for each distro.
2. Set apt/yum to point at only the local repository.
3. Create a cron job on each box to automatically update daily.
4. When you want to push a package update out to all boxes, copy it from the public repository to the local one.
5. Profit!

Re:Tools exist

[Jurily]

When you want to push a package update out to all boxes, copy it from the public repository to the local one.

Assuming of course all boxes have the same version of the OS, the same packages installed, etc.

I suggest tentakel, and that OP could have found it in 2 minutes with Google. I did.

http://www.google.co.uk/search?q=multiple+linux+remote+administration The first hit mentions it.

Re:Tools exist

[value_added]

Assuming of course all boxes have the same version of the OS, the same packages installed, etc.

And segregating things on the system that hosts the public repository is impossible?

I don't think any of this is exactly rocket science. On my home LAN where I use FreeBSD, for example, I have a motley collection of hardware ranging from Soekris boxes to Opterons. Everything gets built on a central build server and distributed automagically from there using a setup similar to what's suggested the OP. Not a single box has the same collection of userland software installed, while certain boxes do get their own custom world/kernel. None of this really requires more effort or involvement on my part than some careful thought beforehand.

One of the nice advantages of a centralised setup is that it accommodates a clean way of testing things beforehand. Rolling out the latest but broken version of "foo" to multiple systems is something to be avoided.

Re:Tools exist

[comcn]

As another "sub 7-digit guy" - there is a reason for this... There is no way I'm going to let over 60 servers automatically install patches without me checking them first! Download, yes. Install, no.

At work we use cfengine to manage the servers, with a home-built script that allows servers to install packages (of a specified version). Package is checked and OK? Add it to the bottom of the text file, GPG sign it, and push it into the repository. cfengine takes care of the rest (our cf system is slightly non-standard, so everything has to be signed and go through subversion to actually work).

Re:Remote admin of a UNIX box?

[backwardMechanic]

Maybe that works for your home network, but SSH'ing to 25 or (maybe a lot) more different boxes to repeat the same task is a bit tedious. Hey, doesn't this sound like the kind of automated task a computer might be good at?

Webmin

[trendzetter]

I recommend Webmin which 100% FOSS. I have found it reliable, flexible and feature-rich.

Re:Webmin

[ParanoidJanitor]

I have to second this. Webmin has everything you ask for and then some. If you have an update script on each machine, you could easily update all of your machines at once with the cluster management tools. I know it works well with APT (having used it myself), but I can't speak for any of the other package managers. In the worst case, it's still easy to push an update command to the non-apt machines through the Webmin cluster tools.

clusterssh

[circlingthesun]

allows you to ssh into multiple machines and execute the same command on all of them from one terminal window. So if you set up a shell script that detects a host's distro and then execute the relevant update command you should be sorted.

You don't want it

[mcrbids]

I admin several busy CentOS servers for my company. You don't probably want a fully web-based application:

1) what happens when some RPM goes awry to borken your server(s)? Yes, it's pretty rare, but it DOES happen. In my case, I WANT to do them one by one in asc order of importance so that if anything is borked, it's most likely to be my least important systems!

2) How secure is it? You are effectively granting root privs to a website - not always a good idea. (rarely, never)

Me? I have a web doohickey to let me know when updates are available. Cron job runs nightly to yum and a pattern match identifies whether or not updates are needed, to show on my homepage. So it doesn't DO the update, butit makes it ez to see has been done.

Re:You don't want it

[galorin]

Depending on how uniform your servers are, keep one version of CentOS and one version of Ubuntu running in a VM, and have these notify you when updates are available. When updates are available, test against these VMs, and do the local repository thing suggested by another person here. Do one system at a time to make sure something doesn't kill everything at once.

Web based apps with admin privs are fine as long as they're only accessable via the intranet, strongly passworded, and no one else knows they're there. If you need to do remotely, VPN in to the site, and SSH into each box. You're an Administrtor, start administratorizing. Some things just shouldn't be automated.

Puppet or CFEngine + Version Control

[hax0r_this]

Look into Puppet or CFEngine (we use CFEngine but am considering switching to Puppet eventually). They're both extremely flexible management tools that will trivially handle package management, but you can use them to accomplish almost any management task you can imagine, with the ability to manage or edit any file you want, running shell scripts, etc.

The work flow goes something like this:
1. Identify packages that need update (have a cron job run on every box to email you packages that need updating, or just security updates, however you want to do it)
2. Update the desired versions in your local checkout of your cfengine/puppet files (the syntax isn't easily described here, but its very simple to learn).
3. Commit/push (note that this is the easy way to have multiple administrators) your changes. Optionally have a post commit hook to update a "master files" location, or just do the version control directly in that location.
4. Every box has an (hourly? Whatever you like) cron job to update against your master files location. At this time (with splay so you don't hammer your network) each cfengine/puppet client connects to the master server, updates any packages, configs, etc, runs any scripts you associated with those updates, then emails (or for extra credit build your own webapp) you the results.

Re:Remote admin of a UNIX box?

[Nursie]

Uh, right. Like putting ssh commands into a script?

ssh user@host aptitude update

Set up key based login and you don't even have to type passwords. By the sounds of it he needs to pay some attention to each individual machine anyway, as he has multiple distros and wants to determine which patches he needs for each box.

Re:Can You Script?

[dns_server]

The corporate product is http://www.canonical.com/projects/landscapeLandscape

yum-updatesd is meant for that

[MrMr]

1) yum -e whateveryoudontneed
2) chkconfig yum-updatesd on
3) Make sure do_update = yes, download_deps = yes, etc are set in yum-updatesd.conf
4) /etc/init.d/yum-updatesd start
This makes your yum system self-updating.

Great!

[Frogbert]

I for one look forward to the rational, well thought out, debate on the various pros and cons of linux distributions and their package managers, that this story will become.

Re:Tools exist: we do it this way.

[nick_urbanik]

I work in a large ISP, and this is the way we manage updates for the various Linux platforms we use. Quite simple, really. You can build tools that help: diff between the downloaded updates and what you have in your own repository, and mail you the ones that you are not using. I find lwn.net's security pages useful in keeping track of what security updates matter to us.

Re:In centos you could try

[cerberusss]

the updates are installed automagically without any manual intervention

I'm not sure that's a good idea on a server. Why would you mindlessly update packages on a server when there's no actual reason to do so?

Re:Remote admin of a UNIX box?

[supernova_hq]

Sorry to reply to my own post, but circlingthesun actually posted the name of it below!

clusterssh

Re:Remote admin of a UNIX box?

[BruceCage]

Set up key based login and you don't even have to type passwords.

Since you basically need root access to do updates this definitely poses a security hazard as when your client is compromised there is direct access to the server. Then again, an attacker could always use a keylogger to capture the password anyways.

If you even attempt to do this I'd setup a different user account specifically for the process of updating and limit the rights accordingly and then I'd restrict the commands that can be executed (you can do this per key).

There may actually be better ways but I'm not a very experienced sysadmin. Most experience I have is from managing a single web server and my local desktop obviously. Be sure to correct me (in a friendly manner) if I'm wrong.

Then again, if you do this from the same machine as your normal account is located on you'll still have the same issues in case of a compromised client. Probably just best to limit every single account to just that what is specifically needed and setup proper host based intrusion detection (OSSEC?) to be notified when something goes wrong. This stuff is hard...

Re:In centos you could try

[supernova_hq]

Because as any decent linux-server-farm admin, you have a closely controlled local repository mirror that only has updates you specifically add.

Re:Remote admin of a UNIX box?

[walt-sjc]

It's called "dssh". Google is your "search" friend (we will ignore the evil side of Google at the moment... :-)

Re:Tools exist: we do it this way.

[Bert64]

You could also use nagios and check_apt/check_yum to alert you of out of necessary security updates, put a script for installing updates on every box (different script for centos/ubuntu, but same syntax), create a user who is added to sudoers for only that script, and create an ssh key for authentication...
Then you can feed the list of hosts that need updating into a script which will ssh to each one in sequence and execute the update script followed if necessary by a reboot..

Re:In centos you could try

[BruceCage]

I'd say that it depends on a lot of factors really.

First of all it depends on how mission critical the services that run on that system are considered and what kind of chances you're willing to take that a particular package might break something. The experience and available time of your system administrator also plays a significant role.

There's also the very highly unlikely scenario that a certain update might include "something bad", for example when the update servers are compromised. See Debian's compromises at Debian Investigation Report after Server Compromises from 2003, Debian Server restored after Compromise from 2006, and Fedora's at Infrastructure report, 2008-08-22 UTC 1200.

I currently manage just a single box (combination of a public web server and internal supporting infrastructure) for the company I work at and have it automatically install both security and normal updates.

I personally trust the distro maintainers to properly QA everything that is packaged. Also, I don't think any single system administrator has the experience or knowledge to be able to actually verify whether or not an update is going to be installed without any problems. The best effort one can make is determine whether or not an update is really needed and then keep an eye on the server while the update is being applied.

In the case of security updates it's a no-brainer for me, they need to be applied ASAP. I haven't had the energy to setup a proper monitoring solution and I've never even seen Red Hat Network in action. So if I had to manually verify available updates (or even setup some shell scripts to help me here) it would be just too much effort considering the low mission criticality of the server. If there does happen to be a problem with the server I'll find out about it fast enough then I'll take a peak at the APT log and take it from there.

Re:Remote admin of a UNIX box?

[MikeBabcock]

IMHO, a good sysadmin is the one who sees such issues and finds or writes a script to do it for them.

(That's "you're both right")

for address in addresses; do
      ssh $address 'touch /var/lib/triggers/update'
done

With an obvious job on the machines watching for such trigger files (to avoid root access, etc.)

if [ -f $TRIGGERFILE ]; then
      yum -y update 2>&1| tee /tmp/yum_trigger.log \
      && rm -f $TRIGGERFILE
fi