Cross-Distro Remote Package Administration?

tobiasly writes "I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches. I currently have to log in to each system, run the appropriate apt-get or yum command to list available updates, determine which ones I need, then run the appropriate install commands. I'd love to have a distro-independent equivalent of the Red Hat Network where I could do all of this remotely using a web-based interface. PackageKit seems to have solved some of the issues regarding cross-distro package maintenance, but their FAQ explicitly states that remote administration is not a goal of their project. Has anyone put together such a system?"

Comments

Tools exist

[PeterBrett]

1. Create a local package repository for each distro.
2. Set apt/yum to point at only the local repository.
3. Create a cron job on each box to automatically update daily.
4. When you want to push a package update out to all boxes, copy it from the public repository to the local one.
5. Profit!

Re:Tools exist

[Jurily]

When you want to push a package update out to all boxes, copy it from the public repository to the local one.

Assuming of course all boxes have the same version of the OS, the same packages installed, etc.

I suggest tentakel, and that OP could have found it in 2 minutes with Google. I did.

http://www.google.co.uk/search?q=multiple+linux+remote+administration The first hit mentions it.

Re:Tools exist

[comcn]

As another "sub 7-digit guy" - there is a reason for this... There is no way I'm going to let over 60 servers automatically install patches without me checking them first! Download, yes. Install, no.

At work we use cfengine to manage the servers, with a home-built script that allows servers to install packages (of a specified version). Package is checked and OK? Add it to the bottom of the text file, GPG sign it, and push it into the repository. cfengine takes care of the rest (our cf system is slightly non-standard, so everything has to be signed and go through subversion to actually work).

Puppet or CFEngine + Version Control

[hax0r_this]

Look into Puppet or CFEngine (we use CFEngine but am considering switching to Puppet eventually). They're both extremely flexible management tools that will trivially handle package management, but you can use them to accomplish almost any management task you can imagine, with the ability to manage or edit any file you want, running shell scripts, etc.

The work flow goes something like this:
1. Identify packages that need update (have a cron job run on every box to email you packages that need updating, or just security updates, however you want to do it)
2. Update the desired versions in your local checkout of your cfengine/puppet files (the syntax isn't easily described here, but its very simple to learn).
3. Commit/push (note that this is the easy way to have multiple administrators) your changes. Optionally have a post commit hook to update a "master files" location, or just do the version control directly in that location.
4. Every box has an (hourly? Whatever you like) cron job to update against your master files location. At this time (with splay so you don't hammer your network) each cfengine/puppet client connects to the master server, updates any packages, configs, etc, runs any scripts you associated with those updates, then emails (or for extra credit build your own webapp) you the results.

Re:Can You Script?

[dns_server]

The corporate product is http://www.canonical.com/projects/landscapeLandscape

Re:You don't want it

[galorin]

Depending on how uniform your servers are, keep one version of CentOS and one version of Ubuntu running in a VM, and have these notify you when updates are available. When updates are available, test against these VMs, and do the local repository thing suggested by another person here. Do one system at a time to make sure something doesn't kill everything at once.

Web based apps with admin privs are fine as long as they're only accessable via the intranet, strongly passworded, and no one else knows they're there. If you need to do remotely, VPN in to the site, and SSH into each box. You're an Administrtor, start administratorizing. Some things just shouldn't be automated.

yum-updatesd is meant for that

[MrMr]

1) yum -e whateveryoudontneed
2) chkconfig yum-updatesd on
3) Make sure do_update = yes, download_deps = yes, etc are set in yum-updatesd.conf
4) /etc/init.d/yum-updatesd start
This makes your yum system self-updating.

Re:Remote admin of a UNIX box?

[supernova_hq]

Sorry to reply to my own post, but circlingthesun actually posted the name of it below!

clusterssh

Re:Remote admin of a UNIX box?

[walt-sjc]

It's called "dssh". Google is your "search" friend (we will ignore the evil side of Google at the moment... :-)

Re:In centos you could try

[BruceCage]

I'd say that it depends on a lot of factors really.

First of all it depends on how mission critical the services that run on that system are considered and what kind of chances you're willing to take that a particular package might break something. The experience and available time of your system administrator also plays a significant role.

There's also the very highly unlikely scenario that a certain update might include "something bad", for example when the update servers are compromised. See Debian's compromises at Debian Investigation Report after Server Compromises from 2003, Debian Server restored after Compromise from 2006, and Fedora's at Infrastructure report, 2008-08-22 UTC 1200.

I currently manage just a single box (combination of a public web server and internal supporting infrastructure) for the company I work at and have it automatically install both security and normal updates.

I personally trust the distro maintainers to properly QA everything that is packaged. Also, I don't think any single system administrator has the experience or knowledge to be able to actually verify whether or not an update is going to be installed without any problems. The best effort one can make is determine whether or not an update is really needed and then keep an eye on the server while the update is being applied.

In the case of security updates it's a no-brainer for me, they need to be applied ASAP. I haven't had the energy to setup a proper monitoring solution and I've never even seen Red Hat Network in action. So if I had to manually verify available updates (or even setup some shell scripts to help me here) it would be just too much effort considering the low mission criticality of the server. If there does happen to be a problem with the server I'll find out about it fast enough then I'll take a peak at the APT log and take it from there.

Re:Remote admin of a UNIX box?

[MikeBabcock]

IMHO, a good sysadmin is the one who sees such issues and finds or writes a script to do it for them.

(That's "you're both right")

for address in addresses; do
      ssh $address 'touch /var/lib/triggers/update'
done

With an obvious job on the machines watching for such trigger files (to avoid root access, etc.)

if [ -f $TRIGGERFILE ]; then
      yum -y update 2>&1| tee /tmp/yum_trigger.log \
      && rm -f $TRIGGERFILE
fi