Slashdot Mirror
Cross-Distro Remote Package Administration?
tobiasly writes "I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches. I currently have to log in to each system, run the appropriate apt-get or yum command to list available updates, determine which ones I need, then run the appropriate install commands. I'd love to have a distro-independent equivalent of the Red Hat Network where I could do all of this remotely using a web-based interface. PackageKit seems to have solved some of the issues regarding cross-distro package maintenance, but their FAQ explicitly states that remote administration is not a goal of their project. Has anyone put together such a system?"
No, nobody ever tried that before.
Hmmm... let's see. SSH ring any bells? Or are you actually going up and sitting at the box to do these updates?
yum -y update
Pirate Party UK
I recommend Webmin which 100% FOSS. I have found it reliable, flexible and feature-rich.
allows you to ssh into multiple machines and execute the same command on all of them from one terminal window. So if you set up a shell script that detects a host's distro and then execute the relevant update command you should be sorted.
I admin several busy CentOS servers for my company. You don't probably want a fully web-based application:
1) what happens when some RPM goes awry to borken your server(s)? Yes, it's pretty rare, but it DOES happen. In my case, I WANT to do them one by one in asc order of importance so that if anything is borked, it's most likely to be my least important systems!
2) How secure is it? You are effectively granting root privs to a website - not always a good idea. (rarely, never)
Me? I have a web doohickey to let me know when updates are available. Cron job runs nightly to yum and a pattern match identifies whether or not updates are needed, to show on my homepage. So it doesn't DO the update, butit makes it ez to see has been done.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Assuming that you want to do the remote thing with SSH as suggested above (and not actually sit down at each desk), you might find the "Tips and Tricks" section here interesting: http://www.gentoo.org/news/en/gmn/20080930-newsletter.xml
If you can script it should be fairly easy. Here is how I would do it (we run mostly gentoo servers and a mixture of windows, Ubuntu (and Ubuntu based) and RPM distros, but the guys using Linux customise so heavily and is tech savvy enough to keep themselves up to date.)
Set up sshd on every desktop, with key authorization (we do this with the gentoo servers.)
With a script and cron job you should be able to push them to run updates regularly. But you can just use the normal update tools and a local repo that is on a server on the lan to keep the packages updated.
Canonical also has a tool to do Ubuntu boxen over a network... cannot remember it's name though.
Any slashdotters happen to remember the name of the utility?
Seven Days with Ubuntu Unity
Look into Puppet or CFEngine (we use CFEngine but am considering switching to Puppet eventually). They're both extremely flexible management tools that will trivially handle package management, but you can use them to accomplish almost any management task you can imagine, with the ability to manage or edit any file you want, running shell scripts, etc.
The work flow goes something like this:
1. Identify packages that need update (have a cron job run on every box to email you packages that need updating, or just security updates, however you want to do it)
2. Update the desired versions in your local checkout of your cfengine/puppet files (the syntax isn't easily described here, but its very simple to learn).
3. Commit/push (note that this is the easy way to have multiple administrators) your changes. Optionally have a post commit hook to update a "master files" location, or just do the version control directly in that location.
4. Every box has an (hourly? Whatever you like) cron job to update against your master files location. At this time (with splay so you don't hammer your network) each cfengine/puppet client connects to the master server, updates any packages, configs, etc, runs any scripts you associated with those updates, then emails (or for extra credit build your own webapp) you the results.
/etc/init.d/yum start
and what do you know - the updates are installed automagically without any manual intervention
https://fedorahosted.org/func/
I know it's get Fedora in it's name but it's been accepted into as a package into Debian (and thus ubuntu).
It's pretty cool, designed to control alot of systems at once and avoid having to ssh into them all at once, has a build in certification system, a bunch of modules written for it already , usable from the command line so you can easily add it into your scripts and has a python api so if you really wanted some you could throw together some django magic if you wanted a web front end. OpenSymbolic is a webfront end for it already although I haven't checked it out.
Not exactly what you wanted as there's a bunch of work you'd need to do to get it to do the things you want.
Why don't you determine which packages you DON'T want updated automatically and add them to an exclude list on each machine. Then you can run yum update from cron.daily and update the accepted packages, then set up a cron job to run an hour or so after the update which checks for other available package updates. It's pretty simple to run yum check-update and pipe the output into an email.
I have no idea if you can do this with apt but I don't see why not.
Instead of a fancy web solutions, you could use the script and scriptreplay commands on each system. Do whatever you need to do once on 1 system but record that using script. After that replay the scripts on each of the other systems. You could either manually or automatically log on to each system and start the replay or you could set up a cron job which fetches and replays the script you publish somewhere.
Not very fancy, but it will get the job done, and it will work for more than just updates, you could also use it to e.g. changes setting or add packages. Or basically anything else you can do from a shell in a repeatable way.
Check man 1 script and man 1 scriptreplay for details.
The easiest way would probably be to use clusterssh or pssh and define the one cluster for each distro. Set up key-based login and then just have a cron job that does pssh -h ubuntu-machines -l username sudo apt-get update >> /var/log/ubuntu-update.log
Add a cron-entry and hosts file for every distro you need.
I think Spacewalk from Redhat (http://www.redhat.com/spacewalk/faq.html) would at least work with the CentOS machines. I believe it adds something similar to the update and configuration management stuff on RHN. It won't work the the Ubuntu desktops though.
1) yum -e whateveryoudontneed /etc/init.d/yum-updatesd start
2) chkconfig yum-updatesd on
3) Make sure do_update = yes, download_deps = yes, etc are set in yum-updatesd.conf
4)
This makes your yum system self-updating.
I for one look forward to the rational, well thought out, debate on the various pros and cons of linux distributions and their package managers, that this story will become.
I work in a large ISP, and this is the way we manage updates for the various Linux platforms we use. Quite simple, really. You can build tools that help: diff between the downloaded updates and what you have in your own repository, and mail you the ones that you are not using. I find lwn.net's security pages useful in keeping track of what security updates matter to us.
Puppet is a tool made to do exactly what you're asking for by abstracting the specific operating system into a generic interface. It might be worth checking out. Also there's a newcomer called chef. And then there's the oldies like cfengine.
How about a sample? I use Gentoo. My servers do this every night: /usr/bin/emerge --quiet --color n --sync && update-eix --quiet && glsa-check -l affected
I could just as well apply every fix automatically, but I like to see it before it goes in.
Acts 17:28, "For in Him we live, and move, and have our being."
I used aptitude for a few months after reading that it installed recommended packages in addition to true dependencies. Well, bollocks to that. There are more times when I don't want "recommended" packages (installing exim sucks when postfix is desired, for example).
As to the script to maintain servers ... below a trivial example for debian/ubuntu -based systems. Setup ssh keys to make life easier from an internal, protected system. If you have more than 6 systems, you probably want to mirror the repositories locally. Also, set your TERM environment variable to make interactive install/updates nicer to use.
===== Weekly Updates ======
#!/bin/sh
SRVS="s1 s2 s3 s4 s5 s6 s7 s8 s9"
for D in $SRVS ; do
ssh root@$D "apt-get update; apt-get upgrade;"
done
red carpet used to be excellent for that, it was developed by ximian back in the days, now this is supposed to be novell. but i never heard much about it anymore.
You could also use nagios and check_apt/check_yum to alert you of out of necessary security updates, put a script for installing updates on every box (different script for centos/ubuntu, but same syntax), create a user who is added to sudoers for only that script, and create an ssh key for authentication...
Then you can feed the list of hosts that need updating into a script which will ssh to each one in sequence and execute the update script followed if necessary by a reboot..
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
On Debian type systems cron-apt is extremely useful for having remote machines notify via email and/or syslog of available updates. By default it downloads but does not install new packages, though it can be set up to do anything you can do with apt-get, so for example you could set it up to automatically install security patches but not other packages. I don't have enough similar machines to benefit from using clusterssh but it cron-apt+clusterssh would seem to be ideal for remote package management of multiple similar Debian type systems.
Puppets and Puppet Master (Actual application names) works for RPM and Debian Package Management, allows push type update I think.
You hit the nail on the head. I have a cron job to exactly that and for 3 days after an "auto update", my Mythbuntu box shuts down on its own after running for about 54 hours. I am now wondering whether an update I should have avoided is the culprit.
By the way, can one point me to a resource that could help me determine what's going on on my box? Thanks.
The apticron package will e-mail you any pending updates on Ubuntu machines...
try using capistrano. a small script over the weekend will simplify your task.
And is there such a repository for Windows that has roughly the same amount of software?
Although I'm sure there is a *nux version.
After all, it's ready for the desktop and the enterprise. Patch management and application distribution would have been one of the first things they sorted.
The key is not to look for "ubuntu and centos solutions" (which sounds more like something typically asked a consultant), but to apply your supposedly broad system administration skills and adapt the already existing administration tools to your environment.
Typically done with small snippets of shell script building on your environments build- and packaging tools, and so on, but more mature things do exist. I mention two:
arusha (ark) http://ark.sourceforge.net/
puppet http://reductivelabs.com/products/puppet/
I would say that FAI is worth looking at. You will have full control over which updates are applied.
I think that you are not putting your efforts where it matters.
What is important is that the critical services run properly on each server. Sure that can be affected by patching but also by many other factors. So don't focus solely on the patching, focus instead on making sure all the services are running properly.
You should have your own scripts that check that each server is responding as required. Make your test suite as strong as you can and improve it each time a new problem crops up that wasn't caught by your spying tools.
Once you have this in place, you can safely do daily automatic updates and stop second guessing the package maintainers. You will have a more reliable system and you will save yourself a lot of work too over the years.
Cfengine is your firend ;-)
You can launch update command every day and manage packages installations with the package command.
Sorry for my bad english
Altiris can do what you ask. You'll have to spend money to get it, but you can do software delivery and patching for Windows, *NIX, and Apple from it (and many other things, depending on the size of your wallet).
There are oodles of configuration management tools out there that do at least most of what you want. My personal recommendation is Bcfg: http://www.bcfg2.org/ . It doesn't quite have the entire web interface (yet), but it is fantastic for keeping everything up to date and clean and telling you when you have outliers. I currently use it for the 350 or so support machines for the 5th fastest computer in the world, and I know _much_ larger installations are using too.
Posted from the wireless couch.
That's your job. Bash CLI, the CLI toolkit, CLI Emacs, key-based ssh and a well-maintained, well-documented pack of own scripts in your favourite interpreted PL are just what it takes to do this sort of thing. No fancy bling-bling required or wanted. It would make your worse, not easyer in the long run.
We suffer more in our imagination than in reality. - Seneca
"I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches"
My advice is, if it ain't broke don't fix it, especially on a production server. Have two identical systems and test the latest bugfix on that, before you roll it out to the live system. You don't know what someone elses bugfix is going to break and would have no way of rolling it back.
This will let you log into mutiple machines at the same time. We use this to manage several hundred workstations and servers since they want us to log into each for updating.
Using cssh with multiple monitors and you have access to each.
1) a configuration system like CFengine or Puppet. It will take care of configuring and updating the machines.
2) a web based monitoring interface. It will show you the system status. Something like Munin, Cacti, etc.
Resolving the large numbers of packages and their minor variations such as i386, x86_64, etc. across all the systems means a potentially large number of distinct Nagios query templates. Generating and managing those directly is a fascinating problem. Building it from scratch for a single in-house usage is a lot of work, usually better spent elsewhere.
It really makes me wish that RedHat Satellilte Server did _not_ require an Oracle backend. For RedHat themselves, with so many thousands of clients, it makes sense to use Oracle for 'RedHat Network', on which the Satellite Server is based. But for the rest of us, almost any lightweight database would have served better and required far fewer resources to run. But now that Oracle has bought Sun, I think we can rely on Oracle to insist that such corporate level supported services require Oracle, not MySQL.
You're probably looking for:
http://www.novell.com/products/zenworks/linuxmanagement/
And yes, it does support some flavors of RedHat. Nigbuntu you're SOL on. If you're developers are running that you should fire them anyway.
If a system is working properly, then applying updates cannot make it better. It can leave it the same or make it worse. I Install my servers and leave them alone for about 3 years, then re-install and run them for another 3 years, then I toss them away and install new ones. Most Linux prolems are caused by finger trouble, so a security problem has to be very serious before I would apply it.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
In all honesty I wouldn't set up a solution. Keep in mind that with different release schedules and different release criteria many times there may be a day or two lag between them and you still need to validate the patch\fix before putting it into a production environment. So given Ubuntu and CentOS, let's say the CentOS patch comes out 2 days later you are either doing a regression test twice on the two systems, or waiting till both are release and running a single regression test.
If you are hell bent on doing this the simplest solution I've found is simply a CRON job script that does the following:
One each system, every day at a given time do a check for package updates and send the results via an email to a shared mailbox called PENDING UPDATES. You can also send them to a SQL database if you choose (that's what I do)
The next thing is to hav a shell script that will SSH into a system and based on the platform grab any pending updates in the database that are marked with a FETCH flag turned on (I use PERL for all shell scripts so tapping a database is pretty damn easy).
Now that shell script will read from the DB of all the systems I manage (before I retired from Geekery that is I handled as many as 2300 systems split between Linux, Unix, NT, AS400, a pair of DEC Alpha workstations, and oddly enough 22 OS/2 boxes) that are flagged for Linux and execute the update script on each box.
As you posted your dillemma you need to work smarter not harder. There is NO REASON for you to "run around". A simple PERL script to execute the same command on multiple systems should be the norm. You can even make the script multi-threaded with the "use thread" or "use threads" depending on which 'flava' of rthe threading library you use (Use the one support QUEUES, easier to stuff the QUEUE with system names, set a MAXTHREADS count and just suck out servers and dispatch them to worker threads)
If you need to on Ubuntu pull an APT-GET INSTALL BLAH and in CENTOS do an update install BLAH it's a simple IF statement on the worker thread.
MY BIGGEST ADVICE: ALWAYS, ALWAYS HAVE HUMAN INTERVENTION INVOLVED BEFORE UPDATING ANYTHING!!
Again whip up a quick MYSQL\POSTGRES database, parse the output of the pending updates for each platform, and have a boolean field that indicates if the update is approved. Then let CRON handle pulling down approved updates or, as I reccomend, have an admin run a script that pulls down updates so it can coincide with regular matinance windows...
-=[ Who Is John Galt? ]=-
I've been using openpkg to do this, works well, keeps my services up to day and works across multiple platforms.
Salut,
Jacques
I know everyone hates Novell but they got a product called ZLM which aims to control servers and workstations during all their life cycle (including network installation of OS, deployment and some cool management tools) and its not restricted to RPM based Linux. You can separate hosts by groups and apply stuff per group or folder. Check it: http://www.novell.com/products/zenworks/linuxmanagement/
I can recommend this book (it uses cfengine extensively): Automating Linux and Unix System Administration, Second Edition
Natxo Asenjo
At the behest of my manager, I've been working on implementing Spacewalk here at my current contract. It's the open-sourced version of Redhat's Satellite.
Downside: they've only recently released 0.5, and there are still lots of bugs (help doesn't, in general, work).
But you can inventory, and push, updates to rpms; ditto for configuration files, and there's kickstart support. It also supports other than RH and Fedora; I've been working with CentOS (yeah, I know, just the serial numbers filed off), but it appears to me that it should support any distro with yum.
Someone noted it requires Oracle: true, the free version of Oracle XE (10g).
I don't like it - way too many problems yet - but it does work (mostly), and is explicitly for what the poster was looking. You can put multiple systems into a group, and do the group (I think), and you define your own software channels and child channels, and subscribe systems to them. It should tell you when they need upgrades (as long as you keep your internal repository up to date).
mark
You can set LDAP to offer the name of each repository server for each machine.
Then roll out that update to that machine.
Install Windows.
If you or someone you work with can write scripts, the Expect scripting language is specifically designed to communicate to numerous machines through telnet sessions - it may prove valuable. Input: a list of ip addresses & logins & passwds. Output: the appropriate interactive commands to search out and kick-off ubuntu updates. There is a security risk to consider, because Expect code visibly contains user passwords for automatic login. Advantage: Expect runs from one machine, establish a telnet sessions to all other machines, and updatew them seemlessly and with an interface for the user to pick and choose if programmed correctly.
Regarding the CentOS side, you could try func or if you have an oracle installation lying around, spacewalk.
This may help you: http://www.canonical.com/projects/landscape
Are you guys really admins?
Script what you would do manually. It's not that hard to determine your distro and thus which app to use to update. Determine how you manage manually and just start BASH'ing it out in a script to do the same thing remotely. SSH keys will be required. Then set up sudo on each system to allow you to run certain commands passwordless. Just be sure to use ssh -t when connecting so that sudo will work properly.
Really, this is admin 101. Any good admin should be able to script. If you can't, then start learning; Or you will only ever be a mediocre admin.
The truth is usually just an excuse for lack of imagination.
http://reductivelabs.com/products/ That will be $4000 in consulting fees, thanks.
Can you setup a cron job that executes on each system? The only problem is that you have to update this script a bunch of places should it change. So you may want to keep the script stored on a "master server" and then have the clients just download the file and execute it each night etc...
This question alone is probably the most compelling reason to not adopt Linux as a desktop solution.
Why bother
I agree with most of the tips here... Just one last tip that I think will really help. Having a Staging / Testing Environment that is similar to your production setup. Whatever you do (i.e. install tools, new scripts, new packages), run it through your Staging Network and let it cook for a few days. If it blows up, that sucks, but at least it didn't blow up production.
Lastly, I've read a couple of people who endorse CFEngine / Puppet. I've been running slack, which is great, it's easy to set up, runs great, and easy to understand. I recommend people check it out.
http://www.sundell.net/~alan/projects/slack/
as a good python project. Just simply forward the x session and have a nice gui round it. get a uname to see the kernel running and you're off to the races.
My Babylon
How about ocsinventory to collect information on your machines. Stores all in MySQL. You can write custom queries to see what you need to update where. Use puppet to manage the updates.
Also work hard to reduce the variations. Each machine should start with a common base and only add the must haves which make them unique. Then you can identify classes of machines so you can focus on updates per class, hopefully reducing your work load in the process.
What kind of Unix admin are you?
That is all.
This may be unsafe depending on which environment variables "yum" reads and if it propagates the environment to programs that it executes. What you want is sudo -- it can be configured to run specific tasks securely.
Why not use a cronjob, that checks what you can update on your system, and mail it to you? Then you will always know what packages you need to update. And the update job can be done with scripts. Like someone else her wrote. Just have a master server, or maybe two, one for each distro. And they can give out the scripts to the machines that needs them. At least you can use that for the less "important" updates. Just my thought on the subject.
Strongly recommend Puppet (http://reductivelabs.com) for this. It's built to do exactly this and a lot more - managing users, groups, packages, cron jobs, and a wide variety of other configuration. It also scales well - Google uses it to manage about 8000 desktops for example and a bunch of ISPs/corporates use it to manage large numbers of production hosts.
Some of my tasks involve administering a 10+ debian/ubuntu machines. Sadly our administrational resources are pretty limited, so manual security patching is not an option, and neither is not-patching.
What I've done is set up cron-apt to only use the security updates repo, exclude the kernel updates, and update automatically. It mails all actions to me, so I have a mailbox of when what was updates if something goes wrong (not once in 1,5 years). There's always a risk with this of course, but it depends on how critical service uptime is.
It should be noticed that lamapper is a twitter sockpuppet taking his empire to 16 sock accounts on /.
yum-updatesd in CentOS & Fedora and /etc/apt/apt.conf.d/50unattended-upgrades in Ubuntu are your friends.
If you configure these properly in combination with a local mirror, you can get it to deploy automatic updates seamlessly.
__________________
Leo
webmaster@007sdomain.com
wsus requires domain and shavlik doesn't work in win anyway. so is there a single one like opsware but (for home) that can handle linux variants and windows versions?
mrepo does exactly what you want. It will mirror any package sources you want (including Redhat Network if you have a subscription) and automatically format a repository for yum or apt clients.