Slashdot Mirror
Cross-Distro Remote Package Administration?
tobiasly writes "I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches. I currently have to log in to each system, run the appropriate apt-get or yum command to list available updates, determine which ones I need, then run the appropriate install commands. I'd love to have a distro-independent equivalent of the Red Hat Network where I could do all of this remotely using a web-based interface. PackageKit seems to have solved some of the issues regarding cross-distro package maintenance, but their FAQ explicitly states that remote administration is not a goal of their project. Has anyone put together such a system?"
Pirate Party UK
Maybe that works for your home network, but SSH'ing to 25 or (maybe a lot) more different boxes to repeat the same task is a bit tedious. Hey, doesn't this sound like the kind of automated task a computer might be good at?
I recommend Webmin which 100% FOSS. I have found it reliable, flexible and feature-rich.
allows you to ssh into multiple machines and execute the same command on all of them from one terminal window. So if you set up a shell script that detects a host's distro and then execute the relevant update command you should be sorted.
I admin several busy CentOS servers for my company. You don't probably want a fully web-based application:
1) what happens when some RPM goes awry to borken your server(s)? Yes, it's pretty rare, but it DOES happen. In my case, I WANT to do them one by one in asc order of importance so that if anything is borked, it's most likely to be my least important systems!
2) How secure is it? You are effectively granting root privs to a website - not always a good idea. (rarely, never)
Me? I have a web doohickey to let me know when updates are available. Cron job runs nightly to yum and a pattern match identifies whether or not updates are needed, to show on my homepage. So it doesn't DO the update, butit makes it ez to see has been done.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Look into Puppet or CFEngine (we use CFEngine but am considering switching to Puppet eventually). They're both extremely flexible management tools that will trivially handle package management, but you can use them to accomplish almost any management task you can imagine, with the ability to manage or edit any file you want, running shell scripts, etc.
The work flow goes something like this:
1. Identify packages that need update (have a cron job run on every box to email you packages that need updating, or just security updates, however you want to do it)
2. Update the desired versions in your local checkout of your cfengine/puppet files (the syntax isn't easily described here, but its very simple to learn).
3. Commit/push (note that this is the easy way to have multiple administrators) your changes. Optionally have a post commit hook to update a "master files" location, or just do the version control directly in that location.
4. Every box has an (hourly? Whatever you like) cron job to update against your master files location. At this time (with splay so you don't hammer your network) each cfengine/puppet client connects to the master server, updates any packages, configs, etc, runs any scripts you associated with those updates, then emails (or for extra credit build your own webapp) you the results.
Uh, right. Like putting ssh commands into a script?
ssh user@host aptitude update
Set up key based login and you don't even have to type passwords. By the sounds of it he needs to pay some attention to each individual machine anyway, as he has multiple distros and wants to determine which patches he needs for each box.
The corporate product is http://www.canonical.com/projects/landscapeLandscape
/etc/init.d/yum start
and what do you know - the updates are installed automagically without any manual intervention
https://fedorahosted.org/func/
I know it's get Fedora in it's name but it's been accepted into as a package into Debian (and thus ubuntu).
It's pretty cool, designed to control alot of systems at once and avoid having to ssh into them all at once, has a build in certification system, a bunch of modules written for it already , usable from the command line so you can easily add it into your scripts and has a python api so if you really wanted some you could throw together some django magic if you wanted a web front end. OpenSymbolic is a webfront end for it already although I haven't checked it out.
Not exactly what you wanted as there's a bunch of work you'd need to do to get it to do the things you want.
Instead of a fancy web solutions, you could use the script and scriptreplay commands on each system. Do whatever you need to do once on 1 system but record that using script. After that replay the scripts on each of the other systems. You could either manually or automatically log on to each system and start the replay or you could set up a cron job which fetches and replays the script you publish somewhere.
Not very fancy, but it will get the job done, and it will work for more than just updates, you could also use it to e.g. changes setting or add packages. Or basically anything else you can do from a shell in a repeatable way.
Check man 1 script and man 1 scriptreplay for details.
1) yum -e whateveryoudontneed /etc/init.d/yum-updatesd start
2) chkconfig yum-updatesd on
3) Make sure do_update = yes, download_deps = yes, etc are set in yum-updatesd.conf
4)
This makes your yum system self-updating.
I have a question about Landscape.
Is it possible to run your own server?
If not, isn't it just another piece of vendor lock-in?
I'm interested in using it but I don't want to depend on Canonical. For example, what if my internet connection goes down? I'd lose the ability to use Landscape at all, right?
The disappearing pencil trick. Let me show you it.
I for one look forward to the rational, well thought out, debate on the various pros and cons of linux distributions and their package managers, that this story will become.
I work in a large ISP, and this is the way we manage updates for the various Linux platforms we use. Quite simple, really. You can build tools that help: diff between the downloaded updates and what you have in your own repository, and mail you the ones that you are not using. I find lwn.net's security pages useful in keeping track of what security updates matter to us.
Puppet is a tool made to do exactly what you're asking for by abstracting the specific operating system into a generic interface. It might be worth checking out. Also there's a newcomer called chef. And then there's the oldies like cfengine.
Sorry to reply to my own post, but circlingthesun actually posted the name of it below!
clusterssh
Set up key based login and you don't even have to type passwords.
Since you basically need root access to do updates this definitely poses a security hazard as when your client is compromised there is direct access to the server. Then again, an attacker could always use a keylogger to capture the password anyways.
If you even attempt to do this I'd setup a different user account specifically for the process of updating and limit the rights accordingly and then I'd restrict the commands that can be executed (you can do this per key).
There may actually be better ways but I'm not a very experienced sysadmin. Most experience I have is from managing a single web server and my local desktop obviously. Be sure to correct me (in a friendly manner) if I'm wrong.
Then again, if you do this from the same machine as your normal account is located on you'll still have the same issues in case of a compromised client. Probably just best to limit every single account to just that what is specifically needed and setup proper host based intrusion detection (OSSEC?) to be notified when something goes wrong. This stuff is hard...
Perfect is the enemy of done.
that won't quite work - most likely, submitter does not want a particular list of packages to never update, but instead wants to evaluate individual patches, so decision is based on the exact patch, not made for all possibeel patchs to aa prticular package.
Rich
It's called "dssh". Google is your "search" friend (we will ignore the evil side of Google at the moment... :-)
You could also use nagios and check_apt/check_yum to alert you of out of necessary security updates, put a script for installing updates on every box (different script for centos/ubuntu, but same syntax), create a user who is added to sudoers for only that script, and create an ssh key for authentication...
Then you can feed the list of hosts that need updating into a script which will ssh to each one in sequence and execute the update script followed if necessary by a reboot..
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
If your internet connection goes down, where will you get updates from?
Congrats, you just volunteered to mail him the floppies.
I think that you are not putting your efforts where it matters.
What is important is that the critical services run properly on each server. Sure that can be affected by patching but also by many other factors. So don't focus solely on the patching, focus instead on making sure all the services are running properly.
You should have your own scripts that check that each server is responding as required. Make your test suite as strong as you can and improve it each time a new problem crops up that wasn't caught by your spying tools.
Once you have this in place, you can safely do daily automatic updates and stop second guessing the package maintainers. You will have a more reliable system and you will save yourself a lot of work too over the years.
Never use ssh+password. Always use ssh+PKI. I think you missed "key" and focused on "[no] password"
From Ubuntu wiki SSHHowto:
If your SSH server is visible over the Internet, you should use public key authentication instead of passwords if at all possible.
That's your job. Bash CLI, the CLI toolkit, CLI Emacs, key-based ssh and a well-maintained, well-documented pack of own scripts in your favourite interpreted PL are just what it takes to do this sort of thing. No fancy bling-bling required or wanted. It would make your worse, not easyer in the long run.
We suffer more in our imagination than in reality. - Seneca
"I administer several Ubuntu desktops and numerous CentOS servers. One of the biggest headaches is keeping them up-to-date with each distro's latest bugfix and security patches"
My advice is, if it ain't broke don't fix it, especially on a production server. Have two identical systems and test the latest bugfix on that, before you roll it out to the live system. You don't know what someone elses bugfix is going to break and would have no way of rolling it back.
IMHO, a good sysadmin is the one who sees such issues and finds or writes a script to do it for them.
(That's "you're both right")
for address in addresses; do /var/lib/triggers/update'
ssh $address 'touch
done
With an obvious job on the machines watching for such trigger files (to avoid root access, etc.)
if [ -f $TRIGGERFILE ]; then /tmp/yum_trigger.log \
yum -y update 2>&1| tee
&& rm -f $TRIGGERFILE
fi
- Michael T. Babcock (Yes, I blog)
or "clusterssh"
http://www.debian-administration.org/article/Use_ssh_on_multiple_servers_at_one_time
Using keyfiles doesn't mean no password. Most ssh guides that talk about keyfiles actually suggest to protect them with a password. Why? Well, like the GP said (what you obviously somehow missed), as soon as one client is somehow compromised you would have root access to every server.
Using passwords doesn't mean you don't use keyfiles. And using keyfiles doesn't mean you don't have to enter passwords. The way to go is ssh+PKI+password+ssh_agent.
The internet. /var/log/messages to find the issue then google to see who else has the same or similar issue. Or troll a few web hosting forums. Seriously, you need to take more interest in what's happening to your machines. My servers send me status reports by email every morning regarding patches, rootkit detection, unauthorised accesses, tripwire incursions etc etc. I have a server which has had minimal patching since 2002 and still runs fine with no rootkits. I have had no unavoidable downtime since 2002. Yes, I run Redhat Enterprise.
Actually,
This is what disturbs me about the push for linux on the desktop. Most people can't be bothered to research and implement simple monitoring measures, then complain that everything's gone tits up. The whole point of linux is - YOU are in control, ultimate control. Use the power wisely or suffer the consequences. Don't go blaming the distro, the maintainers, the world for your inability to control something designed to react to your hand and your hand alone. A poor workman blames his tools, and GNU/linux is merely a tool.
If you have root, fucking act like it !
Exactly, the confusion here might be in the terminology. Password versus passhrase.
Anyways, just using keys doesn't magically make everything more secure, it just negates brute force password attacks. From the few high profile cases I remember the compromise was the result of somebody's private key being compromised (e.g. the Debian compromises).
The only true solution is a combination of the principle of least privilege, sandboxing (SELinux etc.), proper monitoring and a whole host of other security measures.
Perfect is the enemy of done.
You must not handle any credit cards, financial data, or health records then. Some of us are forced to keep servers updated to within a couple weeks of the latest patches or risk a dreaded "uncompliance" status.
Personally, I wouldn't want any of my data trusted to an infrastructure that is only updated once every three years, but in some places that approach makes sense as well. We've certainly taken that approach with SLES servers since Novell's updates are usually more malicious that whatever hole they are patching.
Actually, I remember reading a year or so ago about a program that would allow you to run a specified command via ssh on a list of machines. You could do this with a shell-script (pass arguments), but I think the program also did it all in parallel and showed some output as well.
I think many rootkits do the same thing. You can run a command (via irc) on a list of machines and return output to the channel (in parallel). The best update and control solution is to rootkit your own boxes and maintain them via IRC. Problem solved.
The clash of honour calls, to stand when others fall.