Remote, Automated Configuration of Unix Boxen?

drift factor asks: "I work for a small company with about 200 Linux desktops, all identical aside from hostname and IP information running Redhat from a kickstart install. Currently, when something needs to be changed on all of them (say, the default gateway) we either have to ssh into all of them and do it manually, or write a Perl script to do so for us. Is there a good remote configuration system that will also allow some level of automation, i.e., I tell it what IP I want the default gateway to be on each machine and it updates them for me?" If you manage a large server farm and have used or written utilities that do this, please share your experiences.

Comments

DHCP is obvious

[bluGill]

For your network dhcp is the obvious solution. (Congradulations to the first post comment that was on topic an correct. lacking details, but that is a different story)

For a more general approach, NIS comes to mind, though it wasn't designed to be secure. NIS+? kerboses? Your not the only one with the problem you state, in fact you hardly have a program compared to many installations. I've seen systems with 300 Solaris/aparc machines, 200 sunOS 4.x (much nicer then solaris IMHO), 75 IRIX, 200 Linux, and probably some HPUX and AIX scattered in there two. This at a university for student accessable accounts in CS or other engineering areas. Your 200 machines is nothing compared to the 20,000 users they managed. And most would not consider that close to large.

My point is that others have seen your same problem, and worked on solutions.

Almost the same idea

[tolldog]

I use SystemImager to manage all of my machines.
I build the image on a box and the propogate the changes via a rsh/perl script that tells the clients to update.

For large system updates and configuration, it isn't that bad of way to go.

Depends on the complexity of the task

[Tet]

For changing a nameserver, for example, you could just rdist or rsync /etc/resolv.conf to all the machines. However, it gets more complex when you want to modify a file rather than overwrite it. To change the gateway, as in your example, you'll want to modify the GATEWAY parameter in /etc/sysconfig/network, while leaving the HOSTNAME parameter untouched. The only practical way to do this is to write a script, but then you're doing that already. Writing scripts is The Unix Way (tm) :-)

Warped thought - cron and CVS.

[Christopher Thomas]

Here's a warped thought - set up a CVS server with all config files and applications that you care about updating checked in.

Set up a cron job on each machine to check out the latest version of the installation every day at 5am. Make the cron job shut down and reinitialize anything important, too (or just have it reboot the machine and let init/shutdown scripts take care of it).

This isn't a remote admin solution, but it _does_ let you easily make sure that all packages and config files on the machines are in synch. Upgrade Netscape on the master server, for instance, and the other installations migrate over in a few hours.

You could even rig it so that the cron job calls a specific script before completion, and check that script out of CVS for any specific shell-ish things that have to be done for maintenance to complete the update (test these scripts carefully, though).

cfengine works pretty good for me.

[Sir Spank-o-tron]

We use cfengine and rsync.
we have a master server that get's config files from CVS, and pushes out to rsync servers.

every host calls rsyncs once an hour (or as often as you can get cron to do it) from the rsync servers, and EVERY hosts config is in cfengine, even the rsync servers and cvs servers.

we make heavy use of CNAMEs so that if a machine disappears, we can magically recreate it and none of the workhorse hosts know the difference.

go check out infrastructures.org for some good reading.

Expect can be your best friend in automation.

[meldroc]

I work in a testing lab where we have to run large numbers of regression tests that would be incredibly time consuming if we didn't use automation tools.

I've been using Expect for test automation on the latest series of tests. Expect is an automation scripting tool that runs on top of Tcl. It is very useful for automating command-line based applications using "send &ltstring> expect &ltstring&gt" sequences.

As far as licensing is concerned, Expect was developed at the NIST, and has been released into the public domain. That's about as free as you can get.

Re:Yes

[Zurk]

i use scp to copy the rpm files over, use ssh to login to the box with a script and install files by running rpm -Uvh etc etc by piping the stuff in thru the script. its pretty secure that way and works nicely. also easy to have a set of commands for doing various simple things and letting the script pipe those with a command option. i.e. script install whatever.rpm installs the whatever.rpm file, script cp file1 file2 copies the file etc etc.
I also use procmail to do some grunt work (weird huh?)...for example if i want to clean the print q of server 26 i can mail root@server26 with the header as COMMAND: DELETE PRINTQ and a procmail script will see the COMMAND: in the header and do a simple lprm -Plp -...ok..im half asleep as i type this..but i hope im making some sense. im a uni admin BTW. i've got machine with weird configs to look after so it might not suit your problem...none of my machines are the same.

cfengine!!!!

[The Madpostal Worker]

cfengine is meant for this sort of thing, with it you can easily mirror config files accros the network, have it tidy files, and do trip wire like things. Its a great product, and its free (in both senses). Basically, you tell it how to do tasks in a class sense, this is how you upgrade perl on a solaris box, this is how you do it on a linux box. As a result, you can share code between classes(on a solaris or linux box, do this). Its a *great* system.

/*
*Not a Sermon, Just a Thought
*/

Powerful tools...

[Ryan Kirkpatrick]

There are two tools that I know of that might be of help. They are:

• tut: "Tell Unix To..." by Jim Barbour. A command that will run a non-interactive shell command on a user-defined class of machines. (No URL handy, sorry).
• cfengine: A generalized, powerful, shell like language for dealing with large numbers of machines. (http://www.iu.hioslo.no/cfengine/)

I have never used either, but have read some documentation on cfengine. cfengine looks like it would be more of an automative tool than tut, while tut might be handy for quick and dirty updates. Anyway, my two cents.
------------------------------------------ ----------------------------------