Slashdot Mirror

I managed a deployment of roughly 800 Macs across the campus of a large university using Radmind. I've also managed the campus Linux, Solaris and OpenBSD kerberos servers, web servers and file servers with the same software. Radmind's learning curve is a little steeper at first, but it's one of the most flexible deployment options out there once you get the hang of it.

Radmind's not really a competitor with tools like NetRestore. When used correctly, NetRestore is great for total reimaging of deployed hardware: nothing beats a block-copy installation for speed. Where NetRestore falls down is when dealing with deployment entropy. After imaging, the machine is in an unknown state ("post-image"), and the only way to be sure all machines are in the same state is to blow away the entire disk and reimage, usually at a cost of gigabytes of bandwidth per machine.

This is where Radmind excels. It's basically a tripwire with software deployment and roll back, all based on the differences between what should be installed and what's actually on the disk. The core utility, fsdiff, looks at all files and directories designated as managed by the administrator and generates a list of differences. You can capture those changes as a loadset and upload them to the Radmind server for deployment to other machines, or you can undo any changes detected by fsdiff and restore the client to a known good state.

The great thing about this method of management is that there's minimal bandwidth used. If fsdiff detects no changes on the filesystem, there's no reason to download anything: your system is in a known good state. On the other hand, it makes deploying Apple's system and security updates pretty damn easy. Grab the updater from Apple's website, install, and run the Radmind tools to capture the changes. Store the changes on the server, add the new loadset to your machines' profile (command file), and let your clients pull down the changes.

The Radmind community is very helpful. Most questions to the mailing list (hosted on SF.net, Google groups mirror here) are answered very quickly, and people are eager to share details about local setups and scripting solutions. A typical setup for a Radmind-managed Mac OS X client usually involves a few possible methods for initiating updates, most of which involve iHook as the UI:

1. Check for updates on Radmind server during logout, update client if found.
2. Run a nightly tripwire regardless of updates from server.
3. Run a Radmind update during boot if a special flag file is found on the disk.

Since we relied on students to help run our labs, we also deployed a special, unprivileged local user account, whom the students could log in as. This also triggered a Radmind update. And of course you can trigger updates over ssh (which works well in combination with something like pdsh).

We combined Radmind with NetBoot for rapid, consistent deployments. Once the hardware was in place and on the network, we netbooted, used ASR to install a minimum and relatively recent system, and let Radmind bring everything up to date, including per-host license files and location specific software.

Radmind's not perfect. It manages at the file level. If you want something to manage, say, config files on a line-by-line basis, Radmind isn't going to fit the bill (yet). Generally speaking, though, Radmind manages Mac OS X with ease. Once you've got Radmind managing your Macs, you'll find you have a lot of extra time to do interesting things instead of troubleshooting problems brought on by stale deployments.

The Radmind wiki is a decent place to start looking. Good luck.

Mod parent UP. The OP is thinking about it wrong: ie how to manage unix in the style of windows. Don't give them root and they can't install software. Make sure ... there is NO WAY that they can run programs which aren't already installed. ... Remove all executables that they shouldn't run ...

You can create an RPM to do this for you, then set up the whole thing automagically using Redhat's or SUSE's tools (one is called kickstart). ...

Alternatively, you can bodge it with shell scripts and a cron job :-)

You can screw it up with RPM just as well. If you want to Do It Right, get a filesystems management tool like Radmind or cfengine, use your package manager du jour to make a template for all that good stuff, and use the filesystem management tool to capture, deploy, and manage the template.

Don't confuse package management for filesystems management. They overlap in areas, and so you can cheat by using one tool for the other purpose when nobody is looking, but that's a short-term optimization that will lead to long-term pain.

Radmind: http://radmind.org/. Radmind's is designed for this purpose exactly. It's a tripwire with the ability to roll back changes, or capture them and store them for deployment to other systems.

That's exactly what Radmind does:

http://radmind.org/

Radmind leverages tripwire-like information to provide very large scale configuration management. It supports certificate-based authentication of servers and managed machines. The latest release supports compress in the network layer for cases where CPU is more plentiful than network bandwidth. :w

... and you'll have something that looks a lot like Radmind.

Recovering from this sort of sabotage, and from the setuid root /bin/sh trick described in another reply to the parent, is easy when you're managing the system with radmind.

radmind detects changes to files (size, modtime and optionally checksum), owner, group and mode. The administrator then has the option to reverse those changes, or capture them and apply them elsewhere.

Radmind is a better tool if you want to update just one server and then have that update applied to a number of other machines.

A rooted box with a kernel module installed to hide itself, has to be completely restored.

You can repair such a root kitted box by 1) booting from clean media, and 2) running a tool like radmind that knows how the system is supposed to be and can fix it.

:w

Sure, this might be a secure OS, and you might be using "Systems Management," but unless you are using something like radmind to fully tripwire your machine, you really don't know what's there.

Are you going to take him at face value and continue using the system as is, after patching the security hole that let him in?

Am I a competent sysadmin in this scenario? If "yes," then I guess I'm probably running a tripwire of some sort. So I boot from CD, take a look at what's been changed, and fix it. If I'm really on the ball, I'm using something like radmind, in which case I still boot from CD, but I let radmind reverse any damage that had been done.

:w

I agree. For many years radmind was the only robust solution for OS X. It is by far the most widely used for this task. Though radmind is slow. It will take 2-8 hours to update your computers from scratch. However, why does it matter? Plus radmind will allow you to push incremental upgrades in as little as two minutes. It's somewhat difficult to learn, but the radmind user list is really really great. Most of the people on the list run big university labs or corporate labs and I find them to be really smart and creative. Check it out: http://www.radmind.org/

Thoug I have to say sometimes radmind sucks, like if you go from Jaguar to Panther it can break. Though generally for minor system updates and security fixes it's okay. This is why you TEST! And if you need a full restore you use apple software restore or netrestore from Mike Bombich. I like that guy: http://www.bombich.com/ But then you need an Mac OS X Server http://www.apple.com/server/macosx/ as I recall, in which case, you might as well by an xserve http://www.apple.com/xserve/ since it comes with the software. But again you will only need Bombich once a year; so you can just visit every machine with a cd ad it might be as effective as all the ASR which I found to be difficult to implement. We had to get an Apple Engineer to set it up for us. heh.

You mean the computers all update themselves automatically from a central server, or is it something else?

Check out radmind. It's sort of an imaging and tripwire tool all rolled into one. Runs on Linux, Solaris, *BSD, and Mac OS X.

:w

The Trouble with Tripwire is that you can't tell whether the reports reflect work done by sysadmins or hackers. Radmind is the cure for this issue. Not only does it differentiate between patches & hackers, but you can use it to actually deploy the patches (or other software).

:w

From the redo_prebinding man page: "Redo_prebinding is used to redo the prebinding of an executable or dynamic library"

Shouldn't the prebinding on programs only change at major OS updates or changes to the libraries they access?

Yes, but I'd even go further to say that the OS should never change a system file without my intervention.

Shouldn't Application Packages already be excluded from backups for the most part anyway?

Sure, but when you are talking about a large scale deployment, you can't always control what end users backup.

More importantly to me, prebinding breaks tripwire. When a file is rebound, it triggers a tripwire event, adding noise to my reports. With so much noise, it's hard to tell what's important what's not. Tools like radmind from U of Mich alleviates this problem somewhat by integrating tripwire and system updates. But unless it, and other tools that use checksums are taught the horrors of prebinding, they won't work right.

If you're responsible for the machines you run how can you abdicate that responsibility by using whatever some package maintainer decides to give you? At the University of Michigan we use Linux from Scratch to manage hundreds of machines that provide everything from web servers to IMAP servers to user Desktops & Laptops. The trick is leveraging the work used to administer one machine well out to hundreds of machines. The tool for this is radmind. Radmind doesn't require that you build your software from source, but it leverages the work you put into one machine to manage all of your machines. It also integrates a tripwire with your management software which means you can detect unwanted filesystem changes in addition to managing software.

Rather than claim that closed source products don't incur these costs, I'd say the are invariant.

1. support

Have you used the (offshore) support that comes with shrink-wrapped software? Give me a break. In addition to the mailing list that every software package comes with, the mailing list that the authors reads, if you're interested in paying someone for support, try IBM.

4. documentation

O'Reilly?

2. installation
3. deployment
5. deploying updates

These three can be done on a massive scale with radmind, a piece of open source software.

:w

• Workgroup Manager lets me set user preferences and system policies to control almost every aspect of the client computers;
• Apple Remote Desktop lets me manage all of my machines remotely, even from home over our VPN;
• Carbon Copy Cloner, NetRestore, and NetBoot make imaging & deploying workstations a breeze;
• and Radmind handles disk maintenance and file distribution in the lab.

I've been using Red Hat 8 in a lab setting with 16 workstations and 1 server for over a year now, with no complaints ... well, no BIG ones.

However, the University I work for is preparing to have a meeting for which version of Linux to standardize on and get support for... Red Hat (I'm assuming Enterprise), SuSe, or Fedora.

Back when I did IT support for a large university, we had a problem of disting Windows images to each machine (for those who are not familiar, it's to synchronize each machine with a master image upon logout). It was easy on the Mac, but the best we could do in the Windows world was to use PC-RDist, a piece of software written probably by high school kids in their parents' garage. It did not handle Microsoft software very well. Even with a fully-functional image set up, we had to manually go to each machine and install the MS software (WindowsUpdate patches, Office, etc.) BEFORE we can download the updates from the master image. Plus, any updates to the registry would not be copied because of Windows Protection. Eventually, we just gathered up enough funding (it was hard) to get disk imaging software whenever we needed to hose down a machine and start from scratch. I'm glad I don't have to work in IT anymore. :-)

As the system administrator for the project, that best part is I can roll back any changes. Say, if apple were to release a bad update, I could just remove the overload and everyone would be back at say, a working 10.2.7.

Let's see you do that with windows.

You should check out Radmind for Mac OS X. It won second place for Best Server Solution at Apple's Design Awards. While the underlying technology works for Linux (and Solaris), the wiz-bang GUI is only available for the Mac.

:w

As far as how we manage our clusters, the answer is the same for Solaris, Linux, and for the matter Mac OS X: radmind. Check it out. It integrates tripwire and filesystem management. We use it for installation, patching, and updating.

:w

What I really would like to know is how they install and configure all those machines. Their method of doing that will be very useful for even the (relatively) smaller networks that don't necessarily have to be clusters.

I really hope they describe how they maintain the operating system on them.

ssh -l root remotehost "cd /; tar -X /etc/nobackup cf - *" | bzip2 -9 >remotehost.tar.bz2

You might try radmind. It's used pretty popularly in the Mac OS X world, but was originally written for Solaris, Linux, and *BSD. There's a reasonably sized community using it, and a supportive mailing list.

:w

SystemImager is one of the most useful tools I've ever seen, however, I believe that it would be an enterprise "killer app" if it could do MacOS X, *BSD, Windows etc.

You could roll your own, ala Gentoo or LFS.

This is great, because making an OSX boot disk can be a pain in the arse. I could use this to run a program like Radmind to image a mac from a CD. With Unix(tm) tools able to run cross-platform, I can use Linux as a repair cd.

Very happy.

Try running 1000+ Linux boxes with hundreds of different workloads and configurations

Use NetRestore and NetBoot on your OS X server for rollout, then maintain them with Radmind. NetRestore is much like Apple Software Restore, but better, and Radmind is a replacement for RevRDist or Assimilator, but again, much improved. I've used them all and managing OS X this way is so easy is ridiculous.

Another good resource is Mac OS X Labs. Full of good information about this sort of stuff.

YMMV. Good luck.

Three days after release of a patch or other software update, our entire 20,000+ client network is 85% or more patched. With about 20 man hours of work across three staff. Linux absolutely can not touch that.

For our desktops, it's more like 45 minutes to capture and test the change, and you're done. 20 hour? Oh man!

:w

Here are some open source projects that I've been involved with. Fugu is a graphical wrapper for SFTP & SCP on Mac OS X. I consider it successful because many universities (my peer group) recommend it to their users. Also, one of our success "feathers" is the number of localizations that people have contributed: Spanish, Japanese, German, and Dutch.

radmind is a combination filesystem integrity checker (tripwire) and manager in one package. Again, many (a couple hundred) universities use it. It's important to note that it's less important to me that other groups also use it. It's my peer group that interests me.

I'm also the original author of netatalk. I consider it a success for a couple of reasons. First, it's old. Second, I no longer work on it at all, but there's an active group that continues to make releases. Those are both success "feathers".

Finally, my group wrote the reference implementation of LDAP, and our software is the basis of openldap. The "feather" in this case is having been part of the group defining the LDAP standard, something that many vendors and many packages now use.

:w

Which goes another step further, and stores not only the checksums, but also copies of the file data centrally, so you can undo changes that have been made. OR you can change the data on the central server, and effectively push out updates to hundreds of machines. That's RadMind

:wes

Carbon Copy Cloner is pretty good for getting OS X onto a machine initially, but would be a pain for regular maintenance. I actually use ASR for initial install (macosxlabs.org talks about it here).

I use radmind for regular maintenance of the machines in the the labs I run. It's a powerful unixy tool, a little tricky to get the hang of but it's well worth the effort.

I have to configure hundreds of desktops in Linux, and many of the time the configurations are the same.

:w