Slashdot Mirror

... And tools like RPM, dpkg, as well as scripting systems like cfengine provide ways of readily deploying upgrades and of otherwise maintaining "system cleanliness." After all, you don't want to have the box go down when a log file fills up, and then need to ship out new hard drives to fix things if they do...

• Stupid Users are Bad.
• Stupid Users are Bad for Debian. therefore:
• Stupid Users should be ignored.

With that less-than-hugely-friendly beginning, it moves on to real points that are not nearly as unfriendly as that initial thesis. (Which I think is there to shock the gentle reader...)

The opinions do not remain any less strong; try out:

Eventually these maintainers, although they know they provide a powerful, complex tool, would believe that they need to convince lure more users over and gain the approval of those who are too myopic of lazy to RTFM. These great maintainers would confer and agree that perhaps the users are right and it is time to develop an easier way to install. This is the point where I say "foo!"

The author then proceeds to suggest that "keeping Debian powerful" is Job #1, whilst "making Debian more friendly" is a task that others can worry about.

I tend to agree with this; the proper priority seems to me to be to keep building a powerful and useful (and evermore vastly increasing) loosely-but-sufficiently-tightly integrated set of packages.

Making installation easy isn't forcibly part of that.

I agree with the suggestion that it might be a good idea to have an unambiguously BETTER scheme that could include queueing up installation configuration information, so that you essentially configure as much as possible before getting to the Commit This Configuration To Your System Now point, at which point disks get partitioned, filesystems get mounted, packages get dropped into place, and the likes, rather than configuring it piecemeal, one step committed after another.

It's probably always fair to say that it's a Good Thing to have a partitioning of package sets (ala "Network Server" versus "X-Developer" versus "Desktop System" versus "Web Server" ...) that perfectly agrees with whatever you, the user, wanted; as preferences are in the eye of the beholder, it's nearly impossible to get this perfect.

I'd LOVE to see a lot of the "other-than-packages-and-partitioning" configuration deployed via the install tools creating scripts that do the configuration, and which could be used to redo the configuration... My personal prejudices involve cfengine; I've set up enough cfengine scripts that any time I install Linux on a system I plan to hook up at home, the first thing I do after getting the system running is to install cfengine, do an NFS mount of my "standard" scripts, and then execute them to set up my favorite maze of NFS mounts, shell configuration, and network services. For a distribution to do this sort of thing would be pretty cool.

• tut: "Tell Unix To..." by Jim Barbour. A command that will run a non-interactive shell command on a user-defined class of machines. (No URL handy, sorry).
• cfengine: A generalized, powerful, shell like language for dealing with large numbers of machines. (http://www.iu.hioslo.no/cfengine/)

Ok, so giving the default gateway as an example was a bad idea, but you miss the bigger picture: I need to be able to change a lot more than just network configurations on all the machines. This could a pam configuration file, installing a new package, virtually anything. So far, the thing that looks the best to me is cfengine.

In good "MVC" manner, things would be quite clearly separated out between "front end" versus "server." (That oversimplifies MVC; so sue me...)

With cfengine, the "back end" is a configuration management engine that does things like renaming files, mounting partitions, running network config utilities, and such. The thing that is nice about it over, say, Perl, is that it directly contains abstractions made for doing "config stuff." It's not a hive of Turing-complete bits of code that could be doing anything; if you want to shut off certain services, you might use:

editfiles:
{ /etc/inetd.conf
HashCommentLinesContaining "telnet"
HashCommentLinesContaining "finger"
HashCommentLinesContaining "tftp"
}

Then, when you run the GUIed tool, it doesn't simply run some Perl code that does the system configuration for you, this time; it instead generates a cfengine script, which may assortedly be:

• Run immediately, to have the desired effect;
• Logged, so that a "system audit" has something to work with;
• Read by the gentle user, as a route to Greater Understanding;
• Added into the regular system configuration regime, so that every once in a while, the system makes sure that the change you put in place continues to be there.

  I've used this property of cfengine to help in configuring new systems. If I have cfengine scripts set up that properly configure the local network topology, it's rather slick to drop them on a floppy, put that in the new box, run them, and suddenly have that new box linked up nicely, complete with NFS mounts and security fixes.

  It's possible that cfengine isn't the perfect answer; the point is that by providing a "language for system configuration," it is a whopping lot more suited to large scale use for this task than, say, some set of Perl scripts that someone hacked up.

I've seen people above advocating centralizing your user data, and making the boxes all cookie-cutter installs. Excellent advice.

Once you have them up and running, however the question becomes. "How do I make changes to the environment en-masse?".

Thats where GNU cfEngine comes in. It's a great tool for maintaining heterogeneous networks. You should consider implementing this on the rollout, as it will allow you some means to "push out" changes to all of the hosts.

Check out:

http://www.iu.hioslo.no/cfengine/

Its a very powerful tool, so much forethought and planning is in order with it, but it pays off in the long run in being able to make changes to the machines in large chunks.

-- PoochieReds

I've seen people above advocating centralizing your user data, and making the boxes all cookie-cutter installs. Excellent advice.

Once you have them up and running, however the question becomes. "How do I make changes to the environment en-masse?".

Thats where GNU cfEngine comes in. It's a great tool for maintaining heterogeneous networks. You should consider implementing this on the rollout, as it will allow you some means to "push out" changes to all of the hosts.

Check out:

http://www.iu.hioslo.no/cfengine/

Its a very powerful tool, so much forethought and planning is in order with it, but it pays off in the long run in being able to make changes to the machines in large chunks.

-- PoochieReds

Or even better, use a program like cfengine (http://www.iu.hioslo.no/cfengine/) and automate all your sysadmin tasks. We use a similar system at work written by our developers, and maintaining our 150+ servers spread out in 3 countries is *easy*. Need to upgrade Apache? Do it in one place, and it'll be distributed out during the night or on demand. Need to apply a patch of some kind? Yet again, it's done in one place and pushed out to the servers.

There's no end to the possibilities!

For instance, try out cfengine for a way to handle some of your distributed administration pains. As networks grow ad hoc administration becomes more and more difficult. If all you need is to keep your software in sync maybe you should try out Debian with dselect, apt-get, etc. If you need more rdist/rsync or even cfengine will give you more power. Perhaps you will end up at the point where you need to use Kerberos, NIS+, LDAP, etc. There is a long continuum between using a standalone machine and administering a large heterogenous network of systems. Sometimes it's hard to know when to apply new administration techniques -- and often even harder to know which techniques and tools to use (each time you go through the process you get more "experience" and become more qualified to actually administer systems). It sounds like you are at the point where you need to reevaluate your needs and decide on a more powerful scheme to administer your systems.

Just be glad you've got a Unix workalike at your disposal for administration purposes.

• There's the UNIX Way

  Where Real Men use vi to manually edit all the text files.

• There's the Windows Registry Way

  Where Modern Administrators point'n'drool their way through menus, searching for the configuration.

These are the commonly-perceived stereotypes, and, too often, represent peoples' attitudes, whether pro or con.

Unfortunately, by focusing on this particular dichotomy, people miss the mark, in that neither approach is particularly manageable. Moreover, by thinking they are the only alternatives, people miss the directions for true improvement.

• There's a place for having a centralized "registry" of access methods to access configuration information.

  It may be true that:

  "Lumping configuration data, security data, kernel tuning parameters, etc. into one monstrous fragile binary data structure is really dumb." -- David F. Skoll

  That doesn't mean that there wouldn't be value to building up a hierarchical "tree" that knows how to look up configuration information. The data can sit where it is now; the "tree" is useful for providng the administrator with a comprehensive way of getting at it.

• Automation means You don't have to touch it again.

  Configuration work that the system does for you is the true labour savings; if the system cleans up after itself, and I don't have to do it, that is an automated system.

  I'm glad to drop in an extra cfengine rule into /etc/cfengine and have the system do more work for me.

cfengine is a sort of generic "configuration control" languge. You define things like lines that should be in /etc/hosts , or things that should be mounted, or files that ought to be kept up to date with central repositories, rotating system logs, fixing file permissions, and other similar sorts of things.

A daily/hourly run will go through and "clean up" whatever isn't set according to the instructions.

There's a client/server variation called cfd that allows you to push configuration across a network on demand.

The big point to this is that it treats system set up more like "immunology" than anything else. From a security perspective, this is very good. You get some security rules set up, run them regularly, and fix/prevent holes.

Perhaps more useful, once you set up some common rules for a site, installing a new system becomes real easy: you just need to get as far as installing cfengine, get some config files over, whether via floppy, CD, or NFS mount, and then type # cd /etc/cfengine; cfengine and depending on the sophistication of the rules, you may never need to log on as root again.

For instance, you might set up a location where machines mount a filesystem containing package upgrades or configuration file upgrads, and automagically install them.

The regrettable thing is that cfengine doesn't have the "barneyfication" that naive users may want/need.

On the other hand, it has the major virtue over things like Linuxconf that it is a tool for building configuration systems rather than being a front end that is tightly connected to the back end.

I could see:

• Building a GUI tool that manipulates some combination of templates and data. That allows providing a relatively friendly front end for what could be a bit offputting.
• Using cfengine as the tool for pushing package configuration into place on one's system.

  Thus, rather than merely doing a "cp foo /etc/foo; chmod 774 /etc/foo", the configuration process might include asking the user for input of critical bits of configuration, and constructing a cfengine script that might even be usable to "clean up" if you've done something icky and want to fix the package.

  This would also make it natural to create a little script for a given package that might do security checks, perhaps automagically turning off dangerous options or the like.

What I'm not sure of is the economic merits of Corel Linux.

Note: I installed it last night on my laptop to replace a SuSE install. That went quite well; it took not much more than a cfengine run combined with dropping a previously-tuned XFree86Config file into place to get it acceptably configured, which was a whole lot more satisfactory than an attempt over Christmas holidays to install Debian on it.

(Aside: This laptop has had TurboLinux, SuSE, Debian 2.1, Red Hat, and now Corel Linux installed on it. With the happy merit that I have more-or-less generalized the set of stuff I need to fiddle with after install. Reinstalling means installing a base system + cfengine and then running a cfengine script to get networking fixed up. I probably ought to see if this all copes well with FreeBSD, NetBSD, and OpenBSD too, as I have CDs handy...)

Based on the "Day 1" results, I'm reasonably pleased with Corel Linux, as this was the least painlful install. (Well, grumble, grumble, Corel's package selection tool required a whole lot of mousing around, and having sprained a wrist the night before, the word "painless" may only be treated as true in a conceptual sense...)

You might expect that to bode well for the economics, but that is a questionable assumption.

• I didn't pay Corel anything for the install, as this was a $2 CD from LinuxCentral.

  I'd be game to send Corel a little something; I expect that sending them $10 would be a better deal for them than spending $40 on a boxed set...

  (More likely is the option of buying some shares in Corel... One of the few entertaining things I could do with the cash sitting in my SD-RRSP account when I was forced to sell off some telecom stocks, gripe, gripe, fascist CRTC...)

• I then proceeded to NFS mount a cache with chunks of Debian/Unstable to upgrade it. Mostly complete, and almost a seamless upgrade.

  Which implies that if the Debian Project does a good job of upgrading the "public" stuff, there will be little reason for there to be continuing revenue streams for Corel. Unlike the situation where people really do need to get upgraded CDs for RHAT or Caldera or SuSE.

Something like cfengine would be usable to this end; make install should generate a cfengine script that validates the system configuration, with the option of either warning of problems or of fixing them.

If not cfengine, then something else may be usable.

The critical point here is for the tool used to not merely be "a shell script," as those may get diverse in style to the point of unreadability. The validation needs to be in more of a descriptive style so that it doesn't get unreadable.

Something like cfengine would be usable to this end; make install should generate a cfengine script that validates the system configuration, with the option of either warning of problems or of fixing them.

If not cfengine, then something else may be usable.

The critical point here is for the tool used to not merely be "a shell script," as those may get diverse in style to the point of unreadability. The validation needs to be in more of a descriptive style so that it doesn't get unreadable.

I took a look at this last week with intent to turn a "partly RHAT 5.0, 5.1, and other stuff" box into a Debian system.

Unfortunately, the process is fairly dependent on some particular packages (notably libg++ ) and it looks like it is a distinctly nontrivial process to make this work in practice.

The second problem that you'll hit is that you'll wind up with a whole lot of "cruft," library-like stuff hiding here and there larding up your filesystem.

My inclination would be, instead, to:

• Do a backup of /etc
• Make sure that /home and /usr/local are on suitable partitions such that they will survive the transition unscathed.
• Nuke out /usr , /var , and / by doing an installation from scratch of Debian. Do not format or automagically mount /home or /usr/local ; that can wait 'til after the system is semi-working.
• I took a look at the portions of networking config that I wanted to keep working from old OS to new OS; I set up cfengine rules that I tested on the old RHAT install before installing fresh to manage this.

At this point, the box happens to be running RHAT 6.1; there is a good likelihood that I'll set up some even "smarter" cfengine rules than I have now and run the box through a few distributions just for the sake of regaining familiarity before letting it settle down with Debian.

There's not a precise equivalent to Kickstart; what I would do, if I wanted 13 identical boxes, would be:

• Pick one box as a "master," and install everything that you want installed on it.

  I would mount my Debian CD on a separate box and download via HTTP; this has the result of pushing all the packages that got installed into /var/cache/apt/archives

• Then, add /var/cache/apt/archives to /etc/exports so that it gets exported to other machines that want it.
• Install the "base" Debian stuff (about 6 floppies worth of stuff) on those other machines.
• Copy over some base networking files ( e.g. - /etc/hosts , /etc/fstab and such) and drop them into place so that each machine has some basic common configuration.

  I would tend to want to use cfengine for this; I have yet to get it configured to distribute files itself, which is something it claims to be able to do...

  Inserting extra needed lines in config files like /etc/fstab is the sort of thing that cfengine is ideal for...

• Mount the relevant filesystems on the remote machines, and then have them install via: # dpkg -i /mnt/common-apt-cache/*.deb

  This gets all the machines to have the "common" stuff.

• It would make a lot of sense to either build a .deb package containing any common config scripts/files or provide a common NFS-mounted "export" directory like the "apt package archive" as a conduit to push stuff to the remote boxes.
• Set up a cron job on each box that (let's say) runs a script in /mnt/common-config/ that installs any .deb files in /mnt/common-apt-cache/ that are less than a day old (or check against what's installed).

  I would most definitely try to implement this using cfengine as it's designed to do this sort of thing...

Look at cfengine; the Usenix journal ;login has had a series on it recently; it is really powerful.

• Java

  There seems to have been something of a "trainwreck" with respect to Java. There are lots of "nearly done" Java environments out there, including Kaffe, GCJ, Jikes, "Blackdown," and likely others.

  Unfortunately, none are truly useful without some combination of classes (ala GNU Classpath) and some combination of AWT/Swing. And that has been rather less rapidly forthcoming in the "reasonably free form" that is necessary in order for it to be ubiquitous enough for people to really use it to deploy applications, or to use it as a layer on which to build further infrastructure like EJB.

  Is anybody near to deploying a complete "libre" Java for Linux?

• System Config Tools

  There's Linuxconf. There's COAS. There's cfengine. And Ganymede (tho it needs Java; see above...) and bunches of other system config tools one one degree of incompleteness or another.

  Big, expensive things like UniCentre are also getting ported, although they're not likely of great interest on the home front.

  Is there any intent to try to have some useful protocols to allow intercommunications of some of these systems, or to perhaps pick an existing one rather than recreating the wheel?

• Testing/Standards

  There has been some lipservice about Linux Standard Base (LSB), but it is not evident that anyone has either deployed substantially changed systems as a result of attempting to conform to some common guidelines, nor to actually provide ways of conforming systems to standards.

  There are lots of tools out there to run systems through automated test suites; that is apparently one of the major tasks of one ACLs for Linux project. In other contexts, we find ANSI Common LISP Conformance Tests. The folks at Cygnus run EGCS through testing, and provide EGCS Test Suite Results. Greg is being used to validate that GnuStep conforms to its documentation.

  ... And every "dot zero" release of Red Hat Linux fills many with fear as it tends to at least appear undertested.

  And then there's the Extreme Programming approach (particularly associated with Smalltalk) where one of the core requirements is of Continuous Integration Tests that are integrated in with the development process.

  But it is, often enough, not clear that people are depending in much more than merely the notion that Because it's Open Source, naturally bags of people will want to spend their weekends testing my code.

  We badly need to have some regression tests so that some testing takes place as distributions are constructed. Debian does some of this with dpkg-related tools; it is highly unfortunate that similar tools have not cropped up around RPM.

  Question: What are you doing to help contribute to the public body of test suite code?

The main place where differences between Linux distributions are persistent are with regard to two things:

1. Installation tools

   ... In the case of "initialization" stuff, the custom tools built by Caldera versus RHAT versus SuSE versus ... may be permanently different, but this is relatively uninteresting since you only run this stuff once.

2. System Management tools

   This is arguably a matter for more concern.

   Tools include rpm/dpkg, and the recent proliferation of distributions based on Debian is results in RPM no longer being quite as "worshipped" as it used to be.

   I regard the increase in interest in Debian-based distributions as a good thing since Debian has more automated tools for managing and validating validity of packages, which is an area where RPM had "gotten pretty stuck" for a long time.

   Aside from package management, there is then "system management," with tools like COAS and Linuxconf, where different distributions are promoting different tools. (And I'd put in a plug for the OS-independent tool cfengine that's good for lots of purposes...)

There's some fragmentation, but my old essay Linux and Decentralized Development has the thesis that the net results are positive. I haven't seen compelling evidence to the contrary yet.

cfengine is a GNU project which easily replaces rdist. It uses its own protocol rather than relying on a seperate program (ie: rsh or ssh) to transfer the data. Encrypted transfers are an option in the most recent (v1.5) version.

Check it out at the cfengine home page