How Do You Create Config Files Automatically?

An anonymous reader writes "When deploying new server/servergroup/cluster to your IT infrastructure, deployment (simplified) consist of following steps: OS installation: to do it over network, boot server must be configured for this new server/servergroup/cluster; configuration/package management: configuration server has to be aware of the newcomer(s); monitoring and alerting: monitoring software must be reconfigured; and performance metrics: a tool for collecting data must be reconfigured. There are many excellent software solutions for those particular jobs, say configuration management (Puppet, Chef, cfengine, bcfg2), monitoring hosts and services (Nagios, Zabbix, OpenNMS, Zenoss, etc) and performance metrics (Ganglia, etc.). But each of these tools has to be configured independently or at least configuration has to be generated. What tools do you use to achieve this? For example, when you have to deploy a new server, how do you create configs for, let's say, PXE boot server, Puppet, Nagios and Ganglia, at once?"

Comments

A Database w/ Config File Generators

[Anonymous Coward]

At my institution, we run a MySQL database which we use to store information (such as their IP address, SNMP community) about network devices, linux servers, etc. We then have config file generators that query the database and generate the appropriate configs for Nagios and our other tools, and will restart them if needed. The idea is once you seed the initial information in the database, the config generators will pick them up and do their work so we won't have to remember to add the new hosts everywhere.

Generate config files

[atomic-penguin]

That is what configuration management is supposed to do, as far as I know puppet and cfengine do this already. I believe puppet compiles configuration changes and sends its hosts their configuration automatically, every 30 minutes.

Don't know what Unix or Linux vendor you're using puppet with. Whenever you do your network install, assuming you have some unattended install process, there should be some way to run post installation scripts. Create a post install script that will join your newly installed hosts to your puppet server. Run this post install script with kickstart, preseed, etc. at the end of the install process. Once newly installed hosts are joined to your central puppet server, then puppet can manage the rest of the configurations.

M4 baby, M4

[cerberusss]

Everyone seems to have forgotten about M4, an extremely handy standard Unix tool when you need a text file with some parts changed on a regular basis. I'm a developer and I used M4 in my projects.

In a build process for example you often have text files which are the input for some specialized tool. These could be text files in XML for your object-relational mapping tool. These probably won't support some kind of variable input and this is where M4 comes in handy.

Create a file with the extension ".m4" containing macro's like these (mind the quotes, M4 is kind of picky on that):

    define(`PREFIX', `jackv')

Then let M4 replace all instances of PREFIX:

    $ m4 mymacros.m4 orm-tool.xml

By default, m4 prints to the screen (standard output). Use the shell to redirect to a new file:

    $ m4 mymacros.m4 orm-tool.xml > personalized-orm-tool.xml

Sometimes, it's nice to define a macro based on an environment variable. That's possible too. The following command would suit your needs:

    [jackv@testbox1]$ m4 -DPREFIX="$USERNAME" mymacros.m4 orm-tool.xml
The shell will expand the variable $USERNAME and the -D option tells M4 that the macro PREFIX is defined as jackv.

Gentoo Ebuilds, CVS

[lannocc]

I run Gentoo on all my systems, and since the .ebuild file format was easy for me to understand (BASH scripts) I started creating Ebuilds for everything I deploy. These ebuilds are separated into services and machines, so emerging a machine will pull in the services (and configs) that machine uses.

Here's an example:
- lannocc-services/dhcp
- lannocc-services/dns
- lannocc-servers/foobar

On machine "foobar" I will `emerge lannocc-servers/foobar`. This pulls in my dhcp and dns profiles.

I use CVS to track changes I make to my portage overlay (the ebuilds and config files). I keep config files in a files/ subdirectory beneath the ebuild that then follows the root filesystem to place the file in the right spot. So lannocc-services/dhcp will have a files/etc/dhcp/dhcpd.conf file. I've been doing this for the last few years now and it's worked out great. I get to see the progression of changes I make to my configs, and since everything is deployed as a versioned ebuild I can roll it back if necessary.

RedHat Satellite Server

[giminy]

RedHat's satellite server has some pretty options for this, if you dig deeply enough.

RHSS lets you create configuration files to deploy to all of your machines. It lets you use macros in deployed configuration files, and you can use server-specific variables (they call them Keys iirc) inside of the configuration files to be deployed on remote servers. For example, you create a generic firewall configuration with a macro block that queries the variable SMBALLOWED. If the value is set, it includes an accept rule for the smb ports. Otherwise, those lines aren't included in the deployed config. Every server that you deploy that you expect to run an SMB server on, you set the local server variable SMBALLOWED=1. Satellite server can also be set up to push config files via XMPP (every server on your network stays connected to the satellite via xmpp, the satellite issues commands like 'update blah_config' to the managed server, and the managed server retrieves the latest version of the config file from the satellite server).

Satellite is pretty darned fancy, but also was pretty buggy back when I used it. Good luck!

Reid

Re:Templates

[johnlcallaway]

We did something even simplier on our Sun servers. We used a master server with directories that held the different app and web servers we had. Everything that needed a configuration file that had server specific items, like Apache, had a server-specific script to generate environment variables. A configuration script was created using the template:

. servEnv.sh
cat <<EOD >realConfigFile
## put config file here replacing any server specific items
## with $envVariable from the servEnv.sh script
EOD

We could redeploy a server in 10 minutes from an empty hard drive. Creating a new one took about 10 more minutes to create the servEnv.sh file.

This also gave us the ability to take scripts from dev to qc to production without having to change anything. Part of the servEnv.sh script set things like home directories and such. We could even have multiple environments on one machine.

Re:Here, let me google that for you

[TooMuchToDo]

http://www.redhat.com/spacewalk/

Re:Here, let me google that for you

[jvillain]

I put all my config stuff into a noarch RPM and install it when I kickstart the box. When the configs need to be updated I update the rpm and roll it out as an update. That way we know what version of every thing we have and you can use the RPM tools to check if any thing has been changed.

Reminds me of a sysadmin koan...

[ghostis]

Reminds me of a sysadmin koan I once found...

Junior admin: "How do I configure this server?"
Master: "Turn it on"

http://bashedupbits.wordpress.com/2008/07/09/systems-administration-koans/

Re:standard VM image?

[Decker-Mage]

Actually this is one of the goals VMWare is proposing to meet with their vSphere. vCenter, ad nauseum initiatives. [full disclosure I've beta'ed VMWare software since v1]. This also presupposes full P2V, V2P cross machine conversions if required. The goal here is be anywhere, and run anywhere.

Now if I had the money, I'd toss full de-dup into the storage array mix as well, so much of the image file size essentially disappears unless there is simply no duplication anywhere. And if you are in that situation, take my advice. Quit, or just shoot yourself and get it over with.

It's been a long time since I played at that level (six mainframes, eighteen mini's, 575 desktops, and I never got an accurate count of the 100+ laptops) but at some point you have to ask yourself, when does the customization end? Standardization was the only thing that kept myself and my team of four !relatively! sane.

If you seriously need customization of that level, then you aren't doing things right. Reduce each VM to a single app (Apache, MySQL, IIS, network appliance, whatever) and use virtual switches to create a topology as required. Think of each VM as a particular Lego block, or IC: Systems Componentization as it were. And this is where de-dupe will also shine.

Which explains why a certain storage company bought VMWare, and a certain switching company has created a virtual switch. Now if you don't have the big bucks, you have a slight problem. However you can create this kind of topology if each box has more than one physical network adapter AND you get creative. Now that job I also wouldn't mind trying here. Time to resuscitate some old boxes and see what I can come up with. Been a while since I setup an enterprise class simulation :-).

It's high time that we all realize that the lines between the various (computer) engineering disciplines are now blurred. Sure, be a subject matter expert but know How the other people think and work.

Anyone know of a F/OSS de-dupe?!