Slashdot Mirror
How Do You Create Config Files Automatically?
An anonymous reader writes "When deploying new server/servergroup/cluster to your IT infrastructure, deployment (simplified) consist of following steps: OS installation: to do it over network, boot server must be configured for this new server/servergroup/cluster; configuration/package management: configuration server has to be aware of the newcomer(s); monitoring and alerting: monitoring software must be reconfigured; and performance metrics: a tool for collecting data must be reconfigured. There are many excellent software solutions for those particular jobs, say configuration management (Puppet, Chef, cfengine, bcfg2), monitoring hosts and services (Nagios, Zabbix, OpenNMS, Zenoss, etc) and performance metrics (Ganglia, etc.). But each of these tools has to be configured independently or at least configuration has to be generated. What tools do you use to achieve this? For example, when you have to deploy a new server, how do you create configs for, let's say, PXE boot server, Puppet, Nagios and Ganglia, at once?"
And I type the stuff I need.
(And I start a war on /. )
At my institution, we run a MySQL database which we use to store information (such as their IP address, SNMP community) about network devices, linux servers, etc. We then have config file generators that query the database and generate the appropriate configs for Nagios and our other tools, and will restart them if needed. The idea is once you seed the initial information in the database, the config generators will pick them up and do their work so we won't have to remember to add the new hosts everywhere.
How about Debian, which automatically includes dpkg, aptitude and synaptic?
From my experience it would take care of most aything.
And with a good admin, even more.
.
How do I automate away a sysadmin position?
Love,
Industry
--
Heh, the Captcha word is "unions"
That is what configuration management is supposed to do, as far as I know puppet and cfengine do this already. I believe puppet compiles configuration changes and sends its hosts their configuration automatically, every 30 minutes.
Don't know what Unix or Linux vendor you're using puppet with. Whenever you do your network install, assuming you have some unattended install process, there should be some way to run post installation scripts. Create a post install script that will join your newly installed hosts to your puppet server. Run this post install script with kickstart, preseed, etc. at the end of the install process. Once newly installed hosts are joined to your central puppet server, then puppet can manage the rest of the configurations.
/^([Ss]ame [Bb]at (time, |channel.)){2}$/
but at my work we use PXE boot and cfengine on one of our centos clusters. The nodes PXE boot off of the disk array of the cluster, after the install the next stage of the PXE/kickstart script installs and runs cfengine which gives the node all its NFS mounts, etc. I don't see why you couldn't do a similar thing for nagios configuration and ganglia. In fact for clusters I think that Rocks which uses centos, PXE, and Sun Grid Engine just like our cluster has the option of having ganglia for monitoring too so you probably can steal their setup and see how they automated it.
Eh, has Linux server administration really come into this? Hire knowledgeable admins that can script stuff. Linux is perfect for scripting such configuring and set up. You just need to do those scripts once and you're ready to deploy them on all systems after minimum installation.
If you're a large company, just develop your own solutions, its far better than using someones elses. Just look at google or any other succesfull company.
We have XCAT and post scripts setup to do the majority of our work. Images the machine (PXE generation, DHCP config), installs files based on group, sets the ganglia config. I don't have any monitoring setup on compute nodes as I have ganglia open daily to watch for cluster node failures. Zenoss is done afterwards as I have yet to find a good way to automate that.
#!/bin/sh X -configure \ cp /root/xorg.conf.new /etc/X11/xorg.conf
Politics is Treachery, Religion is Brainwashing
I've had good results with some home-grown scripts that grab the project-specific details from a database and then generate the relevant config files using a templating system like Genshi. Run it periodically against the database, check in changes and email diffs to the admin.
Bogtha Bogtha Bogtha
I have successfully used FAI to install Debian servers in the past. For what I needed it worked great. It is supposed to support other distributions and automatic updates as well but I haven't tried it for either of those uses.
Keep all your config information in LDAP.
Configure your servers to get their information from LDAP wherever possible. Then the config files are all fixed, they basically just point to your LDAP server.
If you have servers apps that cannot get their configuration from LDAP, write a Perl script that generates the config file by looking up the information in LDAP.
If you are tricky you can replace the config file with a socket. Use a perl script to generate the contents of the config file on the fly as the the app asks for it, and make sure the the app does not call seek() on the config file.
Everyone seems to have forgotten about M4, an extremely handy standard Unix tool when you need a text file with some parts changed on a regular basis. I'm a developer and I used M4 in my projects.
In a build process for example you often have text files which are the input for some specialized tool. These could be text files in XML for your object-relational mapping tool. These probably won't support some kind of variable input and this is where M4 comes in handy.
Create a file with the extension ".m4" containing macro's like these (mind the quotes, M4 is kind of picky on that):
define(`PREFIX', `jackv')
Then let M4 replace all instances of PREFIX:
$ m4 mymacros.m4 orm-tool.xml
By default, m4 prints to the screen (standard output). Use the shell to redirect to a new file:
$ m4 mymacros.m4 orm-tool.xml > personalized-orm-tool.xml
Sometimes, it's nice to define a macro based on an environment variable. That's possible too. The following command would suit your needs:
[jackv@testbox1]$ m4 -DPREFIX="$USERNAME" mymacros.m4 orm-tool.xml
The shell will expand the variable $USERNAME and the -D option tells M4 that the macro PREFIX is defined as jackv.
8 of 13 people found this answer helpful. Did you?
In the small shops where I have worked, I find the uses and specific hardware a little too variable to easily automate configurations. One machine is a database server, another is part of a file server cluster, another is a web server, and yet another is a firewall and spam filter. One will have a single large hard drive, another will use software RAID, the others will have hardware RAID. Some have multiple network connections. A large organization that sets up many identical servers every day might find automatic configuration useful. But in that case, why not just use imaging? Much faster than installing an OS over and over.
If that isn't enough, things change so quickly. New versions of OSes come out a few times a year. Specific hardware might be available only in a 6 month window. Expect any automatic configuration to take lots of maintenance or quickly rot.
Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
Boot to ramdisk... Depending on how big your image is and how much ram you've got.
The problem with puppet, debian/apt etc is the inevitable gradual divergence of systems as time passes; scripts fail, packages don't get installed etc. It's exactly the same problem that life faces, you'll notice that all large multicellular organisms go through a stage where there is initially only a single cell. That's because mutations creep in otherwise and the cells diverge from one another over time. Eventually you're left with a random slime which is widely divergent in code.
Apply all your updates to a single image, boot the image on all the machines you want to run it on, they are now all running identical code. Guaranteed. Arrange your clusters such that any one machine can be offline. Plus, if you have an image you're booting, you can roll back to older versions trivially.
Deleted
Eh, has Linux server administration really come into this?
Nope, it hasn't. But I did ask the question in the first place to check if I was missing something. Scripting is fun, love it, but doing everything from scratch (althought I am fan of it, as it gives me the knowledge and total control) is a bit time-consuming. So, if there is a simple software with nice web and API interface for this, and with the ability to create custom scripts which do the actual work, I would like to know about it.
Here's an example:
- lannocc-services/dhcp
- lannocc-services/dns
- lannocc-servers/foobar
On machine "foobar" I will `emerge lannocc-servers/foobar`. This pulls in my dhcp and dns profiles.
I use CVS to track changes I make to my portage overlay (the ebuilds and config files). I keep config files in a files/ subdirectory beneath the ebuild that then follows the root filesystem to place the file in the right spot. So lannocc-services/dhcp will have a files/etc/dhcp/dhcpd.conf file. I've been doing this for the last few years now and it's worked out great. I get to see the progression of changes I make to my configs, and since everything is deployed as a versioned ebuild I can roll it back if necessary.
-IOVAR Web Dev Platform
I found! Its already on slashdot! Heres the link. Oh, wait...
RedHat's satellite server has some pretty options for this, if you dig deeply enough.
RHSS lets you create configuration files to deploy to all of your machines. It lets you use macros in deployed configuration files, and you can use server-specific variables (they call them Keys iirc) inside of the configuration files to be deployed on remote servers. For example, you create a generic firewall configuration with a macro block that queries the variable SMBALLOWED. If the value is set, it includes an accept rule for the smb ports. Otherwise, those lines aren't included in the deployed config. Every server that you deploy that you expect to run an SMB server on, you set the local server variable SMBALLOWED=1. Satellite server can also be set up to push config files via XMPP (every server on your network stays connected to the satellite via xmpp, the satellite issues commands like 'update blah_config' to the managed server, and the managed server retrieves the latest version of the config file from the satellite server).
Satellite is pretty darned fancy, but also was pretty buggy back when I used it. Good luck!
Reid
The Right Reverend K. Reid Wightman,
And if the servers are of more heterogenic nature and/or distributed across multiple datacenters?
"Brings them in configuration..."
For monitoring? Or for other things also, like configuration management?
Nope, a Slackware user, and on those servers I manage every software that interacts with external world (clients) is compiled from source as well as all the required libraries. But hey, I might be getting lazy just by not posting this from some Slackware shell telnet client, but from - you have guessed it - Ubuntu :)
Do you know I one can add a new host for monitoring to openNMS via some sort of API?
I = if
How can you steal a free software?
Anyway, what are the pros of Cfengine compared to Puppet, in your opinion?
http://www.redhat.com/spacewalk/
Looks promising! Tnx!
Good luck. It's still not 1.0 release grade, but we're using it with several thousand servers without many problems.
Excellent point. We admin 2500+ linux servers, and while we use several open source toolkits to do a lot of the hefty lifting, they're all glued together with bash scripts and python code (and a SQL backend).
I put all my config stuff into a noarch RPM and install it when I kickstart the box. When the configs need to be updated I update the rpm and roll it out as an update. That way we know what version of every thing we have and you can use the RPM tools to check if any thing has been changed.
Reminds me of a sysadmin koan I once found...
Junior admin: "How do I configure this server?"
Master: "Turn it on"
http://bashedupbits.wordpress.com/2008/07/09/systems-administration-koans/
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
If you want inspiration about automated configuration management done right, take a look at SME Server. It's got a template-based, event-driven configuration management system with a mature, well-documented API that could easily be appropriated for in-house use.
The SME Server distro itself is a general-purpose small office server, so it's likely not appropriate for your shop, but their approach to configuration management is simple, well-designed and extremely well-implemented.
Full disclosure: I worked for the company that developed SME Server for a couple of years, and I continue to deploy and support it widely.
Crumb's Corollary: Never bring a knife to a bun fight.
Puppet can do all of that for you, including adding the host to nagios if you manage nagios's configuration with Puppet that is.
For my installations I'm currently using Cobbler to deploy a base install, which handles installing the OS and its configuration (IP, hostname, etc.) Cobbler also installs a number of post-install scripts which then run on first boot to install things like vendor specific drivers/packages (eg the HP PSP) and does an initial run of puppet, which automatically registers with puppermaster. The node will pull down everything else it needs based on its standard configuration and any assigned classes. Cobbler can also control Puppet, via external files, to allow all of this to be configured via Cobbler on the command line when you add a host. If you control Nagios via Puppet, it can generate all of the nagios configurations for it as well.
As far as I'm concerned generating configuration files lies solely with the configuration management system, eg Puppet or your own tools (stored in version control!) I use Puppet for everything possible and for things that I am too lazy to put together in Puppet I generate them via custom tools and have the output stored in svn (apache vhosts, etc.)
It's also important to make things as generic as possible and try to use standard tools wherever possible, eg SNMP for monitoring.
Actually this is one of the goals VMWare is proposing to meet with their vSphere. vCenter, ad nauseum initiatives. [full disclosure I've beta'ed VMWare software since v1]. This also presupposes full P2V, V2P cross machine conversions if required. The goal here is be anywhere, and run anywhere.
:-).
Now if I had the money, I'd toss full de-dup into the storage array mix as well, so much of the image file size essentially disappears unless there is simply no duplication anywhere. And if you are in that situation, take my advice. Quit, or just shoot yourself and get it over with.
It's been a long time since I played at that level (six mainframes, eighteen mini's, 575 desktops, and I never got an accurate count of the 100+ laptops) but at some point you have to ask yourself, when does the customization end? Standardization was the only thing that kept myself and my team of four !relatively! sane.
If you seriously need customization of that level, then you aren't doing things right. Reduce each VM to a single app (Apache, MySQL, IIS, network appliance, whatever) and use virtual switches to create a topology as required. Think of each VM as a particular Lego block, or IC: Systems Componentization as it were. And this is where de-dupe will also shine.
Which explains why a certain storage company bought VMWare, and a certain switching company has created a virtual switch. Now if you don't have the big bucks, you have a slight problem. However you can create this kind of topology if each box has more than one physical network adapter AND you get creative. Now that job I also wouldn't mind trying here. Time to resuscitate some old boxes and see what I can come up with. Been a while since I setup an enterprise class simulation
It's high time that we all realize that the lines between the various (computer) engineering disciplines are now blurred. Sure, be a subject matter expert but know How the other people think and work.
Anyone know of a F/OSS de-dupe?!
"[I]t is a wise man who admits the limits of his knowledge or skill, and that pretending either causes harm." --Terry Go
So you're looking for enterprise capabilities like automated deployment and configuration management, and yet you chose a setup that doesn't have any vendor providing them, and requires you to build them yourself, why?
Of course you can cobble something together by writing custom scripts, and setting up puppet, bcfg2, or cfengine.
Which also involves some custom scripting. No matter how you slice it, there's going to be some initial manual programming work to get it working.
There's really no end-to-end pre-made CM solution you will find for Linux, for free, that's not tied to an Enterprise Linux offering of some sort, and doesn't require you to do manual scripting at least, and some initial manual config writing on your own.
Do/you/speak/english and/or any/other/language? AYFKM!!!
We use a robust configuration management/provisioning system consisting of puppet, cobbler and koan.
Puppet is easily scaleable for just about any sort of server need, cobbler and koan take care of the heavy lifting for provisioning. It's also fairly easy to write your own puppet types and modules for various tasks.
With one command we are able to provision a server from bare metal (or vm) to a fully working server, complete with SAN/NAS storage, fully operational daemons and authentication.
Have a squat over at the hobo house.
cfengine is great for what it does. It really just depends on your use case. The only downside is that I am not certain cfengine is still actively maintained.
If you want to customize cfengine you are going to use perl, if you are going to customize puppet you are going to use ruby.
Both are fine, you need to figure out your infrastructure and scalability needs - I have found puppet scales a bit better for large, complex stacks but cfengine is easier for more static, less changing environments.
Have a squat over at the hobo house.
With a properly setup configuration management system you can have it all.
One button, dummy-mode provisioning - os install, configuration files, daemons, monitoring and metrics, authentication and external NAS/SAN storage in one swoop.
I would recommend checking out cobbler/puppet/koan or a tuned cfengine/pxe+kickstart setup.
Have a squat over at the hobo house.
Right, because Debian isn't a mature operating system, and Ubuntu couldn't possibly be based on Debian...
That aside, good luck with your pretty point-and-click crud on servers that don't have X installed (about 99% of deployed Linux servers, probably).
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
The very first thing that came to mind was "Isn't this what you lazy bastards were hired for?" Jeez, if you don't wan't to the the marginlly interesting stuff, I would hate to see your performance on the day in, day out tedium that can be IT.
Well, Duh!!! I haven't thought of Solaris in years although I recall it's a BSD derivative of some sort. That'll work since I still have the Daemon book and experience running it on mi Amiga back in the 80's.
Thanks!
"[I]t is a wise man who admits the limits of his knowledge or skill, and that pretending either causes harm." --Terry Go
not maintained? they just released a total rewrite as v3 and a commercially supported version as well. cfengine is designed for large, complex environments. mar burgess talks cfengine to google: http://www.youtube.com/watch?v=4CCXs4Om5pY
At the risk of sounding like some sort of an advertisement for EMC, If you are working for a company with money... Voyence is a WAY cool product. It will do just about anything you could possibly want to network devices. It will even tell you if you screw up something.
ZFS has de-dupe and it's free and open source. There are some companies making (some even open source and free) storage appliances using ZFS with all it's amazing capabilities. Then, you can connect to it via iSCSI for virtualization or FTP, SMB, etc for the rest.
The hip way to get your IP. No ads, ever.
Yeah, .pls and php.
Also, anyone wanting to build a moonbase using an army of robots should start with a single robot arm, some materials, and a compiler. ;)
Reading the original post again - I'm a little unclear what the question is.
If the question is "How can I manage all this stuff" - you can manage it through puppet.
If the question is "Is there something that can automaticaly do EVERYTHING for me" then the answer is "No" - no matter how much you want to abstract things, at some point, you are going to have to plan and put the system together.
You could roll something sweet with OpenQRM to make it all drag and drop - but you'd have to put in the wrench time to model it after the types of things your organisation has/needs, and you'd have to roll quite a bit of infrastructure out underneath it to make it work.
What you are really asking, I think, is are you missing something in the big picture - and I don't think you are - it's just a matter of scale.
There's a Zenoss/Puppet integration here: http://github.com/mamba/puppet-zenoss/tree/master
There is an open source cluster management stack called UniCluster available at http://grid.org. (disclosure: I work for the company that makes UniCluster). Its intended for managing HPC clusters but it can do everything that you're looking for in one tool. It has support for ganglia, nagios, cacti already built in and adding new third party components is pretty simple. It has a tool to push config files around and will do bare metal provisioning (ie. setup PXE and kickstart for you).
Tom
The unstable version (what will be come stable 1.8) does have a RESTful API for adding nodes. Additionally, 1.6.x and higher have an API for specifying your nodes manually, which can be called from external tools. This feature has been enhanced in what will be 1.8 to still scan interfaces on the nodes you specified, and such.
WWJD? JWRTFM!!!
But each of these tools has to be configured independently or at least configuration has to be generated.
You write that like its bad or something. Decentralized is always more reliable overall.
The correct way is to work it thru in reverse. Automated tools should find things they can monitor, and then humans think about what to do.
NMAP periodically dumps its results in a DB. Watch your CDP too. Maybe sample your ARP cache on your switches. And keep an eye on your RANCID router configs.
One simple script analyzes the nagios config and emails a complaint to either one individual, a mailing list, or a gateway that autogenerates a ticket. The script sends one alert for each issue it finds, something like "WTF nmap found a device at 10.11.12.13 that is not configured or commented as ignore in Nagios". I haven't met a plain text config file yet, that doesn't allow comments, so if you desire not to monitor something you have a syntax in the config file "# ignore 10.11.12.14" and your script understands that.
Nothing wrong with your script generating alerts that contain sample "cut-n-paste" info to add to your configs.
Repeat for reverse DNS, munin monitoring system, MRTG polling of anything with an open SNMP port, etc.
Also you need well backed up and replicated wiki with a page for every device your network monitoring tool detects.
Finally don't forget that if something has been "red" in nagios for perhaps a week and/or its gone from the ARP table for a week, maybe it's time to formally delete it, also necessitating alert emails.
Conveniently this scheme also "forces" people to explain what they think they are doing, to at least one other sentient being, which can be very educational for all concerned if the end users are doing something crazy.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Thanks for the info.
Don't get me wrong, ZFS is a nice, modern file system. But the hype around it is just bizarro. I don't think most folks really get what it can do today and what Sun *says* it will do at some undefined point in the future. It is certainly better than anything previously available as part of the core Solaris OS. People shelled out megabucks to Veritas to deal with the lack LVM and a decent file system in older versions of Solaris.
the growth in cynicism and rebellion has not been without cause
Eh, has Linux server administration really come into this? Hire knowledgeable admins that can script stuff. Linux is perfect for scripting such configuring and set up. You just need to do those scripts once and you're ready to deploy them on all systems after minimum installation.
If you're a large company, just develop your own solutions, its far better than using someones elses. Just look at google or any other succesfull company.
I agree.
We have our own home-grown configuration management system; an open source version of it is available here.
In large systems, a system administrator is a developer. You write software that integrates your configuration management with Nagios, with your kickstart system, with your auditing system, that writes your firewalls.
I use cobbler and cfengine to deploy and maintain a couple of clusters including Xen virtual machines and a ... cfengine a pretty good job at management ...
... uh ... I guess cobbler takes the edge off of configuring dhcp/pxe/dns/yum servers ;-) (not really)
... well, It has its upsides, but it is not ... buses
... that's my rant.
few labs with workstations.
Cobbler does a pretty good job at deployment
Automatic configuration
for deployment and updates. Kickstart scripts can be obtained by building one machine, grabbing the anaconda
script from the root directory and fudging it to taste.
That's almost automatic
On the downside, with cobbler, you get the overenthusiastic release sequences typical of Fedora related
projects (if it compiles and runs, it is production ready; major features introduced within a minor release and
all that good stuff), so updating is a bit of a adrenalin rush time.
But, such is price of freedom (and free beer).
Configuring machines using cfengine is a dog (and I learned to love the pup), but it is the best dog we have.
That is all but automatic. I also have puppet deployed to compare
better than cfengine. Frankly, I do not benefit much from the main concepts and features behind cfengine
and would probably be as well off with puppet, or even func and such.
Having a company backing cfengine
makes me feel a little better now. (I was a little nervous about Mark crossing the streets every day
stop for no one).
I don't think that in the current state of affairs automatic configuration is not even desirable as all of the
components involved very rapidly reach configuration complexity that needs auditing.
I have my working setup, but the next step in improving and upgrading it is a bit of a mystery to me
given the options out there.
Anyway
{Snort} Got it. SysV would be just fine as I have a ton of sysadmin and other documentation for it as well and I've met kin in the past and dealt with it with no training or documentation for that matter at the time. ZFS does seem to sound a bit like what some 'softie zealots shout about the fabled sql-based FS we'll get Real Soon Now.
I'll evaluate my options when I have some more hardware to play with back online. Even bare metal hypervisors don't give a true picture of reality, although I wish they did. Sometimes the HV can mask real-world issues, or create contention (I/O for instance) of their own.
"[I]t is a wise man who admits the limits of his knowledge or skill, and that pretending either causes harm." --Terry Go