Slashdot Mirror
Improving Software Configuration Management?
Elvis77 writes "I am managing a project looking at some software to help us with
software change management. There are numerous good applications around for doing this and our purchasing is complete, but I have been amazed during my investigations at how many organizations rely on good manners, good intentions and good luck to manage the configuration management of their organization's software assets, even in light of the Sarbanes-Oxley Act of 2002 (SOX) that affects US companies (I am in Australia). Organizations outside of the USA, without SOX implications, are usually still concerned about the quality of their software. What do my fellow Slashdot readers consider to be the best practices for configuration management?"
I'd say the practices depend on the risk involved with each change, and that depends on the business -- where a single bad change might cost you $5mil, you're going to behave differently than a place that doesn't even pull in $100k in a year.
My rule of thumb -- good backups. Make sure they're good before a chance (ie, you can restore from them, not just that they ran), and don't make changes on Friday -- no one wants to spend their weekend cleaning up something that went wrong. (and if management wants me to come in and do a change on the weekend -- I want 2 weeks notice
And management signoff on changes is basically useless -- you get bitched at when something goes wrong, no matter how many many notes you make in change management that it's a bad idea, and it's going to break things.
Build it, and they will come^Hplain.
First, try to buy software that logs this for you. That makes it easier to just point the auditors to the logs when they come sniffing around, and keeps your operators and system custodians from wasting time double-entering notes about what they did on some other third-party system.
Combine a Tripwire-like tool with a computer-immune-system like CFEngine, add in a Change Control workflow that isn't too painful and you'll be in good shape. Reconcile the Tripwire reports with the change control paperwork to check that changes are being properly recorded in the workflow.
If you get to the point where you realize how vital souce control is to your project(s) then I'll assume that you're already bogged down with work and deadlines. In this case start small. Just get your data into CVS or Subversion (for teams put the repository on the LAN/web/Sourceforge/etc)
This first step will lead you to the next - increased communication in the team and good documentation. For documentation I've been please with the ReadySet project at Tigris.org. Start with the basics and work up from there. Bringing a whole team along for the ride is time consuming, challenging and in the end - absolutely necessary (make it part of their objectives if you can).
Continue from there
"The difference between stupidity and genius is that genius has its limits." -- Albert Einstein
There are really two paths you're talking about here, and folks tend to confuse the two:
1) Software *development* change mangement. Meaning: tracking things like changes to software code.
2) Software *configuration* change management. Meaning: tracking changes to the configuration of the software. I presume you're talking about this one, but it's not completely clear.
Development change management is well served by tools of varying complexity from ClearCase to Subversion/CVS. Subversion/CVS seem to be very common, as they're free, but I've worked in offices using ClearCase before (not that anyone was terribly happy about it, though).
Configuration change management is much harder, as you're talking about managing the configuration of applications across potentially hundreds of machines. The tools for this vary widely, depending on how hard-core you want to be. They vary from CFengine to full-on provisioning systems (openQMS, for example)...none of them tend to be easy to set up or manage, which makes them less common.
I would recommend using RCS for tracking changes of config files. CVS is based on it, but RCS allows you to work better with just single files instead of entire directories and sets of source codes. Just create an RCS directory in the directory that the config file is in, and do this:
ci -u configfile
when you want to edit:
co -l configfile
check it back in:
ci -u configfile
Easy enough.
First step, forget about the tools.
..
..
Your organization has a process that it uses. The process may not be very formal, but it exists, and your employees are used to it, whether they like it or not.
Most people go out and find a tool that they think will solve all their problems, and then they try to force the tool (and the tool's internally defined process) on the organization as a single step.
Find the control process you want to be using, and work on getting your people to change their processes to match with your desires. Nope, this ain't easy. But guess what? Anything else will end up with rogues who learn the tool well enough to end-run it. People who break the process you depend on because they figured out how to do so.
Find the process you want in place. Teach it. Reinforce it. Encourage it.
While you are doing that, examine the tools, and find one you like that matches your process. As the process becomes institutionalized, as the people in your organization are not only using it, but are helping to keep each other in line with the process, then you can introduce the tool which MATCHES the process they are already using.
There will still be some resistance to the tool. But a good tool makes following the process it supports easier and simpler. If you have a good process and tool match, selling the tool actually becomes MUCH easier.
If you pick a tool, and then use implementation of that tool to force process changes, you will get to face all the resisters of both changes at the same time AND you will generate extra opposition because of the larger amount of change you are requiring in one step . .
Former USAF Capability Maturity Model Software Process Improvement Team member . .
NetDirector does *NIX service configuration management across multiple servers on multiple Linux distros, Solaris, and BSD. And for SOX, it does change tracking and audit trailing, offers role-based management, change scheduling and rollback - all from a nifty Web-based GUI!
I've seen it in action and it's pretty cool. They have an online demo on their site. Personally, I've always just hand-edited config files (vi is your friend), but for a multiple-server environment, this looks like it would be a winner.
There's a whole other layer to CM, which is the issue tracking, requirements tracking, resource management and documentation management piece that probably is more critical than whether you can checkin/checkout changes and be able to back out a change.
We supported a government client where issues were managed with word document templates and emails. It was a disaster, with things getting lost and weekly meetings being the only times decisions were made. They were spending a boatload of money developing something that looked suspiciously like a hobbled version of bugzilla, so we recommended and got approval to stand up bugzilla for issue tracking. It's been a big success and has been expanded to handle requirements management as well.
The key was to set up "products" that matched operational areas for a product rather than thinking that a "product" was defined as a single deliverable. We set up the standard "product" as deliverable, with subcomponents which somewhat matched the functional areas where developer responsibilities were broken down. Then we established a "product" which was essentially an area where management issues could be handled, and had subcomponents for tester access control, requirements definition, external coordination issues and the like. So when we went into testing and found an issue that we decided to defer to a subsequent release, it was moved to the admin area and the requirements subcomponent. This kept policy and requirements control out of the hair of the developers and allowed parallel workflows for requirements, design, development and testing.
It's not a perfect tool for all of that, but it's pretty close to "good enough", and the price is right.
What would I add to this mix if I were God? I would wiki the work-in-process user's manual so developers could flag issues that should be addressed in documentation rather than in code. I would probably wiki and/or subversion the test plans, as word documents utterly suck for test plan management. And I'd spend some time with wiki "special pages" and bugzilla customized components to integrate it all -- linking and content sharing between a wiki and bugzilla would bring the solution set into the 90%+ range of requirements matching.
Coders rarely are the problem in software CM. It's the management, architects, functionals and tester coordination that really has the potential to kill a project. But if you can coordinate all of them well, the flow of functional requirement - use case - design - develop - test - debug - requirement generation & traceability comes together cleanly and raises a development process into a portfolio management/enterprise architecture execution process.
The general process required is that a change can be tracked from a customer request, through the design, to a code file, and back up the V (or round the spiral) to the testing/verification/validation used to make sure that the design and code do what the customer asked for. And then that you can track all the changes that went into a particular version build, which means recording what files you used to build it.
:-)
How do you do that? I don't care, and nor does Sarbanes-Oxley. We used to use Access databases at our place (and still do on some projects). Other projects use a tool called Visual Intercept, which I'd recommend people avoid like a rabid maneating tiger. Some projects have even used Excel (heavens preserve us!). Or simply use multiple text files (checked into your version control system of choice), one per customer request. Or Bugzilla or any number of other tools will serve your purpose. Tracking versions of files could involve some hugely complex build system, or it could just involve slapping a label across the files and doing "get all".
But if you don't grok the basic premise that every line in your code must be attributable to a customer request, then no tool you can buy will help you. This is a process you need to follow, not a magic-bullet tool you can install and everything will be fine. Repeat after me - There Is No Quick Fix For Quality.
The most important thing you can do make sure your system works is therefore to review each change before it gets signed off to make sure changes are complete. That doesn't just mean reviewing that files are changed, but that they're all checked in under version control, AND that the change details (however you do it) are fully filled in. Even if there's automated fields to list all files and their versions in, review them anyway - there's bound to be cases where someone says "well I've got one test file listed, so that'll keep the tool happy, never mind about the other 20".
Grab.
Any effort such as yours should begin with clear thinking, which is aided by purging yourself of such vague, overloaded, and literally nonsensical terms as "configuration management". In case you think this is a troll, note that the posters so far have given your message a range of completely unrelated senses. Describe precisely what you're trying to accomplish if you want useful answers.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
It's downright foolish to believe that a tool will solve your problems. But you sound like a smart guy, so I'm sure you've figured that out. However when in doubt, it's always a safe bet to peruse the SEI site (here's the section on SCM). As far as tools go, I'm always partial to the free ones, but here's my take:
HTH.
Step 1: Get the best staff you can. "10x" developers do exist, but you should be aiming for a staff of "3x" to "5x" developers of varying experience who work well together. The best developers won't really want to manage other people, but will want to be trusted with significant responsibility (i.e. they don't want to be fed detailed "specs"). Being good team members is at least as important as having top-notch skills.
Step 2: Get out of their way. At this point, you need to make it easier for them to get their jobs done. Most software development processes are about making it harder to do the wrong thing, which inevitably makes it harder to do the right thing. For many managers, software processes are like violence: "If some isn't good enough, more must work better." Don't fall into this trap.
Various things that will help (and not hinder) your developers:
When you get to this point, you'll have read a lot more about software development (your good developers can recommend some fantastic books) and you'll have much more precise questions.
If you don't read at least "The Mythical Man Month" by Fred Brooks before managing a software team, you will fail and you won't understand why.
Regards,
Ross
http://www.cmcrossroads.com/ Everything you would want to ask.
This is the most interesting Ask Slashdot thread I've seen in years.
..and take a look at the bigger picture: CoBIT and ITIL
Hi.
I worked in a team trying to adapt XP to their work. I cared especially about nightlies automation. About the process, what I kind of remember ( a bit fuzy though) is: yes do have a theoretical process pretty clear in mind but trying to build the real solution without an "experimenting game" with the developpers is pretty hard. We had regulars meeting in order to change the process we where following. My point: I agree with previous post, start simple, and expend. The tools : ClearCase + a bug/service_request tracking system I don't remember the name. Some issues : both where not integrated. Not easy to track changing causes. We where using a file as a flag for mainline merges, and as a chage log. It worked ok, though was relying on good will. The bug tracking system was traditional heavy client : developpers hated it for it was providing a lot of things but not the essential in an easy manner. Some other point: we had the nightlies to automaticaly label everything ( all sources, test material *and* all the built material ). Great ! ( We were building middleware stuff which has to be compiled on combination of OS versions, compilers versions, JDK versions... about 15 platforms) I'd say yes: label well and strive for the "single label" for one release, even for multi components, multi team, apps. And automate evrything.
Now. I started 3 weeks ago a new job as ConfigMAnager. I am faced with Dimension tool from Serena. This tool aim at version control _and_ change management. Theoreticaly possible to track things up and down, as well as providing lifecycle for items, delegation, co/ci... I never heard about this tool before. And the thread here does not mention it. It gets me a bit more uncomfortable. Seeing so little in terms of forums, community. Hum. Any opinions on that tool ?
In general, one thing I am concerned of, is I feel some people imagine change tracking comes for free (not speaking about tool here). But if you do want to follow procedures, it actually _is_ complex, it takes some time and can not be done as a side task. Let me rephrase that: it _will_ somehow come in the way of the developpers. Painfull but real. How else could it be ? Obviously, we can make it easier, more comfy...
An other point. I would like to have more opinions on version tracking system + change tracking system. Any worked on real scenarios here ? Any integrated solutions ? Or integratable ones ? Subversion + JIRA for example ? I don't know Subversion. I liked JIRA.
Z.
I feel that making a clean well preconfigured install is the first step in configuration management. It is also crucial to your backup plan, as it relieves you from making complete system backups. This is not to say that you shouldn't be tracking your installed files via IDS, but the actual files should be already in your repository. I use debian with apt-repositories, but the general idea should be universal. This method lets me make more selective (and smaller) backups.
/etc directory. I made a simple program for this too. http://developer.berlios.de/projects/etcsvn /etc from being a working copy, and keeping track of ownership and permissions of those files.
I started with FAI - http://www.informatik.uni-koeln.de/fai/
which is really good. FAI shares its configuration style with cfengine. You can even use fai and cfengine in tandem with some sort of install/update strategy. I would highly recommend taking a good look at both of these systems.
Both fai and cfengine are written in perl. I can't stand perl, and since I have desperate need of similar tools, I decided to roll my own in python. The project is here -> http://paella.berlios.de/ . This code is still immature and isn't fit to be used for any activites deemed to be critical.
Another method I am using is simple tracking of changes in the
This program really shines a little more if you have multiple similar hosts, because you can manage some config files with a working copy, patch the corresponding files in the relevant host config directories, commit the changes, and then restore/update the config on those hosts. Its really nothing more than a simple tool to keep your
I am currently looking at bcfg2 http://www.mcs.anl.gov/cobalt/bcfg2/, as a replacement for cfengine. I just found out about it recently, but it's also written in python and has limited client-side dependencies.
Probably the most important thing is to be prepared to spend a great deal of time in planning, implementing and testing your system. Every tool I have seen so far makes assumptions, or has requirements, that don't match yours. Mine do too, and there is really no way to get around this.
As a general rule, you will want to look for a system that stores the configuration in a manner that you can deal with it the easiest, regardless of the configuration that it exports. The mechanics of the configuration processing should be implemented by a language that you are comfortable enough with to make the changes necessary for future strategies.
This is the first thing that needs to get established. I have too often seen people trying to equate these things and it's bound for failure. I've been there and I'm in Config let me tell you it's not pretty.
Software development is one process in of itself.
The goal of software development is to produce a stable release that is suitable to be installed on a "live" or realworld or production system. In this sense a Software Config Mgmt system keeps track of configuration items which are commonly known as source code and documentation. All revisions to source and documentation should be checked into your SCM database. From this database you build and label your source to produce a software and then send that to testing and eventually you release your software to be Generally Available. It's a hand off to whoever owns the application (the customer or your sysadmin) who will eventually install your software. If the end user has a problem they take it up with support (internal or external).
On the hardware or systems side you want to keep track of the exact setup of your production systems. Configuration items in this sense are your overall systems and software and all of the various settings that are possible. Any change to those systems need to be tracked and recorded.
These are entirely different processes/controls yet some shmoe coined the phrase over both disciplines (and hopefully he rots in hell for it). In the end they are entirely different skillsets, processes and disciplines. Take for example parallel development and code merges. There's nothing like that in a systems sense.
Oops, how did this get here?
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
Radmind leverages tripwire-like information to provide very large scale configuration management. It supports certificate-based authentication of servers and managed machines. The latest release supports compress in the network layer for cases where CPU is more plentiful than network bandwidth. :w
SCM is one part of your processes, they all have to work reasonably well to hold. What you write makes me feel little is in place.
Some suggestions:
- document how things work (or do not work) today, no matter how painful the truth is
- write down what you want to achieve. Look into ISO9000, CMM and CMMI for ideas. Particularly CMMI is very hands on, based on real life experiences (best practices) of real life companies and academia.
- make dure your test crew and quality assurance crew have organisatorical independence sufficient to tell managemnt the current status stinks when it does.
- at each iteration and at the end of each project document status, what went right, what went wrong, lessons learned, and how to fix it
- at the start of new projects look over lessons learned and make sure you do learn from your costly mistakes
- make the tools for your processes and your processes fit your needs, not the other way round
- make sure the QA crew audits everything and that their integrity remains at the highest stanrads, seriously. This is a high risk job and I lost mine quite possibly due to my integrity...
- never ever allow Excel to be used for your processes. So far in my career it has bee a surefire sign of a broken process, not even once has that failed to indicate process troubles.
- record metrics but use common sense
- make sure all infomation gained is used. Information pipelines that are open in either end is a sign of trouble.
- allow for your processes and tools to change but don't change those for a running project.