Slashdot Mirror
How Do You Manage a Product Based on Linux?
Ryan writes "Following my advice, my company has decided to base it's new appliance on Linux. So far, it's worked out great. Linux gave us a huge jumpstart on development because of it's open nature and the information we've garnered from public mailing lists. We've added software, modified startup files, and have built our own kernel. Now the question is: How do you manage it all? Do you put it all in CVS or Subversion? Do you use the distro's packaging system (we're using Debian)? What does your build system look like?"
If you're releasing a product to the public, the one word you need to keep in the back of your mind at all times is "reproducability".
.deb file). You can provide "answers" files for debconf so that no questions are asked during installation, and you can tweak various settings as you go along. If you've taken the VMWare approach, you can always login to the image and make final adjustments (just make sure they're scripted and version controlled) after the Debian install is complete.
:-)
e ment) is what you want to search for. Librarianship for software systems, kind of dry and boring, but oh-so-necessary.
Can you, at any point in the future, reproduce whatever version it is that customer X is having trouble with?
There are many ways to do this, ranging from taking complete snapshots of each "build" (requires lots of space, but fast to reproduce), to keeping a short list of the Debian packages installed (not much space, but slower to reproduce). It's a classic space-vs-time tradeoff.
I'd suggest you attempt to automate the system build as much as you can. Use virtualization tools like VMWare to help perform "builds" of your OS images. Most Linux distros have automated install processes. RedHat has "kickstart", Debian has "fai" (http://www.informatik.uni-koeln.de/fai/). At the minimum, you should version control the script you use to build your vmaware images, and the configuration script for fai/kickstart. This should let you re-build everything at some later stage.
When it comes to customising Debian systems, customised Debian packages are the way to go. If you're adding new files, package them up and deploy them as part of the automated install. If you're customising existing packages, edit their source and rebuild them with
customised version numbers, and list those versions of the packages in your fai script. You'll need to go through the whole version control process with each customised package too. (i.e. check it's source into a version control tool, tag it, apply your changes, tag it again, then build your
Do a search for "customized debian", there are quite a few people doing similar things already.
Basically, make sure that the end product requires nothing more than a button push to produce. Anything less and you'll introduce the risk of someone forgetting to perform a step, or doing it wrong. That'll create a support nightmare down the track.
If you can reproduce easily what your customer has, you can also easily make a minimally invasive fix for them. That'll make them happy
If you're looking for resources on this stuff in general, "configuration mangement" (http://en.wikipedia.org/wiki/Configuration_manag
I would definitely put everything you do into Debian packages -- nothing should be done on testing and production systems by hand and the package manager provides a known good framework. There's a bit of a learning curve on how to produce Debian packages, but I believe there are some 'hardening' packages that can be used as models for how to handle the type of sysadmin tasks you're looking at.
... you are using make-kpkg, right?
You're using make-kpkg to build your kernel, of course, so it's already kicking out packages for your locally-built kernels.
I have to agree with the others that the fact that you're asking about version control tools is scary. That's something that should have been decided a long time ago.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
No points today, and I loathe "mod parent up" postings, but that's the perfect response.
Nobody is going to be able to provide any reasonable advice, other than perhaps for the submitter to hire a consultant (or employee) that has proven experience in large scale software development projects.
To submitter Ryan: This is highly non-trivial; you don't seem to have any idea how very much you're missing. If you don't know what you don't know, you need outside help. And because you've already started down the path without a plan, you need help fast. Very fast.
One important point not yet raised is that you need to control the platform for your appliance. Every customer should be on a release of your code, tested and deployed on a release of the platform - hardware as well as OS configuration.
Security and features patches are helpful, but can also bring complexity and confusion when it comes to troubleshooting.
Your best bet is to tweak a build of the OS (shut down unneeded services, automatic updates, etc) then GHOST (or equivalent) the disk so that EVERY customer gets the same thing.
If you choose to rev the hardware or the OS, how will you make sure that your installed base has the same stuff? I can't emphasize enough how important this is to long-term support.
You'll need to consider how to slipstream patches in (connection to your website, flash drive, CD, etc) for both the OS and your code.
You'll need to design it so that you can upgrade the OS install without affecting your application (perhaps a separate filesystem?) What will happen after your customer has installed and used your application for 12 months and you decide to upgrade your code? Do they have customizations? Will your upgrade work?
Hope these ideas help.
Regards,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?
Well written.
To the OP, here's what I've personally seen: a large company with no processes in place doing ad hoc builds on local dev's machines with random tools going straight to production. This was replaced with a build machine and ant with much reluctance, as several developers had their own "job security" babies, as they were mysteriously the only ones that could build certain components, at least until the build machine became the only source of move to production code. However, the business went from ad-hoc builds to weekly builds. Yep - that's weekly builds. 72 hours for dev, 24 hours for UAT and functional testing, 48 hours for QA, then hand-off to ops for migration to production. Every week. Add up those hours and you'll understand why almost no one will work there that's been there.
The cesspool just got a check and balance.
I have written extensively on this problem at my Blog. I use Morphix which is a system for building live cd's. They provide a core and you build on top of it. I have lightly modified the core (with a custom kernel and custom modules). Then you create a main module (which is just an xml file of debian packages). Morphix tools work out all the dependiences etc. I do all of my development in VmWare as it gives me a separate process space/machine to do all my work in. I will be presenting on it this Saturday at the Cerroitos Lug and the SFVLUG. It will be recorded and I will put the (video)podcast online. Along with notes and configs. Its mostly on VmWare but also how I am using it for development work. Feel free to reply to this post with questions and I will be more then happy to answer them. So vmware+morphix(highly customized) = great results. I am also building an appliance (an exchange replacement with MAPI support). See my signature for links.
Charles Wyble System Engineer