How Do You Manage a Product Based on Linux?

Ryan writes "Following my advice, my company has decided to base it's new appliance on Linux. So far, it's worked out great. Linux gave us a huge jumpstart on development because of it's open nature and the information we've garnered from public mailing lists. We've added software, modified startup files, and have built our own kernel. Now the question is: How do you manage it all? Do you put it all in CVS or Subversion? Do you use the distro's packaging system (we're using Debian)? What does your build system look like?"

Comments

If this is really the question

[Anonymous MadCoe]

You really should be stopping and look at what you are doing. How you want to manage it should be part of the strategy, and actually should have been part of deciding to use Linux (not in detail, but strategically).

So my advice, hold on, sit down and look at what you expect to produce and what you would need to get there. From there you can find out what you would need.

You will probably run into some issues, but that's just what happens, there is no ideal situation.

I don't understand the question

[Anonymous Coward]

You're building a whole new appliance, but your software engineers don't know how to manage a development process?

I mean... I'm not being nasty here, but you're in trouble, and I don't even know where anybody could start to give you advice. It would be one thing if you were looking for guidance on a regular small scale software project, but if you're jumping in feet first with a whole new large scale application and no idea how to guide the process...

Re:I don't understand the question

[arivanov]

A round of applause...

While I understand your spleen and I even applaud it, there are a few things which you are missing.

In many settings the initial people with the idea are not capable of software process management (and nearly always incapable of support process management). On top of that the people who they initially hire are usually of the "implement at any cost breaking all rules". That works great up to a prototype and sometimes even slightly later. In fact this "break all rules" culture is a model which most successfull startups in the industry have followed.

After that, once the prototype is up and working and the initial euforia has settled in, the company comes to a point where it needs to grow up. The initial people who have ideas and who are of the "implement at any cost" variety must either move into positions where they do not prolong the company growing pains or leave. The company also needs to hire people (or less likely - find within its ranks) who are capable of long term software development process management. These are the people who must become the team leaders and managers in order for the company to be successful.

Unfortunately, very few companies make it through this stage. Most promote the "implement by any means necessary" or the "initial idea" people into positions where they cannot cope as their mentality is incompatible with the actual requirements. They become increasingly frustrated with the mere fact that there is a process, break it all the times and explain to people who follow it that they are tails that wag the dog. In addition to that they blow any long term planning to hell and gone all the time and push the company into an endless spiral of firefighting crisis management. This all becomes a big mess and it all goes to hell sooner or later.

Anyway, it is not so uncommon for a successful initial stage startup to have no culture of software development process and especially customer support process. In fact that is to be expected, as to get through the initial hurdles you sometimes need to walk on the dead bodies of the rules and processes that have been broken.

Frankly, the ask-slashdotter should be applauded that they have realised that they lack this process. Now he will definitely do not like the answer to his question. It is very simple: hire someone who does and swallow the fact that you will hate him to the point where some of the veterans may have to leave. Tools like subversion, clearcase, CVS, MKS, Bugzilla do not make a process. They implement a process defined by a human. What is needed is the person who has defined the process to have a long term view as well as a view of how the process fits with support, business and the rest of the company. If he does not - no tool will help. There is nothing worse then short-termism in process definition and tool selection forced by short-termism. It will come to bite you in the arse again, and again, and again.

Version Control

[bigattichouse]

Your build should be in some sort of Versioning system (CVS, whatever). SOMETHING that allows you to cover your butt with you `rm` that folder and realize you just tanked the whole thing. Somehow you should be able to rebuild any version of your project back to day 1.

Re:Version Control

[RAMMS EIN]

And you should know who introduced a certain change, and how they motivated that.

Re:depends on the scope of the project

[LiENUS]

Actually even with 1 developer SVN or CVS are huge benefits, what if that developer introduces a bug? with subversion you can go through and look to see when that bug was introduced and roll it back. You'd be hard pressed to roll it back with just tarballs.

Reproducability

[danpat]

If you're releasing a product to the public, the one word you need to keep in the back of your mind at all times is "reproducability".
Can you, at any point in the future, reproduce whatever version it is that customer X is having trouble with?

There are many ways to do this, ranging from taking complete snapshots of each "build" (requires lots of space, but fast to reproduce), to keeping a short list of the Debian packages installed (not much space, but slower to reproduce). It's a classic space-vs-time tradeoff.

I'd suggest you attempt to automate the system build as much as you can. Use virtualization tools like VMWare to help perform "builds" of your OS images. Most Linux distros have automated install processes. RedHat has "kickstart", Debian has "fai" (http://www.informatik.uni-koeln.de/fai/). At the minimum, you should version control the script you use to build your vmaware images, and the configuration script for fai/kickstart. This should let you re-build everything at some later stage.

When it comes to customising Debian systems, customised Debian packages are the way to go. If you're adding new files, package them up and deploy them as part of the automated install. If you're customising existing packages, edit their source and rebuild them with
customised version numbers, and list those versions of the packages in your fai script. You'll need to go through the whole version control process with each customised package too. (i.e. check it's source into a version control tool, tag it, apply your changes, tag it again, then build your .deb file). You can provide "answers" files for debconf so that no questions are asked during installation, and you can tweak various settings as you go along. If you've taken the VMWare approach, you can always login to the image and make final adjustments (just make sure they're scripted and version controlled) after the Debian install is complete.

Do a search for "customized debian", there are quite a few people doing similar things already.

Basically, make sure that the end product requires nothing more than a button push to produce. Anything less and you'll introduce the risk of someone forgetting to perform a step, or doing it wrong. That'll create a support nightmare down the track.

If you can reproduce easily what your customer has, you can also easily make a minimally invasive fix for them. That'll make them happy :-)

If you're looking for resources on this stuff in general, "configuration mangement" (http://en.wikipedia.org/wiki/Configuration_manage ment) is what you want to search for. Librarianship for software systems, kind of dry and boring, but oh-so-necessary.

Development, or Distribution?

[KillerBob]

For distribution to the end-user, you will definitely want a package of some kind. I'm assuming that your end users won't be able to log in with a prompt, but may have some kind of web-based management, right? If you distribute your upgrades/patches as .deb packages (maybe renamed to .bin since that's what users have been conditionned to expect), then it makes things a whole lot easier... among other things, it would facilitate downloading the upgrade from a location other than where the product is: not all users have Internet connections at home, even in this day and age. You may also want to look into implementing something like Slackpackages, since they ignore dependancy. (They're basically just a tarball... you can install them manually by unzipping/tarring the file from / and then looking in the /install directory and manually executing any scripts there)

For actual development... you're gonna *need* to use SubVersion or CVS. Cover your ass. Also, not having it makes managing a project a royal pain in the ass.

I am not an embedded devices development manager

[Ramses0]

...but if you're using Debian, I would highly recommend that you spend a quality week or two *READING* the wonderful documentation debian has and read / ask a few questions on their mailing lists.

Once you understand the package-management system of the SOFTWARE YOU ARE BASING YOUR BUSINESS OFF OF, the answer to your question will become clear... nay- simple.

  - MyCompanySoftware-1_0.deb, MyCompanyKernel-1_0.deb, MyCompanyOtherStuff-1_0.deb

  - Generous use of depends, requires, conflicts, provides, etc. (or maybe up-rev eg: kernel-image-2.6.8-1.deb to kernel-image-2.6.8-1-MyCompany-1.deb, these are the things you can ask for advice on Debian / Ubuntu lists).

  - Source control all files used in any of those *.deb packages, and make an automated build process that can take your source-control tree and generate your packages at any time of the day or night.

  - Set up internal repositories, ie: http://apt.mycompany.com/stable/ .../testing/ .../nightly/, etc. and integrate that with your testing / deployment infrastucture. ...but most of all, please READ the documentation that Debian has put together. In few words, it allows mostly volunteers in their spare time to do exactly what you are trying to do and with a high degree of reliability. The documentation in Debian Policy is the first stop (and most likely the last) for almost anything you are trying to do. When you see the types of bug-reports that are filed against packages that go against policy (ie: incorrect depends, provides, etc) you will see what types of mistakes are possible, and you should seriously consider how to check the work that you've done to make it more likely that your work would not have the same types of bugs filed against it.

--Robert

Re:depends on the scope of the project

[krumms]

No no no no no. :)

Whether you're working in a team or working by yourself: Use Subversion Anyway. Or svk. Or Darcs. Any reputable revision control system will kick the pants out of any ad-hoc solution you come up with. Revision control should be automatic and easy. The value of being able to easily merge changesets alone is reason enough for any non-trivial project. Keeping track of branches for experimental/delicate changes, tagging releases, LOG MESSAGES for all your changes - all of these things, use them, learn to love them. It's a bitch to get in the habit, but when you do it's absolutely worth it.

It's taken me over seven years to truly learn the worth of version control. These days I'd dare not live without it. It really is that good. Honest!

Our situation

[Basje]

I work for an ASP. We've got a web application (built with Perl), running on Debian. At the moment we've about 15 servers (some dedicated to one large customer, some with over 50 customers) live, have 4 full time developers on the product, 15 people in total, and are quite succesful in our niche.

This is the short version of how we do things.

* We looked for an ISP where we rent the servers. They administer the servers (Debian stable), and install the perl modules, apache, etc. We don't have (want) root access to these machines: the ISP is responsible for the stability, and they do a good job. We ask them for changes/additional perl modules to be installed when needed. We've less than satisfactory experiences with several ISPs, make sure you find a good one.
* For a repository we use cvs. This is flexible enough for our needs, and there was some experience with the app. If you haven't got any experience with cvs, also take a look at subversion or mercurial, as you could benefit from the improvements there.
* As a cvs client we use eclipse. Great product, but unfortunately it is Java, and therefore slow. Some of the developers use the editor of eclipse, others use external editors (vi baby :)
* Our work environment is mixed. We all have a windows workstation, but for the actual development we have a server with a dedicated debian VPS for each developer. We connect to the VPS (which is hosted on our lan, and not accessible externally) through ssh, samba and x. The VPS are UML based, but nowadays when setting things up, we'd probably use Xen. The advantage of using VPSs is that it's easy to set up a clean developement/test environment.
* Have a release cycle, and try to stick with it. Most bugs are introduced when improperly tested code is implemented on live servers. Never edit directly on a live machine.

Our current shortcomings (i.e. pitfalls):
* Hardly any automated testing, and no formal testing procedures. Testing the application takes a lot of work, so it is often skimped. This is a risk, and introduced bugs are occasionnally missed.
* The release policy is not always honored due to deadlines. This puts a strain on the organisation, because, as noted above, it needs to be tested manually. This is when testing is skipped most of the times, and most bugs are introduced. It's a commercial tradeoff: let a customer wait, or take the risk. Depending on who and when you ask you get different answers.

Re:Now that's Insightful

[torpor]

What part of "How do You Manage a Product Based on Linux?" do you not understand?

He's not asking for help .. he's interested in the ways /.'ers are maintaining their linux-based products, perhaps (naively) hoping that the peanut gallery might provide an interesting result. This does not necessarily mean he wants help with his lame system; read closely, and you might realize that Ryan seems quite happy with his approach so far .. but this is still an interesting topic worth objective attention. Its not a screaming/crying/spoiled-brat cry for help that some of the similarly inclined responses have implied, anyway ..

Me, I've been building linux-based systems for my own use since the days of the minix-list (and before that, RISCOS distimages). My current approach is quite simple, old-fashioned, but workable nevertheless. I simply apply the following general guide-lines for sysbuilding: complete source-control (using SVN/whatever-the-package-maintainer-uses), avoid cross-compiling, build everything on-board, one Makefile to tie together whatever components are required (linux-kernel/base-image/sysbins/libs/my_app), 'cscope -R' at the root tree when something needs to be worked out, and set it all up so that you can just type 'make' and watch the bootable .img form .. Fortunately the more you do this, the less you need to worry about package maintenance, but of course if the 'final deliverable' is a simple, plain sysimage containing all software onboard required for your embedded app, then package maintenance isn't such an issue. Its kind of fun to have a "single-image deliverable" too ..