How Do You Manage a Product Based on Linux?

Ryan writes "Following my advice, my company has decided to base it's new appliance on Linux. So far, it's worked out great. Linux gave us a huge jumpstart on development because of it's open nature and the information we've garnered from public mailing lists. We've added software, modified startup files, and have built our own kernel. Now the question is: How do you manage it all? Do you put it all in CVS or Subversion? Do you use the distro's packaging system (we're using Debian)? What does your build system look like?"

Comments

I am not an embedded devices development manager

[Ramses0]

...but if you're using Debian, I would highly recommend that you spend a quality week or two *READING* the wonderful documentation debian has and read / ask a few questions on their mailing lists.

Once you understand the package-management system of the SOFTWARE YOU ARE BASING YOUR BUSINESS OFF OF, the answer to your question will become clear... nay- simple.

  - MyCompanySoftware-1_0.deb, MyCompanyKernel-1_0.deb, MyCompanyOtherStuff-1_0.deb

  - Generous use of depends, requires, conflicts, provides, etc. (or maybe up-rev eg: kernel-image-2.6.8-1.deb to kernel-image-2.6.8-1-MyCompany-1.deb, these are the things you can ask for advice on Debian / Ubuntu lists).

  - Source control all files used in any of those *.deb packages, and make an automated build process that can take your source-control tree and generate your packages at any time of the day or night.

  - Set up internal repositories, ie: http://apt.mycompany.com/stable/ .../testing/ .../nightly/, etc. and integrate that with your testing / deployment infrastucture. ...but most of all, please READ the documentation that Debian has put together. In few words, it allows mostly volunteers in their spare time to do exactly what you are trying to do and with a high degree of reliability. The documentation in Debian Policy is the first stop (and most likely the last) for almost anything you are trying to do. When you see the types of bug-reports that are filed against packages that go against policy (ie: incorrect depends, provides, etc) you will see what types of mistakes are possible, and you should seriously consider how to check the work that you've done to make it more likely that your work would not have the same types of bugs filed against it.

--Robert

Our situation

[Basje]

I work for an ASP. We've got a web application (built with Perl), running on Debian. At the moment we've about 15 servers (some dedicated to one large customer, some with over 50 customers) live, have 4 full time developers on the product, 15 people in total, and are quite succesful in our niche.

This is the short version of how we do things.

* We looked for an ISP where we rent the servers. They administer the servers (Debian stable), and install the perl modules, apache, etc. We don't have (want) root access to these machines: the ISP is responsible for the stability, and they do a good job. We ask them for changes/additional perl modules to be installed when needed. We've less than satisfactory experiences with several ISPs, make sure you find a good one.
* For a repository we use cvs. This is flexible enough for our needs, and there was some experience with the app. If you haven't got any experience with cvs, also take a look at subversion or mercurial, as you could benefit from the improvements there.
* As a cvs client we use eclipse. Great product, but unfortunately it is Java, and therefore slow. Some of the developers use the editor of eclipse, others use external editors (vi baby :)
* Our work environment is mixed. We all have a windows workstation, but for the actual development we have a server with a dedicated debian VPS for each developer. We connect to the VPS (which is hosted on our lan, and not accessible externally) through ssh, samba and x. The VPS are UML based, but nowadays when setting things up, we'd probably use Xen. The advantage of using VPSs is that it's easy to set up a clean developement/test environment.
* Have a release cycle, and try to stick with it. Most bugs are introduced when improperly tested code is implemented on live servers. Never edit directly on a live machine.

Our current shortcomings (i.e. pitfalls):
* Hardly any automated testing, and no formal testing procedures. Testing the application takes a lot of work, so it is often skimped. This is a risk, and introduced bugs are occasionnally missed.
* The release policy is not always honored due to deadlines. This puts a strain on the organisation, because, as noted above, it needs to be tested manually. This is when testing is skipped most of the times, and most bugs are introduced. It's a commercial tradeoff: let a customer wait, or take the risk. Depending on who and when you ask you get different answers.