Slashdot Mirror
How Do You Manage a Product Based on Linux?
Ryan writes "Following my advice, my company has decided to base it's new appliance on Linux. So far, it's worked out great. Linux gave us a huge jumpstart on development because of it's open nature and the information we've garnered from public mailing lists. We've added software, modified startup files, and have built our own kernel. Now the question is: How do you manage it all? Do you put it all in CVS or Subversion? Do you use the distro's packaging system (we're using Debian)? What does your build system look like?"
...but if you're using Debian, I would highly recommend that you spend a quality week or two *READING* the wonderful documentation debian has and read / ask a few questions on their mailing lists.
.../testing/ .../nightly/, etc. and integrate that with your testing / deployment infrastucture. ...but most of all, please READ the documentation that Debian has put together. In few words, it allows mostly volunteers in their spare time to do exactly what you are trying to do and with a high degree of reliability. The documentation in Debian Policy is the first stop (and most likely the last) for almost anything you are trying to do. When you see the types of bug-reports that are filed against packages that go against policy (ie: incorrect depends, provides, etc) you will see what types of mistakes are possible, and you should seriously consider how to check the work that you've done to make it more likely that your work would not have the same types of bugs filed against it.
Once you understand the package-management system of the SOFTWARE YOU ARE BASING YOUR BUSINESS OFF OF, the answer to your question will become clear... nay- simple.
- MyCompanySoftware-1_0.deb, MyCompanyKernel-1_0.deb, MyCompanyOtherStuff-1_0.deb
- Generous use of depends, requires, conflicts, provides, etc. (or maybe up-rev eg: kernel-image-2.6.8-1.deb to kernel-image-2.6.8-1-MyCompany-1.deb, these are the things you can ask for advice on Debian / Ubuntu lists).
- Source control all files used in any of those *.deb packages, and make an automated build process that can take your source-control tree and generate your packages at any time of the day or night.
- Set up internal repositories, ie: http://apt.mycompany.com/stable/
--Robert
I don't work for them and haven't used their product much, but there's a company called R-Path (founded by former Red Hat early employees) that seems like its designed for "appliances" just like yours.
The idea is that you build your platform on thier system, then you add your programs on top. The system merges updates from them and your system and places it onto the target system. The system they've built is called Conary. Conary itself is Free Software, but RPath sells services along with it that seem attractive.
It looks very well put together and if I were looking at building an appliance, it's certainly something I'd be considering.
http://www.rpath.com
I work for an ASP. We've got a web application (built with Perl), running on Debian. At the moment we've about 15 servers (some dedicated to one large customer, some with over 50 customers) live, have 4 full time developers on the product, 15 people in total, and are quite succesful in our niche.
:)
This is the short version of how we do things.
* We looked for an ISP where we rent the servers. They administer the servers (Debian stable), and install the perl modules, apache, etc. We don't have (want) root access to these machines: the ISP is responsible for the stability, and they do a good job. We ask them for changes/additional perl modules to be installed when needed. We've less than satisfactory experiences with several ISPs, make sure you find a good one.
* For a repository we use cvs. This is flexible enough for our needs, and there was some experience with the app. If you haven't got any experience with cvs, also take a look at subversion or mercurial, as you could benefit from the improvements there.
* As a cvs client we use eclipse. Great product, but unfortunately it is Java, and therefore slow. Some of the developers use the editor of eclipse, others use external editors (vi baby
* Our work environment is mixed. We all have a windows workstation, but for the actual development we have a server with a dedicated debian VPS for each developer. We connect to the VPS (which is hosted on our lan, and not accessible externally) through ssh, samba and x. The VPS are UML based, but nowadays when setting things up, we'd probably use Xen. The advantage of using VPSs is that it's easy to set up a clean developement/test environment.
* Have a release cycle, and try to stick with it. Most bugs are introduced when improperly tested code is implemented on live servers. Never edit directly on a live machine.
Our current shortcomings (i.e. pitfalls):
* Hardly any automated testing, and no formal testing procedures. Testing the application takes a lot of work, so it is often skimped. This is a risk, and introduced bugs are occasionnally missed.
* The release policy is not always honored due to deadlines. This puts a strain on the organisation, because, as noted above, it needs to be tested manually. This is when testing is skipped most of the times, and most bugs are introduced. It's a commercial tradeoff: let a customer wait, or take the risk. Depending on who and when you ask you get different answers.
the pun is mightier than the sword
A round of applause...
While I understand your spleen and I even applaud it, there are a few things which you are missing.
In many settings the initial people with the idea are not capable of software process management (and nearly always incapable of support process management). On top of that the people who they initially hire are usually of the "implement at any cost breaking all rules". That works great up to a prototype and sometimes even slightly later. In fact this "break all rules" culture is a model which most successfull startups in the industry have followed.
After that, once the prototype is up and working and the initial euforia has settled in, the company comes to a point where it needs to grow up. The initial people who have ideas and who are of the "implement at any cost" variety must either move into positions where they do not prolong the company growing pains or leave. The company also needs to hire people (or less likely - find within its ranks) who are capable of long term software development process management. These are the people who must become the team leaders and managers in order for the company to be successful.
Unfortunately, very few companies make it through this stage. Most promote the "implement by any means necessary" or the "initial idea" people into positions where they cannot cope as their mentality is incompatible with the actual requirements. They become increasingly frustrated with the mere fact that there is a process, break it all the times and explain to people who follow it that they are tails that wag the dog. In addition to that they blow any long term planning to hell and gone all the time and push the company into an endless spiral of firefighting crisis management. This all becomes a big mess and it all goes to hell sooner or later.
Anyway, it is not so uncommon for a successful initial stage startup to have no culture of software development process and especially customer support process. In fact that is to be expected, as to get through the initial hurdles you sometimes need to walk on the dead bodies of the rules and processes that have been broken.
Frankly, the ask-slashdotter should be applauded that they have realised that they lack this process. Now he will definitely do not like the answer to his question. It is very simple: hire someone who does and swallow the fact that you will hate him to the point where some of the veterans may have to leave. Tools like subversion, clearcase, CVS, MKS, Bugzilla do not make a process. They implement a process defined by a human. What is needed is the person who has defined the process to have a long term view as well as a view of how the process fits with support, business and the rest of the company. If he does not - no tool will help. There is nothing worse then short-termism in process definition and tool selection forced by short-termism. It will come to bite you in the arse again, and again, and again.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Does your PXE process automatically partition/format the disk with the OS?
.tar.gz of each filesystem.
I used PXE boot on Linux a few years ago with great success, but when I was considering doing an appliance-type solution, I created a customized system rescue CD which included a
This would have allowed me to script the partitioning process, as well as the extraction of the filesystems to end up with a bootable CD which would create an appliance hands-free. (At least that is what I was on target to complete when the project was cancelled by management.) Our original goal was to be able to distribute the media to a remote location and have unskilled people create applicances from commodity hardware.
Theoretically the same thing could be done with a PXE and a boot menu. Is that what you've done?
Regards,
Anomaly
But Herr Heisenberg, how does the electron know when I'm looking?
(to the parent)
Agreed, reproducability is key. Not only for your customers, but just for releases in general (your QA guys love you if you can deliver builds to them and actually know which build you gave them by the time they're done testing it). Also, having stuff wrapped in deb (or, I suppose rpm) packages is really nice too, as it makes field-upgrades a snap (even on embedded appliances, provided you have a deliver mechanism), and again, your QA guys will love you if they don't have to wait for a whole new build to test out a fix; just pass them a handful of debs that you rolled on your own dev machine (source deltas checked in, of course).
(to the OP)
Just make sure all of your build constituents are under version control (subversion is *fine*, really... you probably want to stay away from CVS, they are different, trust me). That should include all source files, all config files, all make files, all make automation scripts, and even any notes you kept while setting the damn thing up. The worst problem I've seen in source control/build automation is magic files that do magic things (not so much fun when hardware fails). Even if it requires some manual process to deploy some of the build scripts from the source tree to a build slave, at least there is a central point from which to collaboratively edit them.
FAI is bad news, big time. It's complicated and frustrating. If you want to customise and deploy Debian on servers, then it's probably what you want, however. FAI is really good for a server or desktop deployment environment, like, for example, if you had a standard corporate server platform, or standard corporate desktop. You use your FAI rig to clone out machines, and when you upgrade certain debs, you just remotely command your already deployed machines to 'apt-get upgrade' (or better yet, feed 'apt-get install' a programmatically-tailored & delivered list of packages). You can use 'apt-get dist-upgrade' too, but you'd really need a deep understanding of the dependency tree and you'd need to make a new 'distro' release so that apt knows what it's upgrading too (not to mention field-updating the sources.list). It's a non-trivial problem, but the tools will do it, and once it's set up, you have a single point of maintainance and control for the machines you're managing.
For the poster's particular appliance, something like FAI or "kickstart" may not be what you need. In my last position (also embedded Debian, custom kernel, custom glibc, etc) installation wasn't simply slipping a FAI cd into the cdrom drive (there *was* no cdrom drive... embedded appliance, remember?). We actually had a flash device and had our own flash burn tool. In addition, our device had custom FPGA bits (on Xilinx chips), so we actually built the tools themselves out of source control, as well as the FPGA bits, and then used the tools to write the filesystem to the flash via the bootloader we wrote into the FPGA code.
We built our install tree using debootstrap and a custom package list (which I think FAI would have used anyway, IIRC), and then trucked the debootstrap'ed directory tree on to the flash using a binary of the tool that was built. So, our build actually consisted of more than just the flash filesystem, it was a snapshot of all the associated tools as well. I definitely suggest investing in both source control and a RAID machine big enough to store snapshots (svn branching and tagging is the de facto way to associate a particular build with it's source files, frozen in time). That way (and I promise this will happen) when you are nearing a release, and the feature list just grew, and you can't quite get there by your delivery date, you can backport your last few weeks of bugfixes to the release branch (which you've already tagged), from which you can produce a release build.
Certainly store any release snapshots, but I'd suggest also keeping any snapshots you pass on to your QA department. If you don't have a QA dept., then you should choose an approximate time interval and keep the b
When I grow up, I want to have Christopher Walken hair.
Why didn't you just say "STFU and RTFM!!!!!!!!1" ('1' intended) and get back to your <sarcasm>thrilling</sarcasm> life? People come here, ask a serious question that's troubling them, and once they make it past the editorial interest filter, they get this bull. This isn't just one more stupid forum, this is Slashdot.
What examples can you provide of these "some issues" you're talking about? The asker is trying to understand the whole process, and you're just telling him to expect trouble? What a worthless comment!