Slashdot Mirror
Lack of Testing Threatening the Stability of Linux
sebFlyte writes "Andrew Morton, a Linux kernel maintainer, has said that he thinks that the lack 'credit or money or anything' given to those people who put in long hours testing Linux releases is going to cause serious problems further down the line. In his speech at Linux.Conf.Au he also waded into the ongoing BitKeeper debate, saying 'If you pick a good technology and the developers are insane, it's all going to come to tears.'"
I'm really surprised no company really has used this as a business model.
I think it'd be awesome to run a software debugging/testing firm, where basically you have a bunch of computers and a bunch of users come in and try their best to break the software. Cheap labor and a good variety in machines, and you could quite quickly clean up even some of the nastiest code.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
I don't think that is the issue at hand. The issue is the way he is behaving in public. The flames, the "fuck off" attitude towards people working on the kernel, etc...
The kernel did not get where it is with his current attitude.
As I said, the pressure can get to anyone and the kernel is now a mighty beast of a project to maintain. He just needs to get his head screwed on straight. Either that or risk turning into another Theo.
I have successfully used Test Driven Development in several of my projects and it is a uniquely satisfying experience. Writing test cases before writing the code then completing each test case one after another in steady progression gives a constant stream of small victories. It also means you can run all test cases at a later time and see that "yep, everything still works" or "doh! that change just broke 10 things I already had working."
There are several other benefits to writing tests first as well. The experts in the link above explain it all better than I could, I'm sure.
Many open source projects are taking this approach already and usually boast the number of unit tests along with the lines of code included in the distribution. Anyone can type in "build test" for example and it will show the program run and pass some odd thousand tests.
Is it time for the Kernel to embrace this methodology? I certainly think it is a genuine best practice. But is it applicable to OS development as well? I don't see any reason why it wouldn't be, but I am not a kernel developer myself.
BTW, we have not one, but two of my colleagues down under right now listening to Andrew in person. It should be interesting to get a first-hand account of what was said.
- Necron69
that is NOT Microsoft's approach to testing.
;) These really dont happen that often during the product cycle, because ad-hoc testing doesn't catch that much stuff if you've got well developed automation suites. However, it's still very worthwhile because it is a good feedback mechanism to explain why your other testing missed something, and it's the best way to notice the odd "that's funny..." sort of issues that are not functinoally incorrect but are still user annoyance type issues.
Where did you hear or get the impression that that was the MS "approach" to QA ?
I've written test suites for the following Microsoft Products
- Visual Basic Compiler, 7.0
- Microsoft Business Framework 1.0 (unreleased)
None of them involved just using the compiler or the business framework over and over in day to day work to find bugs.
We have a variety of test approaches, including a few that _might_ be construed as what you describe - There are a few ways that we get test coverage via product usage
- stress
- bug bashes
- app weeks
Stress is funnier than it sounds. Did you know we're not allowed to ship windows until the exact build of windows under ship consideration has been running on hundreds (thousands, usually) of machines continuously with no problems while enlisted in a distributed "stress" client... where they're pounded and pounded with automated tests that do things like starve memory whilst performing other work, etc? Same with ASP.NET and the CLR - they have to _survive_ for a pre-determined time period before the build can be considered shippable. We dont think there are any show-stopper bugs at this point - but we just want to be reasonably sure. Note that if we find a bug (even an unrelated one, like the documentation has a typo) and take a fix for it, the stress cycle resets because the bits have changed. Better safe than sorry. In the end game of a product release it can literally be the case that taking a bug fix means delaying ship for another week or more.
- bug bashes
this is probably most like what you're describing. Everyone on the team sits down for a couple of days and really just beats on a specific area of the product. Security Bug Bashes have become popular int he last couple years (wonder why
- app week
For developer tool products (like Microsoft Business Framework) we like to do an app week with each milestone, where everyone on the team builds some sort of end to end application, using as much of the toolchain as possible. This sort of testing really makes the employees better (we're usually pretty compartmentalized on our areas of functionality ownership). It also lets unreleated parties take a look at peices of the product they don't own (so don't have preconceived notinos about). Finally, it lets us simulate the end-to-end customer experience on our product stack. If we can build the sort of apps a customer might build with our tools, then the tools are probably alright. Where we run into problems, we know the tools need help.
bug bashes and app weeeks happen perhaps 1-2 weeks per milestone (which is on the order of 2 months). It is a small part of our testing, time, effort, and results wise. It's still important to do, but it is not the _focus_ of QA at microsoft.
My opinions are my own, and do not necessarily represent those of my employer.
Well, that would really be a problem, but despite Theo's personality (which I think might have its own charms) doesn't necessarily get in the way of development. Just think of the huge contributions OpenBSD made. Common Address Redundancy Protocol (CARP) for instance. Or their excellent firewall, pf (now present in all BSDs). Not to mention OpenSSH. And beside these standalone or highly portable applications, they released a secure and stable OS. Not 'just' a kernel. They write their own libc. They maintain a lot of software in their base system. Apache 1.3.x can be almost considered a fork, with their security/stability related patchset. Which comes down to my main point: The problem is not lack of resources, monetary or otherwise
Currently there are ~100 developers payed fulltime just to work on the kernel (at various organizations). There are none in FreeBSD. There are perhaps a dozen devs whose employers let them work on FreeBSD part-time, or there are various works that are sponsored by companies (pair network comes to mind) from time-to-time. But all in all, FreeBSD, that writes its own kernel, its own C library, and generally speaking maintains an OS (userland apps like their package management and ports system for instance, burncd - the native cd burning app of freebsd, etc.) does that with 1/50 of the resource Linux & co has just to develop the kernel.
This is not about linux vs. freebsd btw. I chose to use the latter, you chose the former, I really don't care, and I'm not willing to engage in yet another linux vs. bsd flamefest. You can argue endlessly about why linux is better, and I can do the same about FreeBSD, but I think we can agree on one point: either way, neither is that much better (lets cut down that figure to 10x - you can't possibly claim that linux is 10x better or something). In other words, my point is that it is not about (monetary) resources. It is a problem of organization imho. Less frequent releases, more API/ABI stability, a controlled release engeneering process might be a solution. Perhaps a branch split like it was done during 2.5.x (current 2.6) development. Pronounce the current 2.6.x branch STABLE, meaning introducing a POLA (policy of least astonishment in freebsd) and forbid API/ABI changes, then continue development in a new, 2.7 branch at the current pace.
I don't mean to imply that there is no release engeneering in linux kernel development whatsoever. But somehow FreeBSD's (and I assume the other BSD's as well) release engeneering seems to me a lot more transparent. Click the first few links at the top of this page to see what I mean by "controlled release engeneering process."
Actually, I'd say that giving proper credit and public recognition for bug reports is good enough for most of the end-users. Case in point (interestingly enough, from LKML and by Andrew Morton himself):
Getting an answer like that should lift anyone's spirits. Not only has the bug been fixed, it was also recorded for posterity that a certain user discovered it and helped to his ability in fixing it. And to top it all off, the reporter was given an honest praise and a thank you. The last part alone is usually enough for most users, to see that the developers actually care.
As for resumes? If you have a verifiable record of reporting back bugs and helping to test their fixes, you should be able to use that for your advantage in CV or at least in an interview. If nothing more, it shows that you can communicate with different kinds of people and have enough technical ability to follow through with their requests for further details. You might have even gotten a better product for yourself to use.
There is no such thing as good luck. There is only misfortune and its occasional absence.