Slashdot Mirror
Lack of Testing Threatening the Stability of Linux
sebFlyte writes "Andrew Morton, a Linux kernel maintainer, has said that he thinks that the lack 'credit or money or anything' given to those people who put in long hours testing Linux releases is going to cause serious problems further down the line. In his speech at Linux.Conf.Au he also waded into the ongoing BitKeeper debate, saying 'If you pick a good technology and the developers are insane, it's all going to come to tears.'"
I thought good technology required insane developers...
Isn't any developer insane?
...does it seem like Linus might need a vacation?
TFA states that he's starting to take as much pride in rejecting patches as he does accepting them, and with this whole BitKeeper thing, it seems to me like he might need a small break.
Of course, I'm not one to really talk, as I don't do nearly as much as he does with Linux...
Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing.
Also, maybe slowing down the kernel releases a bit might help. I know that I do an emerge world on my Gentoo boxes about once a week, and it seems like there's a new kernel release every week. If there's a need for more testing, perhaps a little less time releasing and more time testing is in order.
DBA? Software Engineer? My company is hiring! Click
I know it's early, but do we really have to mod everything flamebait, even if it's hilarious??? Come on..
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
where the testers (a.k.a. users) get to pay $$$ for the privilege of testing OS stability.
just imagine what'll happen if Linux actually makes a dent in the non-geek desktop market, and widespread use by "appliance operators" ensues.
Must be why there is a huge popularity of mugs at MS bearing the obnoxious logo:-
"You don't have to be a developer to work here, but it helps".
Coming from someone who was at that talk, he specifically said NOT to give money to testers. His words were actually 'give them credit, fame and loose women'.
This drew laughs from the audience.
That's the converse. The contrapositive is, after a quick application of de Morgan's law:
"If it doesn't come to tears, then you didn't pick a good technology or your developers are sane."
Osnews had an article a while ago about some of the testing Sun do on Solaris - http://www.osnews.com/story.php?news_id=10178
Testing of Linux might be easier if it contained some automated features for sending crash reports back to a central database. Gathering some basic data on the stack trace, thread states, processes, etc. might help troubleshoot the OS in the context of the wide array of systems, configurations, and usage patterns. I know that both Microsoft and Apple have benefited strongly from this feature. Some tin-foil-hat wearers might object to their box phoning home. Tin foil hatters can just disable the feature but it might mean that the types of bug they experience never get fixed.
If developers are going to fix the bugs that occur in the real world, they need data from the real-world.
Two wrongs don't make a right, but three lefts do.
I thought the point of Open Source was to allow more people to read through the code. You mean thousands of people aren't really doing that for fun? I'm shocked.
More seriously... I think many of the people who DO eyeball the code are looking for security problems these days (where you do get recognition, etc.). For the record, I know I won't get any HR props for putting OS bugs that I've uncovered on my resume, but the security bugs I've found are always good conversation pieces.
"Bugzilla is fine for tracking bugs, but as it's currently set up, it's not very good for resolving bugs."
Hmm... I'd be interested to understand what alternatives to a web-based system he has in mind. Any thoughts?
"This process, where individuals communicate via a Web site, is very bad for the kernel overall."
Someone you trust is one of us.
I thought the whole Fedora project WAS mass testing of "cutting edge technology for Linux". Have I been wasting my time submitting bugs? Most have been fixed that I submitted so far.
Two roads diverged in a wood, and I - I took the one the bus load of girls just went down.
Long answer; kinda. You can use core dumps and system logs to interpret what's going on, but you can never really know for sure. Besides, the kind of errors that are in the kernel are the kinds of errors that really don't return error codes; they're the kind that crash the computer and make you reboot.
Microsoft's method is for some of the higher up software, and so is Apple's. If there's a bug in the kernel it's very unlikely that their code will catch it. Or at least that's been my experience.
If the problem is that Linux is so buggy, we just need to run it on a bunch more machines, and start randomly poking it as hard as we can until we break something. Once we've broken it, do it again to make sure it's not hardware, and then go to work fixing it. Good old brute force repairs.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
I knew you visited Slashdot. Why not sign up for an account?
Morton is correct.
Even at commercial companies, QA isn't a "sexy" task. People would rather bang out code than write testing harnesses and run benchmarks.
Also, free software is driven by programmers, who tend to hate QA. Like any artist or craftsman, a programmer hates having their work critiqued. They spent hundreds (or thousands) of hours on a program, only to have someone nit-pick the details and point out the flaws. But for art, "quality" is a subjective quality -- and with software, quality and reliability are tangible quantities that can be measured.
My Acovea project demonstrated the problem. Users of GCC love Acovea; many developers of GCC, on the other hand, seem to treat it is an annoying distraction. Acovea identified more than a dozen errors (some quite serious) in the GCC 3.4/4.0 compilers -- and yes, I did report them to bugzilla. Only a couple of GCC's maintainers have said "thanks."
Not that the cool reception deters me. I have a new version of Acovea in the wings, and will be unleashing it on GCC 4.x Real Soon Now. ;)
As a consultant, I've been paid to perform QA work on commercial software packages -- but only one company, and a big one at that, has ever contracted me to QA a free software project.
Right now, free software is about many things, but quality is not job 1. And that needs to change.
All about me
I have successfully used Test Driven Development in several of my projects and it is a uniquely satisfying experience. Writing test cases before writing the code then completing each test case one after another in steady progression gives a constant stream of small victories. It also means you can run all test cases at a later time and see that "yep, everything still works" or "doh! that change just broke 10 things I already had working."
There are several other benefits to writing tests first as well. The experts in the link above explain it all better than I could, I'm sure.
Many open source projects are taking this approach already and usually boast the number of unit tests along with the lines of code included in the distribution. Anyone can type in "build test" for example and it will show the program run and pass some odd thousand tests.
Is it time for the Kernel to embrace this methodology? I certainly think it is a genuine best practice. But is it applicable to OS development as well? I don't see any reason why it wouldn't be, but I am not a kernel developer myself.
I remember in the early days there was a program called 'crashme' that threw randomly-generated executables at the system, and it was credited bolstering stability. Do tests like this still hapen frequently by the unappreciated? Is there a good place online to read about these tests and their results for different point-releases? Along similar lines, I recall someone throwing random input at the various gnu utilities, and it was discovered that they were more robust against this sort of abuse than the commercial unix equivalents. Are there any other interesting tests that anybody knows about? Breaking stuff is fun.
The "cue the foo posts in 3, 2, 1..." posts will commence with no subsequent foo posts in 3, 2, 1...
It may not be a Linux issue per se (more of a distro issue, I think), and it's purely anecodatal, but I've been seeing some QA problems lately in the mainstream distro I use. They include a bug that requires me to hand-edit the X11 config file to get my mouse to work, having to manually rebuild the routing table after every boot, and a so-far baffling total freeze of the system after rand() hours, only when it's serving web pages. I've been using Linux to do this job for six years, and never had these kinds of problems before.
http://alternatives.rzero.com/
When I heard that I nearly fell off my ostrich.
Linux is constantrly improving, but that means it is also constantly changing, and that makes it a constantly moving target.
That applies to most distros as well as the kernel itself.
It's hard to put a lot of effort into testing something when it's possible those tests will be invalidated a few months down the road...
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
If anybody reading this is interested in participating in the test procedure, check out the Linux Test Project.
My lame blog.
Software testing (usually) isn't monkeys pounding on keyboards until the box BSOD's.
:)
:). I remember dling the Gaim tarball (this was loooong ago) and seeing about getting it built on my SGI machine. IIRC, there were some makefile / #include problems getting it to even build, and once it was built there were some other issues with its runtime. Ultimately i submitted a patch to the gaim folks that more or less "enabled" gaim on IRIX. There is no way anybody had ever used Gaim on an SGI without making these fixes, so it seems reasonable to suggest the authors had never tried it before. This lack of a platform test matrix is pretty common amongst smaller F/OSS apps, even when they say "works on *nix" they mean "works on the distribution of linux i run at home".
It is difficult to test software without adequately understanding what it is supposed to do. Varying the underlying machine type is almost irrelevant for binary distributed software unless you're testing an operating system kernel or looking for race conditions in software (which is really just a stab in the dark)
How are you going to have 3rd party people debug software they know nothing about?
Where users help find bugs is by using the software. It honestly takes a certain mentality to be an effective software breaker, and it's not very common. It takes something else entirely to be a software tester; you've got to be a good developer (because software testing is about automation these days unless you're insane) but you've got to not get sucked into the developers way of thinking.
I assure you - letting normal users play with software doesn't clean it up. we can show that this is true in the following way:
- more users use Microsoft software for more hours a day than any other software in the world
- slashdotters say Microsoft software is the buggest software made
clearly if users using software was sufficient to find all the bugs, MS stuff would be bug free, based on its frequency of use alone. I know this isn't the case, because im a software tester at Microsoft.
(The appropriate response is "well then, stop posting and get back to work; you're clearly not done yet!"
W.r.t. linux kernel testing: this is something that's always amazed me - linux works surprisingly well for something with so little formal testing. On the other hand, when there are edge case problems my experience has been that nobody is much interested in fixing them. One example i had was at a consulting gig. the client was looking to move his web hosting business onto linux boxes if he could get more sites per box then he could on windows. He had a problem where his linux server would start dying after a few days. I started to look into it and the box would basically panic() in low memory situations. I asked Alan Cox about it (via irc) and the response was "buy more memory". Nice.
Another sore point with me growing up was xserver crashes. The Xserver was 99% reliable, but then you'd get some random crash and lose everything you were doing, and you knew there was no real way of getting it fixed or investigating it.. you just had to hope it magically got better somehow.. maybe when you switched hardware or something.
Then there's the just plain lack of testing of some F/OSS projects in general. When i was in college i had NeXT, Sun, and SGI boxes in my dorm room (but no linux
Another baby patch i submitted was for the openBSD kernel.. this time for the wdc driver. Back when UDMA 100 was newish, i bought 2 UDMA 100 disks a month or so apart.. so they were different sizes and different vendors, but on the same bus. The UDMA rollback code in openBSD would drop the DMA level from 5 (UDMA100) to 2 (something much slower, i dont remember what) after a certain number of DMA errors. This obviously sucked since you can run UDMA devices at different speeds on the same bus, and you can also fallback to UDMA66 and UDMA33, both of which are better than mode 2.
My opinions are my own, and do not necessarily represent those of my employer.
ing," he said.
Wait a minute here...
I thought the whole scheme was structured thusly...
I crank up the latest greatest kernel. I find a bug. I report it. My bug gets fixed. THAT's MY REWARD! The friggin bug gets squashed. What more could one ask for, with a clear conscience and a straight face.
As for those guys who fix the stuff. Well sanity is a relative term as we should all realize in light of the Japanese influence and emergence of cargo cults in WW-2 Niu Guinea. AFAIK, most Linux users view the kernel developers as some mysterious force from which benefit is derived through clever creation of effigy's.
BTW, we have not one, but two of my colleagues down under right now listening to Andrew in person. It should be interesting to get a first-hand account of what was said.
- Necron69
I was under the impression that by using Linux, I was, in a sense, testing Linux.
Whenever you read this sig someone's refrigerator light turns on.
that is NOT Microsoft's approach to testing.
;) These really dont happen that often during the product cycle, because ad-hoc testing doesn't catch that much stuff if you've got well developed automation suites. However, it's still very worthwhile because it is a good feedback mechanism to explain why your other testing missed something, and it's the best way to notice the odd "that's funny..." sort of issues that are not functinoally incorrect but are still user annoyance type issues.
Where did you hear or get the impression that that was the MS "approach" to QA ?
I've written test suites for the following Microsoft Products
- Visual Basic Compiler, 7.0
- Microsoft Business Framework 1.0 (unreleased)
None of them involved just using the compiler or the business framework over and over in day to day work to find bugs.
We have a variety of test approaches, including a few that _might_ be construed as what you describe - There are a few ways that we get test coverage via product usage
- stress
- bug bashes
- app weeks
Stress is funnier than it sounds. Did you know we're not allowed to ship windows until the exact build of windows under ship consideration has been running on hundreds (thousands, usually) of machines continuously with no problems while enlisted in a distributed "stress" client... where they're pounded and pounded with automated tests that do things like starve memory whilst performing other work, etc? Same with ASP.NET and the CLR - they have to _survive_ for a pre-determined time period before the build can be considered shippable. We dont think there are any show-stopper bugs at this point - but we just want to be reasonably sure. Note that if we find a bug (even an unrelated one, like the documentation has a typo) and take a fix for it, the stress cycle resets because the bits have changed. Better safe than sorry. In the end game of a product release it can literally be the case that taking a bug fix means delaying ship for another week or more.
- bug bashes
this is probably most like what you're describing. Everyone on the team sits down for a couple of days and really just beats on a specific area of the product. Security Bug Bashes have become popular int he last couple years (wonder why
- app week
For developer tool products (like Microsoft Business Framework) we like to do an app week with each milestone, where everyone on the team builds some sort of end to end application, using as much of the toolchain as possible. This sort of testing really makes the employees better (we're usually pretty compartmentalized on our areas of functionality ownership). It also lets unreleated parties take a look at peices of the product they don't own (so don't have preconceived notinos about). Finally, it lets us simulate the end-to-end customer experience on our product stack. If we can build the sort of apps a customer might build with our tools, then the tools are probably alright. Where we run into problems, we know the tools need help.
bug bashes and app weeeks happen perhaps 1-2 weeks per milestone (which is on the order of 2 months). It is a small part of our testing, time, effort, and results wise. It's still important to do, but it is not the _focus_ of QA at microsoft.
My opinions are my own, and do not necessarily represent those of my employer.
If the average Linux user were educated on how to recognize a bug, and file a meaningful bug report it would mean a lot to developers, and likely speed up development and stability. ...and scare away 99.9% of potential new users.
Game! - Where the stick is mightier than the sword!
From TFA:
"A lack of commitment to testing by the Linux community may ultimately threaten the stability..."
The content of the article is much better than the headlines and excerpts being quoted. I was there and felt that what he was geting at was that we need to start thinking about updating QA procedures. The ratio of bugs to features is decreasing, but the rate of features is (maybe?) growing that much faster. The point of his talk was to outline a number of options for improving QA, thre are issues, but the sky certinly isn't falling either. It was an excellent follow on from Tridge's keynote the previous day on how to do quality system programming (overshadowed by his very brief coverage of the BK thing).
Xix.
"Everything is adjustable, provided you have the right tools"
This article is about kernel development. While I appreciate the development being done to make the kernel faster/better/cheaper (well, it doesn't get any cheaper), it's already a Pretty Damn Good kernel. It sounds to me like the most crucial thing would be to solidify it and test the bejeezus out of it, then largely freeze it, because that's not where the problems are.
When people complain about MS Windows, they're not (usually) complaining about the kernel. They're talking about all of the stuff built on top of it: window manager, IE, networking, configuration. If the Linux kernel is receiving too little testing to be stable, what about the millions of lines of code that go into X windows, Gnome, CUPS (as mentioned the other day), etc.
If MS didn't have to make kernel changes to bettter support security, I suspect they wouldn't be touching it at all. BSODs are still more common than they should be, but most users find them extremely rare, and the kernel is Fast Enough relative to the work that needs to be done. The improvements in Longhorn are largely about changes above the kernel, especially in its spiffy interface.
While I'm grateful to Linus and all of the other developers for the kernel improvements, and while Open Source means never being told what to work on, kernel improvements other than stability are probably a terrible use of manpower. The kernel is a tiny fraction of the lines of code that go into a Linux distro. They are basic, and need to be rock-solid, but while performance improvements there benefit everybody, they don't benefit you at all if X, or KDE, or Konqueror, or any of the hundreds of other higher-level apps crash.
Then, there's the mysterious Stanford Code Validator, used to great effect for a while. I feel certain that a few sweeps of that would uncover many of the more troublesome problems.
For those without SCV (99.9999% of the planet), there are some Open Source code validators out there. It should be possible, at the very least, to use those to identify the more blatant problems.
If you're not sure about using code validators, then it's simple enough to write programs that hammer some section of the kernel. For example, if you have some large number of threads mallocing, filling and freeing random-sized blocks of memory, can you demonstrate memory leaks? How well does the VMM handle fragmented memory? What is the average performance like, as a function of the number of threads?
Likewise, you can write disk-hammering tools, ethernet tests, etc. For the network code, for example, what is the typical latency added by the various optional layers? Those interested in network QoS would undoubtably find it valuable to know the penalties added by things like CBQ, WFQ, RED, etc. Those developing those parts of the code would likely find the numbers valuable, too.
If you don't want to write code, but have a spare machine that isn't doing anything, then throw on a copy of Linux and run Linpack or the HPC Challenge software. (Both are listed on Freshmeat.) The tests will give kernel developers at least some useful information to work with.
If you'd rather not spend the time, but want to do something, map a kernel. There's software for turning any source tree into a circular map, showing the connections within the program. If we had a good set of maps, showing graphically the differences between kernel versions (eg: 2.6.1 through to 2.6.12-pre3) and between kernel variants (eg: standard tree, the -ac version and the -mm version), it would be possible to get a feel for where problems are likely. (Bugs are most likely in knotty code, overly-complex code, etc. Latency is most likely in over-simplified code.) You don't have to do anything, beyond fetch the program, run it over the kernels, and post the images produced somewhere.
None of this is difficult. Those bits that are time-consuming are often mostly time-consuming for the computer - the individual usually doesn't need to put in that much effort. None of this will fix everything, but all of it will be usable in some way to narrow down where the problems really lie.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
What are they expecting? It's based on voluntary work.
If anybody needs some guaranteed service, or commercial-grade testing, maybe they should hire some programmers to do it?
I was there, and the quote was taken _absolutely_ out of context: 'If you pick a good technology and the developers are insane, it's all going to come to tears.' He was not refering to BK in this instance; he was in fact talking more generally about SCM systems, and how he had noticed that these projects tended to attract "insane" developers (also the ide drivers do this too).
This was all part of a larger, very insightful remark, saying that had Linus chosen a free SCM tool three years ago, we would now have a fantastic SCM in the free software world. In this instance, it is not so much the _tool_ that would need to be good, but that the _team_ behind the tool needs to be solid, responsive etc.
Simon.