Slashdot Mirror
Lack of Testing Threatening the Stability of Linux
sebFlyte writes "Andrew Morton, a Linux kernel maintainer, has said that he thinks that the lack 'credit or money or anything' given to those people who put in long hours testing Linux releases is going to cause serious problems further down the line. In his speech at Linux.Conf.Au he also waded into the ongoing BitKeeper debate, saying 'If you pick a good technology and the developers are insane, it's all going to come to tears.'"
I thought good technology required insane developers...
Isn't any developer insane?
'If you pick a good technology and the developers are insane, it's all going to come to tears.'"
Too right; Bitkeeper is developed by McVoy. That should have been warning enough.
...does it seem like Linus might need a vacation?
TFA states that he's starting to take as much pride in rejecting patches as he does accepting them, and with this whole BitKeeper thing, it seems to me like he might need a small break.
Of course, I'm not one to really talk, as I don't do nearly as much as he does with Linux...
Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing.
Also, maybe slowing down the kernel releases a bit might help. I know that I do an emerge world on my Gentoo boxes about once a week, and it seems like there's a new kernel release every week. If there's a need for more testing, perhaps a little less time releasing and more time testing is in order.
DBA? Software Engineer? My company is hiring! Click
I know it's early, but do we really have to mod everything flamebait, even if it's hilarious??? Come on..
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
"If you pick a good technology and your developers are insane, then it's going to come to tears."
Contrapositive is:
"If you don't pick a good technology or your developers are not insane, then it won't come to tears?"
Just some food for thought.
'If you pick a good technology and the developers are insane, it's all going to come to tears.'
There never has been a better description of Linux written in history!
buhahahaha!
where the testers (a.k.a. users) get to pay $$$ for the privilege of testing OS stability.
It is the new trend now the developers pay you for using their technology....
just imagine what'll happen if Linux actually makes a dent in the non-geek desktop market, and widespread use by "appliance operators" ensues.
Must be why there is a huge popularity of mugs at MS bearing the obnoxious logo:-
"You don't have to be a developer to work here, but it helps".
Coming from someone who was at that talk, he specifically said NOT to give money to testers. His words were actually 'give them credit, fame and loose women'.
This drew laughs from the audience.
For the slashdot editors :
"may ultimately threaten" != "threatening"
But, hey, the second one is more dramatic looking, so by all means run the scary one.
Truly, you are now ready to join the mainstream media.
Decentralised SCM. Testing framework as part of the release cycle (now easy for most kernel dev with UML or Xen, contrary to stable-release Aegis manual...)
Osnews had an article a while ago about some of the testing Sun do on Solaris - http://www.osnews.com/story.php?news_id=10178
man it must suck working at those companies...
Testing of Linux might be easier if it contained some automated features for sending crash reports back to a central database. Gathering some basic data on the stack trace, thread states, processes, etc. might help troubleshoot the OS in the context of the wide array of systems, configurations, and usage patterns. I know that both Microsoft and Apple have benefited strongly from this feature. Some tin-foil-hat wearers might object to their box phoning home. Tin foil hatters can just disable the feature but it might mean that the types of bug they experience never get fixed.
If developers are going to fix the bugs that occur in the real world, they need data from the real-world.
Two wrongs don't make a right, but three lefts do.
I thought the point of Open Source was to allow more people to read through the code. You mean thousands of people aren't really doing that for fun? I'm shocked.
More seriously... I think many of the people who DO eyeball the code are looking for security problems these days (where you do get recognition, etc.). For the record, I know I won't get any HR props for putting OS bugs that I've uncovered on my resume, but the security bugs I've found are always good conversation pieces.
"Bugzilla is fine for tracking bugs, but as it's currently set up, it's not very good for resolving bugs."
Hmm... I'd be interested to understand what alternatives to a web-based system he has in mind. Any thoughts?
"This process, where individuals communicate via a Web site, is very bad for the kernel overall."
Someone you trust is one of us.
I thought the whole Fedora project WAS mass testing of "cutting edge technology for Linux". Have I been wasting my time submitting bugs? Most have been fixed that I submitted so far.
Two roads diverged in a wood, and I - I took the one the bus load of girls just went down.
Long answer; kinda. You can use core dumps and system logs to interpret what's going on, but you can never really know for sure. Besides, the kind of errors that are in the kernel are the kinds of errors that really don't return error codes; they're the kind that crash the computer and make you reboot.
Microsoft's method is for some of the higher up software, and so is Apple's. If there's a bug in the kernel it's very unlikely that their code will catch it. Or at least that's been my experience.
If the problem is that Linux is so buggy, we just need to run it on a bunch more machines, and start randomly poking it as hard as we can until we break something. Once we've broken it, do it again to make sure it's not hardware, and then go to work fixing it. Good old brute force repairs.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
They're also telling me that it's time to go home and clean the guns.
Best Slashdot Co
I knew you visited Slashdot. Why not sign up for an account?
Linux+Qtopia is very popular in the handheld/embedded market.
Best Slashdot Co
but i think that when you start developping open source stuff, may it be linux-related projects or any open source languages (php, perl, etc) you're basically saying that you're doing it because you like it and not for the money. If you would want the money, you'd develop your shit in languages such as ASP, VB or Java.
I guess that when you've been doing it for as long as those guys, it gets depressing at some point to work for free. Coding linux related stuff is (in my book) meant to be a hobby, not a profession (yet). So if you do this for money, you'll be disappointed.
A computer makes it possible to do, in half an hour, tasks which were completely unnecessary to do before.
wtf is up with the headline? aren't they talking about lack of credit/money? "Putting in long hours of testing"... ?
If the average Linux user were educated on how to recognize a bug, and file a meaningful bug report it would mean a lot to developers, and likely speed up development and stability.
Morton is correct.
Even at commercial companies, QA isn't a "sexy" task. People would rather bang out code than write testing harnesses and run benchmarks.
Also, free software is driven by programmers, who tend to hate QA. Like any artist or craftsman, a programmer hates having their work critiqued. They spent hundreds (or thousands) of hours on a program, only to have someone nit-pick the details and point out the flaws. But for art, "quality" is a subjective quality -- and with software, quality and reliability are tangible quantities that can be measured.
My Acovea project demonstrated the problem. Users of GCC love Acovea; many developers of GCC, on the other hand, seem to treat it is an annoying distraction. Acovea identified more than a dozen errors (some quite serious) in the GCC 3.4/4.0 compilers -- and yes, I did report them to bugzilla. Only a couple of GCC's maintainers have said "thanks."
Not that the cool reception deters me. I have a new version of Acovea in the wings, and will be unleashing it on GCC 4.x Real Soon Now. ;)
As a consultant, I've been paid to perform QA work on commercial software packages -- but only one company, and a big one at that, has ever contracted me to QA a free software project.
Right now, free software is about many things, but quality is not job 1. And that needs to change.
All about me
I have successfully used Test Driven Development in several of my projects and it is a uniquely satisfying experience. Writing test cases before writing the code then completing each test case one after another in steady progression gives a constant stream of small victories. It also means you can run all test cases at a later time and see that "yep, everything still works" or "doh! that change just broke 10 things I already had working."
There are several other benefits to writing tests first as well. The experts in the link above explain it all better than I could, I'm sure.
Many open source projects are taking this approach already and usually boast the number of unit tests along with the lines of code included in the distribution. Anyone can type in "build test" for example and it will show the program run and pass some odd thousand tests.
Is it time for the Kernel to embrace this methodology? I certainly think it is a genuine best practice. But is it applicable to OS development as well? I don't see any reason why it wouldn't be, but I am not a kernel developer myself.
I remember in the early days there was a program called 'crashme' that threw randomly-generated executables at the system, and it was credited bolstering stability. Do tests like this still hapen frequently by the unappreciated? Is there a good place online to read about these tests and their results for different point-releases? Along similar lines, I recall someone throwing random input at the various gnu utilities, and it was discovered that they were more robust against this sort of abuse than the commercial unix equivalents. Are there any other interesting tests that anybody knows about? Breaking stuff is fun.
The "cue the foo posts in 3, 2, 1..." posts will commence with no subsequent foo posts in 3, 2, 1...
It may not be a Linux issue per se (more of a distro issue, I think), and it's purely anecodatal, but I've been seeing some QA problems lately in the mainstream distro I use. They include a bug that requires me to hand-edit the X11 config file to get my mouse to work, having to manually rebuild the routing table after every boot, and a so-far baffling total freeze of the system after rand() hours, only when it's serving web pages. I've been using Linux to do this job for six years, and never had these kinds of problems before.
http://alternatives.rzero.com/
Or does this Guy seem like a complete prick ? And will be the end to Linux if Linus doesn't smack him in the face and kick his arse out the of the door ?
"Sweet llamas of the Bahamas !"
I'm (honestly) a bit confused by the juxtaposition of 'appliance operator' and 'desktop'.
Best Slashdot Co
When I heard that I nearly fell off my ostrich.
Have you checked? Today? It never hurts to be sure.
Best Slashdot Co
And started seeing credibility problems with the article. Is there a transcript of what he actually said anywhere?
Linux is constantrly improving, but that means it is also constantly changing, and that makes it a constantly moving target.
That applies to most distros as well as the kernel itself.
It's hard to put a lot of effort into testing something when it's possible those tests will be invalidated a few months down the road...
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
... obviously hasn't hurt Microsoft. Oh wait, I take that back. Since MS treats their users as beta testers, there's plenty of testing ... just not at Microsoft.
If anybody reading this is interested in participating in the test procedure, check out the Linux Test Project.
My lame blog.
Software testing (usually) isn't monkeys pounding on keyboards until the box BSOD's.
:)
:). I remember dling the Gaim tarball (this was loooong ago) and seeing about getting it built on my SGI machine. IIRC, there were some makefile / #include problems getting it to even build, and once it was built there were some other issues with its runtime. Ultimately i submitted a patch to the gaim folks that more or less "enabled" gaim on IRIX. There is no way anybody had ever used Gaim on an SGI without making these fixes, so it seems reasonable to suggest the authors had never tried it before. This lack of a platform test matrix is pretty common amongst smaller F/OSS apps, even when they say "works on *nix" they mean "works on the distribution of linux i run at home".
It is difficult to test software without adequately understanding what it is supposed to do. Varying the underlying machine type is almost irrelevant for binary distributed software unless you're testing an operating system kernel or looking for race conditions in software (which is really just a stab in the dark)
How are you going to have 3rd party people debug software they know nothing about?
Where users help find bugs is by using the software. It honestly takes a certain mentality to be an effective software breaker, and it's not very common. It takes something else entirely to be a software tester; you've got to be a good developer (because software testing is about automation these days unless you're insane) but you've got to not get sucked into the developers way of thinking.
I assure you - letting normal users play with software doesn't clean it up. we can show that this is true in the following way:
- more users use Microsoft software for more hours a day than any other software in the world
- slashdotters say Microsoft software is the buggest software made
clearly if users using software was sufficient to find all the bugs, MS stuff would be bug free, based on its frequency of use alone. I know this isn't the case, because im a software tester at Microsoft.
(The appropriate response is "well then, stop posting and get back to work; you're clearly not done yet!"
W.r.t. linux kernel testing: this is something that's always amazed me - linux works surprisingly well for something with so little formal testing. On the other hand, when there are edge case problems my experience has been that nobody is much interested in fixing them. One example i had was at a consulting gig. the client was looking to move his web hosting business onto linux boxes if he could get more sites per box then he could on windows. He had a problem where his linux server would start dying after a few days. I started to look into it and the box would basically panic() in low memory situations. I asked Alan Cox about it (via irc) and the response was "buy more memory". Nice.
Another sore point with me growing up was xserver crashes. The Xserver was 99% reliable, but then you'd get some random crash and lose everything you were doing, and you knew there was no real way of getting it fixed or investigating it.. you just had to hope it magically got better somehow.. maybe when you switched hardware or something.
Then there's the just plain lack of testing of some F/OSS projects in general. When i was in college i had NeXT, Sun, and SGI boxes in my dorm room (but no linux
Another baby patch i submitted was for the openBSD kernel.. this time for the wdc driver. Back when UDMA 100 was newish, i bought 2 UDMA 100 disks a month or so apart.. so they were different sizes and different vendors, but on the same bus. The UDMA rollback code in openBSD would drop the DMA level from 5 (UDMA100) to 2 (something much slower, i dont remember what) after a certain number of DMA errors. This obviously sucked since you can run UDMA devices at different speeds on the same bus, and you can also fallback to UDMA66 and UDMA33, both of which are better than mode 2.
My opinions are my own, and do not necessarily represent those of my employer.
ing," he said.
Wait a minute here...
I thought the whole scheme was structured thusly...
I crank up the latest greatest kernel. I find a bug. I report it. My bug gets fixed. THAT's MY REWARD! The friggin bug gets squashed. What more could one ask for, with a clear conscience and a straight face.
As for those guys who fix the stuff. Well sanity is a relative term as we should all realize in light of the Japanese influence and emergence of cargo cults in WW-2 Niu Guinea. AFAIK, most Linux users view the kernel developers as some mysterious force from which benefit is derived through clever creation of effigy's.
If you actually engineer something, you can design in testability.
But Linus T. in an interview back last centruy, said that 'Linux is organic' - not engineered.
So who is surprised that testing is 'hard'?
What's all this then? Developers need money to produce quality products?!? Richard Stallman would be rolling in his grave... Once he dies and gets to his grave... err... yes.
BTW, we have not one, but two of my colleagues down under right now listening to Andrew in person. It should be interesting to get a first-hand account of what was said.
- Necron69
Good trade, if you ask me.
I'm making a
I was under the impression that by using Linux, I was, in a sense, testing Linux.
Whenever you read this sig someone's refrigerator light turns on.
that is NOT Microsoft's approach to testing.
;) These really dont happen that often during the product cycle, because ad-hoc testing doesn't catch that much stuff if you've got well developed automation suites. However, it's still very worthwhile because it is a good feedback mechanism to explain why your other testing missed something, and it's the best way to notice the odd "that's funny..." sort of issues that are not functinoally incorrect but are still user annoyance type issues.
Where did you hear or get the impression that that was the MS "approach" to QA ?
I've written test suites for the following Microsoft Products
- Visual Basic Compiler, 7.0
- Microsoft Business Framework 1.0 (unreleased)
None of them involved just using the compiler or the business framework over and over in day to day work to find bugs.
We have a variety of test approaches, including a few that _might_ be construed as what you describe - There are a few ways that we get test coverage via product usage
- stress
- bug bashes
- app weeks
Stress is funnier than it sounds. Did you know we're not allowed to ship windows until the exact build of windows under ship consideration has been running on hundreds (thousands, usually) of machines continuously with no problems while enlisted in a distributed "stress" client... where they're pounded and pounded with automated tests that do things like starve memory whilst performing other work, etc? Same with ASP.NET and the CLR - they have to _survive_ for a pre-determined time period before the build can be considered shippable. We dont think there are any show-stopper bugs at this point - but we just want to be reasonably sure. Note that if we find a bug (even an unrelated one, like the documentation has a typo) and take a fix for it, the stress cycle resets because the bits have changed. Better safe than sorry. In the end game of a product release it can literally be the case that taking a bug fix means delaying ship for another week or more.
- bug bashes
this is probably most like what you're describing. Everyone on the team sits down for a couple of days and really just beats on a specific area of the product. Security Bug Bashes have become popular int he last couple years (wonder why
- app week
For developer tool products (like Microsoft Business Framework) we like to do an app week with each milestone, where everyone on the team builds some sort of end to end application, using as much of the toolchain as possible. This sort of testing really makes the employees better (we're usually pretty compartmentalized on our areas of functionality ownership). It also lets unreleated parties take a look at peices of the product they don't own (so don't have preconceived notinos about). Finally, it lets us simulate the end-to-end customer experience on our product stack. If we can build the sort of apps a customer might build with our tools, then the tools are probably alright. Where we run into problems, we know the tools need help.
bug bashes and app weeeks happen perhaps 1-2 weeks per milestone (which is on the order of 2 months). It is a small part of our testing, time, effort, and results wise. It's still important to do, but it is not the _focus_ of QA at microsoft.
My opinions are my own, and do not necessarily represent those of my employer.
From TFA:
"A lack of commitment to testing by the Linux community may ultimately threaten the stability..."
The content of the article is much better than the headlines and excerpts being quoted. I was there and felt that what he was geting at was that we need to start thinking about updating QA procedures. The ratio of bugs to features is decreasing, but the rate of features is (maybe?) growing that much faster. The point of his talk was to outline a number of options for improving QA, thre are issues, but the sky certinly isn't falling either. It was an excellent follow on from Tridge's keynote the previous day on how to do quality system programming (overshadowed by his very brief coverage of the BK thing).
Xix.
"Everything is adjustable, provided you have the right tools"
Assume I was drunk when I posted this.
Morton also emphasised that he didn't agree with Torvalds' longstanding philosophy that rejecting patches from the kernel was just as important as accepting them.
... slashdot?
So imagine criticising an editor because he felt cutting the bad was as important as including the good. Then you'd get a publication with the quality of
I believe, there are real alternatives to linux. As linux is just a kernel. What we use and are comfortable - the numerous programs and utilities that run in the OS space are the GPLed GNU softwares which can run on linux alternatives too like FreeBSD, MacOS, OpenBSD, NetBSD and so on.
For the diehard fans of GPLed software , there is the GNU Hurd which can be embraced instead of Linux kernel. And the end user will never know the difference. This scenario of lack of testing will not occur for Hurd because it is a purely non-profit venture whereby even the developers do not rely on the project for their livelyhood. For them, it is a pure hobby and pleasure to work on the project and that is incentive enough for them to put in man hours in bettering the software.
Linux Help
for all things on Linux
replace testers with script
(I know, I know humans make mistakes and discover bugs )
but hey just try building all my modules into a static kernel and it fails (cxx drivers)
basically hardware manufacturers would love to send their bits to a place to get a stamp on it thats what they should do
get a logo
e.g. drivers for linux
if your hardware works and passes the automagic tests on linux then you get to put that logo on your product
job done
regards
John Jones
noo, this is
-- it's ridiculous how many people misspell ridiculous... (damn, damn, damn...)
The issue may have been already covered on Slashdot. At any rate, these were the findings of the study "Maintainability of the Linux Kernel" http://www.vuse.vanderbilt.edu/~srs/preprints/linu x.longitudinal.preprint.pdf at the Vanderbilt University, Nashville:
"We have examined 365 versions of Linux. For every version, we counted the number of
instances of common (global) coupling between each of the 17 kernel modules and all the other
modules in that version of Linux. We found that the number of instances of common coupling
grows exponentially with version number. This result is significant at the 99.99% level, and no
additional variables are needed to explain this increase. On the other hand, the number of lines
of code in each kernel modules grows only linearly with version number. We conclude that,
unless Linux is restructured with a bare minimum of common coupling, the dependencies
induced by common coupling will, at some future date, make Linux exceedingly hard to
maintain without inducing regression faults."
So? Microsoft have teams of testers and their stuff is still unstable as hell. Spend the time on new features is what I say! It's *cultural* now to not expect stuff to be stable. Not that that is a good thing mind....
This article is about kernel development. While I appreciate the development being done to make the kernel faster/better/cheaper (well, it doesn't get any cheaper), it's already a Pretty Damn Good kernel. It sounds to me like the most crucial thing would be to solidify it and test the bejeezus out of it, then largely freeze it, because that's not where the problems are.
When people complain about MS Windows, they're not (usually) complaining about the kernel. They're talking about all of the stuff built on top of it: window manager, IE, networking, configuration. If the Linux kernel is receiving too little testing to be stable, what about the millions of lines of code that go into X windows, Gnome, CUPS (as mentioned the other day), etc.
If MS didn't have to make kernel changes to bettter support security, I suspect they wouldn't be touching it at all. BSODs are still more common than they should be, but most users find them extremely rare, and the kernel is Fast Enough relative to the work that needs to be done. The improvements in Longhorn are largely about changes above the kernel, especially in its spiffy interface.
While I'm grateful to Linus and all of the other developers for the kernel improvements, and while Open Source means never being told what to work on, kernel improvements other than stability are probably a terrible use of manpower. The kernel is a tiny fraction of the lines of code that go into a Linux distro. They are basic, and need to be rock-solid, but while performance improvements there benefit everybody, they don't benefit you at all if X, or KDE, or Konqueror, or any of the hundreds of other higher-level apps crash.
I'm both.
Welcome to the world of software testing. Be prepared to work ten times as hard to get the same recognition as a developer.
And you wonder why Debian is so far behind and yet far more stable than any other distro? Then again, Debian is maintained across multiple platforms.
Personally, I liked the earlier practice of running stable even releases (2.4) and testing/developing on odd releases (2.5).
I realize that they have abandoned that in 2.6, but I don't really understand why they did that.
In a windows world you have a manufacturer from whom you can get detailed specs on hardware, etc.
In the linux world it's a rarity to get such information, which means that you can only test what you have. So yes, the devs can test a "brand X rev 1.02" video card up the wazoo and have it working perfectly, but it becomes something of a feedback-from-users loop when the r1.03 card breaks the compatability of the 1.02 driver.
Let's face it, there are way too many configurations out there to test for all combinations, or even for all common ones given the cost of hardware/etc
Yes, there are stupid non-hardware related bugs as well, those could perhaps be tested better. But with the advent of constant new hardware people are clamoring for something that works at all, nevermind just works well. Hell, I'd suffer bugs happily just to get the integrated cardreader on my zd7000 laptop to work in 'nix.
Companies like MS can get specs for hardware, get samples of hardware, and get support from manufacturers. Companies want the gorilla to support their hardware (generally they write their own drivers too, but OS compatability is still an issue).
For open-source projects a question like "hey, our RE'd linux drivers for your card X is behaving oddly under this circumstance, can you explain" is quite likely to get a response of 'sod off' if any at all.
Something I'd like to see is a place where I could pay a kernel dev for work on a specific driver etc that is important to me. Say if I want the soundcard drivers fixed up, or a driver that runs my cardreader... I'm happy to pay a bit for something that works if somebody is happy to put in the effort (and understand that I'm not rich, but perhaps a few contributors can make it more worthwhile).
For them, it is a pure hobby and pleasure to work on the project and that is incentive enough for them to put in man hours in bettering the software.
Can you say "naive?"
...don't run a vanilla kernel that they tweaked themselves, they run the subfork (if I can use that as a term) kernel that their distro uses, so in that sense it's useful to -again-most people. Vanilla kernels, no idea, day to day *practical* kernels, the way it is now with bug reports is a lot better than nothing.
Then, there's the mysterious Stanford Code Validator, used to great effect for a while. I feel certain that a few sweeps of that would uncover many of the more troublesome problems.
For those without SCV (99.9999% of the planet), there are some Open Source code validators out there. It should be possible, at the very least, to use those to identify the more blatant problems.
If you're not sure about using code validators, then it's simple enough to write programs that hammer some section of the kernel. For example, if you have some large number of threads mallocing, filling and freeing random-sized blocks of memory, can you demonstrate memory leaks? How well does the VMM handle fragmented memory? What is the average performance like, as a function of the number of threads?
Likewise, you can write disk-hammering tools, ethernet tests, etc. For the network code, for example, what is the typical latency added by the various optional layers? Those interested in network QoS would undoubtably find it valuable to know the penalties added by things like CBQ, WFQ, RED, etc. Those developing those parts of the code would likely find the numbers valuable, too.
If you don't want to write code, but have a spare machine that isn't doing anything, then throw on a copy of Linux and run Linpack or the HPC Challenge software. (Both are listed on Freshmeat.) The tests will give kernel developers at least some useful information to work with.
If you'd rather not spend the time, but want to do something, map a kernel. There's software for turning any source tree into a circular map, showing the connections within the program. If we had a good set of maps, showing graphically the differences between kernel versions (eg: 2.6.1 through to 2.6.12-pre3) and between kernel variants (eg: standard tree, the -ac version and the -mm version), it would be possible to get a feel for where problems are likely. (Bugs are most likely in knotty code, overly-complex code, etc. Latency is most likely in over-simplified code.) You don't have to do anything, beyond fetch the program, run it over the kernels, and post the images produced somewhere.
None of this is difficult. Those bits that are time-consuming are often mostly time-consuming for the computer - the individual usually doesn't need to put in that much effort. None of this will fix everything, but all of it will be usable in some way to narrow down where the problems really lie.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
What are they expecting? It's based on voluntary work.
If anybody needs some guaranteed service, or commercial-grade testing, maybe they should hire some programmers to do it?
>Microsoft have teams of testers and their stuff is still unstable as hell...
That just means that OSS is a disaster waiting to happen.
If you go to www.osdl.org you will see that limited testing is being done on the current kernel. It look mostly database oriented. Yet, it is some testing.
For the diehard fans of GPLed software , there is the GNU Hurd which can be embraced instead of Linux kernel.
Ah, yes. GNU Hurd - the OS of the future. Always has been, always will be.
I am not a tester in the windows division, but based on what i understand to be accurate, windows appcompat testing is a HUGE time / resource sink.
A good person on this subject is Raymond Chen, a reasonably famous MS blogger. He's written a number of times on all the crap he personally did in Windows 95 to make certain apps run _at all_. Yeah, there was code in Win95 for the sole purpose of letting certain pre-Win95 games run properly - when those games had bugs in them that we couldn't have counted on the develoeprs to fix. We took the hit of fixing our OS to maintain runtime compat with the games, because when the customer sees "crap win95 broke my games" it doesn't matter that the game author had bugs or was using things incorrectly; what they see is that windows 95 broke * for them.
Raymond is a prolific writer. try looking at http://blogs.msdn.com/oldnewthing/
you're right on the money w.r.t. 3rd party stuff being a big source of hangs. Obviously the resilience to 3rd party apps taking down the OS has gotten better with the NT family, but we've got data internally about where the customer-reported crashes come from (i.e. via support and people clicking the "send crash dump to MS" stuff) and by and large it comes from non-WHQL video drivers, anti-virus software, and other installable filesystem peices. Basically, where we've let 3rd parties plug into kernel space (drivers / FS integration points), which is what you'd expect.
I don't see many user space apps taking down the OS these days- do you have any in particular that seem to do it reliably?
My opinions are my own, and do not necessarily represent those of my employer.
I have a friend whom i deeply respect who has been programming for over twenty years. He read Kent Beck's book and exclaimed that it "changed his life." I'm now working on a project and integrating my project with CppUnit. It's mega sexy cool. So, yeah, I think it's time for the Linux Kernel to embrace TDD. However, it may be a bit of a challenge. I have a cave-man grasp of the Linux Kernel. (Grog know kernel monolithic.) Thus devising effective unit tests could be a reel beech!
If some young Turk wants to earn his chops within the OSS community and snag a lifetime's worth of wiffie, he can create a "LinuxUnit" framework with a suite of tests that demonstrates key parts of the kernel are correct. With "LinuxUnit" in place, anybody with a kernel patch proposal could demonstrate his patch breaks nothing, and if it extends kernel functionality, demonstrate that it performs as promised. Moreover, if (heaven forfend) a bug is discovered in Linux, a "LinuxUnit" test case demonstrating the issue would serve as a point of focus.
Though I think this is a Real Good Idea, since I am neither young, nor Turkish, I place it before the Slashdot community as an unimplemented suggestion.
"Common Address Redundancy Protocol (CARP) for instance."
I've been calling it Common Redundancy Address Protocol (CRAP).
My bad!
I'm surprised to hear the testing even exists in the Linux community. From my perspective in the BSD world, it looks as though linux is stuck in a perpetual Alpha state. It may even make it to Beta, but then another "often release" hits.
The only project that appears to have quality/testing clearly included in their goals is OpenBSD. They release every 6 months. There is a period of testing that leads up to each release. Heck, they even use -Wall to compile the entire OS.
As an IT manager, I have big issues deploying Linux. And the daily fragmentation in distributions furthers the undermining of the project.
The Linux community needs to get their act together, grow up, and manage/test the software they build. Until then, your doomed to obscurity only celebrated on Slashdot.
Cheers,
Jim
Absolutely, that is your reward, as things stand now. Andrew is saying this isn't enough, and he has good reason to believe so.
AKPM puts out the -mm series of kernels, where major/non-obvious/controversial patches go to get tested before going into the vanilla tree. A lot of the subsystem (ACPI, USB, DRM, etc) maintainers go through Andrew (and -mm) as well, before hitting mainline.
The problem is that -mm isn't getting nearly enough testing. I tracked down and reported a bug that completely broke initrds in one release, *a week after it came out.* There was some grumbling that that particular release didn't boot for somebody, but it was a week before someone picked up on the fact that a major kernel function was completely non-functional. Had I not taken the time to track it down, it might have gone to mainline before someone noticed what was broken. And this isn't the only time things like this have happened.
AKPM is a stand-up kernel maintainer, and a lot of the advances in the way the kernel is being developed have come from him (it *is* an advance; remember what 2.4.11 was like?). When dealing with bug reports, and you know, users, he's wonderfully responsive and easy to work with.
But I don't have the time or inclination to build every -mm release on i386 and ppc like I used to, and he stability of the Linux kernel is suffering for it, not because I'm a great coder (I'm certainly not!) but because -mm is so short of testers that every single one counts.
If there were some more reward for the hours I put in to tracking down some missing #include or just pinning down the symptoms of a bug I'd probably still be doing it. But it just doesn't seem worth my time, and my conscience is clear.
"That's all I have to say about that" --Forrest Gump
Then please explain the abortion that is Visual Source Safe?
That thing is a gigantic collection of bugs and untested code.
there are mechanisms now for getting bugs back to MS - the "upload dump" stuff from more or less all of our apps actually goes somewhere. real people look at it, investigate the crashes, etc.
Also, w.r.t. MS not releasing source - you can install the Windows Debugging tools and get the symbols for windows. you can run whatever copy of windows you have now under a kernel debugger and single step through instructions. You dont see source code, but you see fully decorated information in stack frames, etc. It's enough to get a pretty good idea of where a problem is, even though there's no source code available. Case in point - i've found debugging the windows kernel with only symbols to be easier than trying to debug the linux kernel even though full source is available. My unix kernel debugging is more about dropping printk()'s or wahtever in places i think the bug is and then converging on it with recompile/reruns. The windows kernel debugger(s) are actually pretty slick comparatively.
In any event, my point wasn't about people not being able to file bugs or nail them down - it's about reliably finding them. Undirected, Ad-Hoc testing just isn't a cost effective approach.
As far as some of the hole poking stuff - unless you can suggest that the behavior has security implications, that's not as valuable. Deleting random files in %system32% and then saying "look, notepad doesn't run! ha ha!" aren't really the kind of bugs that we'd be interested in fixing, because that just isn't a real customer scenario (especially since System File Protection will try and keep you from doing such things). If you go deleting or changing stuff you shouldn't and stuff starts breaking, unless this is a customer scenario, or unless it's easy for an attacker to utilize this to acheive some aim, the response will be akin to a doctor saying "if it hurts, stop doing it!"
My opinions are my own, and do not necessarily represent those of my employer.
I've used VMWare to test windows drivers on linux boxes; its great for testing device independent bits (like filesystems). Buti it has limits
-threading/race conditions dont surface (nonexistent/different thread model)
-limited set of devices
-no power events
They are still fantastic; you can treat a bluescreen or a toasted filesystem as an app crash, not a major distaster. But eventually you need to move to real hardware.
If I'm gonna slave away in front of a computer developing some hunk of code, I'd better get some money or money-making credit for it. I don't work for peanuts. "Thanks" and general praise don't put beer in my fridge and steaks on the grill.
If there's only hobby incentive to do this, you ain't get'n much work. Plus, if there's no job to keep, no paycheck riding on my performance, yer going to just get whatever half-arsed crap I put out that works (if it works to begin with).
So for all paid programmers doing work for pay, where's the incentive to do it all for free?
Look, everyone is real sorry you never got picked until last in gym class, but this whole mental masturbation you have over Slashdot borders on psychosis.
For a couple of years now, I have realized that testing is the 'unsexy' part of open-souce development that doesn't 'scratch any itch'. As such, QA can sometimes not be adequate, and what's left is a few brave souls trying to sort through a myraid of user bug reports. It's not the way things should be done.
What the Open Source world needs is coordinated testing. The involves a well-designed bug tracking and QA system, with enough attention and kudos to keep people interested in working on things.
My vision is a comprehensive test lab, set up somewhere on cheap real estate, with numerous system representing the broad spectrum of hardware configurations, shepherded by some talented sysadmins, and used locally or remotely by a vast, well coordinated league of testers.
Without going into great details, I believe that the confluence of inexpensive real estate, good infrastructure, and reasonably easy reach of talented admins and programmers, leaves two cities at the top of the list, one for each two continents with which I feel I have the knowledge to make some rough qualifications.
- Pittsburgh, Pennsylvania, USA
- Leipzig, Saxony, Germany
People and resources are needed. First, we need people to rally around and work out that this is a good idea. Second, we need QA testing prople with a lot of expertise to design systems for testing. Third, we need space for the computers, fourth, we need the computers (I'll donate most of the pile in my basement!), fifth, we need the on-site sysadmins and builders, sixth, we need bandwidth and IP space (IP space may end up being more important than high bandwidth. Seventh, we need a way to financially support the equipment and people onsite.
What do the rest of you think?
From the article:
Morton also emphasised that he didn't agree with Torvalds' longstanding philosophy that rejecting patches from the kernel was just as important as accepting them.
"I diametrically disagree with him on that stuff," he said.
"If we just drop the patch on the floor . . . it's the kernel that ends up missing out. It's the maintainers' function to get patches into the kernel, rather than taking pride in rejecting them."
So on one hand, we should reward those who do the testing. On the other hand, when the testers say "Hey! This patch broken by design!" we should ignore them and add the patch to the kernel.
Taking a step back, I'd also suggest that he seems to lack a broad perspective on the issue. Cellular death is a requirement for nearly all multi-cellular organisms; the tadpole's taile must die before it becomes a frog. Brevity characterizes good writing, not wordiness. Lincoln's "Gettyspurg Address" is noted for its brevity; it followed a two-hour speech which is not remembered by anyone today. A sculptor's skill is reflected just as much in what he takes away as in what he leaves.
Why should code be any different?
With the BSDs you get none of these geeky head games and BS licensing schemes. Linux will eventually either flop or take a major back seat because it's developers aren't recognized and/or getting some sort of compensation. I think it's pretty "nervy" of Linus to say (in such a condescending manner) that some of the patches should be dropped. Like he's forking over major $$$ for the effort. Typical open source geek attitude.
BSD, on the other hand, I feel is much better model. Much less stricter licensing (do whatever you want). ONE source for the entire OS (not just kernel). Easier upgradability. No wonder Apple chose BSD over Linux to use in it's OS X.
"We've made big changes to the standard kernel, and haven't broken it any more than it was before."
For those who are going to take the headline and run with it to denounce the current Linux kernel as "unstable"...like CA and certain
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
That'll keep me chuckling all day.
Bless you.
Linux is dying!
Last time I checked I was able to write a program in just about any language I wanted for free. Yes, even ASP, VB, and Java.
[100% ISO 646 Compliant]
SVM, ERGO MONSTRO.
OK, so this is what you do. Hire some kernel level programmers for you work and have them develop the source as they see fit. Thats the beauty you can do what you want with it. The current "stable" releases of the kernel are just that "Stable". And I would put them up against any OS out there. It's is truly a community thing. I am sure there are improvments that need to be made but hey===" Morton said the process had generally worked well." The kernel can always be better and testing can be improved. His concerns about the kernel are valid but all he is saying is he is worried thats all.
Naive is so inadequate. Utterly clueless doesn't adequately cover that statement.
Devoid of comprehension of the demanding nature of microkernel communication based software construction, or basic project management methodology. Technical, and yet not overly insulting.
There is no America. There is no democracy. There is only IBM and AT&T and DuPont, Dow, General Electric, and Exxon
I'm sure to get flamed for this, but one reason I left Linux for FreeBSD was that I was being treated as an upaid software test monkey. Heck, it almost seemed as if the more I *paid* for a distro, the more QA work I was expected to do volunteer.
The number one response when complaining about an issue was "did you file a bug?" I'm not paying you for the privilege of filing bugs! Redhat and Novel have the money and resources to do proper QA work on the kernel and their distros as a whole. And I'm sure they do. So why does it seem like they don't?
Don't get me wrong, I *DO* file bugs. But I file bugs on my own initiative, and on projects that interest me. It is the labeling of untested, unverified and unvalidated software as "stable" that irks me.
Don't blame me, I didn't vote for either of them!
Here's what I'm thinking; there are huge companies such as IBM, HP, and others that have the resources to QA/Test the kernel before blessing it as enterprise quality. I say leave the hard-core testing to those who have the resources ($$$) to do so. After all, who really cares if the kernel is stable? Is it Joe Hacker at home running Linux on their desktop for fun? I say not so much. Is it these huge elephants who have at some level bet the farm on Linux being stable enough to run huge production-level enterprise systems who care? I say yes indeedy. So I say, testing is always a great thing, but leave the true hardening to the ones who sell Linux-based systems for profit. I mean, the GNU General Public License leaves the code open for anyone (including IBM thorugh maybe RedHat) to fix any problems, just as long as they commit their changes back to the main branch. This means we as Linux users benefit from Big Blue's hard work in making the kernel enterprise ready.
I think this is the way it should to work going forward.
I opened up a web petition, we are going to make a distro of Linux without any code written by Andrew Morton, and the petition is to have Andrew fired, tarred and feathered, and all reference to him and his work expunged from Linux.
Ok, you have formalized testing. When did this start, during the Windows 2000 development cycle? Because I will admit, the stability of 2000 is much better than any other Microsoft OS.
However, Microsoft's products are still shoddy by design. Microsoft needs to rearchitect their entire code base to get rid of the poor security, size and speed bloat, semi-coupled software objects and gross incompatibility other operating systems.
But I guess that will never happen as long as the highest priority at Microsoft is abusing their monopoly position to leverage other markets and break other tools instead of sane, stable and comprehensible design.
I was there, and the quote was taken _absolutely_ out of context: 'If you pick a good technology and the developers are insane, it's all going to come to tears.' He was not refering to BK in this instance; he was in fact talking more generally about SCM systems, and how he had noticed that these projects tended to attract "insane" developers (also the ide drivers do this too).
This was all part of a larger, very insightful remark, saying that had Linus chosen a free SCM tool three years ago, we would now have a fantastic SCM in the free software world. In this instance, it is not so much the _tool_ that would need to be good, but that the _team_ behind the tool needs to be solid, responsive etc.
Simon.
I'm interested in this... can you give a specific F/OSS example?
I'm one of those testers AM talks about. Please see this. I haven't been paid squat.
If someone wants to help, rather than just talk about it, I accept PayPal donations via rlrevell at joe-job.com.
I'll setup a machine here just for kernel testing. All I need is an automated test suite and an RPM that puts in the right cron.whatever files.
Let it download new kernel versions, compile them, modprobe 'till it dies, send back result tests, etc.
I'll pay for the electricity and data costs, though it's going to get lower QoS on my firewall. Maybe I'll run SETI on the same machine niced to 19.
I bet we could convince ten thousand other people to do the same thing. Suddenly you have a decent sampling of hardware and more tests than people are willing to run or stumble across in daily work for every nightly kernel build with useful bug reports. Regressions should even be easier to identify, and breakage can be reported on a scoreboard.
So... let me know where do download it.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
No pre-formulated opinion turned away! Lease an opinion now, low RTFA rates (OAC)! Don't know what you're talking about? It's okay with our Get Modded Out Of Troll Payment Plan (tm).
"Thank you. Please spellcheck your genitalia references though.
In fact, I think it did happen with Win2K. There was an article somewhere describing how Microsoft finally stuck 1000 or so testers onto Win2K to make it something actually half-respectable.
Still, it's contemporary is Solaris 8, which is one freakin' stable OS. I think Ace's Hardware recent reported passing 400 days uptime on their Solaris 8 web server (surviving several Slashdottings, too). So, Microsoft still has some catching up to do.
Doesn't it seem odd that in a discussion that is about a threat to Linux, there are lots of posts from Microsoft developers? Usually, Microsoft keeps a low key at Slashdot.
Hmmmm....no I'm sure it's just a coincidence.
Hey Microsoft, how about you do like everyone else and reduce the price on your operating system and office suite? Oh, that would make you go bankrupt, sorry to have mentioned it.
http://www.piratebay.org/torrents-details.php?id=3 320297 Test away!
Obvious solution to two Linux problems: Turn bug testing into an MMORPG. Linux bugs get squashed and Linux gets an unique game with near-infinite content. And selling a Xbox port could support kernel development.
Some hack spewing out FUD writes "John Doe, a Windows SDK maintainer, has said that he thinks that the lack 'credit or money or anything' given to those people who put in long hours testing Windows releases is going to cause serious problems further down the line. In his speech at a recent MSDN conference he also waded into the ongoing Visual Studio debate, saying 'If you pick a good technology and the developers are insane, it's all going to come to tears.'"
I strongly doubt this is the case.
What I suggest instead is that lack of testing will hurt smaller linux distros. but larget distros (fedora, debian, etc) are constantly in use by thousands of people at all stages and with all levels of competancy. I see no reason to suspect that these people won't provide their own testing grounds.
If linux distros had a sales release scedule (like windows) then this would be an issue, but this is the nature of linux. new stuff comes out, the linux community picks it apart and finds the flaws, and a new release fixes those problems.
What's wrong with that? it's jsut a different scale.