Posted by
CmdrTaco
on from the debugging-the-beast dept.
The Qube writes "As a followup to the in-depth story posted back in February regarding the history and the development of Windows NT, part 3 of the series of articles is now online.
It discusses the software testing procedures inside Microsoft."
Microsoft has testing procedures?
by
Anonymous Coward
·
· Score: 5, Funny
Well, I guess you have to make sure those "features" work correctly
Microsoft's testing:
by
Anonymous Coward
·
· Score: 4, Funny
QA Engineer 1: "Does it compile?" QA Engineer 2: "Yup." QA Engineer 1: "Okay, I'm declaring it GM and releasing it to manufacturing." QA Engineer 2: "It's 'Miller Time'!"
Re:Microsoft's testing:
by
Anonymous Coward
·
· Score: 4, Funny
Is it strange that 'F7' is spell check in M$ word and compile in dev studio?
Re:Microsoft's testing:
by
Grease
·
· Score: 5, Funny
QA Engineer 2: "It's 'Miller Time'!"
As an ex-MS employee, I take great offense at this statement. In fact, we drank Red Hook, not Miller.
Heres another one
by
ArchieBunker
·
· Score: 5, Funny
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
-- Only the State obtains its revenue by coercion. - Murray Rothbard
I'm definitely sticking with gnome
by
1nv4d3r
·
· Score: 4, Funny
From the article (not kidding!):
Originally, the KDE worked to upgrade its infrastructure to Windows 2000 on its own. But after spending 18 months evaluating and testing, Cornett realized they'd need some help. The department contacted Microsoft Consulting Services (MCS) to ask about architectural guidance, and hired a full-time technical account manager from Microsoft's Enterprise Services group. Eventually, the KDE joined Windows Server 2003 Rapid Adoption Program (RAP), which allowed them to begin working with the product early in its development process.
(ok, so when you look into it you're likely to realize that it's the Kentucky Dept of Education, but when skimming the article it caught my eye and I was really confused!)
The video of this is out...
by
Anonymous Coward
·
· Score: 3, Funny
It starts off with a room. a huge room... the largest one you've ever seen...
The camera flies down, zooming in & out, between dozens of the ten million monkeys at ten million PCs, and back up to a control desk manned by straw-chewing yokels.
A screen flashes red
"Sir! Monkey number Y435A23J has come up with something that boots!"
The camera pans around to Bill Gates' face
"I call it... Windows 2008. Release it"
or something
Re:Text of the article (fixed formatting)
by
Omkar
·
· Score: 4, Funny
You know you're on slashdot when a post begins
Windows Server 2003: The Road To Gold
Part Three: Testing Windows
and ends
Can anyone recommend a way to get my cat off heroin? It would be much
appreciated.
All jokes aside...
by
KFury
·
· Score: 5, Insightful
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Re:All jokes aside...
by
asa
·
· Score: 4, Insightful
"Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes...."
Except that this article says nothing about the testing methodology that Microsoft uses. It describes how Microsoft helps certain customers test deployment. Deployment testing has little or nothing to do with software testing.
This is an article about how Microsoft has the budget to help "special" customers with a free "service" (not software) and frankly, the bits about offering cash-strapped school systems free consulting and test deployments sounds a lot more like a Microsoft press release than a software testing case study.
I was genuinely hoping to read about their software QA process. What a waste of 5 minutes.
No way this article is about software testing. This is about an evaluation lab where customers bring in their applications to show to Microsoft. It's a marketing puff-piece, that's all.
Where is the description of the test methodologies used? The bug escalation and change control systems? What sort of configuration control is used?
Development costs
by
BWJones
·
· Score: 3, Interesting
O.K., can anyone here tell me why Microsoft is spending an order of magnatude more $$'s to develop Windows than Apple is spending on developing OS X? It can't be testing because the Apple products appear so much more refined.
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Seriously....
by
wowbagger
·
· Score: 5, Insightful
All the Q/A in the world does you no good if you don't act upon what Q/A has found.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
{ {{edit,compile} while errors} link } while errors
Download to victim
boot victim. Watch hundreds of assertion failures from the OS scroll by on debugging console.
Shut down debugger as it has leaked all available memory
Start debugger again
Load my driver and test.
locate bugs in my driver, begin again
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code.
With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do.
Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money.
As part of the Microsoft culture, it appears that you've missed the point.
The problem is the 50 million lines of code itself.
I would have "managed" NT's testing by "not managing it" at all, and instead would have clipped out all those bells and whistles to make a much more trim and modular OS.
The code base is unecessarily large, from a functional point of view.
But just like the current SUV problem in America, it appears that Microsoft is dancing a tango with the consumers.
Microsoft produces shitty code that looks good on the screen, and the consumers say "ohh" and "ahh" while not minding the crashes and restrictions, and then Microsoft gets encouraged to produce more "pretty code".
I don't consider this problem to be fixable... we who know better and are less mediocre simply have to fend for ourselves and rely on the influence of our leadership to promote the Better Way.
This is the slow method of providing a good example for others to follow, which is the only leadership that matters.
Microsoft's billions are just a facade; consumer mediocrity is another facade; what will matter in the long run is bullet-proof code that more serves public needs instead of software-industry investors.
-- [You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
I'll try posting something original as opposed to the MS-bashing and the MS-bashing-bashing whilst remaining at least a little ontopic.
I think Microsoft would do well to test more and make less. Each incarnation of Windows seems to have brought disproportionately large improvements (or hindrances if you like) in the user interface, features, and resource consumption. Whilst a gradual accumulation of features and a slow increase in resource use is inevitable for any operating system I think Microsoft has been making their systems grow too much too quickly.
Microsoft seems to be running out of some new features to add to each new version of Windows to entice consumers are resorting to making their own features (notably,.NET and the like) in order to keep sales high. Unfortunately I think that in terms of features and UI they can't push the boundary too much further for the next few years (though obvious beyond that there will no doubt be new ideas).
As such I feel that MS would benefit from focusing on testing instead of adding new things. Consolidation is often just as helpful as (if not better than) augmentation, particularly for larger systems. I feel that sales would remain high if Windows had no new features or UI but could genuinely be considered as stable as alternatives.
--
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
Comparing with the UNIX model
by
mnmn
·
· Score: 4, Interesting
Having a monolithic kernel + garbage, monolithic reigstery, no way to control the revisions of DLLs, worse than RPM package management etc is what NT is all about. The UNIX model is making small highly useable tools that you understand perfectly, and making them work together. In win32, if you cant work with an API, you make a new API, and we now have several generations of APIs to deal with just to access the screen, resources, widgets etc in windows. You have a standard API that has worked for over a decade for most UNIXen, and the ones that do change their API purge the last version and replace it with a similar system.... think sysv vs BSD, Solaris sysctls vs Linux sysctls.
Thats why the win32 system spirals into complexity, no matter how much money is pumped into development or testing. Of course one of the best things about windows is also one of the worst, that vendors developing their own drivers for their hardware might make incompatible or bad drivers, or ones that step on the feet of other installed drivers in the system. In the Linux kernel, all the drivers are present before the testers and are considered while any major change takes place.. such as the VM or switching to 64-bit cpu. This is true for most other UNIXen where drivers are sent to the unix vendor for testing as well, but thats not as efficient as the Linux model.
And then the number of eyeballs testing Linux and FreeBSD is a phenomenon Microsoft cant copy. The free software community does not work for a paycheck, but theres more sincerity towards the software than there would be for a proprietary software. Free Software can be a matter of ego and gives a sense of competition with Microsoft. You cant buy that. This I believe is the biggest reason why colossal manhours are poured into free software development, while some of these developers work the rest of their days as data entry or office clerks, even mcdonalds.
-- "Give orange me give eat orange me eat orange give me eat orange give me you."
-Nim Chimpsky
Interesting article, but not really about testing.
by
deranged+unix+nut
·
· Score: 3, Informative
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer. 2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle. 3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test. 4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in. 5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes. 6) The build lab picks up the changes for the day and starts to compile. 7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue. 8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests. 9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such. 10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
Re:Ah, yes, the obligatory Linux advocacy
by
jareth-0205
·
· Score: 3, Informative
And that would be Linux, I suppose? Because no bugs ever creep into Linux, and there's never been a security flaw found.
That's the point! Bugs are much more likely to be found in an open system such as Linux because of the nature of Open source development - all people using the software can reporting / fixing bugs, not just the limited few inside a company. The parent poster is actually complimenting MS testing, just saying that it can never be as good as open source because of the numbers involved.
Slashdot Mirror
Well, I guess you have to make sure those "features" work correctly
QA Engineer 1: "Does it compile?"
QA Engineer 2: "Yup."
QA Engineer 1: "Okay, I'm declaring it GM and releasing it to manufacturing."
QA Engineer 2: "It's 'Miller Time'!"
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
Only the State obtains its revenue by coercion. - Murray Rothbard
From the article (not kidding!):
Originally, the KDE worked to upgrade its infrastructure to Windows 2000 on its own. But after spending 18 months evaluating and testing, Cornett realized they'd need some help. The department contacted Microsoft Consulting Services (MCS) to ask about architectural guidance, and hired a full-time technical account manager from Microsoft's Enterprise Services group. Eventually, the KDE joined Windows Server 2003 Rapid Adoption Program (RAP), which allowed them to begin working with the product early in its development process.
(ok, so when you look into it you're likely to realize that it's the Kentucky Dept of Education, but when skimming the article it caught my eye and I was really confused!)
It starts off with a room. a huge room... the largest one you've ever seen...
The camera flies down, zooming in & out, between dozens of the ten million monkeys at ten million PCs, and back up to a control desk manned by straw-chewing yokels.
A screen flashes red
"Sir! Monkey number Y435A23J has come up with something that boots!"
The camera pans around to Bill Gates' face
"I call it... Windows 2008. Release it"
or something
You know you're on slashdot when a post begins
Windows Server 2003: The Road To Gold
Part Three: Testing Windows
and ends
Can anyone recommend a way to get my cat off heroin? It would be much appreciated.
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Kevin Fox
No way this article is about software testing. This is about an evaluation lab where customers bring in their applications to show to Microsoft. It's a marketing puff-piece, that's all.
Where is the description of the test methodologies used? The bug escalation and change control systems? What sort of configuration control is used?
O.K., can anyone here tell me why Microsoft is spending an order of magnatude more $$'s to develop Windows than Apple is spending on developing OS X? It can't be testing because the Apple products appear so much more refined.
Visit Jonesblog and say hello.
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
www.eFax.com are spammers
The annoying thing about NT (4.0), is that now that it is almost rock-solid, MS no longer support it.
:)
Those years of public beta testing certainly paid off.
\\ Mitch
"We're hoping to get a long-term lifespan out of Windows Server 2003 without having to do major upgrading."
These guys obviously aren't students of "Licensing 6.0".
Everyone will start to cheer when you put on your sailin' shoes.
People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money.
... we who know better and are less mediocre simply have to fend for ourselves and rely on the influence of our leadership to promote the Better Way.
This is the slow method of providing a good example for others to follow, which is the only leadership that matters.
Microsoft's billions are just a facade; consumer mediocrity is another facade; what will matter in the long run is bullet-proof code that more serves public needs instead of software-industry investors.
As part of the Microsoft culture, it appears that you've missed the point.
The problem is the 50 million lines of code itself.
I would have "managed" NT's testing by "not managing it" at all, and instead would have clipped out all those bells and whistles to make a much more trim and modular OS. The code base is unecessarily large, from a functional point of view.
But just like the current SUV problem in America, it appears that Microsoft is dancing a tango with the consumers. Microsoft produces shitty code that looks good on the screen, and the consumers say "ohh" and "ahh" while not minding the crashes and restrictions, and then Microsoft gets encouraged to produce more "pretty code". I don't consider this problem to be fixable
[You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
I'll try posting something original as opposed to the MS-bashing and the MS-bashing-bashing whilst remaining at least a little ontopic.
.NET and the like) in order to keep sales high. Unfortunately I think that in terms of features and UI they can't push the boundary too much further for the next few years (though obvious beyond that there will no doubt be new ideas).
I think Microsoft would do well to test more and make less. Each incarnation of Windows seems to have brought disproportionately large improvements (or hindrances if you like) in the user interface, features, and resource consumption. Whilst a gradual accumulation of features and a slow increase in resource use is inevitable for any operating system I think Microsoft has been making their systems grow too much too quickly.
Microsoft seems to be running out of some new features to add to each new version of Windows to entice consumers are resorting to making their own features (notably,
As such I feel that MS would benefit from focusing on testing instead of adding new things. Consolidation is often just as helpful as (if not better than) augmentation, particularly for larger systems. I feel that sales would remain high if Windows had no new features or UI but could genuinely be considered as stable as alternatives.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
Having a monolithic kernel + garbage, monolithic reigstery, no way to control the revisions of DLLs, worse than RPM package management etc is what NT is all about. The UNIX model is making small highly useable tools that you understand perfectly, and making them work together. In win32, if you cant work with an API, you make a new API, and we now have several generations of APIs to deal with just to access the screen, resources, widgets etc in windows. You have a standard API that has worked for over a decade for most UNIXen, and the ones that do change their API purge the last version and replace it with a similar system.... think sysv vs BSD, Solaris sysctls vs Linux sysctls.
Thats why the win32 system spirals into complexity, no matter how much money is pumped into development or testing. Of course one of the best things about windows is also one of the worst, that vendors developing their own drivers for their hardware might make incompatible or bad drivers, or ones that step on the feet of other installed drivers in the system. In the Linux kernel, all the drivers are present before the testers and are considered while any major change takes place.. such as the VM or switching to 64-bit cpu. This is true for most other UNIXen where drivers are sent to the unix vendor for testing as well, but thats not as efficient as the Linux model.
And then the number of eyeballs testing Linux and FreeBSD is a phenomenon Microsoft cant copy. The free software community does not work for a paycheck, but theres more sincerity towards the software than there would be for a proprietary software. Free Software can be a matter of ego and gives a sense of competition with Microsoft. You cant buy that. This I believe is the biggest reason why colossal manhours are poured into free software development, while some of these developers work the rest of their days as data entry or office clerks, even mcdonalds.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer.
2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle.
3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test.
4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in.
5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes.
6) The build lab picks up the changes for the day and starts to compile.
7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue.
8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests.
9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such.
10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.