Posted by
CmdrTaco
on from the debugging-the-beast dept.
The Qube writes "As a followup to the in-depth story posted back in February regarding the history and the development of Windows NT, part 3 of the series of articles is now online.
It discusses the software testing procedures inside Microsoft."
Microsoft has testing procedures?
by
Anonymous Coward
·
· Score: 5, Funny
Well, I guess you have to make sure those "features" work correctly
Heres another one
by
ArchieBunker
·
· Score: 5, Funny
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
-- Only the State obtains its revenue by coercion. - Murray Rothbard
All jokes aside...
by
KFury
·
· Score: 5, Insightful
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Seriously....
by
wowbagger
·
· Score: 5, Insightful
All the Q/A in the world does you no good if you don't act upon what Q/A has found.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
{ {{edit,compile} while errors} link } while errors
Download to victim
boot victim. Watch hundreds of assertion failures from the OS scroll by on debugging console.
Shut down debugger as it has leaked all available memory
Start debugger again
Load my driver and test.
locate bugs in my driver, begin again
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
Slashdot Mirror
Well, I guess you have to make sure those "features" work correctly
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
Only the State obtains its revenue by coercion. - Murray Rothbard
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Kevin Fox
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
www.eFax.com are spammers