Posted by
CmdrTaco
on from the debugging-the-beast dept.
The Qube writes "As a followup to the in-depth story posted back in February regarding the history and the development of Windows NT, part 3 of the series of articles is now online.
It discusses the software testing procedures inside Microsoft."
Check this PPT to.
by
Karpe
·
· Score: 2, Informative
There is a very interesting presentation on the design and development of windows NT/2000 presented on USENIX here (google HTML rendition here). I love to bash Microsoft too, but reading it, there are at least some decisions that I think they did right.
Re:Let's see-
by
Anonymous Coward
·
· Score: 1, Informative
I don't know about the rest of you, but I am pretty sick of comments like this.
I use both Linux and Windows XP, and I for one have to say that Linux on the desktop is not exactly rock solid stable either. Parts of KDE crash all of the time (_especially_ KATE), OpenOffice.org is dog-slow. And all in all, I would say windows is substantially _more_ responsive than Linux is (on the desktop again).
I know that this post will probably be modded as trolling, but hell.. thats the state of things right now.
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
Interesting article, but not really about testing.
by
deranged+unix+nut
·
· Score: 3, Informative
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer. 2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle. 3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test. 4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in. 5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes. 6) The build lab picks up the changes for the day and starts to compile. 7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue. 8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests. 9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such. 10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
I run the following system: Pentium III @ 1ghz 256MB RAM GeForce2 MX RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel KDE 3.0.3-8 RedHat
And I almost never have problems with KDE. I use Kate for almost all my programming, and I can count the number of times it's crashed on one hand with fingers left over.
You know that you can adjust how much CPU time KDE uses, right? I don't know about other distros, but for RedHat 8 it's under Extras - Preferences - Desktop Settings Wizard.
Re:Development costs
by
BWJones
·
· Score: 2, Informative
Nonetheless, the user base of MacOS as a server OS is trace.
That may be, but until recently Apple has not had an OS capable of large scale serving. I have used IRIX, Solaris and Windows in the past, but I find OS X to be the best of breed in terms of a do it all OS.
There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated. These guys don't mess around - they expect to have industrial strength support during the upgrade, and they expect there to be no regressions.
Ummmm, well, seeing as how the xserve is a very recent entry into the field, I would not expect many large deployments yet. However, the ability to provide over 5000 hits/second and saturating 3 T3 lines is pretty damned impressive. Or how about streaming 3000 live connections all at once? Or netbooting hundreds of machines all at once? As for sites that are using OS X, off the top of my head, I believe that there is certainly Apple.com, all of the Quicktime streaming they are doing for the movies on Apple.com. I am sure there are others that I am not aware of.
Apple is in an entirely different league - they can ship a trivial OS update that accidentally deletes entire hard disks worth of data and can get off by paying for a few hundred disk recoveries and having an "everybody makes mistakes" attitude. That kind of thing isn't really acceptable for the desktop, but the extreme loyalty of Apples customers means they can essentially get away with it. That simply doesn't fly when you run most of the worlds servers.
Well, Microsoft has proven you wrong here on innumerable occasions. I can't tell you how many times I have had to advise folks to reinstall Windows because of file/registry corruption, or had to deal with the implications of an infection by a virus or worm in someones database or email program that got in becuase of commonly known and well documented security holes. You are sounding like a Microsoft apologist here.
Re:Ah, yes, the obligatory Linux advocacy
by
jareth-0205
·
· Score: 3, Informative
And that would be Linux, I suppose? Because no bugs ever creep into Linux, and there's never been a security flaw found.
That's the point! Bugs are much more likely to be found in an open system such as Linux because of the nature of Open source development - all people using the software can reporting / fixing bugs, not just the limited few inside a company. The parent poster is actually complimenting MS testing, just saying that it can never be as good as open source because of the numbers involved.
Re:Development costs
by
afantee
·
· Score: 2, Informative
>> 2003 is a server OS. MacOS X is not, despite Apples best attempts.
There is a thing called Mac OS X Server, you Windows idiot.
>> The only parts of MacOS that are used for serving stuff is the open source code, which effectively is built and tested by the community.
You are talking pure shit through your fat ass. What about WebObjects, NetInfo, Apple Remote Desktop, NetBoot and a host of other Apple sysadmin tools?
Re:Comparing with the UNIX model
by
mnmn
·
· Score: 2, Informative
Could you please explain why sendmail had over a decade of exploitable security bugs?
Yeah. there were'nt enough eyes on it, and Sendmail Inc really left debugging to the community. Dont forget it was eventually fixed. Noone is claiming Windows 3.1 is not buggy.. all software contains bugs somewhere. Only some sendmail bugs happened to live so long and were eventually fixed. That bug became famous because sendmail is so commonly used because its so stable.
I guess some other eye will need to report those bugs...
My pair are included in the many eyeballs working on free software. Expect bugs in redhat's own applications since they belong to one company and their profits. And then, compare OpenBSDs stability with anything else out there.
I understand your time limitations and lack of enthusiasm towards free software, I was submitting bugs and porting network drivers between linux kernel versions when I was working at the factory as general labour. Nowadays I'm working as tech support and busy with a bunch of cisco routers for certifications; building a future and making sure I somehow get paid for my efforts, but I respect and admire the free software movement. I will not be removing my eyeballs from that pool. And you can expect Linux to keep growing without a paid testing department.
-- "Give orange me give eat orange me eat orange give me eat orange give me you."
-Nim Chimpsky
Re:Development costs
by
afantee
·
· Score: 2, Informative
>> That they support probably two or three orders of magnitude more hardware is reason enough
What the fuck are you talking about?
Other than different CPU architecture, Mac OS X and Windows both support the same sort of hardware: ATI and nVidia GPU, Ethernet, USB, FireWire, 802.11b, SCSI and ATA Drive. Apple usually is years ahead of MS in adopting new technology: USB, FireWire, FireWire 800, BlueTooth, 802.11b, 802.11g, gigabit Ethernet, Rendezvous.
Apple is 60 times smaller than MS, but actually makes more software as well as better ones, in addition to lots of sexy hardware innovations.
Slashdot Mirror
There is a very interesting presentation on the design and development of windows NT/2000 presented on USENIX here (google HTML rendition here). I love to bash Microsoft too, but reading it, there are at least some decisions that I think they did right.
I don't know about the rest of you, but I am pretty sick of comments like this.
I use both Linux and Windows XP, and I for one have to say that Linux on the desktop is not exactly rock solid stable either. Parts of KDE crash all of the time (_especially_ KATE), OpenOffice.org is dog-slow. And all in all, I would say windows is substantially _more_ responsive than Linux is (on the desktop again).
I know that this post will probably be modded as trolling, but hell.. thats the state of things right now.
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer.
2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle.
3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test.
4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in.
5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes.
6) The build lab picks up the changes for the day and starts to compile.
7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue.
8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests.
9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such.
10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
I run the following system:
Pentium III @ 1ghz
256MB RAM
GeForce2 MX
RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel
KDE 3.0.3-8 RedHat
And I almost never have problems with KDE. I use Kate for almost all my programming, and I can count the number of times it's crashed on one hand with fingers left over.
You know that you can adjust how much CPU time KDE uses, right? I don't know about other distros, but for RedHat 8 it's under Extras - Preferences - Desktop Settings Wizard.
Nonetheless, the user base of MacOS as a server OS is trace.
That may be, but until recently Apple has not had an OS capable of large scale serving. I have used IRIX, Solaris and Windows in the past, but I find OS X to be the best of breed in terms of a do it all OS.
There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated. These guys don't mess around - they expect to have industrial strength support during the upgrade, and they expect there to be no regressions.
Ummmm, well, seeing as how the xserve is a very recent entry into the field, I would not expect many large deployments yet. However, the ability to provide over 5000 hits/second and saturating 3 T3 lines is pretty damned impressive. Or how about streaming 3000 live connections all at once? Or netbooting hundreds of machines all at once? As for sites that are using OS X, off the top of my head, I believe that there is certainly Apple.com, all of the Quicktime streaming they are doing for the movies on Apple.com. I am sure there are others that I am not aware of.
Apple is in an entirely different league - they can ship a trivial OS update that accidentally deletes entire hard disks worth of data and can get off by paying for a few hundred disk recoveries and having an "everybody makes mistakes" attitude. That kind of thing isn't really acceptable for the desktop, but the extreme loyalty of Apples customers means they can essentially get away with it. That simply doesn't fly when you run most of the worlds servers.
Well, Microsoft has proven you wrong here on innumerable occasions. I can't tell you how many times I have had to advise folks to reinstall Windows because of file/registry corruption, or had to deal with the implications of an infection by a virus or worm in someones database or email program that got in becuase of commonly known and well documented security holes. You are sounding like a Microsoft apologist here.
Visit Jonesblog and say hello.
>> 2003 is a server OS. MacOS X is not, despite Apples best attempts.
There is a thing called Mac OS X Server, you Windows idiot.
>> The only parts of MacOS that are used for serving stuff is the open source code, which effectively is built and tested by the community.
You are talking pure shit through your fat ass. What about WebObjects, NetInfo, Apple Remote Desktop, NetBoot and a host of other Apple sysadmin tools?
Could you please explain why sendmail had over a decade of exploitable security bugs? Yeah. there were'nt enough eyes on it, and Sendmail Inc really left debugging to the community. Dont forget it was eventually fixed. Noone is claiming Windows 3.1 is not buggy.. all software contains bugs somewhere. Only some sendmail bugs happened to live so long and were eventually fixed. That bug became famous because sendmail is so commonly used because its so stable. I guess some other eye will need to report those bugs... My pair are included in the many eyeballs working on free software. Expect bugs in redhat's own applications since they belong to one company and their profits. And then, compare OpenBSDs stability with anything else out there. I understand your time limitations and lack of enthusiasm towards free software, I was submitting bugs and porting network drivers between linux kernel versions when I was working at the factory as general labour. Nowadays I'm working as tech support and busy with a bunch of cisco routers for certifications; building a future and making sure I somehow get paid for my efforts, but I respect and admire the free software movement. I will not be removing my eyeballs from that pool. And you can expect Linux to keep growing without a paid testing department.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
>> That they support probably two or three orders of magnitude more hardware is reason enough
What the fuck are you talking about?
Other than different CPU architecture, Mac OS X and Windows both support the same sort of hardware: ATI and nVidia GPU, Ethernet, USB, FireWire, 802.11b, SCSI and ATA Drive. Apple usually is years ahead of MS in adopting new technology: USB, FireWire, FireWire 800, BlueTooth, 802.11b, 802.11g, gigabit Ethernet, Rendezvous.
Apple is 60 times smaller than MS, but actually makes more software as well as better ones, in addition to lots of sexy hardware innovations.