Inside The Development of Windows NT: Testing

The Qube writes "As a followup to the in-depth story posted back in February regarding the history and the development of Windows NT, part 3 of the series of articles is now online. It discusses the software testing procedures inside Microsoft."

Comments

All jokes aside...

[KFury]

All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.

Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.

Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.

Re:All jokes aside...

[asa]

"Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes...."

Except that this article says nothing about the testing methodology that Microsoft uses. It describes how Microsoft helps certain customers test deployment. Deployment testing has little or nothing to do with software testing.

This is an article about how Microsoft has the budget to help "special" customers with a free "service" (not software) and frankly, the bits about offering cash-strapped school systems free consulting and test deployments sounds a lot more like a Microsoft press release than a software testing case study.

I was genuinely hoping to read about their software QA process. What a waste of 5 minutes.

--Asa

Re:All jokes aside...

[Anonymous Coward]

How long have you been with Microsoft? Do they have better benefits than they used to?

How is blind anti-microsoft arrogance any better than FUD? My comment, if you read beyond the first line, is that these are smart people doing a hard job, and yet it still doesn't work. It's a little too easy to say that Windows breaks because its engineers are stupid, but that's not the reason. And no, I've never worked for them and never will. I work for Google.

Testing??? Not at all.

[the eric conspiracy]

No way this article is about software testing. This is about an evaluation lab where customers bring in their applications to show to Microsoft. It's a marketing puff-piece, that's all.

Where is the description of the test methodologies used? The bug escalation and change control systems? What sort of configuration control is used?

Re:I'm definitely sticking with gnome

[JJahn]

I think its funny that it took them 18 months to figure out they needed help. Obviously Cornett is not all that gifted in the computer realm, given 18 months I'm pretty sure I could come up with something better than installing an MS beta on all my servers.

Besides the fact that as far as I can tell Windows 2003 offers very few benefits over the finally acceptably stable Windows 2000.

Re:Development costs

[IamTheRealMike]

2003 is a server OS. MacOS X is not, despite Apples best attempts. The only parts of MacOS that are used for serving stuff is the open source code, which effectively is built and tested by the community. MS include things like IIS/Active Directory as part of the Windows product, so more testing is needed.

It's also a lot more popular.

Seriously....

[wowbagger]

All the Q/A in the world does you no good if you don't act upon what Q/A has found.

I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.

A normal cycle of development went something like this:

1. { {{edit,compile} while errors} link } while errors
2. Download to victim
3. boot victim. Watch hundreds of assertion failures from the OS scroll by on debugging console.
4. Shut down debugger as it has leaked all available memory
5. Start debugger again
6. Load my driver and test.
7. locate bugs in my driver, begin again

Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.

Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.

So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.

Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.

But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.

Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"

And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?

I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."

(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."

Re:Seriously....

[Anonymous Coward]

ISO-9000 cert mearly states that you HAVE a process, and have it documented. Oh and process can change easy enough, it is after all just a doc file.

Worked at one place where they mandated that on us. One dude documented how to format a floppy disk in DOS. Then another doc how to put a sticker on it.

Unless its taken seriously its neato to have but other than that...

NT4 was basicly NT3.51 with the graphical shell. NT3.51 was 2 years old by the time NT4 shipped... Few were serious about using it at the time. It was OS2/BSD/AIX/Novel or something like that. Put NT as my main server in 95? You would have heard 'ARE YOU NUTS' three miles away.

Its taken them this long to make NT work. Not to surprising. Mature OS's take awhile. Linux has had a quicker life cycle simply becuase there are more people working on it because they care. Not because its their job. And the 'new' mac os is built on top of an already stable OS BSD. One could make an argument that it IS a BSD variant. Much easier to make something that works when youve had 20 years of someone banging on it.

But back to your point of the asserts. If it was run like most qa depts. They didnt even LOOK at it. They had hundreds of qa monkeys and they ran their 'scenerio' scripts. They probably didnt even know HOW to look at a debug port. Where I work we MAKE our testers look at the error logs and get concerned about ERRORS.

NT

[bobm17ch]

The annoying thing about NT (4.0), is that now that it is almost rock-solid, MS no longer support it.

Those years of public beta testing certainly paid off. :)

Ah, yes, the obligatory Linux advocacy

[Anonymous Brave Guy]

Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.

And that would be Linux, I suppose? Because no bugs ever creep into Linux, and there's never been a security flaw found. Except if you read Bugtraq, of course.

This wasn't even the point of the article -- though it might have been more interesting if it had been -- and yet still someone comes in here and starts making out that Microsoft have bad QA and the open source model is vastly superior. How sad.

Quote from the Ky. Dept. of Ed. customer.

[hndrcks]

"We're hoping to get a long-term lifespan out of Windows Server 2003 without having to do major upgrading."

These guys obviously aren't students of "Licensing 6.0".

Re:Testing

[LaCosaNostradamus]

People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money.

As part of the Microsoft culture, it appears that you've missed the point.

The problem is the 50 million lines of code itself.

I would have "managed" NT's testing by "not managing it" at all, and instead would have clipped out all those bells and whistles to make a much more trim and modular OS. The code base is unecessarily large, from a functional point of view.

But just like the current SUV problem in America, it appears that Microsoft is dancing a tango with the consumers. Microsoft produces shitty code that looks good on the screen, and the consumers say "ohh" and "ahh" while not minding the crashes and restrictions, and then Microsoft gets encouraged to produce more "pretty code". I don't consider this problem to be fixable ... we who know better and are less mediocre simply have to fend for ourselves and rely on the influence of our leadership to promote the Better Way. This is the slow method of providing a good example for others to follow, which is the only leadership that matters. Microsoft's billions are just a facade; consumer mediocrity is another facade; what will matter in the long run is bullet-proof code that more serves public needs instead of software-industry investors.

Hmmph.

[Dthoma]

I'll try posting something original as opposed to the MS-bashing and the MS-bashing-bashing whilst remaining at least a little ontopic.

I think Microsoft would do well to test more and make less. Each incarnation of Windows seems to have brought disproportionately large improvements (or hindrances if you like) in the user interface, features, and resource consumption. Whilst a gradual accumulation of features and a slow increase in resource use is inevitable for any operating system I think Microsoft has been making their systems grow too much too quickly.

Microsoft seems to be running out of some new features to add to each new version of Windows to entice consumers are resorting to making their own features (notably, .NET and the like) in order to keep sales high. Unfortunately I think that in terms of features and UI they can't push the boundary too much further for the next few years (though obvious beyond that there will no doubt be new ideas).

As such I feel that MS would benefit from focusing on testing instead of adding new things. Consolidation is often just as helpful as (if not better than) augmentation, particularly for larger systems. I feel that sales would remain high if Windows had no new features or UI but could genuinely be considered as stable as alternatives.

Re:Development costs

[BWJones]

2003 is a server OS. MacOS X is not, despite Apples best attempts.

I don't know what you are talking about. I have been using OS X as a server OS for some time now and it has got to be the easiest server OS to manage. It is more stable than W2003 server, easier to manage less expensive etc...etc...etc... I am running it here and in several other places in addition to my primary workstation that also hosts a couple of small bandwidth websites.

apple.com is the #hardware site on the Web

[afantee]

>> There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated.

I am not so sure about that. In a recent survey of hardware site, apple.com is the #1 site with about 3.5 mln unique visitors, while hp.com is a distant second with 2.7 mln. Apple Store is probably the best online store with annual sale in billions of dollars, Apple also host the most popular QuickTime movie trailers, and iTunes Music Store sell over a mln songs a week.

In fact, I haven't found another Web site with the combination of e-commerce, movie trailer and music download anywhere near the sophistication of apple.com. And guess what, apple.com is powered by Mac OS X Server.

Re:Let's see-

[TheNetAvenger]

I run the following system:
Pentium III @ 1ghz
256MB RAM
GeForce2 MX
RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel
KDE 3.0.3-8 RedHat

And I almost never have problems with KDE

Key words 'almost never'...

How about a 200mhz 80mb Laptop running Windows 2000 since Beta 1 back in fall of 1997 up to currently running WindowsXP.

And to this day it has only crashed once, when the user popped the IDE Hard Drive out accidentally while the system was running.

However, with a nice journalled file system, putting back in the hard drive and just turning it back on was all that was needed(no integrity check :) )

People can debate stability and who's OS is better and runs the best all day here. Neither WindowsNT/XP or any *nix is perfect, PERIOD.

So give up the my OS is the best and never will crash and runs faster stuff. This just won't be true across the vast array of hardware these OSes run on and the devices they support. THERE IS NO SUCH THING, SORRY.

And BTW, the story about the little 200mhz laptop is 100% true. It was my personal test system for years (low end test system for the past few years) and it has yet to see a BSOD or lockup, except when (admittedly) 'I' accidentally popped out the Hard Drive while it was doing some serious number crunching.

It also is used far more than an average person's PC. I still use it as a development test platform for software speed analysis, since it is now on the low end of the spectrum.

And it is still a cute notebook that runs well, is great for Music, Video, and Audio books and at one time I loved more than any other computer. *tear in eye* :)

Re:Comparing with the UNIX model

[acarey]

Interesting "comparison" you make.

• "Monolithic" kernel: NT kernel is object based ("everything is an object" [file, ACL, semaphore, process, etc.] VS Unix "everything is a file"), mach-style modular and (as of W2K) fully reentrant. Allows for multiple independent subsystem operation (e.g. Win32 and OS2/1.0 and POSIX); significantly more advanced than (e.g.) Linux and BSD.
• "Monolithic" registry: while I do not particularly agree with the implementation of the Windows Registry I do find some merit in the concept; a central, format-common storage system for hardware, OS, subsystem and application-level configuration settings. The Windows Registry is divided into several separate databases that appear as a single entity to the user (hence your mistaken assessment as "monolithic") but in actuality can be swapped in and out and backed up separately. Microsoft presumably decided on a binary based system for reasons of performance and encryption, particularly in the Windows 9x line where ACLs are not available to secure the files (on an NT-line system a text-only Registry would be feasible, if perhaps slow, because ACLs could be used to allow fine-grain control to the Registry).
• No way to control the revision of DLLs; more or less fixed in NT 4 with Add/Remove Programs control panel applet, so long as applications obeyed the model; unfortunately, many didn't. Situation further improved (i.e. basically eliminated for the casual user) with Windows Installer 1.0 in 1998. Fixed foreever with the release of .NET side-by-side assemblies in 2000. Significantly better package management than the abominable RPM system used by Redhat, although I must confess a personal preference for Debian's APT.
• Small but highly useable tools: COM components and object linking and embedding mean that virtually every application installed on a Windows system can make its functionality available programmatically. Object reuse and rapid application development significantly more advanced than CORBA based systems. The promise of open source may eventually eclipse the binary-based COM model (and its successors, e.g. .NET assemblies) but for the meantime open source code reuse is basically limited to cut and paste which is not exactly what object oriented proponents had in mind. Not helped by open source's fascination with C (as opposed to object oriented languages).
• API evolution: agreed. But the problem is somewhat alleviated by use of COM interface separation; for instance, it is still possible (although not recommended) to target, e.g. DAO for desktop database work rather than ADO or ADO.NET.
• "Many eyes" philosophy of open source: agreed, in theory this should render Microsoft obsolete, in practice it has merely spurred Microsoft to create a better product.

There, that should provoke some flames :)