Posted by
CmdrTaco
on from the debugging-the-beast dept.
The Qube writes "As a followup to the in-depth story posted back in February regarding the history and the development of Windows NT, part 3 of the series of articles is now online.
It discusses the software testing procedures inside Microsoft."
Heres another one
by
ArchieBunker
·
· Score: 5, Funny
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
-- Only the State obtains its revenue by coercion. - Murray Rothbard
I'm definitely sticking with gnome
by
1nv4d3r
·
· Score: 4, Funny
From the article (not kidding!):
Originally, the KDE worked to upgrade its infrastructure to Windows 2000 on its own. But after spending 18 months evaluating and testing, Cornett realized they'd need some help. The department contacted Microsoft Consulting Services (MCS) to ask about architectural guidance, and hired a full-time technical account manager from Microsoft's Enterprise Services group. Eventually, the KDE joined Windows Server 2003 Rapid Adoption Program (RAP), which allowed them to begin working with the product early in its development process.
(ok, so when you look into it you're likely to realize that it's the Kentucky Dept of Education, but when skimming the article it caught my eye and I was really confused!)
Re:I'm definitely sticking with gnome
by
JJahn
·
· Score: 2, Insightful
I think its funny that it took them 18 months to figure out they needed help. Obviously Cornett is not all that gifted in the computer realm, given 18 months I'm pretty sure I could come up with something better than installing an MS beta on all my servers.
Besides the fact that as far as I can tell Windows 2003 offers very few benefits over the finally acceptably stable Windows 2000.
The video of this is out...
by
Anonymous Coward
·
· Score: 3, Funny
It starts off with a room. a huge room... the largest one you've ever seen...
The camera flies down, zooming in & out, between dozens of the ten million monkeys at ten million PCs, and back up to a control desk manned by straw-chewing yokels.
A screen flashes red
"Sir! Monkey number Y435A23J has come up with something that boots!"
The camera pans around to Bill Gates' face
"I call it... Windows 2008. Release it"
or something
Re:Text of the article (fixed formatting)
by
Omkar
·
· Score: 4, Funny
You know you're on slashdot when a post begins
Windows Server 2003: The Road To Gold
Part Three: Testing Windows
and ends
Can anyone recommend a way to get my cat off heroin? It would be much
appreciated.
All jokes aside...
by
KFury
·
· Score: 5, Insightful
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Re:All jokes aside...
by
asa
·
· Score: 4, Insightful
"Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes...."
Except that this article says nothing about the testing methodology that Microsoft uses. It describes how Microsoft helps certain customers test deployment. Deployment testing has little or nothing to do with software testing.
This is an article about how Microsoft has the budget to help "special" customers with a free "service" (not software) and frankly, the bits about offering cash-strapped school systems free consulting and test deployments sounds a lot more like a Microsoft press release than a software testing case study.
I was genuinely hoping to read about their software QA process. What a waste of 5 minutes.
No way this article is about software testing. This is about an evaluation lab where customers bring in their applications to show to Microsoft. It's a marketing puff-piece, that's all.
Where is the description of the test methodologies used? The bug escalation and change control systems? What sort of configuration control is used?
Development costs
by
BWJones
·
· Score: 3, Interesting
O.K., can anyone here tell me why Microsoft is spending an order of magnatude more $$'s to develop Windows than Apple is spending on developing OS X? It can't be testing because the Apple products appear so much more refined.
Re:Development costs
by
IamTheRealMike
·
· Score: 2, Insightful
2003 is a server OS. MacOS X is not, despite Apples best attempts. The only parts of MacOS that are used for serving stuff is the open source code, which effectively is built and tested by the community. MS include things like IIS/Active Directory as part of the Windows product, so more testing is needed.
It's also a lot more popular.
Re:Development costs
by
Ciderx
·
· Score: 3, Funny
So you didn't use OS X prior to version 10.1.5?
Re:Development costs
by
BWJones
·
· Score: 2, Insightful
2003 is a server OS. MacOS X is not, despite Apples best attempts.
I don't know what you are talking about. I have been using OS X as a server OS for some time now and it has got to be the easiest server OS to manage. It is more stable than W2003 server, easier to manage less expensive etc...etc...etc... I am running it here and in several other places in addition to my primary workstation that also hosts a couple of small bandwidth websites.
Re:Development costs
by
BWJones
·
· Score: 2, Informative
Nonetheless, the user base of MacOS as a server OS is trace.
That may be, but until recently Apple has not had an OS capable of large scale serving. I have used IRIX, Solaris and Windows in the past, but I find OS X to be the best of breed in terms of a do it all OS.
There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated. These guys don't mess around - they expect to have industrial strength support during the upgrade, and they expect there to be no regressions.
Ummmm, well, seeing as how the xserve is a very recent entry into the field, I would not expect many large deployments yet. However, the ability to provide over 5000 hits/second and saturating 3 T3 lines is pretty damned impressive. Or how about streaming 3000 live connections all at once? Or netbooting hundreds of machines all at once? As for sites that are using OS X, off the top of my head, I believe that there is certainly Apple.com, all of the Quicktime streaming they are doing for the movies on Apple.com. I am sure there are others that I am not aware of.
Apple is in an entirely different league - they can ship a trivial OS update that accidentally deletes entire hard disks worth of data and can get off by paying for a few hundred disk recoveries and having an "everybody makes mistakes" attitude. That kind of thing isn't really acceptable for the desktop, but the extreme loyalty of Apples customers means they can essentially get away with it. That simply doesn't fly when you run most of the worlds servers.
Well, Microsoft has proven you wrong here on innumerable occasions. I can't tell you how many times I have had to advise folks to reinstall Windows because of file/registry corruption, or had to deal with the implications of an infection by a virus or worm in someones database or email program that got in becuase of commonly known and well documented security holes. You are sounding like a Microsoft apologist here.
Re:Development costs
by
afantee
·
· Score: 2, Informative
>> 2003 is a server OS. MacOS X is not, despite Apples best attempts.
There is a thing called Mac OS X Server, you Windows idiot.
>> The only parts of MacOS that are used for serving stuff is the open source code, which effectively is built and tested by the community.
You are talking pure shit through your fat ass. What about WebObjects, NetInfo, Apple Remote Desktop, NetBoot and a host of other Apple sysadmin tools?
Re:Development costs
by
afantee
·
· Score: 2, Informative
>> That they support probably two or three orders of magnitude more hardware is reason enough
What the fuck are you talking about?
Other than different CPU architecture, Mac OS X and Windows both support the same sort of hardware: ATI and nVidia GPU, Ethernet, USB, FireWire, 802.11b, SCSI and ATA Drive. Apple usually is years ahead of MS in adopting new technology: USB, FireWire, FireWire 800, BlueTooth, 802.11b, 802.11g, gigabit Ethernet, Rendezvous.
Apple is 60 times smaller than MS, but actually makes more software as well as better ones, in addition to lots of sexy hardware innovations.
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Seriously....
by
wowbagger
·
· Score: 5, Insightful
All the Q/A in the world does you no good if you don't act upon what Q/A has found.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
{ {{edit,compile} while errors} link } while errors
Download to victim
boot victim. Watch hundreds of assertion failures from the OS scroll by on debugging console.
Shut down debugger as it has leaked all available memory
Start debugger again
Load my driver and test.
locate bugs in my driver, begin again
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
Re:Seriously....
by
Anonymous Coward
·
· Score: 2, Insightful
ISO-9000 cert mearly states that you HAVE a process, and have it documented. Oh and process can change easy enough, it is after all just a doc file.
Worked at one place where they mandated that on us. One dude documented how to format a floppy disk in DOS. Then another doc how to put a sticker on it.
Unless its taken seriously its neato to have but other than that...
NT4 was basicly NT3.51 with the graphical shell. NT3.51 was 2 years old by the time NT4 shipped... Few were serious about using it at the time. It was OS2/BSD/AIX/Novel or something like that. Put NT as my main server in 95? You would have heard 'ARE YOU NUTS' three miles away.
Its taken them this long to make NT work. Not to surprising. Mature OS's take awhile. Linux has had a quicker life cycle simply becuase there are more people working on it because they care. Not because its their job. And the 'new' mac os is built on top of an already stable OS BSD. One could make an argument that it IS a BSD variant. Much easier to make something that works when youve had 20 years of someone banging on it.
But back to your point of the asserts. If it was run like most qa depts. They didnt even LOOK at it. They had hundreds of qa monkeys and they ran their 'scenerio' scripts. They probably didnt even know HOW to look at a debug port. Where I work we MAKE our testers look at the error logs and get concerned about ERRORS.
Clearly you've never done development on a driver for ANY OS.
You don't want your development machine to be the machine your driver runs on. If your driver breaks, it could crash the machine, corrupt the file system, or any number of nasty things.
You also need to be able to inspect OS level objects, which you usually cannot do on a running system. So, most OS's provide a debug capability that "freezes" the system, and through a simple interface (usually serial) allows you to inspect the state of the whole system. However, to do that you need a second computer to do the development on.
So, I would suggest you stop trolling on/. and go out into the wider world, and learn something about the things you try to talk about.
For the record, the device driver I was developing was for a DSP based RF analysis board, which if you had spent any time at all checking my background as posted on/. you would have seen is what I do for a living.
Now, put down the mouse, drop your cock, go out into the world, and start actually learning something other than how to troll/. - we have plenty of those already.
The annoying thing about NT (4.0), is that now that it is almost rock-solid, MS no longer support it.
Those years of public beta testing certainly paid off.:)
--
\\ Mitch
Check this PPT to.
by
Karpe
·
· Score: 2, Informative
There is a very interesting presentation on the design and development of windows NT/2000 presented on USENIX here (google HTML rendition here). I love to bash Microsoft too, but reading it, there are at least some decisions that I think they did right.
Quote from the Ky. Dept. of Ed. customer.
by
hndrcks
·
· Score: 4, Insightful
"We're hoping to get a long-term lifespan out of Windows Server 2003 without having to do major upgrading."
People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code.
With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do.
Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money.
As part of the Microsoft culture, it appears that you've missed the point.
The problem is the 50 million lines of code itself.
I would have "managed" NT's testing by "not managing it" at all, and instead would have clipped out all those bells and whistles to make a much more trim and modular OS.
The code base is unecessarily large, from a functional point of view.
But just like the current SUV problem in America, it appears that Microsoft is dancing a tango with the consumers.
Microsoft produces shitty code that looks good on the screen, and the consumers say "ohh" and "ahh" while not minding the crashes and restrictions, and then Microsoft gets encouraged to produce more "pretty code".
I don't consider this problem to be fixable... we who know better and are less mediocre simply have to fend for ourselves and rely on the influence of our leadership to promote the Better Way.
This is the slow method of providing a good example for others to follow, which is the only leadership that matters.
Microsoft's billions are just a facade; consumer mediocrity is another facade; what will matter in the long run is bullet-proof code that more serves public needs instead of software-industry investors.
-- [You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
I'll try posting something original as opposed to the MS-bashing and the MS-bashing-bashing whilst remaining at least a little ontopic.
I think Microsoft would do well to test more and make less. Each incarnation of Windows seems to have brought disproportionately large improvements (or hindrances if you like) in the user interface, features, and resource consumption. Whilst a gradual accumulation of features and a slow increase in resource use is inevitable for any operating system I think Microsoft has been making their systems grow too much too quickly.
Microsoft seems to be running out of some new features to add to each new version of Windows to entice consumers are resorting to making their own features (notably,.NET and the like) in order to keep sales high. Unfortunately I think that in terms of features and UI they can't push the boundary too much further for the next few years (though obvious beyond that there will no doubt be new ideas).
As such I feel that MS would benefit from focusing on testing instead of adding new things. Consolidation is often just as helpful as (if not better than) augmentation, particularly for larger systems. I feel that sales would remain high if Windows had no new features or UI but could genuinely be considered as stable as alternatives.
--
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
Comparing with the UNIX model
by
mnmn
·
· Score: 4, Interesting
Having a monolithic kernel + garbage, monolithic reigstery, no way to control the revisions of DLLs, worse than RPM package management etc is what NT is all about. The UNIX model is making small highly useable tools that you understand perfectly, and making them work together. In win32, if you cant work with an API, you make a new API, and we now have several generations of APIs to deal with just to access the screen, resources, widgets etc in windows. You have a standard API that has worked for over a decade for most UNIXen, and the ones that do change their API purge the last version and replace it with a similar system.... think sysv vs BSD, Solaris sysctls vs Linux sysctls.
Thats why the win32 system spirals into complexity, no matter how much money is pumped into development or testing. Of course one of the best things about windows is also one of the worst, that vendors developing their own drivers for their hardware might make incompatible or bad drivers, or ones that step on the feet of other installed drivers in the system. In the Linux kernel, all the drivers are present before the testers and are considered while any major change takes place.. such as the VM or switching to 64-bit cpu. This is true for most other UNIXen where drivers are sent to the unix vendor for testing as well, but thats not as efficient as the Linux model.
And then the number of eyeballs testing Linux and FreeBSD is a phenomenon Microsoft cant copy. The free software community does not work for a paycheck, but theres more sincerity towards the software than there would be for a proprietary software. Free Software can be a matter of ego and gives a sense of competition with Microsoft. You cant buy that. This I believe is the biggest reason why colossal manhours are poured into free software development, while some of these developers work the rest of their days as data entry or office clerks, even mcdonalds.
-- "Give orange me give eat orange me eat orange give me eat orange give me you."
-Nim Chimpsky
Re:Comparing with the UNIX model
by
mnmn
·
· Score: 2, Informative
Could you please explain why sendmail had over a decade of exploitable security bugs?
Yeah. there were'nt enough eyes on it, and Sendmail Inc really left debugging to the community. Dont forget it was eventually fixed. Noone is claiming Windows 3.1 is not buggy.. all software contains bugs somewhere. Only some sendmail bugs happened to live so long and were eventually fixed. That bug became famous because sendmail is so commonly used because its so stable.
I guess some other eye will need to report those bugs...
My pair are included in the many eyeballs working on free software. Expect bugs in redhat's own applications since they belong to one company and their profits. And then, compare OpenBSDs stability with anything else out there.
I understand your time limitations and lack of enthusiasm towards free software, I was submitting bugs and porting network drivers between linux kernel versions when I was working at the factory as general labour. Nowadays I'm working as tech support and busy with a bunch of cisco routers for certifications; building a future and making sure I somehow get paid for my efforts, but I respect and admire the free software movement. I will not be removing my eyeballs from that pool. And you can expect Linux to keep growing without a paid testing department.
-- "Give orange me give eat orange me eat orange give me eat orange give me you."
-Nim Chimpsky
Re:Comparing with the UNIX model
by
acarey
·
· Score: 2, Insightful
Interesting "comparison" you make.
"Monolithic" kernel: NT kernel is object based ("everything is an object" [file, ACL, semaphore, process, etc.] VS Unix "everything is a file"), mach-style modular and (as of W2K) fully reentrant. Allows for multiple independent subsystem operation (e.g. Win32 and OS2/1.0 and POSIX); significantly more advanced than (e.g.) Linux and BSD.
"Monolithic" registry: while I do not particularly agree with the implementation of the Windows Registry I do find some merit in the concept; a central, format-common storage system for hardware, OS, subsystem and application-level configuration settings. The Windows Registry is divided into several separate databases that appear as a single entity to the user (hence your mistaken assessment as "monolithic") but in actuality can be swapped in and out and backed up separately. Microsoft presumably decided on a binary based system for reasons of performance and encryption, particularly in the Windows 9x line where ACLs are not available to secure the files (on an NT-line system a text-only Registry would be feasible, if perhaps slow, because ACLs could be used to allow fine-grain control to the Registry).
No way to control the revision of DLLs; more or less fixed in NT 4 with Add/Remove Programs control panel applet, so long as applications obeyed the model; unfortunately, many didn't. Situation further improved (i.e. basically eliminated for the casual user) with Windows Installer 1.0 in 1998. Fixed foreever with the release of.NET side-by-side assemblies in 2000. Significantly better package management than the abominable RPM system used by Redhat, although I must confess a personal preference for Debian's APT.
Small but highly useable tools: COM components and object linking and embedding mean that virtually every application installed on a Windows system can make its functionality available programmatically. Object reuse and rapid application development significantly more advanced than CORBA based systems. The promise of open source may eventually eclipse the binary-based COM model (and its successors, e.g..NET assemblies) but for the meantime open source code reuse is basically limited to cut and paste which is not exactly what object oriented proponents had in mind. Not helped by open source's fascination with C (as opposed to object oriented languages).
API evolution: agreed. But the problem is somewhat alleviated by use of COM interface separation; for instance, it is still possible (although not recommended) to target, e.g. DAO for desktop database work rather than ADO or ADO.NET.
"Many eyes" philosophy of open source: agreed, in theory this should render Microsoft obsolete, in practice it has merely spurred Microsoft to create a better product.
There, that should provoke some flames:)
-- -- "I believe the human being and the fish can coexist peacefully." - George W. Bush, 29 September 2000
Interesting article, but not really about testing.
by
deranged+unix+nut
·
· Score: 3, Informative
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer. 2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle. 3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test. 4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in. 5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes. 6) The build lab picks up the changes for the day and starts to compile. 7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue. 8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests. 9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such. 10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
I run the following system: Pentium III @ 1ghz 256MB RAM GeForce2 MX RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel KDE 3.0.3-8 RedHat
And I almost never have problems with KDE. I use Kate for almost all my programming, and I can count the number of times it's crashed on one hand with fingers left over.
You know that you can adjust how much CPU time KDE uses, right? I don't know about other distros, but for RedHat 8 it's under Extras - Preferences - Desktop Settings Wizard.
Re:Ah, yes, the obligatory Linux advocacy
by
jareth-0205
·
· Score: 3, Informative
And that would be Linux, I suppose? Because no bugs ever creep into Linux, and there's never been a security flaw found.
That's the point! Bugs are much more likely to be found in an open system such as Linux because of the nature of Open source development - all people using the software can reporting / fixing bugs, not just the limited few inside a company. The parent poster is actually complimenting MS testing, just saying that it can never be as good as open source because of the numbers involved.
apple.com is the #hardware site on the Web
by
afantee
·
· Score: 2, Insightful
>> There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated.
I am not so sure about that. In a recent survey of hardware site, apple.com is the #1 site with about 3.5 mln unique visitors, while hp.com is a distant second with 2.7 mln. Apple Store is probably the best online store with annual sale in billions of dollars, Apple also host the most popular QuickTime movie trailers, and iTunes Music Store sell over a mln songs a week.
In fact, I haven't found another Web site with the combination of e-commerce, movie trailer and music download anywhere near the sophistication of apple.com. And guess what, apple.com is powered by Mac OS X Server.
Re:Let's see-
by
TheNetAvenger
·
· Score: 2, Insightful
I run the following system: Pentium III @ 1ghz 256MB RAM GeForce2 MX RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel KDE 3.0.3-8 RedHat
And I almost never have problems with KDE
Key words 'almost never'...
How about a 200mhz 80mb Laptop running Windows 2000 since Beta 1 back in fall of 1997 up to currently running WindowsXP.
And to this day it has only crashed once, when the user popped the IDE Hard Drive out accidentally while the system was running.
However, with a nice journalled file system, putting back in the hard drive and just turning it back on was all that was needed(no integrity check:) )
People can debate stability and who's OS is better and runs the best all day here. Neither WindowsNT/XP or any *nix is perfect, PERIOD.
So give up the my OS is the best and never will crash and runs faster stuff. This just won't be true across the vast array of hardware these OSes run on and the devices they support. THERE IS NO SUCH THING, SORRY.
And BTW, the story about the little 200mhz laptop is 100% true. It was my personal test system for years (low end test system for the past few years) and it has yet to see a BSOD or lockup, except when (admittedly) 'I' accidentally popped out the Hard Drive while it was doing some serious number crunching.
It also is used far more than an average person's PC. I still use it as a development test platform for software speed analysis, since it is now on the low end of the spectrum.
And it is still a cute notebook that runs well, is great for Music, Video, and Audio books and at one time I loved more than any other computer. *tear in eye*:)
I didn't say that Linux was perfect. Hell, the kernel that ships with RedHat 8 (2.4.18-7 I think) would freeze during heavy IDE activity unless I disabled DMA (by passing the kernel the ide=nodma parameter), which is why I upgraded it to 2.4.21-pre5-ac2.
Obviously all OSes are going to have bugs. The question is, how severe are those bugs? How frequent do they manifest themselves? KDE hasn't crashed on me in a very long time, and Linux hasn't crashed since I upgraded to 2.4.21-pre5-ac2 (and prior to "upgrading" to RedHat 8 in March, back when I ran RH 7.2 I think the OS crashed maybe once).
My experiences running Win2k on the same system haven't been nearly as good.
Slashdot Mirror
Well, I guess you have to make sure those "features" work correctly
QA Engineer 1: "Does it compile?"
QA Engineer 2: "Yup."
QA Engineer 1: "Okay, I'm declaring it GM and releasing it to manufacturing."
QA Engineer 2: "It's 'Miller Time'!"
But wasn't it supposed to be Not Tested?
"Testing? What's that? If it compiles, it is good, if it boots up it is perfect." - Linus Torvalds
Only the State obtains its revenue by coercion. - Murray Rothbard
From the article (not kidding!):
Originally, the KDE worked to upgrade its infrastructure to Windows 2000 on its own. But after spending 18 months evaluating and testing, Cornett realized they'd need some help. The department contacted Microsoft Consulting Services (MCS) to ask about architectural guidance, and hired a full-time technical account manager from Microsoft's Enterprise Services group. Eventually, the KDE joined Windows Server 2003 Rapid Adoption Program (RAP), which allowed them to begin working with the product early in its development process.
(ok, so when you look into it you're likely to realize that it's the Kentucky Dept of Education, but when skimming the article it caught my eye and I was really confused!)
It starts off with a room. a huge room... the largest one you've ever seen...
The camera flies down, zooming in & out, between dozens of the ten million monkeys at ten million PCs, and back up to a control desk manned by straw-chewing yokels.
A screen flashes red
"Sir! Monkey number Y435A23J has come up with something that boots!"
The camera pans around to Bill Gates' face
"I call it... Windows 2008. Release it"
or something
You know you're on slashdot when a post begins
Windows Server 2003: The Road To Gold
Part Three: Testing Windows
and ends
Can anyone recommend a way to get my cat off heroin? It would be much appreciated.
All jokes aside, the amount of testing and the build process for NT is one of the most tightly organized and comprehensive testing methodologies in existence.
Rather than take 'miller time' pot shots at Microsoft, the real takeaway is the understanding that, no matter how rigorous the testing and build process, there is a complexity limit where a unified one-organization nightly fix-build-test model simply can't provide a product of suitable quality.
Better to acknowledge the best-of-breed methodology Microsoft uses to test their OSes, and conclude that while this breed works okay for applications, a world-class operating system needs peer review and distributed open source development to create a quality, secure product.
Kevin Fox
No way this article is about software testing. This is about an evaluation lab where customers bring in their applications to show to Microsoft. It's a marketing puff-piece, that's all.
Where is the description of the test methodologies used? The bug escalation and change control systems? What sort of configuration control is used?
O.K., can anyone here tell me why Microsoft is spending an order of magnatude more $$'s to develop Windows than Apple is spending on developing OS X? It can't be testing because the Apple products appear so much more refined.
Visit Jonesblog and say hello.
Testing is a vital part of programming. Tests should always be written PRIOR to the programming. This allows you to think of problems before they arise. In some sense it seems as if MS is avoiding this by having someone "come over to fix the problem within 20 minutes." HHowever, given the diverse environments there does not seem to be a direct solution for them. The EEC seems to be a huge step forward to finding where code breaks for a given customer but it doesnt solve any security holes (which should have been addressed pre-coding when you come up w/ tests for your software). As for all the joking about MS programmers in this forum so far, i find it kinda rediculous that people do that. People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money. Since its all about the money you obviously roll out the product and try to patch it as fast as you can when somone does find a bug that got by Q&A and the testers. You need to find a balance between testing a product completely and releasing a product to make money. Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
I used to do driver development for NT4.0. As such, I had a "victim" machine and a "debugging" machine, linked via a serial cable. The victim runs my driver, and I do my development and debug using the debugging machine to access the kernel debugger on the victim.
A normal cycle of development went something like this:
Note: this was a fresh install of NT4.0 debugging, with SP4. No third party apps (other than my own) installed. This was using Microsoft's WinDBG.
Now, I don't know about Microsoft's developers, but I regard an assertion failure as a failure - i.e. a bug to be fixed. Having HUNDREDS of them in released code is just unacceptable. Using an ASSERT() as a debugging printf() is wrong.
So either a) the MS developers have a different view of things than I do, or b) the MS developers were allowing hundreds of easily identified problems to go into release.
Now, EVERY non-trivial software project's lead engineer must make a decision at every release - "Do I fix these bugs and slip the release, or fix these bugs in the next release?" And EVERY lead will allow some bugs to slip. Usually, those bugs are deemed minor - spelin mestaches (sic), layout errors, things like that.
But to have a) hundreds of assertion failures, which give you file and line number of the error, and b) a memory leak in your debugger bad enough that you can WATCH it leak away hundreds of megabytes of memory each time, and to allow that to go out? Ugh.
Now I am sure that MS Q/A found those errors - if not they are far more incompetent that I am willing to assume they are. So clearly Q/A was overruled by management - "We don't care, ship it anyway!"
And that is the central problem to ANY Q/A department - if management overrules them, and forces a shipment anyway, then how do you blame Q/A?
I've said this before, and I shall say it again now: this is one of the places a real ISO-9000 standard can be useful. If the spec sayth "Lo, and the release candidate code shall have no bugs open against it in the bug tracking system, and any bugs that exist shall be clearly targeted to later revisions, and Q/A shall findth no undocumented bugs in the code, or the release shall be slipped, and the bugs corrected, AMEN!" then Q/A can say "OK, if you want to throw our ISO-9000 cert out the door, then by all means override us and ship."
(Yes, that won't prevent management from simply targeting all bugs to a later revision and shipping, but it at least forces some consideration of the consequences to me made."
www.eFax.com are spammers
The annoying thing about NT (4.0), is that now that it is almost rock-solid, MS no longer support it.
:)
Those years of public beta testing certainly paid off.
\\ Mitch
There is a very interesting presentation on the design and development of windows NT/2000 presented on USENIX here (google HTML rendition here). I love to bash Microsoft too, but reading it, there are at least some decisions that I think they did right.
"We're hoping to get a long-term lifespan out of Windows Server 2003 without having to do major upgrading."
These guys obviously aren't students of "Licensing 6.0".
Everyone will start to cheer when you put on your sailin' shoes.
People that laugh at the products MS produces really do have to look hard at how THEY would manage and TEST 50 MILLION lines of code. With 50 million lines of code you're looking at virtually an infinate number of tests to run, which is obviously impossible to do. Thus you either have to roll out a product that hasn't been 100% tested because of its size or keep testing and never make money.
... we who know better and are less mediocre simply have to fend for ourselves and rely on the influence of our leadership to promote the Better Way.
This is the slow method of providing a good example for others to follow, which is the only leadership that matters.
Microsoft's billions are just a facade; consumer mediocrity is another facade; what will matter in the long run is bullet-proof code that more serves public needs instead of software-industry investors.
As part of the Microsoft culture, it appears that you've missed the point.
The problem is the 50 million lines of code itself.
I would have "managed" NT's testing by "not managing it" at all, and instead would have clipped out all those bells and whistles to make a much more trim and modular OS. The code base is unecessarily large, from a functional point of view.
But just like the current SUV problem in America, it appears that Microsoft is dancing a tango with the consumers. Microsoft produces shitty code that looks good on the screen, and the consumers say "ohh" and "ahh" while not minding the crashes and restrictions, and then Microsoft gets encouraged to produce more "pretty code". I don't consider this problem to be fixable
[You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
I'll try posting something original as opposed to the MS-bashing and the MS-bashing-bashing whilst remaining at least a little ontopic.
.NET and the like) in order to keep sales high. Unfortunately I think that in terms of features and UI they can't push the boundary too much further for the next few years (though obvious beyond that there will no doubt be new ideas).
I think Microsoft would do well to test more and make less. Each incarnation of Windows seems to have brought disproportionately large improvements (or hindrances if you like) in the user interface, features, and resource consumption. Whilst a gradual accumulation of features and a slow increase in resource use is inevitable for any operating system I think Microsoft has been making their systems grow too much too quickly.
Microsoft seems to be running out of some new features to add to each new version of Windows to entice consumers are resorting to making their own features (notably,
As such I feel that MS would benefit from focusing on testing instead of adding new things. Consolidation is often just as helpful as (if not better than) augmentation, particularly for larger systems. I feel that sales would remain high if Windows had no new features or UI but could genuinely be considered as stable as alternatives.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Its a fine line and MS has done a fairly good job given the size of their code base and the pressure on them from the comsumer to get new products out in a timely way.
Whoa, there. Since when is it the consumer who is pressuring MS for new products? It seems to me that it's MS who has been rushing new "features" into production and pressuring consumers to upgrade. I don't know of anyone who had a burning desire to upgrade to Word 2K or Windows XP. The fact that others were upgrading and causing compatibility problems was the compelling reason.
Having a monolithic kernel + garbage, monolithic reigstery, no way to control the revisions of DLLs, worse than RPM package management etc is what NT is all about. The UNIX model is making small highly useable tools that you understand perfectly, and making them work together. In win32, if you cant work with an API, you make a new API, and we now have several generations of APIs to deal with just to access the screen, resources, widgets etc in windows. You have a standard API that has worked for over a decade for most UNIXen, and the ones that do change their API purge the last version and replace it with a similar system.... think sysv vs BSD, Solaris sysctls vs Linux sysctls.
Thats why the win32 system spirals into complexity, no matter how much money is pumped into development or testing. Of course one of the best things about windows is also one of the worst, that vendors developing their own drivers for their hardware might make incompatible or bad drivers, or ones that step on the feet of other installed drivers in the system. In the Linux kernel, all the drivers are present before the testers and are considered while any major change takes place.. such as the VM or switching to 64-bit cpu. This is true for most other UNIXen where drivers are sent to the unix vendor for testing as well, but thats not as efficient as the Linux model.
And then the number of eyeballs testing Linux and FreeBSD is a phenomenon Microsoft cant copy. The free software community does not work for a paycheck, but theres more sincerity towards the software than there would be for a proprietary software. Free Software can be a matter of ego and gives a sense of competition with Microsoft. You cant buy that. This I believe is the biggest reason why colossal manhours are poured into free software development, while some of these developers work the rest of their days as data entry or office clerks, even mcdonalds.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer.
2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle.
3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test.
4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in.
5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes.
6) The build lab picks up the changes for the day and starts to compile.
7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue.
8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests.
9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such.
10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
I run the following system:
Pentium III @ 1ghz
256MB RAM
GeForce2 MX
RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel
KDE 3.0.3-8 RedHat
And I almost never have problems with KDE. I use Kate for almost all my programming, and I can count the number of times it's crashed on one hand with fingers left over.
You know that you can adjust how much CPU time KDE uses, right? I don't know about other distros, but for RedHat 8 it's under Extras - Preferences - Desktop Settings Wizard.
>> There simply are no deployments of the type talked about in the article, with hundreds of domain servers needing to be migrated.
I am not so sure about that. In a recent survey of hardware site, apple.com is the #1 site with about 3.5 mln unique visitors, while hp.com is a distant second with 2.7 mln. Apple Store is probably the best online store with annual sale in billions of dollars, Apple also host the most popular QuickTime movie trailers, and iTunes Music Store sell over a mln songs a week.
In fact, I haven't found another Web site with the combination of e-commerce, movie trailer and music download anywhere near the sophistication of apple.com. And guess what, apple.com is powered by Mac OS X Server.
I run the following system:
:) )
:)
Pentium III @ 1ghz
256MB RAM
GeForce2 MX
RedHat 8, upgraded to the 2.4.21-pre5-ac2 kernel
KDE 3.0.3-8 RedHat
And I almost never have problems with KDE
Key words 'almost never'...
How about a 200mhz 80mb Laptop running Windows 2000 since Beta 1 back in fall of 1997 up to currently running WindowsXP.
And to this day it has only crashed once, when the user popped the IDE Hard Drive out accidentally while the system was running.
However, with a nice journalled file system, putting back in the hard drive and just turning it back on was all that was needed(no integrity check
People can debate stability and who's OS is better and runs the best all day here. Neither WindowsNT/XP or any *nix is perfect, PERIOD.
So give up the my OS is the best and never will crash and runs faster stuff. This just won't be true across the vast array of hardware these OSes run on and the devices they support. THERE IS NO SUCH THING, SORRY.
And BTW, the story about the little 200mhz laptop is 100% true. It was my personal test system for years (low end test system for the past few years) and it has yet to see a BSOD or lockup, except when (admittedly) 'I' accidentally popped out the Hard Drive while it was doing some serious number crunching.
It also is used far more than an average person's PC. I still use it as a development test platform for software speed analysis, since it is now on the low end of the spectrum.
And it is still a cute notebook that runs well, is great for Music, Video, and Audio books and at one time I loved more than any other computer. *tear in eye*
I didn't say that Linux was perfect. Hell, the kernel that ships with RedHat 8 (2.4.18-7 I think) would freeze during heavy IDE activity unless I disabled DMA (by passing the kernel the ide=nodma parameter), which is why I upgraded it to 2.4.21-pre5-ac2.
Obviously all OSes are going to have bugs. The question is, how severe are those bugs? How frequent do they manifest themselves? KDE hasn't crashed on me in a very long time, and Linux hasn't crashed since I upgraded to 2.4.21-pre5-ac2 (and prior to "upgrading" to RedHat 8 in March, back when I ran RH 7.2 I think the OS crashed maybe once).
My experiences running Win2k on the same system haven't been nearly as good.