Big Challenges for Vista Bug Hunters

The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."

Comments

special software

[macadamia_harold]

This is where special software automatically exercises programs rapidly while looking for errors.

and this software, folks, goes by the name "internet explorer".

Re:special software

[suv4x4]

I know a guy who used to work on test suites at Microsoft who has since quit, given their awful attitude towards bugs in Vista, and got a Mac

You'll see this kind of attitude in every bigger software company. I've had personal experiences like this in Adobe and Macromedia with their flagship products. Features are dropped, specs constantly changing and inconsistent between teams.

In some cases, you can spot the same feature implemented twice in source, with different interfaces, in different locations, and code linking randomly to one or the other, or even both (imagine updating this).

The bugs to be fixed are selected first for how obvious they are (likely to occur) and not how critical they are. This is why it's common that bugs that can totally wreck operation and lead to data loss may be left, if the occurence is rare or unlikely.

Everybody is in stress and the main goal is that you get the reviewed bug off your shoulders: if it's mildly reminiscent to something else, it's marked duplicate. If you can't reproduce it quickly, it's marked as fixed or not reproducible. Tricky bugs are marked "fact of life" or "deffered".

Successful companies and their products grow, but the way the products and resources are managed does not scale. Instead, programmers are expected to churn a major release every X months, screw everything else, and keep the cash flowing, the investors happy.

With Windows, we have a successful product that supports a huge ecosystem of applications (including legacy support), localization, usage cases etc. It's natural that in time, updates will become more rare, and will be much slower and more expensive to produce.

The trend of software-as-a-service is not coincidental with this situation. In 5 to 10 years the base software we use might be so complex and tough to work on, that the only way it can be sustained is by small, regular payments, and the updates will be small, incremental, security/performance oriented. No more big releases, no more rushes to fix bugs in the last moment.

This is the way evolution works. The other route is, of course, revolution...

Time

[Kangburra]

This was a similar story for Windows ME, in the end the time to release became more important than the quality of the product. I would like to see Vista delayed until it's ready, even if that's not for six more months. In my view that would earn Microsoft more points than meeting a schedule and then needing to (service) patch it fairly quickly.

my $0.02

Re:Time

[scsirob]

Further delay aint happening. Vista will be out the door, regardless of the remaining bugs. They still have 'patch tuesday' to make updates, and the installation sequence itself already includes an initial update phase. So any really big bugs that remain present in the RTM build can still be fixed later.

Re:Time

[Hennell]

But they will probally figure that delaying it for 6 more months suggests its still not good enough. So even if its not ready they'd prefer the patch route.

All a bit academic really, the adverage person will get it good or not and slashdoters will complain it crashes and is insecure. Its like the circle of life.

Re:Time

[h2g2bob]

Well, I suppose it's not like they've got a reputation to protect...

Though to be fair people will have a go at MS if it's late OR if it has bugs, so they can't win. That said, either way people will be forced to use it.

Re:Time

[Ucklak]

Getting it out the door is more important than if it's ready.

At the time it's released, Mac will have another OS out but that's beside the point. That only matters to people that are `on the fence` OS wise and not a significant number. In the halls of the OS engineers, it matters as it proves what insiders at MS have said that "Microsoft isn't able to ship products anymore."

When SP1 is released, there will be hoopla and hype that Vista will have even more features, be more stable, and even more secure.

Re:Time

[Dan Ost]

either way people will be forced to use it.

I don't know. Few people upgrade their version of Windows
unless they're getting a new machine. However, lots of people
are discovering that a 3 year old computer is perfectly capable
of doing what they need it to do and so doesn't really
need to be replaced unless the hardware is failing.

Even more interesting, for the first time ever, Apple's
offerings are starting to be percieved as a real alternative
that is, arguably, comparably priced.

It will be interesting to see what impact, if any, Vista has
on the sale of Apple's computers.

Re:Time

[mackyrae]

XP is going really cheap now. I think one of my friends got it for $80 or something. With Vista about to be released, they're trying to get the old XP backstock out the door. Just buy your XP disks now and hold them until the quads come out.

Huh..

[Anonymous Coward]

Half a million installs, and 450k of them crashed.

Color me unimpressed.

Re:Huh..

[Angostura]

Depends how you interpret the figures. I have a stable, well configured Mac. Last week, I had a dodgy 3rd party app that crashed 3 times. Each time the Apple crash reporter asked me to send a report to Apple.

If I had been running a beta version of the operating system I would have gone ahead and sent, on the grounds that it might have been a bad interaction between app and OS. In the event I said no.

You need to know more about what is triggering the crash reporter.

Re:Huh..

[alohatiger]

Or maybe 499,999 didn't crash, and one of them crashed 450,000 times.

I wouldn't want to be the guy

[Anonymous Coward]

"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."

That is the kind of information that can get people fired...

Re:I wouldn't want to be the guy

[jonnyj]

The real world is probably worse than the statistics suggest.

I tried to install Vista on three PCs, all of which passed Microsoft's Ready for Vista testing tool, but all three failed before they were able to sent any crash data back to Microsoft. Two installs hung due because Vista didn't like my SATA / motherboard combination. The other got its knickers twisted over my partitioning scheme. And that was before I got a chance to find out if any of my other hardware (printer, scanner, TV card, firewire, network, graphics, CD/DVD, monitor, soundcard, suspend/resume, camera, etc) had any kind of working support.

Recent Linux kernels work perfectly with all this hardware. Could Vista be the first Microsoft OS that lags behind Linux for hardware support?

Statistics!

[shreevatsa]

More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft.

In other words, about nine out of ten systems using Vista crashed at some point. And that's counting just those who sent the crash reports. :-)

Re:Statistics!

[ImaNihilist]

That's really what happened in North Korea, they just don't want to admit it. Some noob installed Vista on one of the nuclear control computers, and then it crashed, and boom.

Now the world will be destroyed, and we'll find out it was really Steve Ballmer's plan all along...then he'll throw a chair at something.

Begun the dark times have.

Re:Statistics!

[petes_PoV]

yes, it is a lot - in fact it's quite disgraceful.

The only reason people are even considering Vista (given this level of failure) is that we have become so used to PCs crashing that we're blind to it.

Imagine if your TV switched itself off and took a minute or two to come back as often as your computer crashes. You'd send it back and demand a proper one - that worked.

I'm aware of all the arguments about 3rd party software/drivers etc. being the real cause. That's as maybe, but if the Vista architecture was designed to be robust and tolerant to these faults, then the problems we see just wouldn't arise. After all, it's not like MS can say that faulty drivers are a new problem

Re:Statistics!

[endemoniada]

I'm getting a little tired of people comparing a computer running an OS to a TV set or similar.

The TV will do one thing, and one thing only. That's displaying an analog signal as moving images and sound. That's all. The day that's all Windows will ever have to do, that's the day you can demand a refund.

Re:Statistics!

[Tim C]

I don't know about Vista, but on XP the default is to submit crash reports for all crashes. That includes software you are yourself developing. Yes, you soon learn to switch that off, but at least some of those reports will be from developers writing code for Vista and submitting crash reports for their own software (or testers doing so).

450,000 of 500,000 people report crashes ?

[Anonymous Coward]

Obiously 50,000 users didn't test anything at all.

Just wanted to thank god for linux.

Re:FUD?

[Dr. Eggman]

Nah, it's just a fancy place to bring visitors to. They probably bought it off a 50's sci-fi movie set, spinning reels and all.

This surely is the solution to sift through it all

[QuatermassX]

Surely Microsoft could use this to sift through such an vast quantity of code: http://www.google.com/codesearch

Just please don't start hurling chairs at my Karma!

Re:FUD?

[nstlgc]

IIRC it's the monitoring room for the Office Crash Assistant, the place where you send your data to after you crash. They analyse this data attempting to find patterns that lead to crashes. (I'm not sure how good this helps Office, but for Windows itself it's an excellent tool to find broken driver releases.)

Yes, you can use hardware to track down bugs...

[klubar]

Hardware is almost required to debug some low-level system code. Real-time stuff, like device drivers and scheduler really requires hardware tracer to determine what happened and when.

With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD. If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.

It's the crappy hardware and badly written drivers that cause the crashes. That's the difference with Apple.... since they control the hardware there's less crashes due to bad hardware and there are fewer third-party drivers for Mac hardware. The software is probably the same quality.

Re:Yes, you can use hardware to track down bugs...

[ookaze]

Hardware is almost required to debug some low-level system code. Real-time stuff, like device drivers and scheduler really requires hardware tracer to determine what happened and when

Huh ? This is called a serial console terminal, and I wouldn't call a terminal a 'hardware tracer'. Other OS just use a serial console to debug device drivers and scheduler, with most debugging done in software.

With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD

BS ! I've had a (documented on MS bugs site) BSOD just after install with WinXP for my Adaptec card, and not a cheap one.
Unfortunately, the WinXP native drivers caused the BSOD, I even had to install Linux on the PC (which worked perfectly) to be able to install the driver from the vendor instead of the MS one, so that WinXP don't crash anymore.
Unless you tell me Adaptec are not quality hardware (the card bought in 1999 is still working great BTW, and was not crashing in Win9x).

If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash

No, but I've lost complete big documents because of Word (he couldn't read its own saved files, thankfully I have OOo now, that saved me several times already), I won't even talk about IE.

It's the crappy hardware and badly written drivers that cause the crashes. That's the difference with Apple.... since they control the hardware there's less crashes due to bad hardware and there are fewer third-party drivers for Mac hardware. The software is probably the same quality

In my case, I know that's complete BS (that crappy hardware causes the crashes), as I've run Linux on the same hardware as my only Windows client.
And while the Linux tagged along just fine for weeks, the WinXP couldn't last more than a week without locking up or becoming unusable.
Is it really badly written drivers or crappy OS ? I had an old Creative Live ! card that was dying (and finally died).
WinXP was BSODing regularly when it was dying, then when it died, just BSOD at every boot. I then installed Linux on the PC, and it just tagged along, never crashed, telling me it couldn't initialise the driver, the driver saying it couldn't initialise the card.

Re:Yes, you can use hardware to track down bugs...

[3vi1]

If you run a system with pure MS drivers and quality hardware you'll never see a BSOD

Does Microsoft make any video drivers that can even run the Aero GUI? Or, by "quality hardware" do you mean really old generic hardware supported by MS's generic drivers? I've never seen any MS drivers for my scanner, printer, webcam... basically nothing in my "quality" system other than the MS keyboard. So, I don't know if what you propose is even possible.

While there are lots of entries in the MS KB that are totally due to 3rd party drivers, there are many that are not. So, even if what you propose is possible, it's unlikely to be correct.

If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.

Okay, you're high, aren't you?

Exploits come out every other day to crash IE And, there are hundreds if not thousands of MS KB articles regarding Outlook crashes. Office? I've seen it crash many times due to internal bugs. And, when it crashes, you can't shut down Windows because it tells you that you must exit all Office apps first - thanks to MS's wonderful OS integration.

Re:Yes, you can use hardware to track down bugs...

[metamatic]

With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD.

I just had XP not merely BSOD, but destroy itself so it wouldn't boot any more, after I installed a set of Microsoft security updates.

It was using no 3rd party drivers, and it wasn't a hardware problem because I was running it under VMware and other Windows images continued to run just fine.

Why try, and fail to, reinvent the wheel...

[rsidd]

Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just

• Take Linux, or one of the BSDs (like Apple did)
• Spend small amounts of money improving it (all that's really needed is improved device drivers)
• Spend some money on improving Wine (it would be really easy for them, compared to anyone else who wants to do it), et voila -- near-perfect backward compatibility (certainly far better than Apple's MacOS 9 -> MacOS X or PowerPC->Intel moves)

From every point of view it seems to make more sense. They spend less money, get a more reliable product that can run very nicely on existing hardware, get some good press for a change, and benefit from the work of unpaid open-source programmers all over the world. But it isn't their way.

Re:Why try, and fail to, reinvent the wheel...

[I'm Don Giovanni]

yeah, because *every* OS must be unix-based because it's perfect in every way, can't be improved. The peak of OS tech was achieved 30 years ago. [face_rollseyes]

As for Apple, I wish that they had succeeded with Copeland, so there would still be at least one mainstream OS that wasn't Unix or NT based. Apple chose NexT (the BSD version (there was also an NT version)) out of desperation, not because they so loved BSD or Unix.

So simple maths

[stewwy]

tells us that 450,000/500,000 gives us a crash rate of exactly 90% seems about right for a windows system , can't think why they haven't released it earlier.

Ignorant

[rbarreira]

You're ignorant. From your post, it seems you think that the crashes were OS crashes, which is not true. Most (or all?) of the crash information is about applications crashing, not the whole OS. Any application, not just OS code or Microsoft apps.

It's more akin to you turning on your TV and finding out that your channels suck.

What about compatibility?

[UnknowingFool]

Something that I haven't heard much recently is about Vista compatibility. MS has said before that it will be compatible and for most software and hardware, it was true in previous versions. But there were enough exceptions. ME was supposed to be backwards compatible. But many specialized drivers had to be written for it. XP definitely required some driver updates. Since Vista is a architectural change, so one would except some compatibility issues especially when DRM and enhanced security is thrown into the mix.

Technically would MS classify incompatibility as not a a bug, especially if is does not cause a crash?

Re:so it's like my Tivo

[Antique Geekmeister]

That sounds like a thermal problem. Have you tried opening the case and cleaning the air vents, and making sure the fans are all working properly?

high crash rate

[192939495969798999]

more than half a million installed, and 450,000 sent back crash data... so even if we assume it was nearly a million, that's 50% crash rate. I'd guess it was way higher even than that. So, over half of the systems were crashing bad enough for Microsoft to care? Wow! What exactly is the problem? I thought this was supposed to be a newer, better version. Wouldn't we see a 10% crash rate, or even a 25% crash rate at this point if things were really getting any better?

High failure rate

[DigitAl56K]

"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft"

So the liklihood of a crash is near 100% ?

Big bang testing

[plopez]

Sounds like typical end of project 'big bang' testing. All those issues they ignored in development? Let's fix them now and fast! I would hate to be the MS QA person.

(Yes I am aware I used singular, it was a joke, OK?)

Haste makes waste.

[dpbsmith]

"Rushing to fix bugs" is like rushing any other meticulous job. It can't be done.

Bugs are the consequence of rushing the job in the first place. (Taking time, is of course, necessary but not sufficient).

If Microsoft knows a way to "rush" bug fixes without compromising quality, they would have been able to "rush" their development without creating the bugs in the first place.