Big Challenges for Vista Bug Hunters

The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."

Comments

Time

[Kangburra]

This was a similar story for Windows ME, in the end the time to release became more important than the quality of the product. I would like to see Vista delayed until it's ready, even if that's not for six more months. In my view that would earn Microsoft more points than meeting a schedule and then needing to (service) patch it fairly quickly.

my $0.02

Re:Time

[Hennell]

But they will probally figure that delaying it for 6 more months suggests its still not good enough. So even if its not ready they'd prefer the patch route.

All a bit academic really, the adverage person will get it good or not and slashdoters will complain it crashes and is insecure. Its like the circle of life.

Re:Time

[h2g2bob]

Well, I suppose it's not like they've got a reputation to protect...

Though to be fair people will have a go at MS if it's late OR if it has bugs, so they can't win. That said, either way people will be forced to use it.

Re:Time

[Ucklak]

Getting it out the door is more important than if it's ready.

At the time it's released, Mac will have another OS out but that's beside the point. That only matters to people that are `on the fence` OS wise and not a significant number. In the halls of the OS engineers, it matters as it proves what insiders at MS have said that "Microsoft isn't able to ship products anymore."

When SP1 is released, there will be hoopla and hype that Vista will have even more features, be more stable, and even more secure.

Re:Time

[Dan Ost]

either way people will be forced to use it.

I don't know. Few people upgrade their version of Windows
unless they're getting a new machine. However, lots of people
are discovering that a 3 year old computer is perfectly capable
of doing what they need it to do and so doesn't really
need to be replaced unless the hardware is failing.

Even more interesting, for the first time ever, Apple's
offerings are starting to be percieved as a real alternative
that is, arguably, comparably priced.

It will be interesting to see what impact, if any, Vista has
on the sale of Apple's computers.

Statistics!

[shreevatsa]

More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft.

In other words, about nine out of ten systems using Vista crashed at some point. And that's counting just those who sent the crash reports. :-)

Re:Statistics!

[petes_PoV]

yes, it is a lot - in fact it's quite disgraceful.

The only reason people are even considering Vista (given this level of failure) is that we have become so used to PCs crashing that we're blind to it.

Imagine if your TV switched itself off and took a minute or two to come back as often as your computer crashes. You'd send it back and demand a proper one - that worked.

I'm aware of all the arguments about 3rd party software/drivers etc. being the real cause. That's as maybe, but if the Vista architecture was designed to be robust and tolerant to these faults, then the problems we see just wouldn't arise. After all, it's not like MS can say that faulty drivers are a new problem

Re:Statistics!

[endemoniada]

I'm getting a little tired of people comparing a computer running an OS to a TV set or similar.

The TV will do one thing, and one thing only. That's displaying an analog signal as moving images and sound. That's all. The day that's all Windows will ever have to do, that's the day you can demand a refund.

Re:Statistics!

[Tim C]

I don't know about Vista, but on XP the default is to submit crash reports for all crashes. That includes software you are yourself developing. Yes, you soon learn to switch that off, but at least some of those reports will be from developers writing code for Vista and submitting crash reports for their own software (or testers doing so).

Re:Huh..

[Angostura]

Depends how you interpret the figures. I have a stable, well configured Mac. Last week, I had a dodgy 3rd party app that crashed 3 times. Each time the Apple crash reporter asked me to send a report to Apple.

If I had been running a beta version of the operating system I would have gone ahead and sent, on the grounds that it might have been a bad interaction between app and OS. In the event I said no.

You need to know more about what is triggering the crash reporter.

Critical Path sunglasses

[Anonymous Coward]

One team I was on had a pair of "critical path sunglasses", whoever was on the critical path got to keep them, the joke being they wouldnt see daylight for so long their eyeballs would suffer when they did go outside -hence the sunglasses.

This meeting looks like a triage session to me: someone goes through the list of bugs, dividing them up into ones to focus on, and which to ignore. Triage has always been a microsof strength: making the decisions as to exactly how buggy something can be and yet still ship successfully.

There's one person with a keyboard (probably hooked to a real vista pc),the foreground laptop is running outlook on what looks rather suspiciously like WinXP (that or vista without Aero, which is roughly how laptops will run it anyway). There arent enough empty soft drink cans or laptops plugged in to AC power for a long lived meeting. Rooms get messier after about six hours, even with less people in.

high crash rate

[192939495969798999]

more than half a million installed, and 450,000 sent back crash data... so even if we assume it was nearly a million, that's 50% crash rate. I'd guess it was way higher even than that. So, over half of the systems were crashing bad enough for Microsoft to care? Wow! What exactly is the problem? I thought this was supposed to be a newer, better version. Wouldn't we see a 10% crash rate, or even a 25% crash rate at this point if things were really getting any better?

High failure rate

[DigitAl56K]

"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft"

So the liklihood of a crash is near 100% ?

Re:Yes, you can use hardware to track down bugs...

[3vi1]

If you run a system with pure MS drivers and quality hardware you'll never see a BSOD

Does Microsoft make any video drivers that can even run the Aero GUI? Or, by "quality hardware" do you mean really old generic hardware supported by MS's generic drivers? I've never seen any MS drivers for my scanner, printer, webcam... basically nothing in my "quality" system other than the MS keyboard. So, I don't know if what you propose is even possible.

While there are lots of entries in the MS KB that are totally due to 3rd party drivers, there are many that are not. So, even if what you propose is possible, it's unlikely to be correct.

If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.

Okay, you're high, aren't you?

Exploits come out every other day to crash IE And, there are hundreds if not thousands of MS KB articles regarding Outlook crashes. Office? I've seen it crash many times due to internal bugs. And, when it crashes, you can't shut down Windows because it tells you that you must exit all Office apps first - thanks to MS's wonderful OS integration.

Re:Why try, and fail to, reinvent the wheel...

[I'm Don Giovanni]

yeah, because *every* OS must be unix-based because it's perfect in every way, can't be improved. The peak of OS tech was achieved 30 years ago. [face_rollseyes]

As for Apple, I wish that they had succeeded with Copeland, so there would still be at least one mainstream OS that wasn't Unix or NT based. Apple chose NexT (the BSD version (there was also an NT version)) out of desperation, not because they so loved BSD or Unix.

Haste makes waste.

[dpbsmith]

"Rushing to fix bugs" is like rushing any other meticulous job. It can't be done.

Bugs are the consequence of rushing the job in the first place. (Taking time, is of course, necessary but not sufficient).

If Microsoft knows a way to "rush" bug fixes without compromising quality, they would have been able to "rush" their development without creating the bugs in the first place.