Big Challenges for Vista Bug Hunters
The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."
This is where special software automatically exercises programs rapidly while looking for errors.
and this software, folks, goes by the name "internet explorer".
Push Button, Receive Bacon
This was a similar story for Windows ME, in the end the time to release became more important than the quality of the product. I would like to see Vista delayed until it's ready, even if that's not for six more months. In my view that would earn Microsoft more points than meeting a schedule and then needing to (service) patch it fairly quickly.
my $0.02
Common sense is not so common
Half a million installs, and 450k of them crashed.
Color me unimpressed.
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."
That is the kind of information that can get people fired...
Obiously 50,000 users didn't test anything at all.
Just wanted to thank god for linux.
Nah, it's just a fancy place to bring visitors to. They probably bought it off a 50's sci-fi movie set, spinning reels and all.
Demented But Determined.
Surely Microsoft could use this to sift through such an vast quantity of code: http://www.google.com/codesearch
Just please don't start hurling chairs at my Karma!
Hmm.. exactly 450,000 machines eh?
What are the chances? Damn.
It looks like a meeting. People rarely seem professional at meetings.
The efficiency of a group of people is not the sum of intelligence, it's the sum of stupidity.
IIRC it's the monitoring room for the Office Crash Assistant, the place where you send your data to after you crash. They analyse this data attempting to find patterns that lead to crashes. (I'm not sure how good this helps Office, but for Windows itself it's an excellent tool to find broken driver releases.)
I'm Rocco. I'm the +5 Funny man.
I think we should take this as the way to release complex software... No matter how many tests you can perform the system will be buggy in some way.. Better to patch it as often as you can, there are simply too many case scenarios out there to live with..
Anyone spot the need for shades in the room? Perhaps someone at Micrsoft does indeed think the future is that bright ...?
And what a team demo like this has to do with bug fixing I don't know?
In terms of directly dealing with the problem(s) and imperfections of a software release ... everybody sitting in a room taking notes on someone's demo ... I am not sure the world is ready for what real software enginnering really looks like. I (and the vast majority of my team mates) do our best and most productive work when keeping such 'mass migrations to a meeting room' to an absolute minimum.
And, however much the software engineering world is predominantly male, why is there only 1 lady in the room?
Anyone else wonder if this picture was taken for padding the media story? If so, then I have to admit they did work hard at making it look messy and busy. Unfocussed and secondary ... but messy and busy, indeed.
AidanapwordI for one would love to know more about the tools they use for automated testing.
In my company, we have a build & testing server running compiler and NUnit tests for all data-layer tests (complete tests like "load all of everything" and more specific tests like "authorise user with known bad credentials - expect login-failure") alongside NUnitForms tests for the application-layer (random, frantic clicking's everywhere and specific user-journey tests).
All in all, it's quite a good system for rooting out the majority of bugs, but I'm always looking for ways of improving this side of things.
I can only imagine Microsoft must use similar techniques, but obviously a tad more intensive; can anyone shed any light?
throw new NoSignatureException();
As usual with microsoft products.. i will wait for SP 2 to arrive before putting any thought into actually getting it...
Hardware is almost required to debug some low-level system code. Real-time stuff, like device drivers and scheduler really requires hardware tracer to determine what happened and when.
With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD. If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.
It's the crappy hardware and badly written drivers that cause the crashes. That's the difference with Apple.... since they control the hardware there's less crashes due to bad hardware and there are fewer third-party drivers for Mac hardware. The software is probably the same quality.
This is where special software automatically exercises programs rapidly while looking for errors.
Maybe they should try hooking WGA up to this thing.
which I can't replace as I can't see paying a monthly fee.
p.s. my TIVO, a LINUX device, freezes now 2 or 3 times a month.. I have to switch it off (unplug it actually) and gosh yes, wait a minute (*5 or 6 actually) to resume my normal broadcast day.
every day http://en.wikipedia.org/wiki/Special:Random
Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just
From every point of view it seems to make more sense. They spend less money, get a more reliable product that can run very nicely on existing hardware, get some good press for a change, and benefit from the work of unpaid open-source programmers all over the world. But it isn't their way.
Is that crash data in general, or crash data on Microsoft applications or OS parts? I ask because windows can send crash data for any application which crashes...
The AACS key is NOT 0xF606EEFD628B1CA427BEA93A9CA9773F
tells us that 450,000/500,000 gives us a crash rate of exactly 90% seems about right for a windows system , can't think why they haven't released it earlier.
C'mon, don't you know anything? It is obviously where their staff of 10,000 monkeys press buttons randomly on keyboards of pcs running Vista. This is how they test for their bugs. If the monkeys can't find it, Joe Sixpack should be safe and secure.
You're ignorant. From your post, it seems you think that the crashes were OS crashes, which is not true. Most (or all?) of the crash information is about applications crashing, not the whole OS. Any application, not just OS code or Microsoft apps.
It's more akin to you turning on your TV and finding out that your channels suck.
The AACS key is NOT 0xF606EEFD628B1CA427BEA93A9CA9773F
Something that I haven't heard much recently is about Vista compatibility. MS has said before that it will be compatible and for most software and hardware, it was true in previous versions. But there were enough exceptions. ME was supposed to be backwards compatible. But many specialized drivers had to be written for it. XP definitely required some driver updates. Since Vista is a architectural change, so one would except some compatibility issues especially when DRM and enhanced security is thrown into the mix.
Technically would MS classify incompatibility as not a a bug, especially if is does not cause a crash?
Well, there's spam egg sausage and spam, that's not got much spam in it.
One team I was on had a pair of "critical path sunglasses", whoever was on the critical path got to keep them, the joke being they wouldnt see daylight for so long their eyeballs would suffer when they did go outside -hence the sunglasses.
This meeting looks like a triage session to me: someone goes through the list of bugs, dividing them up into ones to focus on, and which to ignore. Triage has always been a microsof strength: making the decisions as to exactly how buggy something can be and yet still ship successfully.
There's one person with a keyboard (probably hooked to a real vista pc),the foreground laptop is running outlook on what looks rather suspiciously like WinXP (that or vista without Aero, which is roughly how laptops will run it anyway). There arent enough empty soft drink cans or laptops plugged in to AC power for a long lived meeting. Rooms get messier after about six hours, even with less people in.
more than half a million installed, and 450,000 sent back crash data... so even if we assume it was nearly a million, that's 50% crash rate. I'd guess it was way higher even than that. So, over half of the systems were crashing bad enough for Microsoft to care? Wow! What exactly is the problem? I thought this was supposed to be a newer, better version. Wouldn't we see a 10% crash rate, or even a 25% crash rate at this point if things were really getting any better?
stuff |
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft"
So the liklihood of a crash is near 100% ?
Sounds like typical end of project 'big bang' testing. All those issues they ignored in development? Let's fix them now and fast! I would hate to be the MS QA person.
(Yes I am aware I used singular, it was a joke, OK?)
putting the 'B' in LGBTQ+
Waiting for SP2 will take around 2 years after Vista is released. Assuming an early '07 launch, SP1 should be out by the end of '07. By then, both Microsoft and other hardware vendors should have their bugs & drivers worked out. For XP, I waited until SP1a and it worked fine.
/.'ers would even touch a brand new release of Windows. I just see it as moving from 500,000 beta-testers to 50,000,000 beta-testers.
It's funny, though, that neither you nor I nor a lot of other
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."
...and the other 50,000 uninstalled before it could crash. Seriously, why would you install this before 2009? Or at all?
Has anyone considered that MS is actually trying to make a decent OS here? Sure, it's fun to poke fun at them for all the crashes and bugs, but isn't that the ENTIRE point of the 'release early, release often' mindset?
At least they're trying to find their bugs, at least they are running a widespread beta.
As usual with microsoft products, I will wait for an integrated, WGA-disabled version to arrive via BitTorrent before putting any thought into actually getting it.
You are welcome on my lawn.
... but rather, fixing (or finding) the causes of these crashes are. Even if you assume pessimistically that every 45 crashes are cause by one bug/issue, that's still... 10,000 issues. Onward to January 2007!
Fedorec Core 4, Ubuntu 6.
IANAL but write like a drunk one.
But not Linux, because then they would have to give back to the community.
IANAL but write like a drunk one.
"... half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data ..."
or, "Half a million installs of Vista, and almost half a million have crashed"
This is something that Microsoft likes to do a lot. Wait to fix things after it is out.
You accuse MS of this but go on to praise Apple. Are you forgetting that Apple released what was essentially betaware as the RTM build of OSX 10.0?
And how about a more recent example: iTunes 7.0 was so buggy when it was released that Apple had to release iTunes 7.0.1 within a week. And it's still buggy as hell.
-- "I never gave these stories much credence." - HAL 9000
"Rushing to fix bugs" is like rushing any other meticulous job. It can't be done.
Bugs are the consequence of rushing the job in the first place. (Taking time, is of course, necessary but not sufficient).
If Microsoft knows a way to "rush" bug fixes without compromising quality, they would have been able to "rush" their development without creating the bugs in the first place.
"How to Do Nothing," kids activities, back in print!
You be here -> . (#16363681)
.
.
.
.
.
.
.
.
.
.
.
.
Sarcasm be here -> . (#16363007)
"Blah blah blah." - [citation needed]
The problem with this statistic is that it is almost meaningless without more information. No real conclusion can be drawn. This could mean a number of different things. For instance:
* Vista could have one bug (and only one) that all of the systems encountered and successfully reported
* Same as above, but the conclusion drawn is that Vista 5ux0rz cause it has a huge bug
* Vista has 450,000 different bugs, each system reporting a different bug
* One system crashed 450,000 times and reported each time and the facts were skewed into calling each report a unique system.
The possibilites are endless. Journalists really need to start providing statistics that actually add value to the article rather than adding more mud (or FUD?) to the water.
"It's not whether you win or lose, it's how drunk you get." -- H. J. Simpson
It would've been really funny if the laptop you can see in front was either a Mac (which'd be too obvious to get into the room) or running Linux (which'd still look like a non-Mac PC).
look! it's a bird, it's a plane, it's....a girl? yes, a girl browsing Slashdot on Linux
We don't know there was "only one" lady in the room. We can only be sure that (a) there is at most one lady in the room and (b) that there is exactly one woman in the room. This is assuming the photographer is male.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
So Mario Garzia is Microsoft's "Director of Windows Reliability". Now there's an oxymoron for the ages. Hey Mario, you're fired!
"The chart indicates a decline in Office 2007 flaws from the start of 2006 through last week"
:O im shocked!
since the graph can never be y=0, is the NYT implying that Office 2007 will always be flawed?
Doesn't this require the third-party application's developer to have signed the application's code using a VeriSign code signing certificate? At 499 USD per year plus whatever your state charges for incorporating, I'd imagine that such a certificate is too expensive for many hobbyist developers of freeware and shareware to purchase and renew.
I used to run the Windows Feedback team which built crash reporting into Windows XP (if you don't like "send error reports" you can blame me :-)
There are 2 things that people should know to put the 450K reporting systems into clearer perspective:
1.) Crash reports typically form a clear and very steep pareto curve. In fact, the top 1% of problems on XP accounted for over half of the reports. In other words; in general people tend to hit the same issues. This makes sense from a common sense perspective as well; one would expect a defect in a common code path in a popular application to be encountered more frequently than uncommon code paths in unpopular applications. It also means that emphasis can be placed on fixing *the most important* issues for the majority of customers and working down the curve to cover as much breadth as possible.
2.) The way crash reporting works is that it handles any unhandled exception which results in an app crash, or any kernel bugcheck. This means that 3rd party applications and drivers are reported as well as crashes in the OS and MS applications. When last I worked on this, there were over 6 million unique .exes reported and more than 95% of the reports by volume were in 3rd party code paths. I expect that distribution is a little more skewed towards MS code over the Vista development cycle since it was under devel, but my bet is that it is still predominantly 3rd party.
At the end of the day, the way that I hope people look at this is that it is better to have data about problems and act upon it, than to not collect the data. 450K Vista systems reporting errors means that the Vista team and 3rd party app/driver developers have had the data and opportunity to act upon it. This is a good thing.