Big Challenges for Vista Bug Hunters
The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."
This is where special software automatically exercises programs rapidly while looking for errors.
and this software, folks, goes by the name "internet explorer".
Push Button, Receive Bacon
This was a similar story for Windows ME, in the end the time to release became more important than the quality of the product. I would like to see Vista delayed until it's ready, even if that's not for six more months. In my view that would earn Microsoft more points than meeting a schedule and then needing to (service) patch it fairly quickly.
my $0.02
Common sense is not so common
No matter how much bug fixing they do, their products are, for the most part, still going to suck. Users will still get the lingering bug-fix as a service pack, also enabling Microsoft to butt-fuck it's users with some more DRM.
Half a million installs, and 450k of them crashed.
Color me unimpressed.
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."
That is the kind of information that can get people fired...
WTF is the "Button Room" about? I doubt they actually have special hardware for software testing...
# cat
Damn, my RAM is full of llamas.
Obiously 50,000 users didn't test anything at all.
Just wanted to thank god for linux.
Surely Microsoft could use this to sift through such an vast quantity of code: http://www.google.com/codesearch
Just please don't start hurling chairs at my Karma!
Or more accurately - the USERS.
How much, if any, of this "testing" is to establish whether anyone who will have to use Vista be able to do so without going mad?
We know that Vista and Office Vista (or whatever the next version of Office will be called) will be the most bloated software onslaught known to man. Sure, Clippy is dead, but what about all the other "productivity enhancements" that are there simply to ensure that the MS product managers who pushed through these changes will get their pay rises?
Rant rant rant I could go on all day.
"And the meaning of words; when they cease to function; when will it start worrying you?"
Hmm.. exactly 450,000 machines eh?
What are the chances? Damn.
>>did anyone notice ALL the chairs are taken?
That is done on purpose so Ballmer won't throw them anymore.
It looks like a meeting. People rarely seem professional at meetings.
The efficiency of a group of people is not the sum of intelligence, it's the sum of stupidity.
I think we should take this as the way to release complex software... No matter how many tests you can perform the system will be buggy in some way.. Better to patch it as often as you can, there are simply too many case scenarios out there to live with..
Anyone spot the need for shades in the room? Perhaps someone at Micrsoft does indeed think the future is that bright ...?
And what a team demo like this has to do with bug fixing I don't know?
In terms of directly dealing with the problem(s) and imperfections of a software release ... everybody sitting in a room taking notes on someone's demo ... I am not sure the world is ready for what real software enginnering really looks like. I (and the vast majority of my team mates) do our best and most productive work when keeping such 'mass migrations to a meeting room' to an absolute minimum.
And, however much the software engineering world is predominantly male, why is there only 1 lady in the room?
Anyone else wonder if this picture was taken for padding the media story? If so, then I have to admit they did work hard at making it look messy and busy. Unfocussed and secondary ... but messy and busy, indeed.
AidanapwordI for one would love to know more about the tools they use for automated testing.
In my company, we have a build & testing server running compiler and NUnit tests for all data-layer tests (complete tests like "load all of everything" and more specific tests like "authorise user with known bad credentials - expect login-failure") alongside NUnitForms tests for the application-layer (random, frantic clicking's everywhere and specific user-journey tests).
All in all, it's quite a good system for rooting out the majority of bugs, but I'm always looking for ways of improving this side of things.
I can only imagine Microsoft must use similar techniques, but obviously a tad more intensive; can anyone shed any light?
throw new NoSignatureException();
As usual with microsoft products.. i will wait for SP 2 to arrive before putting any thought into actually getting it...
Hardware is almost required to debug some low-level system code. Real-time stuff, like device drivers and scheduler really requires hardware tracer to determine what happened and when.
With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD. If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.
It's the crappy hardware and badly written drivers that cause the crashes. That's the difference with Apple.... since they control the hardware there's less crashes due to bad hardware and there are fewer third-party drivers for Mac hardware. The software is probably the same quality.
This is where special software automatically exercises programs rapidly while looking for errors.
Maybe they should try hooking WGA up to this thing.
which I can't replace as I can't see paying a monthly fee.
p.s. my TIVO, a LINUX device, freezes now 2 or 3 times a month.. I have to switch it off (unplug it actually) and gosh yes, wait a minute (*5 or 6 actually) to resume my normal broadcast day.
every day http://en.wikipedia.org/wiki/Special:Random
Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just
From every point of view it seems to make more sense. They spend less money, get a more reliable product that can run very nicely on existing hardware, get some good press for a change, and benefit from the work of unpaid open-source programmers all over the world. But it isn't their way.
Is that crash data in general, or crash data on Microsoft applications or OS parts? I ask because windows can send crash data for any application which crashes...
The AACS key is NOT 0xF606EEFD628B1CA427BEA93A9CA9773F
tells us that 450,000/500,000 gives us a crash rate of exactly 90% seems about right for a windows system , can't think why they haven't released it earlier.
You're ignorant. From your post, it seems you think that the crashes were OS crashes, which is not true. Most (or all?) of the crash information is about applications crashing, not the whole OS. Any application, not just OS code or Microsoft apps.
It's more akin to you turning on your TV and finding out that your channels suck.
The AACS key is NOT 0xF606EEFD628B1CA427BEA93A9CA9773F
Something that I haven't heard much recently is about Vista compatibility. MS has said before that it will be compatible and for most software and hardware, it was true in previous versions. But there were enough exceptions. ME was supposed to be backwards compatible. But many specialized drivers had to be written for it. XP definitely required some driver updates. Since Vista is a architectural change, so one would except some compatibility issues especially when DRM and enhanced security is thrown into the mix.
Technically would MS classify incompatibility as not a a bug, especially if is does not cause a crash?
Well, there's spam egg sausage and spam, that's not got much spam in it.
One team I was on had a pair of "critical path sunglasses", whoever was on the critical path got to keep them, the joke being they wouldnt see daylight for so long their eyeballs would suffer when they did go outside -hence the sunglasses.
This meeting looks like a triage session to me: someone goes through the list of bugs, dividing them up into ones to focus on, and which to ignore. Triage has always been a microsof strength: making the decisions as to exactly how buggy something can be and yet still ship successfully.
There's one person with a keyboard (probably hooked to a real vista pc),the foreground laptop is running outlook on what looks rather suspiciously like WinXP (that or vista without Aero, which is roughly how laptops will run it anyway). There arent enough empty soft drink cans or laptops plugged in to AC power for a long lived meeting. Rooms get messier after about six hours, even with less people in.
Most people are not fooled thrice (PHBs excepted) and new crops of folks who have yet to be burned by MS' business ethics and exceptionally bloated software products are shrinking rapidly -- MS is running out of suckers. Which explains much about MS's Playstation and iPod knockoffs: At least MS knows their con is running out of conees in the PC software racket.
New product lines we might see from MS in the near future:
- Pointy, singing tin-foil hats stamped with "PHBs Rule!" and an image of Dilbert's boss -- with ads for sale to the highest bidder
- Lapel pins which sing out screechy fortunes -- with ads for sale to the highest bidder
- Singing tires for your automobile -- with ads for sale to the highest bidder
- Singing Toilet paper -- with ads for sale to the highest bidder (I might buy this one, hmm.)
The list is endless, the point is made, I'm outta here.
Happy Monday to all.
Everything in the Universe sucks: It's the law!
Cue in:
1) Windows is extremely unstable and it BSODs every an hour or so. You have to reinstall it every month, there is no other way around it. Me, I switched over to Linux in 1997 and never looked back.
2) Windows administration is royal PITA; you can't do jack shit and everything is so confusing and convoluted. Every time I have to touch a Windoze box (once in three months or so) I feel so frustrated. Me, I don't run Windows anywhere.
3) NT is a dead end OS, it's inherently flawed. They should switch over to Linux/BSD/Whatever like Apple did. I mean, if Apple could do it, why Microsoft can't?
4) Why is Office so important? OO is getting better. And there is GIMP, yeah.
"Blah blah blah." - [citation needed]
more than half a million installed, and 450,000 sent back crash data... so even if we assume it was nearly a million, that's 50% crash rate. I'd guess it was way higher even than that. So, over half of the systems were crashing bad enough for Microsoft to care? Wow! What exactly is the problem? I thought this was supposed to be a newer, better version. Wouldn't we see a 10% crash rate, or even a 25% crash rate at this point if things were really getting any better?
stuff |
This is something that Microsoft likes to do a lot. Wait to fix things after it is out. Microsoft loves to do everything at the last second. As someone elce was saying look at what happened to Windows ME! It took them this long to release Vista yet this thing sounds like it has not been tested at all. I am thinking that there will be major updates after it comes out. They should just wait for two more months and let the beta testers keep going at it and see if they can't get ahead of the game? Also I think that if they release updates more often then they do now I think that they could get a little better at patching things. Apple will have an update out winthin a week of the problem being found. Anyone who uses NOD32 knows how often that updates By itself might I add. That is how often Micosoft should patch. It should be seemless and no restart required. One of the main reasions I do not use only Windows is becuase of the restarts that are required all the time. Linux never has to reboot unless it is a kernal update and Mac does not take as long as windows to start so I do not mind much. In conclusion the success of this version of windows will hang in the ballance of security and compatibility.
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft"
So the liklihood of a crash is near 100% ?
Sounds like typical end of project 'big bang' testing. All those issues they ignored in development? Let's fix them now and fast! I would hate to be the MS QA person.
(Yes I am aware I used singular, it was a joke, OK?)
putting the 'B' in LGBTQ+
Waiting for SP2 will take around 2 years after Vista is released. Assuming an early '07 launch, SP1 should be out by the end of '07. By then, both Microsoft and other hardware vendors should have their bugs & drivers worked out. For XP, I waited until SP1a and it worked fine.
/.'ers would even touch a brand new release of Windows. I just see it as moving from 500,000 beta-testers to 50,000,000 beta-testers.
It's funny, though, that neither you nor I nor a lot of other
"More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."
...and the other 50,000 uninstalled before it could crash. Seriously, why would you install this before 2009? Or at all?
Has anyone considered that MS is actually trying to make a decent OS here? Sure, it's fun to poke fun at them for all the crashes and bugs, but isn't that the ENTIRE point of the 'release early, release often' mindset?
At least they're trying to find their bugs, at least they are running a widespread beta.
As usual with microsoft products, I will wait for an integrated, WGA-disabled version to arrive via BitTorrent before putting any thought into actually getting it.
You are welcome on my lawn.
N/T
... but rather, fixing (or finding) the causes of these crashes are. Even if you assume pessimistically that every 45 crashes are cause by one bug/issue, that's still... 10,000 issues. Onward to January 2007!
Fedorec Core 4, Ubuntu 6.
IANAL but write like a drunk one.
But not Linux, because then they would have to give back to the community.
IANAL but write like a drunk one.
...but I just wanted check we're talking about the same Microsoft here.
This is THE Microsoft, right? The one who breaks CSS support for IE, refuses to use pdf or odf, and once upon a time, whenever you searched "slashdot" on their search engine, fed you a story about possible Linux viruses as the first result?
Poor naive rsidd.
You know, there is a difference between trolling and pointing out the flaws in your reasoning. Just saying.
For the fun of it (I know I'm a sadist) I downloaded and installed RC2 on my home machine to give it a whirl. Overall it's much slower than XP (my machine is a AMD64 2.2 Ghz with 2GB of RAM) and it has an annoying habit of hanging when it goes to reboot. For example, you install an updated video driver, it asks you to reboot and it tries to, but then the screen goes blank and just sits there. Hitting the reset button is about the only thing to get it to reboot at that point. I just don't see how Vista will be a viable OS anytime soon -- until MS pushes by stopping support of XP in a few years.
"... half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data ..."
or, "Half a million installs of Vista, and almost half a million have crashed"
"Rushing to fix bugs" is like rushing any other meticulous job. It can't be done.
Bugs are the consequence of rushing the job in the first place. (Taking time, is of course, necessary but not sufficient).
If Microsoft knows a way to "rush" bug fixes without compromising quality, they would have been able to "rush" their development without creating the bugs in the first place.
"How to Do Nothing," kids activities, back in print!
The problem with this statistic is that it is almost meaningless without more information. No real conclusion can be drawn. This could mean a number of different things. For instance:
* Vista could have one bug (and only one) that all of the systems encountered and successfully reported
* Same as above, but the conclusion drawn is that Vista 5ux0rz cause it has a huge bug
* Vista has 450,000 different bugs, each system reporting a different bug
* One system crashed 450,000 times and reported each time and the facts were skewed into calling each report a unique system.
The possibilites are endless. Journalists really need to start providing statistics that actually add value to the article rather than adding more mud (or FUD?) to the water.
"It's not whether you win or lose, it's how drunk you get." -- H. J. Simpson
Dude, you need to go re-watch the Lion King...
It would've been really funny if the laptop you can see in front was either a Mac (which'd be too obvious to get into the room) or running Linux (which'd still look like a non-Mac PC).
look! it's a bird, it's a plane, it's....a girl? yes, a girl browsing Slashdot on Linux
We don't know there was "only one" lady in the room. We can only be sure that (a) there is at most one lady in the room and (b) that there is exactly one woman in the room. This is assuming the photographer is male.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
So Mario Garzia is Microsoft's "Director of Windows Reliability". Now there's an oxymoron for the ages. Hey Mario, you're fired!
"The chart indicates a decline in Office 2007 flaws from the start of 2006 through last week"
:O im shocked!
since the graph can never be y=0, is the NYT implying that Office 2007 will always be flawed?
Doesn't this require the third-party application's developer to have signed the application's code using a VeriSign code signing certificate? At 499 USD per year plus whatever your state charges for incorporating, I'd imagine that such a certificate is too expensive for many hobbyist developers of freeware and shareware to purchase and renew.
I used to run the Windows Feedback team which built crash reporting into Windows XP (if you don't like "send error reports" you can blame me :-)
There are 2 things that people should know to put the 450K reporting systems into clearer perspective:
1.) Crash reports typically form a clear and very steep pareto curve. In fact, the top 1% of problems on XP accounted for over half of the reports. In other words; in general people tend to hit the same issues. This makes sense from a common sense perspective as well; one would expect a defect in a common code path in a popular application to be encountered more frequently than uncommon code paths in unpopular applications. It also means that emphasis can be placed on fixing *the most important* issues for the majority of customers and working down the curve to cover as much breadth as possible.
2.) The way crash reporting works is that it handles any unhandled exception which results in an app crash, or any kernel bugcheck. This means that 3rd party applications and drivers are reported as well as crashes in the OS and MS applications. When last I worked on this, there were over 6 million unique .exes reported and more than 95% of the reports by volume were in 3rd party code paths. I expect that distribution is a little more skewed towards MS code over the Vista development cycle since it was under devel, but my bet is that it is still predominantly 3rd party.
At the end of the day, the way that I hope people look at this is that it is better to have data about problems and act upon it, than to not collect the data. 450K Vista systems reporting errors means that the Vista team and 3rd party app/driver developers have had the data and opportunity to act upon it. This is a good thing.
Seriously - then best thing with vista is the new framework .net 3.0. I ' ve tried Windows Workflow Foundation - they have really improved many things. so i'm looking forward to vista