WGA Meltdown Blamed On Human Error

Erris writes "As commentators like Ars Technica slam WGA as deeply flawed, Microsoft is blaming human error and swears it won't happen again. 'Alex Kochis, Microsofts senior WGA product manager, wrote in a blog posting that the troubles began after preproduction code was installed on live servers. ... rollback fixed the problem on the product-activation servers within 30 minutes ... but it didnt reset the validation servers. ... "we didnt have the right monitoring in place to be sure the fixes had the intended effect"' Critics were not impressed. 'A system thats not totally reliable really should not be so punitive, said Gartner Inc. analyst Michael Silver. Michael Cherry, an analyst at Directions on Microsoft in Kirkland, Wash., said he was surprised that it was even possible to accidentally load the wrong code onto live servers ... [and asks], "what other things have they not done?' This is not the first time this has happened, either."

Comments

Zoom

[Anonymous Coward]

http://slashdot.org/comments.pl?sid=280723&cid=203 75627

Re:Zoom

[gatzke]

Slashdot is not about journalistic integrity, it never has been. It is about nerd topics and dupes.

ACs complaining about twitter does look like astroturfing. MS has enough money to pay a few guys to beat back public opinion on well-known public tech sites. Without facts disputing the current article, it looks like you are just pro-MS ranting against a anti-MS article without any substance.

Fact- WGA broke for a while causing many people troubles.

Fact- Some people don't like having to phone MS all the time to keep a product running.

Fact- MS has paid astroturfers to anonymously post pro-MS grassroots stuff online.

Re:Zoom

[ZachPruckowski]

While your point is valid, I don't think that's the ultimate issue. Even if Twitter's rants were useful/interesting (and some of them are), the editors still need to get control over this. If I wanted the Twitter show, I'd read his blog. I mean, Slashdot has a large problem if a single writer is manipulating the front page for his own ends. I don't mind so much when someone gets a lot of stories by writing well about something popular (NewYorkCountryLawyer, for instance), but sock-puppetry has to be dealt with.

Re:Zoom

[iminplaya]

Well, let's be honest. Any program or OS that requires activation deserves a good bashing, and we should not support it in any fashion. And I proudly champion those who develope workarounds. Those who complain about bootleggers while benefiting from them as Microsoft and Adobe do are just as hypocritical as gay bashing republicans.

Re:Zoom

[CrashPoint]

ACs complaining about twitter does look like astroturfing. MS has enough money to pay a few guys to beat back public opinion on well-known public tech sites. Without facts disputing the current article, it looks like you are just pro-MS ranting against a anti-MS article without any substance.

If there's only one Slashdotter on Microsoft's payroll, it's twitter. He effectively smokescreens legitimate criticism of Microsoft with his childish, myopic blame-Microsoft-for-everything posts and his egocentric belief that anyone who calls him out on his bullshit MUST be brainwashed by/working for "M$". He obsessively catalogs every post made by his "enemies", frequently lies about what they actually said, and conveniently disappears when confronted with inconvenient facts or questions. He does more to discredit "his" side in an afternoon than any dozen pro-Microsoft astroturfers could hope to accomplish in a month.

If Steve Ballmer isn't personally giving twitter a handjob right now, then he's neglecting his responsibility to his company.

Have we gone backwards?

[Ckwop]

This sort of ties in with what I was saying on IRC with my friends yesterday. My central point was that all operating system have got worse over the past ten years.

I'm currently reading the Mythical Man Month (which I imagine most of you of heard of and already read) and in it he talks about the OS/360 operating system in great detail. I'm recalling this from memory so I'm sure someone will correct my mistakes but anyway, the machine had 2MB of memory and the operating system cost 400Kb of the memory. They charged something like $9.50 a month for 1Kb of system memory. That meant that every Kilobyte of memory saved was worth hundered or even thousands of dollars over the life time of the machine.

It made me realise what is in retrospect a fairly obvious statement. The cost of the operating system on your hardware is an effect that should be minizimed. The operating system exists as a framework for runs tasks and applications, not for being a self-serving execuse to munch resources.

While Moore's Law technically means something different; the adage has held true that computing power has doubled every eighteen months. This means that my machine which I bought in January should be roughly 100 times more powerful than the machine I had in 1997. Yet do I have hundred times more power to run my applications on a modern Operating System? Absolutely not.

Strictly speaking, there are no tasks I do today that I couldn't do in 1997. I can be honest that computing hasn't really got easier since then either. There's the odd innovation here and there that's nice from a usability point of view, but fundamentally nothing has really changed. For an example, Office 97 and Windows 98 are no harder to use than XP and Office 2003. The addition of an extra monitor to my compute has impacted my productivity more than the choice of software in this period.

In short, where did all these cycles go?

Now Microsoft Vista is a sort of a post-modern operating system. In every sense it is a regression. It does not allow tasks to be managed easier yet requires an enormous amount of extra resources just to operate. WGA in a sense breaks the very stability of the system. The point of the OS is to perform tasks and applications yet Microsoft can take this away from you either by malice or stupidity.

When are we going to demand more from OS vendors? When are we going to demand that future versions do the same as the previous version with less memory and less CPU overhead? Why do we pay to upgrade only to find our upgrades are wiped out by OS bloat? All of these are interesting questions, and while off-topic slightly, I'd like to see what you think!

Simon

Re:Have we gone backwards?

[Anonymous Coward]

Depends hwat hard ware you are using, whilst I'm sure that if you bought the newest hardware at the same time as buying OS X, then OS X would run faster than the previous version, because it has been better optimized to use the extra processing power, but if you use the hardware the original operating system was made for, then run OS X on it, OS X would almost certainly run slower on it. Then once you look at it, with all the hardware advances and the relatively small steps forward (just slight optimization) OS X isn't good either.

Re:Have we gone backwards?

[Dunbal]

When are we going to demand more from OS vendors?

      I would extend this to "software" as a whole. Software seems to be in a special protected class, since companies are able to KNOWINGLY deliver a defective product and be immune from prosecution. Computer games I am looking at you. There seems to be a mentality in the industry of "ship now, patch later".

      I can't let this go without a car analogy (this is slashdot after all):

      It's like buying a new car from a dealership, only to find out it comes with 5 flat tires. But the salesman puts his arm on your shoulder and says "hey, no worries, look - there's a gas station just over there and you can get those tires fixed in no time".

      It's high time the software industry as a whole was held accountable for this sloth. And don't give me the crap about "oh but there are so many different computers and hardware and configurations". After all, ISN'T THAT WHAT WINDOWS WAS SUPPOSED TO FIX? We certainly were sold on that idea in 1995. Windows was supposed to be a common application interface that smoothed over all the hardware differences. But because it's the poorly documented, bloated, kludge that it is, programmers yet again have to rely on little tricks and cheats to get top performance out of it. Resulting in crashes/bugs on non-standard systems.

Re:Have we gone backwards?

[mdwh2]

Yet do I have hundred times more power to run my applications on a modern Operating System?

Yes you do. At least, all the while "power" means "CPU speed". I suspect you are conflating it with a more general meaning of power, but Moore's Law says nothing about this.

Strictly speaking, there are no tasks I do today that I couldn't do in 1997.

Because strictly speaking, a turing machine can do anything that any other turing maching can do. However, you can do them an awful lot quicker (things like video/mp3 encoding, compiling or 3D rendering).

However, I think your problem is that you expect a faster CPU to somehow make you more productive when using a word processor.

Windows Vista may or may not have problems - but note that the Windows line today is a darn sight better and more stable than the hopeless version of Windows most people were using in 1997 (just as Mac OS X is much better and more stable than the primitive OS that shared same name which it replaced).

Re:Have we gone backwards?

[pandrijeczko]

The Amiga was taken down, not because there was not enough demand for it, but because it was too efficient.

Rubbish! The Amiga was a far superior machine to the IBM PC but Commodore/Escom/Gateway/Amiga Inc. did not have a single clue as to how to market it and expand it correctly. It was their total lack of incompetence that caused its death.

Amiga users (and I know because I was one of them once) were the most loyal bunch of users there could possibly be, a bunch of people who remained loyal for years despite being continually f*cked in the arse by unfulfilled promises by David Pleasance and whomever else controlled the Amiga name over the years.

Re:Have we gone backwards?

[teh kurisu]

The cost of the operating system on your hardware is an effect that should be minizimed.

I disagree, because you have to take the feature set into account, and subject it to a cost/benefit analysis. You might think that XP is a better OS than Vista, because it's not bloated by the Aero interface. Fair enough. But someone using DOS might say that DOS is a better OS than XP, because it isn't bloated by a graphical interface at all. Most people these days would disagree, and say that the benefit of having a GUI outweighs the cost in processing power.

The problem is that while 'cost' can be enumerated in terms of system load, 'benefit' is in the eye of the beholder. It will always be a value statement. So Microsoft might think that the Aero interface justifies the cost. A lot of Slashdotters think otherwise. It's for this reason that alternative operating systems exist.

Re:Have we gone backwards?

[Anonymous Coward]

There is another big difference with software. The license.

If you buy a car, its yours. If you buy software, you get a license to use it. The software still belongs to the company that wrote it, you just get temporary permission to use it. The software company often gets to decide when, where and how you can use it. You are often told you cant even sell the software. The software company can decide to change the license *after* you bought it, usually tied in with a software update that fixes a bug or flaw in the software. If Ford or GM sold a car with a faulty seatbelt, then when replacing the seatbelt informed you that you were no longer allowed to let anyone else drive the car or that it couldn't be driven on wednesdays, would that be acceptable?

Windows was supposed to be a common application interface that smoothed over all the hardware differences. But because it's the poorly documented, bloated, kludge that it is, programmers yet again have to rely on little tricks and cheats to get top performance out of it. Resulting in crashes/bugs on non-standard systems.

A lot of the problems with hardware incompatibility are directly due to Microsoft. Microsoft insists on hardware coming with tilt-bits and requires the hardware to be overly complicated to discourage hacking and insists information about this is done to be kept secret. Without this, the hardware cant get WHQL certification. This program is supposed to be about quality but in reality is designed to force hardware to add DRM and to make writing drivers for open-source operating systems harder.

Re:Have we gone backwards?

[WhatAmIDoingHere]

"Also, how fast does Tiger run on a beige Powermac G3?"

Also, how fast does XP run on a P2 233 with 64 megs of ram?

I've actually done the latter, and the answer is "Not well at all."

Re:Have we gone backwards?

[Sancho]

From Win95 to Win98 to Win2000 to WinXP, I've seen nothing but stability and security improvements. Vista has some security improvements, too, but in my experience, it isn't any more stable than XP. What's also come with every single new release of Windows is a changed UI, more eye-candy, and features that many geeks find useless.

That doesn't mean that they're useless to everyone.

Part of the issue is that you're focusing on the operating system. Windows is really quite a bit more than that--it's an operating environment (or a desktop environment, as GNOME/KDE are described.) This means that they aren't just there to provide a framework for performing tasks--the operating environment performs tasks on your behalf, provides feedback, allows the user access to information in a subtle, yet useful way (many OS X widgets, for example, and whatever Microsoft is calling their clone of it in Vista.)

In the Unix world, we separate the operating system (kernel) from the shell (bash/ksh/whatever) from the window maanger (metacity/fluxbox/xwm) from the desktop environment (GNOME/KDE). This separation allows for immense flexibility. I can mix-and-match flavors, and even eliminate some of these layers entirely, depending upon my needs.

Windows, however, caters to the mass market. It needs consistency in order to maintain its marketshare, while simultaneously requiring each version to have a distinct look in order to differentiate itself from the earlier versions. It has to be everything to everyone in order to keep existing users and attract new ones. It makes sense to throw in as much stuff as you can, so that people will want to use their product.

Most people buying a computer will use it for the Internet (browsing, email) and maybe for creating documents and managing finances. Yes, they could do this on a 10 year old machine. The only reason to upgrade, then, is for the new UI or because their old computer broke. In either case, they aren't really losing anything. They're gaining more cycles in their new computer, and they're getting an OS that uses those cycles. If their tasks don't change, their CPU power needs (over what the OS requires) probably haven't changed, either.

In more specialized circumstances, yes, it matters. And that's part of the reason that new OS are adopted fairly slowly in the business world. Not only do we want to ensure that the change is as easy as possible, but we want to make sure that we aren't losing anything.

I think I've rambled a bit much, but the gist is, you aren't the target of Windows Vista, and Microsoft isn't just making an operating system. And that you're bringing Unix-like preconceptions into the Microsoft world.

Re:Have we gone backwards?

[dhasenan]

We're doing things in a reasonable amount of time these days that previously were possible only in an unreasonable amount of time, as far as 3D modeling and such.

I've heard that, in the early days of UNIX, computer time was expensive, so you could be wrong as long as you were fast. But these days, processor time is cheap, so you no longer have to be fast. Some people transfer this into automated verifications -- array bounds checking, for instance, or design-by-contract. Bounds checking is expensive for C/C++ primitive arrays, and not yet practical, but it's used efficiently in other languages.

I really think that good design practices interfere with efficiency in many cases. And I'm happy with that situation. I prefer stability to speed, but we're getting both, to some degree.

Re:Have we gone backwards?

[m50d]

It's a fair bet your program would work on Windows 2000 and Windows Vista. Yet Windows Vista will "tax" your system more to achieve exactly the same result. This is my point - the operating system is gobbling more and more resources that should be used by your applications without giving the user anything in return. In this sense, we are moving backwards.

A few things the OS will do that it didn't used to (I know nothing about vista, I've been a linux guy since winme):

Window transparency - when used properly, this can give a huge increase in productivity, since it's pretty much the same effect as adding another monitor.

Audio mixing in software - again, very useful if used properly

More effective search functions

Plus, of course, all the necessary things to support that fast hardware properly - faster timers, the address space gymnastics necessary to use large amounts of RAM on a processor architecture not really designed for it, filesystems that can handle the larger-but-relatively-slower disks we use today. Ultimately there's not that much for the OS to do, and it certainly shouldn't be grabbing all the processor cycles, but OSes *have* improved.

"won't happen again"?

[haeger]

So, if it's human error that caused the problem, how can the swear that it won't happen again? Will there be no more humans working at microsoft anymore?
I don't get it?
People make mistakes and as long as people are involved in any process they will cock up from time to time.

The point about systems not being so punitive is a valid one and should be brought up more often and louder. People who've paid money for their product should not be punished for an error on microsofts end.

.haeger

Re:It's a fair point

[Anonymous Coward]

Sure, for a 24 hour window pirates would have a free-for-all in getting perfectly valid WGA results.

Actually, pirates would probably very quickly figure out how to set the WGA server failure condition in Windows to get the automatic pass without ever actually contacting the real WGA servers, which would render WGA completely worthless. Well... more so.

I don't use Windows, can't stand Microsoft, and had a hearty laugh at the news of the WGA meltdown, but the problem is not as easy to solve from a technical standpoint as you believe.

Microsoft is blaming human error

[suv4x4]

Microsoft is blaming human error and swears it won't happen again.

Self-contradictory: of all things that could happen out there, one thing will keep happening, and that's human errors.

Realistically, it's just another fail point on your OS that will blow up from time to time.

Monitoring

[Dunbal]

"we didnt have the right monitoring in place to be sure the fixes had the intended effect"'

      This sounds a lot like the Bush administration's excuse... oops!

      Seriously, Microsoft is great at monitoring YOUR computer, but they can't monitor their own?

I've said it before and I'll say it again

[FoolsGold]

If the pirates are having no problems and it's the legit users who are getting fucked in the ass, why the hell does Microsoft continue to bother with WGA?

What do they gain? Was WGA suppose to convince people using illegitimate versions of Windows to turn to the light? Fuck that, they'll just download the latest cracked WGA .DLL and get on with it, while the legit users will get boned because their serial key wasn't recognized or whatever.

WGA does NOTHING to hinder piracy, at least not with any level of success that compensates for the negative affects to legit users. It's a complete joke - and yet Microsoft doesn't have the balls to admit this yet. It pisses me off to see such short-sightedness from a bunch of guys who are suppose to be experienced in business.

Re:I've said it before and I'll say it again

[AxminsterLeuven]

Microsoft has its monopoly on the desktop market because DOS and Windows have been pirated like crazy in the past. I remember when the little local computershops would throw in a little box with freshly copied diskettes with the expensive beige 386 you just bought. It seems stupid to start cracking down on piracy just when free OS'es are starting to become an attractive alternative and the fruit-logo competitor is offering a relatively cheap shiny system with more bling.

Not an acceptable answer

[Anonymous Coward]

Look, most of us here work (directly or indirectly) in software. Who hasn't had a launch fail, or a product go bad, in a way that's negatively impacted customers. Such things DO happen. Usually not out of malice, and even sometimes not from carelessness--there are things that sometimes you can't catch on a test system. So to that extent, I feel for the folks who caused this problem..

So why do I call it unacceptable? Because of the difference in standards. On Microsoft's side, they are holding the user to a high level of scrutiny, and reserve the right to cripple some OS features if Microsoft believes the install is pirated. No discussions. Go directly to "aero jail".

Which is possibly understandable if their stance is "look, we're losing billions here--we need to fight piracy." But if they're going to take such radical and punitive measures as locking down OS features based on their tool, then they have to have an absolutely rock solid fail resistant totally monitored system. Basically, they need to hold WGA to a higher standard than most business software. This needs to be the gold standard if they want people to trust the system (and TFA links to a number of other reasonably well-balanced Ars articles that suggest it is not).

Oops, we forgot to monitor the validation boxes? You can't be organic about this--add monitoring for problems as they're discovered on a system this critical not just to Microsoft, but to their customers. You have to anticipate what MIGHT happen, even if "there's no way that should ever occur." You have to think of things that should never happen, but would be problematic if they did.

The fact that they failed here, if it never happens again, might not be a huge deal. But their answer shreds confidence that this is an isolated issue. The fact that this specific failure might not happen again gives me no comfort. Because their answer indicated that they didn't get it when they designed the system, and the don't get it now.

What they SHOULD have said is "boy, this was something we never thought could happen. We have fixed the issue, and are confident we have the monitoring to prevent this specific issue going forward. And we are undertaking a comprehensive review of our validation and monitoring systems to make sure nothing even remotely close to this could ever possibly happen again." Nothing less should be acceptable.

Re:Not an acceptable answer

[morgan_greywolf]

Additionally, IT organizations need to be able to rely on their operating system software for mission critical applications. This does not instill any sort of confidence in Microsoft operating systems. And Microsoft is scratching their heads wondering why IT organizations insist on running *nix on mission critical servers...

Re:Not an acceptable answer

[SpaceLifeForm]

I agree, their explanation sucks. Plain and simple, this problem
did not occur as a result of the failure to monitor the situation.

No, Microsoft just fucked up, and did not even know what their
recovery procedure should have been.

It was a procedural screwup, not due to lack of monitoring.

If they had been monitoring the systems closer, yes they could
have discovered the fuckup sooner, but they still fucked up.

A simple fix would have restored service quickly

[ei4anb]

if ( Wga_is_Available ) DoWgaValidationTests
else
default = TrustTheCustomer

I wonder if they considered that?

You have a choice, people!

[pandrijeczko]

You do not need to keep yourselves tied to Microsoft's apron strings, there are alternative operating systems you can use.

If WGA or other Microsoft activities are p*ssing you off as a user, then have some strength of conviction and DO SOMETHING ABOUT IT!

Just stop with the continual whining about it...

Re:Why didn't they kill the server?

[Technician]

So why didn't MSFT just kill the server to let people's software default to "genuine" instead of leaving the server connected with faulty software?

It's an anti-piracy feature. It prevents a business from firewalling the WGA server to get "genuine" status. Remember there was an un-authorised software update site? If it works without the real MS saying it's OK, the anti-piracy feature does not work.

Unfortunately for MS is this feature does not prevent users from migrating to the alternatives. It's hard to run a monopoly when Ubuntu is legal and free for the taking. If they had a choice, the first would be that I run Windows fully paid for. Second choice is that I run a pirated copy, but they are using WGA to prevent that to encourage me into the first choice, but the result is I have gone to their worst option.. I've gone legal to the competition. MS is helping themselves break their monopoly by reducing piracy.

What's the Big Deal?

[KwKSilver]

Don't MS customers like being treated like criminals and being abused in other ways? They are getting what they bargained for. Sorry, no sympathy here.

Of course it was human error

[Vexorian]

Humans designed WGA, afterall.

What about the future?

[Riturno]

What I really wonder about is when will these servers go down permanently? While I hate to do it, I can still install NT3.51 on an old machine if there is a critical need to pull something off an old tape. What happens in the future when WGA goes dark? Will they issue a patch to unlock the OS? At some point MS may have to limit or eliminate backward compatibility. Will virtualization be good enough? This WGA debacle leads me to more questions and concerns than comfort. To me it is not about today. Like the fun with MS formats, it is about tomorrow.

Re:It's a fair point

[Nasarius]

Oh please. You're talking as if there aren't already full cracks for every version of Windows and WGA. That horse left the barn a long time ago. It's perfectly reasonable for Microsoft to prevent "casual" piracy by people who don't know any better, but going to absurd measures to foil serious crackers has never yielded anything but a few days' delay.

Same old, same old...

[Coyoteold1]

Over the years, I've watched a zillion methods employed to prevent various forms of digital piracy. Generally, the more comfortable a company is that a method will safeguard their product from piracy, the more annoying it is for their legitimate users to employ it. Anyone remember dongle-protected software?

I've worked at several places that legitimately purchased licenses to software, then used cracked versions of what they'd paid for, simply so that they could work in piece without juggling dongles, CD Keys, and other such.

Essentially, any wall that can keep out invaders also hinders legitimate travelers. Any wall that allows access to legitimate guests also allows for the egress of the unwanted.

When we employ truly draconian or paranoid means to safeguard intellectual property, it carries with it subtle risks. Among them, the risk that it won't work well, or will hinder legitimate users while still being exploitable by illegitimate ones.

It's a little like the death penalty. A lot of people would be more in favor of it, if they didn't fear that it was employed unfairly against people who don't deserve it.

If a system is put into place to protect a company against digital pirates, that randomly hassles the company's legitimate users, or if it is, as many pieces of software are today, just completely buggy and bloated (the product of an industry driven just as much, if not moreso, by marketing and artificial deadlines as by a desire for a properly-working product), is it worth it?