Slashdot Mirror
Failed Win XP Upgrade Wipes Out UK Government Agency
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
I can imagine it now
Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"
The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.
</cynical>
Get paid to search..It's geniune and
Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.
Quidquid latine dictum sit, altum sonatur.
It was EDS that screwed it up. I can't say I'm surprised. For once I find it hard to blame Microsoft - rolling an XP patch out onto a Windows 2000 machine (or 60000) will have the predictable effect of hosing the system. Given what I know about EDS (I worked there for two summers) I don't think running Linux would have helped.
My username does not make me Apathetic. It's irony, get it?
Because Accenture is the other choice!
This sort of cockup would have been impossable with the ex Arther Anderson crowd. They would still be struggling to get the shrink wrap off the CDs without wrinkling thier suits.
Seriously the problem is government procurement procedures. The contract goes to the lowest bidder and a record of past f****ups is not taken into account.
Old COBOL programmers never die. They just code in C.
agreed this is more and EDS issue than a microsoft one.
i'm trying to give up sigs.
I wish I could take one of you Linux "experts" up on your idea. "Here, upgrade these 2000 PCs, all of which are from different manufacturers and different configurations, to Linux. I need it done in the off hours and I need everything to work like it did before.".
*crickets*
Of course someone will reply and say "ok!" knowing it won't happen. It's not because I don't have the ability to make that decision but it's because I know better than to get real information/insight about IT from most /. posters.
It's painfully obvious that a scant few here actually have a clue about running a business that relies on IT. It's more than ripping CDs and DVDs kids. Sure, the company that did the mistake is at fault but the problem is not in the chosen OS, it's in the chosen technicians and management.
If anything, this is the world crying out "what were you _thinking_ having a 60,000 unit network all running the same system???" and perhaps the world will wake up and realize that it _might_ be a good idea to mix systems a bit so that whatever happens to one system, you still have some significant percentage of the network still running.
Doesn't anyone do risk analysis anymore?
sigs are hazardous to your health
Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.
UNIX? They're not even circumcised! Savages!
I once knew a bean-counter (quite senior) on nearly 3 times my engineer's salary. He was sat there in front of a spreadsheet adding up a column of numbers on a pocket calculator.
Welcome to the UK Public Sector. That was your tax money.
Stick Men
Everyone needs a second chance. And a third. And a fourth. And a fifth. And a ...
I'm sure the government has perfectly good reasons for continuing to hand contracts to EDS. It's just probably not a reason they want to tell you because it involves (bribery|nepotism|stupidity|all of the above)
Jedidiah.
Craft Beer Programming T-shirts
OK, so let's see. Someone installs WinXP dll's on Win2K machines. Hmm, I wonder, how come those don't boot now?..
It's true that Microsoft's robustness is rather mirage-like, but there's a thing called human error, and that can bring down any system. All the software did was follow human instructions, after all: that's why we need IT people with brains to decide who is doing what.
However, PXE boot and a server with HDD images ready would've been helpful...
Microsoft sells itself as easy to administer, what in management terms means that the systems are so /user friendly/ that any moron can administer them.
/user friendly/ GUI program.
So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS.
Aside from that, a point-and-click update cannot fail so miserably. A script made by the admin, of course should, because you can assume that someone smart (and bold) enoguh to make a little script should be responsible for their decisions. Some guy clicking checkboxes shouldn't be allowed by those means to break 60000 computers, through a
GUIs for dummies should have enough checks to prevent such underiable effects, they have a sufficiently constrained domain to be able to do so. If the guy wanted to do a legal task that the tools dosnt' allow, he could always write some Visual Basic Script, and then he would be on his own. Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.
"What happened to all the competent people??"
They emigrated, most likely. One of the problems with incompetence is that it's self-reinforcing, the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time.
They could have called Novell or IBM.
Apart from that though - any setup can be screwed-up by an admin, no currently available OS can protect you from that. So for a TCO estimate at least we would have to look at the total loss due to screw-ups like this, and weigh them with the number of installations. Using a single data point can't be valid. That said, my gut feeling is that Linux provides considerably better TCO.
They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker. ;)
Looks like they got a deal; they got the version that also blocks viruses, worms, and abuse of Solitaire!
Writing article about Free iPod. Please help out.
They probably wanted to block assholes who disguise 'Free iPod' links in the sigs. 'TinyUrl' my ass. If you want an iPod, ask your parents to raise your allowance. Otherwise, I heartily encourage you to fuck off.
When you have nothing left to burn you must set yourself on fire
Don't worry knowing Linux and the IT of the public sector they'd have chmoded root to 777 long before any upgrade.
Yes. It's not like the upgrade could detect the version of the program it's being applied to, and only install if the version matches the version it is intended for. That is completely unheard of, and would be impossible technically.
This was sarcasm, FYI.
This situation is more analogous to a wrong signal causing the door to open and then jam. And yes, such a door manufacturer deserves to be blamed.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
...but is there any actual evidence is was a Microsoft error? I like bashing Windows as much as the next guy, but it seems this is at least as likely to be a huge fumble by the admins.
They go over budget because when a project is accurately costed, some idiot manager somewhere goes beserk and says it must be done in time-(large chunk of time) and for cost-(managers' & directors' bonuses). Knowing this most s/w projects are unrealistically timed and funded. Anyway, EDS has right royally screwed up on all the big govt. projects yet the govt. continues to use them. Is that as a result of competence?
Did he inhale?
They're probably using something like Novadigms's Radia. And instead of linking the correct 7 PCs, they linked to all of them (misconfigured group). In that case, it's not a case if installing a patch that is installed using the new mechanisms, the "Patch Manager" simply dumps the files to all the machines that connect up using it's client, and force an overwrite.
Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.
Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.
So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...
The vast majority of tools that do these rollouts dont roll out the patches as supplied by the vendor. The patches are applied to a machine in a known state, and then that machine is scanned by the tool to see whats changed. This changeset is whats rolled out. And yes, jsut tried it, XP SP2 does indeed refuse to install on a Windows2000 system.
Upgrades NEVER work! Not for Windows 95, 98, ME, 2000, XP, Longhorn, whatever! It will never be a good idea to try and replace a MS OS without doing a clean install.
This is first day stuff.
Without any specific details on the failure or what exactly happened, it seems like this is a huge admin error. My guess is they're using something like Altiris to do their builds, and if an admin were to accidentally "drop" the package meant for the the test group on to the production group, wham-o... every PC starts installing a build that probably isn't meant for them, and won't work. And you can imagine how that would go.
As much as I'm sure the zealots among us would like to make this seem like a Windows failure, it looks like it's more of an example of how outsourcing leads to disconnected, incompetent, and unmotivated IT staff. And that, of course, leads to mishaps like this.
Either way, if you work for a company that brings EDS in house in any way, drop your shit and run. And don't look back. The flash could be blinding.
The public sector in the UK is nothing more than unemployment benefit for the middle classes.
In my experience (having worked for both) in terms of inefficieny and stupidity, there's only one thing worse than the British Public sector and that's the British Private sector.
My company used to be part of a large public sector concern and was sold off. Since then we seem to spend nearly of our time/money:
Changing company logo and name every 6-12 months
Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others).
Paying huge bonuses to upper managent.
Paying huge car allowances to middle management including those who refuse to drive.
Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items.
Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later.
Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.
Repeatedly shuffling kit from datacenter to datacenter around the country at vast expense and disruption to our customers.
Ordering expensive buffets for management meetings , 95%+ of which get thrown away.
Managers having a schedule involving meetings all over the country which means that they spend about 25 hours out of 40 driving.
Managers refusing to use video-conferencing for meetings even in the light of the above.
How many of these things happened when I was in the public sector? Virtually none. We didn't have the money to throw around on such things. We were forced to be efficient.
Also if this private sector company I'm referring to was atypically inefficient, presumably it would do so badly it would collapse or be taken over. So this implies that many private sector companies are like this.
It's very easy to slag off the public sector if you use stereotypes, generalizations and distortions.
This isn't Microsoft that ballsed it up, nor is it inherently the fault of DWP. Chances are it's an underpaid sysadmin somewhere who hit the wrong checkbox when rolling out the patch.
If someone can manage this by selecting the "wrong checkbox" then the system is broken by design.
Microsoft sell a complex system with the claim idiots can administer it. The DWP employ/contract idiots to administer a complex, but vital, system. Niether of these are "innocent parties".
OK, fair enough. A lot of the people I worked with when I was in a government position do deserve that.
But a lot of them don't. I would say most state employees work their asses off doing pointless things, rather than screwing off. The problem is more with upper management than with the rank and file... though the problem does bleed over into the lower level employees because, after all, how long can you pour your energy into a task that you know only is neccesary because incompetent managers fail to streamline the operation and give you more real, productive work, before you start to take the job much less seriously?
So those petty state officials who shirk work do so as much due to being beat down, disillusioned, and tapped out as far as trying to do something about it in the face of a "front row" that doesn't like to listen to comments from their inferiors.
When I was working for the state, I considered myself very lucky to be involved in a project that was doing something meaningful, being productive and, while mistakes were made here and there, was relatively efficient overall. I could see how this was not the case in the departments working beside ours.
Eventually, though, the egos of the upper echelon managed to intrude even into our well defended (by caring managers) little island of fortitude and competance, and I had to say screw it. Now, unlike most of the rest of my friends that got laid off and sucked the government unemployement insurance tit, I am fending for myself with the money I saved by not buying useless crap.
So when people try to say I was overpayed at 60% of my fair private-industry salary, I don't shirk from the criticism. Yeah, the benefits were better than the private sector and the environment more permissive, but at least I didn't go looking for a handout like others so they could keep up the credit card payments for their DVD collections and car loan for their gas guzzling S.U.V.
At least I, one of those loathsome, lazy, state workers, had the good conscience not to apply my talents to better the carreer of a gaggle of idiots who aren't overseen adequately by the legislature that created their positions. If you want the state sector fixed, aim at the top. The clock punchers at the bottom are just a symptom of a management that preserves itself by not giving their underlings enough of a reason to revolt.
Someone had to do it.
It's much more reliable to back up your data and do a fresh install. I experimant with upgrades, but even(or especially) with linux, I prefer to clean the disk and start fresh. Apple on the other hand(before OS X anyway, don't know if it still is) was great. It would just create a clean new system folder. With the old one still there, I could just "bless" it if necessary. Oh, well...There's still nothing more trustworthy than pen and paper, and a good ol' mimeograph machine(the hand crank variety) for makin' copies...And they smell great.
What?