Slashdot Mirror
Failed Win XP Upgrade Wipes Out UK Government Agency
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.
Never email donotemail@WeAreSpammers.com
I can imagine it now
Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"
The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.
</cynical>
Get paid to search..It's geniune and
Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?
Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.
Quidquid latine dictum sit, altum sonatur.
Every Desktop Shutdown.
All those moments will be lost in time, like tears in rain.
It's like a thousand solitaire players suddenly cried out in frustration and then silence...
I like muppets.
"On another note, How did upgrading seven machines to XP BSOD 60000"
If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.
I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.
I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.
It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.
Obviously these sysadmins were incompetent. Everybody knows that a BSOD is impossible under Windows XP. If they had simply upgraded the other 60,000 machines to XP first, and then updated these 7 problem systems, this whole problem would easily have been avoided.
It was EDS that screwed it up. I can't say I'm surprised. For once I find it hard to blame Microsoft - rolling an XP patch out onto a Windows 2000 machine (or 60000) will have the predictable effect of hosing the system. Given what I know about EDS (I worked there for two summers) I don't think running Linux would have helped.
My username does not make me Apathetic. It's irony, get it?
the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.
In actual fact, the Register quotes:
According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'
and then below it:
Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.
So, by merging them you get the following story:
There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.
Somewhat slightly different to the Slashdot version wouldn't you say?
In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.
Avantslash - View Slashdot cleanly on your mobile phone.
was the government spokesperson. After the intro to this piece on Radio 4 this morning, her opening sentence was "Let me correct you, 20% of our workstations are functioning". Talk about a positive spin.
Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.
UNIX? They're not even circumcised! Savages!
I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.
There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme. This is one we could lobby our representatives on to ensure they do it right..
Where to have the debate where it might be read by those who mater:
Free service to fax your MP
Boris
Richard Allan
Tom Watson
Shaun Woodward
Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure. as well as the NHS computer system UK
UK Laptops
"What happened to all the competent people??"
They emigrated, most likely. One of the problems with incompetence is that it's self-reinforcing, the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time.
Something that makes me curious, you hear Ballmer lament about the lower TCO of windows. You hear the linux community shriek about it's lower TCO. The bottom line is really this, if your sysAdmins are less than competent and bugger up something like this which system would have a lower cost to recover? This is a really good thing to know when you are considering any enterprise system. Call it, TCCR (total cost of catastrophic recovery). Ballmer, Linux communities answer me this!
All your database are belong to us
you can call them senators if it makes you feel better.
...but is there any actual evidence is was a Microsoft error? I like bashing Windows as much as the next guy, but it seems this is at least as likely to be a huge fumble by the admins.
They're probably using something like Novadigms's Radia. And instead of linking the correct 7 PCs, they linked to all of them (misconfigured group). In that case, it's not a case if installing a patch that is installed using the new mechanisms, the "Patch Manager" simply dumps the files to all the machines that connect up using it's client, and force an overwrite.
Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.
Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.
So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...
The public sector in the UK is nothing more than unemployment benefit for the middle classes.
In my experience (having worked for both) in terms of inefficieny and stupidity, there's only one thing worse than the British Public sector and that's the British Private sector.
My company used to be part of a large public sector concern and was sold off. Since then we seem to spend nearly of our time/money:
Changing company logo and name every 6-12 months
Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others).
Paying huge bonuses to upper managent.
Paying huge car allowances to middle management including those who refuse to drive.
Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items.
Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later.
Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.
Repeatedly shuffling kit from datacenter to datacenter around the country at vast expense and disruption to our customers.
Ordering expensive buffets for management meetings , 95%+ of which get thrown away.
Managers having a schedule involving meetings all over the country which means that they spend about 25 hours out of 40 driving.
Managers refusing to use video-conferencing for meetings even in the light of the above.
How many of these things happened when I was in the public sector? Virtually none. We didn't have the money to throw around on such things. We were forced to be efficient.
Also if this private sector company I'm referring to was atypically inefficient, presumably it would do so badly it would collapse or be taken over. So this implies that many private sector companies are like this.
It's very easy to slag off the public sector if you use stereotypes, generalizations and distortions.