Slashdot Mirror
Failed Win XP Upgrade Wipes Out UK Government Agency
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
Never email donotemail@WeAreSpammers.com
The BBC article mentions that EDS is responsible for the ugprade. They're partnered with Altiris, so I'd be willing to bet that the upgrade was carried out using the Altiris Client Management Suite.
It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.
"On another note, How did upgrading seven machines to XP BSOD 60000"
If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.
I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.
I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.
It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.
So ... 5 working days, 60,000 PCs (= 60,000 employees?)
Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.
=> TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.
Yet the government keep awarding them [EDS] contracts. Why?
I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky - you could spend millions only to not get the tender, and IBM felt that the chance of getting the tender awarded to IBM was too small. So... I'd suggest either it's too costly to play so players are dropping out (the reasonably answer), or someone in government really loves EDS, and IBM know it (the tinfoil hat answer).
Living in the UK, I'm minded to go for option 2.
This is where the serious fun begins.
EDS is one of very few companies that will accept government contracts. US Gov'mt accounting requirements are onerous (hard to comply with) by any standard, so in order to compete for the contract, you have to have a huge team of accountants that know how to produce the kind of records and reports that the Government accounting office(s) expect.
There is a huge hue and cry (outrageous exclamation of disgust and anger) over mismanagment and eggregious spending in government contracts. Having worked in the sector, I'm somewhat familiar. The contractor I worked for made sure there was no waste, fraud, or abuse. However, it spent 10 times as much as the job required, just to do this. The obvious choice for our firm was it would have been far cheaper to run things by GAAP (Generally Accepted Accounting Practices)(the private sector accounting standards), and have both a nice large internal audit division and "internal affairs" watchdog enforcement. Alas, most governments are not run this way, and if they are, they devolve into the current format due to political expediency.
I have friends that work for EDS and they comment on the kinds of hoops they have to jump through just to do simple stuff. They've built up a rather large experience pool in doing this hoop-jumping, so they can do contracts cheaper than some other companies.
EDS also tends to run things according to CMM levels whenever they're developing things, so at least if there's a mess-up (as there obviously was here), there will be some kind of follow-through to improve the process of doing this kind of work. EDS's management doesn't want the black eye any more than the government or Microsoft do, but they'll spend the money to make sure it doesn't happen the same way again. There is, after all, no way to prevent all errors, but I give them credit for trying most of the time.
Unitarian Church: Freethinkers Congregate!
and you missed out big time. 4 years later you could have been naming your own price for Y2k fixes.
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
Something that makes me curious, you hear Ballmer lament about the lower TCO of windows. You hear the linux community shriek about it's lower TCO. The bottom line is really this, if your sysAdmins are less than competent and bugger up something like this which system would have a lower cost to recover? This is a really good thing to know when you are considering any enterprise system. Call it, TCCR (total cost of catastrophic recovery). Ballmer, Linux communities answer me this!
All your database are belong to us
Only theregister appears to talk about Win2k and XP, so lets see what they're saying.
So if this is true then EDS pushed out a partial upgrade. Now come on, if you installed 75% of a new distro over an old one then rebooted would you blame Redhat because it didn't work?
Or there's the other version
So again EDS pushed out XP patches, overwriting Win2k files and the machines crashed
Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.
Of course theregister could be wrong. It might happen. Heh.
Is there another MP who's taken a clearer anti-ID card stance, and is prepared to discuss their positon so openly?
UK Laptops
If you give a chimp an Uzi with a defective trigger mechanism and a bunch of people get shot, whose fault is it: the chimp's or the Uzi's? My first networking experience was with AppleTalk; plug it in and you had a network. I was subsequently required--with co-worker--to learn everything we could about Windows networking so we could implement it in one of our products.
My co-worker and I spent the next period AMAZED that Windows networking even worked at all. The system of domain controllers and WINS servers and browse lists and host files... it's too byzantine to be believed. There is, without doubt, a corporate network somewhere that could be comopletely undone by someone opening a wireless laptop in the wrong place at the wrong time. Add Windows XP and the attendant SP2 fun they're having and you get chaos.
Yes, those delightful folks at EDS are the chimps in this scenario, but Microsoft's products are definitely the defective Uzi. And I note that the BBC News article studiously avoided mentioning either of them. Hmm... Microsoft wouldn't be doing everything it can to tamp down this PR disaster, would it?
Naaah!