Slashdot Mirror
Failed Win XP Upgrade Wipes Out UK Government Agency
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.
Never email donotemail@WeAreSpammers.com
I can imagine it now
Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"
The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.
</cynical>
Get paid to search..It's geniune and
OH SHI-
If only they had reached the conclusion hinted at in this BBC News article a year or two ago, this would not have happened.
It's certainly bad PR for Microsoft though, perhaps this will serve as a wake-up call to other governments that "other options" are out there.
But still I have to say it: "HAHA!"
Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?
The thing is, this sort of thing is expected and accepted by the UK public sector. They'll just find a scapegoat and keep on buying Microsoft. The sad thing is, that's my tax money.
Stick Men
Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.
Quidquid latine dictum sit, altum sonatur.
Every Desktop Shutdown.
All those moments will be lost in time, like tears in rain.
If this was a Linux/Oracle/$flavourOfTheMonth upgrade, would you be just as scathing?
Hey, they may have shite for brains, but the money was worth it! No trouble with the dreaded 228 patents that Linux supposedly infringes!
It's like a thousand solitaire players suddenly cried out in frustration and then silence...
I like muppets.
From the Guardian article: "At this point there is no known solution or ETA"
I RTFA and all I see is a money discussion, not a technical discussion. I would speculate that an SMS or Zenworks push or somthing similar which was supposed to be restriced to the 7 PC's went almost everywhere. It might be a fair bet that the remaining 20,000 might have been upgraded too if those people had been at work and turned on their computers. IT Computer management tools give the department much power, which could do plenty of damage in the wrong hands.
Have you Meta Moderated t
Jon.
This makes me feel so much better about the working mistakes I have made. I would love to see an interview with whoever clicked on "OK" to trigger this one off...
I work in a local government authority myself.
Although we have several xp boxes (mainly used by my development team (along with Windows 2k Pro ones)), there is no way this IT department is going to roll out XP across the entire authority (approximately 400 machines) until at least Mid quarter 2005, there are far far too many problems to even contemplate it.
Heck, half the staff haven't even figured out the difference between a wallpaper and screensaver yet, yet alone giving them more fancy gadgets.
The BBC article mentions that EDS is responsible for the ugprade. They're partnered with Altiris, so I'd be willing to bet that the upgrade was carried out using the Altiris Client Management Suite.
It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.
From the article: Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess." EDS is also thought to be flying in fire brigades."
/.
Brilliant work on the part of EDS, trying to patch the wrong systems, lord only knows what can happen then.
You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it? That seems to be the case here, EDS screwed up, and of course it's Microsoft's fault in the eyes of
Help Brendan pay off his student loans
It seems your firm is costing the British tax payer enormous quantities of cash, predominently through incompetence. Please take any drastic steps necessary to prevent any further IT disasters and consider if your firm deserves the billions of consulting dollars it has already banked.
Yours,
Jim.
i'm trying to give up sigs.
"On another note, How did upgrading seven machines to XP BSOD 60000"
If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.
I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.
I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.
It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.
The installation and update of operating systems is so easy any more, a blind one armed monkey masturbating could do it.
I've worked with EDS people, and the one armed monkey would be a godsend compared to most of them that I've had the "fortune" of working with...
When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.
Yea, I can just see them going bankrupt over this. Their coffin was half closed before, but now they're bound to be pennystock.
Obviously these sysadmins were incompetent. Everybody knows that a BSOD is impossible under Windows XP. If they had simply upgraded the other 60,000 machines to XP first, and then updated these 7 problem systems, this whole problem would easily have been avoided.
So ... 5 working days, 60,000 PCs (= 60,000 employees?)
Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.
=> TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.
the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.
In actual fact, the Register quotes:
According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'
and then below it:
Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.
So, by merging them you get the following story:
There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.
Somewhat slightly different to the Slashdot version wouldn't you say?
In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.
Avantslash - View Slashdot cleanly on your mobile phone.
They upgraded seven machines and 80,000 died? That sounds weird, but maybe they were the AD servers. Why then, on a small number of such critical boxes, didn't they just restore from backups?
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
I wish I could take one of you Linux "experts" up on your idea. "Here, upgrade these 2000 PCs, all of which are from different manufacturers and different configurations, to Linux. I need it done in the off hours and I need everything to work like it did before.".
*crickets*
Of course someone will reply and say "ok!" knowing it won't happen. It's not because I don't have the ability to make that decision but it's because I know better than to get real information/insight about IT from most /. posters.
It's painfully obvious that a scant few here actually have a clue about running a business that relies on IT. It's more than ripping CDs and DVDs kids. Sure, the company that did the mistake is at fault but the problem is not in the chosen OS, it's in the chosen technicians and management.
When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.
No, but that doesn't necessarily mean it's bad for the rest of us!
Let's hope Congress plans to upgrade soon!
See? Even Microsoft is good for something!
Im pretty embarrassed for my country right now. How the fuck did we go from technological pioneers to this? And its only the tip of the iceberg, what with Ken Livingstone's numerous stupid ideas, David Blunkett's insanity and the incompetence of 100's of 'IT' projects (hint: if its called an IT project it means its run by incompetent MCSEs and it will fail catastrophically leaving millions of people without a service or having planes crashing into the ground, time and time again) with tax money falling out of their pockets, fuck them! Why do these idiots get the contracts? What happened to all the competent people??
This comment does not represent the views or opinions of the user.
and you missed out big time. 4 years later you could have been naming your own price for Y2k fixes.
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
was the government spokesperson. After the intro to this piece on Radio 4 this morning, her opening sentence was "Let me correct you, 20% of our workstations are functioning". Talk about a positive spin.
Ofcourse, this is /. so this post will sink fast...
DAMN YOU OCTODOG! DAMN YOU TO HELL!
from the reg article;
"This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess."
So, even more of the money I pay in tax is being diverted to M$ then...
I once knew a bean-counter (quite senior) on nearly 3 times my engineer's salary. He was sat there in front of a spreadsheet adding up a column of numbers on a pocket calculator.
Welcome to the UK Public Sector. That was your tax money.
Stick Men
I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.
There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme. This is one we could lobby our representatives on to ensure they do it right..
Where to have the debate where it might be read by those who mater:
Free service to fax your MP
Boris
Richard Allan
Tom Watson
Shaun Woodward
Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure. as well as the NHS computer system UK
UK Laptops
Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software. It's like saying that a prison guard intending to open one gate to let someone out accidentally opened all of the gates and then they blamed the door manufacturer.
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
I get to do cool stuff with UNIX nowadays. 40 years of cool stuff is better than becoming an EDS pointy-hair for 4 years and having to learn IBM JCL.
Stick Men
I'm guessing that they were attempting to use some fancy MS too to perform some automatic upgrades/updates and while they intended for a limited number of machines to recieve the updates, they went and installed on all the machines that used the service... just a guess mind you but it would fit the circumstances as layed out in the articles. And I would suspect when you add WinXP components to a Win2k installation that bad things would happen.
As much as I would like it to be, it doesn't seem like a "Microsoft" problem exactly and were a parallel Linux situation have happened I'm not sure anything less would have happened... well I guess it would have to depend on a number of things -- for example, if it were an RPM-using distro on the desktops and the wrong RPMs were sent to ALL machines instead of the select few, the machines for which the upgrades were unsuitable would have simply failed due to dependencies unless the --force option were used... okay I'm rambling now but basically, I don't see it as a Microsoft problem as much as I see it as a misuse of tools.
The TCO of their MS installation just went up though... and they shouldn't exclude the cost of firing, hiring and retraining either.
No problem, Just get all the individual users to stick in a Knoppix cd. Remake the network, and let the main root look at all the individual cruft. Then burn a complete backup of all the individual important info. With Windows utilities they would need to fly in 200 WinIT guys to do the same thing. They have not got a hope in hell of doing the same thing on line with a Windows boot CD!
Come now, people, if there was an inherent flaw of this magnitude in the Windows XP upgrade, this would be more widespread.
Windows XP could be a contributing factor, but NOT necessarily the causation. It's hard to speculate on a matter like this, but if I had to put money down I'd put it on shoddy IT work. Microsoft's an easy excuse.
This opinion might be unpopular, but the anti-Microsoft groupthink should be challenged from time to time. Did anybody else entertain this idea or is it just me - I'm not in the IT industry, so this could be more widespread than the article would lead me to believe.
"There's no success like failure, and failure's no success at all."
- Bob Dylan
Microsoft sells itself as easy to administer, what in management terms means that the systems are so /user friendly/ that any moron can administer them.
/user friendly/ GUI program.
So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS.
Aside from that, a point-and-click update cannot fail so miserably. A script made by the admin, of course should, because you can assume that someone smart (and bold) enoguh to make a little script should be responsible for their decisions. Some guy clicking checkboxes shouldn't be allowed by those means to break 60000 computers, through a
GUIs for dummies should have enough checks to prevent such underiable effects, they have a sufficiently constrained domain to be able to do so. If the guy wanted to do a legal task that the tools dosnt' allow, he could always write some Visual Basic Script, and then he would be on his own. Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.
Something that makes me curious, you hear Ballmer lament about the lower TCO of windows. You hear the linux community shriek about it's lower TCO. The bottom line is really this, if your sysAdmins are less than competent and bugger up something like this which system would have a lower cost to recover? This is a really good thing to know when you are considering any enterprise system. Call it, TCCR (total cost of catastrophic recovery). Ballmer, Linux communities answer me this!
All your database are belong to us
Don't worry knowing Linux and the IT of the public sector they'd have chmoded root to 777 long before any upgrade.
'Hopefully just another nail in Microsoft's coffin....' Buddy, you need to wake up and realize Microsoft aint going no where. Just like when faulty parts of cars kills tens of thousands of people a year(if not more), GM isn't going out of business. They are a force to be reckoned with. Microsoft is the same, and everyone needs to learn how to live with it, instead of constantly bitching about what wouldve happened to any leading OS, or software company, were Microsoft not in their place. Sheesh
This sig is o Unfunny o Funny
dude have you never heard of "The Peter Principle". That bloke is probably being promoted right now.
I used to have a better sig than this, but I got tired of it
Problem isn't the platform as much as the implementation. I'd say that someone bollocksed the whole thing up, which could be just as tragic rolling out a linux upgrade or whatever.
We've got to educate the people spending our money on large computer systems to spend part of that money on more testing!
Yes. It's not like the upgrade could detect the version of the program it's being applied to, and only install if the version matches the version it is intended for. That is completely unheard of, and would be impossible technically.
This was sarcasm, FYI.
This situation is more analogous to a wrong signal causing the door to open and then jam. And yes, such a door manufacturer deserves to be blamed.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
Easy, a dialog like this appeared:
"Do you want to update the machines on your network now?"
[Accept]
No cancel button.
--
Wiki de Ciencia Ficcion y Fantasia, un cuento por Fly.
With a service history like this:
p ort_agency_it_failure/ v enue_sacks_eds/ _ abbey_offshore/
http://www.theregister.co.uk/2004/04/26/child_sup
http://www.theregister.co.uk/2003/12/11/inland_re
http://www.theregister.co.uk/2004/05/04/eds_mulls
How do these guys win new contracts from major companies? Amazing, truly amazing... I interviewed there once, got an offer, but that very night when I was thinking about taking the job, I had a pentagram stigmata burns appear on my back! It took 3 months of holy water baths to get it off...
all the clients connected to the network received a partial, but fatal, 'upgrade.'
Is it possible that someone noticed that the updates were going to 60,000 machines instead of just 7, said 'oh shit', and pulled the plug without thinking?
EDS is also thought to be flying in fire brigades.
Yeah, to put out the fires from their smouldering backsides.
you can call them senators if it makes you feel better.
Jeez, sometimes Slashdot readers are blind and zealous like headless chickens...
1. The patch they tried to update with wasn't a complete one for an OS upgrade.
2. Then they deployed it to their entire network by mistake.
This interesting piece of information can be gathered by RTFA.
I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch?
Maybe some... thing... would panic?
Beware: In C++, your friends can see your privates!
...but is there any actual evidence is was a Microsoft error? I like bashing Windows as much as the next guy, but it seems this is at least as likely to be a huge fumble by the admins.
This is an interesting question. Most large companies have at least a few gloriously incompetent people who really should have gone long ago but for whatever reason haven't.
However, I don't know any reports which consider Total Cost of Ownership Assuming Your IT Department Is A Bunch of Blathering Idiots. Most seem to assume a certain degree of competence.
Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software.
This sounds like they were pushing out the upgrade via SMS. Checking that the upgrade was on an appropriate system here would not have mattered since the upgrade path from win2k to WinXP is legitimate. This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container. The real question here is why are so many systems under one system and even better why did the sysadmins who did this application not check to ensure the advertisement was sent to the proper container.
EDS takes the blame for this, not MS.
Keyboard Infantry since 2002
The usual patches from WindowsUpdate do detect operating systems. If that was the case it looks like someone rolled their own patches (easy to do, you can extract the patches from the windowsupdate MSIs, then bundle them into 1 file) and didn't do an OS check.
They're probably using something like Novadigms's Radia. And instead of linking the correct 7 PCs, they linked to all of them (misconfigured group). In that case, it's not a case if installing a patch that is installed using the new mechanisms, the "Patch Manager" simply dumps the files to all the machines that connect up using it's client, and force an overwrite.
Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.
Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.
So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...
= The ohno second - That minuscule fraction of time in which you realize that you've just made a BIG mistake.
"Free software as in beer, copy protection as in racket" - Telsa Gwynne
Upgrades NEVER work! Not for Windows 95, 98, ME, 2000, XP, Longhorn, whatever! It will never be a good idea to try and replace a MS OS without doing a clean install.
This is first day stuff.
Without any specific details on the failure or what exactly happened, it seems like this is a huge admin error. My guess is they're using something like Altiris to do their builds, and if an admin were to accidentally "drop" the package meant for the the test group on to the production group, wham-o... every PC starts installing a build that probably isn't meant for them, and won't work. And you can imagine how that would go.
As much as I'm sure the zealots among us would like to make this seem like a Windows failure, it looks like it's more of an example of how outsourcing leads to disconnected, incompetent, and unmotivated IT staff. And that, of course, leads to mishaps like this.
Either way, if you work for a company that brings EDS in house in any way, drop your shit and run. And don't look back. The flash could be blinding.
The public sector in the UK is nothing more than unemployment benefit for the middle classes.
In my experience (having worked for both) in terms of inefficieny and stupidity, there's only one thing worse than the British Public sector and that's the British Private sector.
My company used to be part of a large public sector concern and was sold off. Since then we seem to spend nearly of our time/money:
Changing company logo and name every 6-12 months
Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others).
Paying huge bonuses to upper managent.
Paying huge car allowances to middle management including those who refuse to drive.
Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items.
Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later.
Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.
Repeatedly shuffling kit from datacenter to datacenter around the country at vast expense and disruption to our customers.
Ordering expensive buffets for management meetings , 95%+ of which get thrown away.
Managers having a schedule involving meetings all over the country which means that they spend about 25 hours out of 40 driving.
Managers refusing to use video-conferencing for meetings even in the light of the above.
How many of these things happened when I was in the public sector? Virtually none. We didn't have the money to throw around on such things. We were forced to be efficient.
Also if this private sector company I'm referring to was atypically inefficient, presumably it would do so badly it would collapse or be taken over. So this implies that many private sector companies are like this.
It's very easy to slag off the public sector if you use stereotypes, generalizations and distortions.
She added that the emergency payments system was "working perfectly."
Jones agreed, "I still have plenty of blank cheques. My pen is at room temperature."
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
Some interesting views here, but I would contend that this was a screw-up waiting to happen because screw-up potential was built in to the setup. A sysadmin has pressed the button here for sure but I wouldnt be too hasty to point the finger.
This is what happens when you have a fat client. There's a lot in a fat client. A lot to go wrong, a lot to be insecure. It therefore needs a lot of looking after. Many updates, many risks. Multiply by many desktops and it only becomes manageable by central updates. Central updates means lots of automation. Lots of automation means someone presses the wrong button and.. BANG.
But for the whole thing to go BSOD... now THAT is bad. It means you can't even back out. The reports I have seen imply that they had to nuke Windows or install stuff manually using some kind of recovery diskette... It's a disaster whichever way up you put it.
Would it have happened if they used Linux? Who knows. Linux is a complicated beastie too.
However, if they had used web apps or thin client for eveything then the issue might not have even come up.
It does make an interesting academic exercise to consider what would happen if the same screw-up hit other installations with many thousands of windows clients. Yes I am referring to the recently announced UK NHS (900,000 nodes) and US AirForce (500,000) Microsoft "wins".
I have seen NHS and DWP apps. Pretty basic stuff. Running these things on XP or W2000 is a bit of a hammer to crack a nut. The only earthly reason I can think of is the MS upgrade machine says they have to.
I wish at was Friday, but I dont want to wish my life away. So I wish it was last Friday.
Interesting set of threads ... "it's not Microsoft's fault that EDS pushed the update out wrongly".
The fundamental error here is deep seated and architectural - they have 80,000 user interface devices which are stateful. By putting the wrong device on the desktop they have set this situation up.
In the olden days when clerks in government agencies used green screens this problem wouldn't happen. If a green screen failed, it would be replaced as a FRU. Today's equivalent is something like a SunRay - the user interface device holds only enough configuration to bootstrap itself and, again, is a FRU.
The situation at the DWP is different: the user interface device is a stateful device which holds configuration itself, and requires this configuration to be consistent before it gets enough connecticity to be remotely managed. The toolkits discussed, which are used to push config around these UI devices, are probably most excellent, but there should be no need for this sort of mularky.
So while I don't necessarily blame Microsoft for this incident, I do blame them for creating a monoculture where this sort of architecture is deployed. I expect the trials underway in government using SunRay devices as the user interface will be watched with more interest after this debacle.
A final question - how on earth do DWP recover 60,000 unbootable PCs?
The last scintilla of doubt just rode out of town
It's this willingness to say "Localised error. That's all. Nothing to see here" that gives IT it's bad reputation. With properly designed processes and appropriate tools, localised error cannot have catastrophic consequences. In a system like this, I can see no excuse for pushing something out to 60K desktops in a nightly update without at least one, and probably both of:
a) Pushing it out to (say) 600 representative desktops a night or two before and monitoring
b) Having a cast-iron, regularly practiced and tested, process for pulling it back again.
Look at somewhere like SEI who make the Space shuttle flight control software. It cannot go wrong and it doesn't. Why, because they have processes! There are checks and testing and simulation and code walk-throughs and whatever, and if a problem NEARLY makes it through, and is caught in late testing or whatever, there are processes to look back and see how it got that far and make sure that the processes are improved so it doesn't happen again. The process writes the software and the people carry out the various roles prescribed by the process. There are processes for monitoring and improving the processes, etc.
The Holy Grail of IT is to reduce bugets by lowering payroll. Because Windows is really easy to install and maintain, the sales pitch goes, you can hire less skilled (expensive) people to do the work. Problem is, Windows isn't so easy to install and maintain anymore, if it ever was. Even before Active Directory, keeping 4,000 or more Windows systems up to date with the latest patches was a challenge. AD introduces even greater complexity, requiring the admins that ride herd over it have *at least* the same skill level as their brother (or sister) UNIX admins (I'd argue they actually need to be *more* skilled). Of course EDS and others have stubbornly refused to recognize this, and so you have foobars like the one reported in the original article here.
eldapo
Radia is a bloomin' pain in the neck, the last place I worked used Radia and it was horrible ( Radia it's self is possibly very nice and useful, it's the way it was implemented that was annoying ).
Company polict stated that everyone should always turn off there PC's when they left for the day and you'd get moaned at if you didn't. The Radia team told everyone they must keep their PC's on at all times but this was never company policy.
Every morning it would take 20mins or so for Radia to install all the nights patches and reboot the PC's a couple of times. At random times during the day it would also reboot your PC automatically for you if you didn't notice what was happening quickly enough to stop it.
Various PC's were being used as servers but not offically classed as such ( due to the excessive hurt and pain involved in that process ) and they also would reboot themselves randomly cause outages on whatever they were doing.
Some PC's were still Windows 95 and Radia would never manage to install anything on them, just keep crashing, rebooting indefinitely.
In the end I managed to delete enough of it that it stopped working and gave me some peace of mind.
I think the lesson here is not to just deploy cool new tools willy nilly without assessing their place in your working practices.
If you give a chimp an Uzi with a defective trigger mechanism and a bunch of people get shot, whose fault is it: the chimp's or the Uzi's? My first networking experience was with AppleTalk; plug it in and you had a network. I was subsequently required--with co-worker--to learn everything we could about Windows networking so we could implement it in one of our products.
My co-worker and I spent the next period AMAZED that Windows networking even worked at all. The system of domain controllers and WINS servers and browse lists and host files... it's too byzantine to be believed. There is, without doubt, a corporate network somewhere that could be comopletely undone by someone opening a wireless laptop in the wrong place at the wrong time. Add Windows XP and the attendant SP2 fun they're having and you get chaos.
Yes, those delightful folks at EDS are the chimps in this scenario, but Microsoft's products are definitely the defective Uzi. And I note that the BBC News article studiously avoided mentioning either of them. Hmm... Microsoft wouldn't be doing everything it can to tamp down this PR disaster, would it?
Naaah!
I disagree, although I am not talking about this in a server situation, 99% of the upgrades I've done with MS Operating Systems went flawlessly. The problem is, is that so many people do not do them properly. They don't uninstall anti-virus software (Disableing is not good enough, it still leaves filters and such in the registry), they try to upgrade from an unstable OS, they don't check application compatibility, they don't uninstall drivers where possible. Geeks do this as well.
I say this as someone who has done hundreds, if not thousands of windows upgrades and Windows installs.
It's much more reliable to back up your data and do a fresh install. I experimant with upgrades, but even(or especially) with linux, I prefer to clean the disk and start fresh. Apple on the other hand(before OS X anyway, don't know if it still is) was great. It would just create a clean new system folder. With the old one still there, I could just "bless" it if necessary. Oh, well...There's still nothing more trustworthy than pen and paper, and a good ol' mimeograph machine(the hand crank variety) for makin' copies...And they smell great.
What?
I tried to sell some software to database stuff to the EDS group in charge of the tax system (based in Telford, England). I kid you not. We asked if they has an ERD of the database. "Whats an ERD?" we got back. You kn ow a database design. How do you design the database? "Well Dave here gets on the console and types SQL statements in to Oracle". On the test system, right? "no, direct in to live". We got up and left. There is no way I am going to be front page news for my software taking down the live tax system in the UK!
First of all, that's precisely what happened here. EDS broke the update packages by bypassing them entirely using a third party product that did not do version checking.
Second, The machines wouldn't boot, therefore there is no way to run any kind of script to fix the problem, thus your third solution is likely what happened in this case as well. However, it takes some time to manually go to 60,000 machines and fix them, even if it only takes 5 minutes per machine.
The exact same thing would have happened with any OS, including Linux, had EDS decided to bypass the normal version checking tools and do things themselves, creating an unbootable system.
If you need web hosting, you could do worse than here
It's well known that EDS are incompetant and unprofessional, costing UK taxpayers hundreds millions of pounds. Examples include tax, welfareand air safety. In fact they seem to be awarded contracts by default despite not a single success with projects running hundreds of millions over budget and those that aren't a couple of years late are junked as a massive write-off.
It's well known that the UK government are in the pocket of EDS and Microsoft. The worst thing is that it's not intentional. The people in charge of making these decisions are complete non-techies and haven't heard of any IT company that aren't a regular in the new headlines of the FT. It's not corruption, it's basically a lack of education.
Phillip.
Property for sale in Nice, France