Failed Win XP Upgrade Wipes Out UK Government Agency

Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.

Comments

The reason for the upgrade

[Anonymous Coward]

They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.

Re:The reason for the upgrade

[linux_maverick]

LOL! Someone, please mod this up :-D

Re:The reason for the upgrade

[Not_Wiggins]

They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.

Looks like they got a deal; they got the version that also blocks viruses, worms, and abuse of Solitaire! ;)

Re:The reason for the upgrade

[phaln]

In other news, productivity is UP 64% since the day before the crash!

Re:The reason for the upgrade

[Anonymous Coward]

That wouldn't have been hard. This is a government department. The only place for productivity to go would be up.

Re:The reason for the upgrade

[Lev13than]

They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.
Looks like they got a deal; they got the version that also blocks viruses, worms, and abuse of Solitaire! ;)

Writing article about Free iPod. Please help out.

They probably wanted to block assholes who disguise 'Free iPod' links in the sigs. 'TinyUrl' my ass. If you want an iPod, ask your parents to raise your allowance. Otherwise, I heartily encourage you to fuck off.

Re:The reason for the upgrade

[msisden]

Here in Tech Support call that 'Firefox'

Re:The reason for the upgrade

[Not_Wiggins]

And for the record... the article is not going to have favorable things to say about the free ipod experience. Maybe it'll make you feel better, but you've contributed to that sentiment. Once it is done, I'll be changing my sig/home url to the article.

Sheesh! Really... get a grip!

Re:The reason for the upgrade

[realdpk]

Hear, hear. My foe list (comprised of freeipod, gmail, etc spams) is getting pretty large.

Re:The reason for the upgrade

[BLAG-blast]

Writing article about Free iPod [tinyurl.com]. Please help out.

If you are seriously writing a article on free ipod crap why don't you link to a page that explains a bit about your project. Otherwise you're not different from the 12 year old kids you are internet begging for ipods (I'm assuming that your older than 12, could be a bad assumption).

Will you give away the ipods you get, or will you keep them? With Xmas coming up there are a lot of poor childern out there who are going to get any much else other than AOL and Live linux CDs.....

And for the record... the article is not going to have favorable things to say about the free ipod experience.

Is you writting going to be bias from the start?

Or are you writting about how the free-ipod fad is causing a lot of REALLY ANNOYING internet begging. "I want an ipod, please give up your privacy so I can have an ipod". Please note that "I want a free-ipod so I write bad things about free-ipods, please give your privacy so I can an ipod" is in no way any different.

Oh wait, I've seen a post where you accuse somebody of being home from highschool, you're most likely 12 years old.

Sheesh! Really... get a grip!

Sheesh! Really... get a grip!

Re:The reason for the upgrade

[Anonymous Coward]

This is a government department. The only place for productivity to go would be up.

Dear Sirs:

I am a highly overpaid government employee with nothing better to do all day than read Slashdot and post AC comments, and frequent gambling and off-track betting sites. I and those of my many co-workers that were good enough to show up for work on a Friday and are not currently on a 3 hour lunch break and are looking at this on my monitor find the above post highly offensive and demand that you moderate it "flamebait " immediately. Otherwise, we shall be forced to file a grievance and call a national strike, affecting dozens.

Re:The reason for the upgrade

[Anonymous Coward]

I've got to remember to request that my jokes be modded up as well.

lol - that was great - you've got to mod that up!

Re:The reason for the upgrade

[Anonymous Coward]

It wasn't his joke, it was my joke. I didn't request anyone mod anything, because I knew it would be modded up anyway. I've got a knack for Slashdot humor.

Re:The reason for the upgrade

[Not_Wiggins]

If you are seriously writing a article on free ipod crap why don't you link to a page that explains a bit about your project. Otherwise you're not different from the 12 year old kids you are internet begging for ipods (I'm assuming that your older than 12, could be a bad assumption).

Good idea. I'll have to get that up as work allows.

Will you give away the ipods you get, or will you keep them? With Xmas coming up there are a lot of poor childern out there who are going to get any much else other than AOL and Live linux CDs.....

That assumes that one gets multiple ipods from the deal. Point is, you don't. That's one of the "negative" reasons behind the offer scheme.

Or are you writting about how the free-ipod fad is causing a lot of REALLY ANNOYING internet begging. "I want an ipod, please give up your privacy so I can have an ipod". Please note that "I want a free-ipod so I write bad things about free-ipods, please give your privacy so I can an ipod" is in no way any different.

That's basically what I've surmised. You believe that I've started my article with a bias, but that wasn't the case; I really wanted to see how people would react to this sort of thing. It is interesting because the response really has been mixed. There are those who think it is cool/interesting to sign-up for offers they were interested in already just to get "free stuff" (albeit not free, because they sell your info to marketing houses). And there are those who spit venom on anyone trying this.

What I'd really like to know is why people are so venomous about it instead of trying to educate those who are participating in what has become an annoying trend. Seems to me that while what I started writing originally was about the marketing phenomenon, it has been slowly transforming itself into a piece about intolerance on the internet: we don't see other people as real people. And that really hits both sides of this issue with the iPod. People who "want one" don't think about if they're annoying other people or what the consequences really are (in terms of privacy/annoyance) to those they get to sign-up. On the other side are those people who'd rather just swear, call names, and flame. The ironic thing is that by doing those things, they're trying to make the problem go away but don't actually do anything to contribute to helping; it just vents feelings of frustration and annoyance.

From where is all that frustration coming? I can't imagine it is just ONE offer ONE time. It has to be a build-up of some sort of abuse internet users generally feel... but what is it? Are we sick of marketing pervading our living spaces? Is it frustration with the economy? Is it that the anonimity and ease of the internet tempts some dark part of human nature too readily? I dunno... and I'd like to find out.

I think I've gotten all I'd care to get out of the link test.

I'm still trying to figure out if the uptick in aggressively negative response was because I switched it from the obvious URL to the "tinyurl.com" one or if people's patience is just wearing thin over this whole marketing thing.

And as for donating all those iPods, there were some people who signed up, but almost nobody "completed" an offer... there is no iPod to donate.

Re:The reason for the upgrade

[Anonymous Coward]

I've got a knack for Slashdot humor.

and oxymorons too, evidently.

Re:The reason for the upgrade

[skids]

OK, fair enough. A lot of the people I worked with when I was in a government position do deserve that.

But a lot of them don't. I would say most state employees work their asses off doing pointless things, rather than screwing off. The problem is more with upper management than with the rank and file... though the problem does bleed over into the lower level employees because, after all, how long can you pour your energy into a task that you know only is neccesary because incompetent managers fail to streamline the operation and give you more real, productive work, before you start to take the job much less seriously?

So those petty state officials who shirk work do so as much due to being beat down, disillusioned, and tapped out as far as trying to do something about it in the face of a "front row" that doesn't like to listen to comments from their inferiors.

When I was working for the state, I considered myself very lucky to be involved in a project that was doing something meaningful, being productive and, while mistakes were made here and there, was relatively efficient overall. I could see how this was not the case in the departments working beside ours.

Eventually, though, the egos of the upper echelon managed to intrude even into our well defended (by caring managers) little island of fortitude and competance, and I had to say screw it. Now, unlike most of the rest of my friends that got laid off and sucked the government unemployement insurance tit, I am fending for myself with the money I saved by not buying useless crap.

So when people try to say I was overpayed at 60% of my fair private-industry salary, I don't shirk from the criticism. Yeah, the benefits were better than the private sector and the environment more permissive, but at least I didn't go looking for a handout like others so they could keep up the credit card payments for their DVD collections and car loan for their gas guzzling S.U.V.

At least I, one of those loathsome, lazy, state workers, had the good conscience not to apply my talents to better the carreer of a gaggle of idiots who aren't overseen adequately by the legislature that created their positions. If you want the state sector fixed, aim at the top. The clock punchers at the bottom are just a symptom of a management that preserves itself by not giving their underlings enough of a reason to revolt.

Re:The reason for the upgrade

[Anonymous Coward]

Its too bad firefox crashed the first time I started reading this.

Re:The reason for the upgrade

[magefile]

People generally don't like tinyURL. No knowing if it links to cnn.com/foo or goatse.cx (or whatever the current mirror is, since that site is offline). There's a negative reaction in general to the freeipod crap, but getting rid of that tinyURL would probably help you.

Re:The reason for the upgrade

[Not_Wiggins]

Actually, it turned out to be a good thing that I had that tinyURL link in there (originally, just done for simplicity, but had a "side-benefit.")

It really helped to highlight some of the issue with the freeIpod stuff; people can ignore marketing, but are absolutely infuriated if they feel "deceived" somehow. The line with the link didn't change, just the url behind it... and I can tell you, it was like throwing a rock at a hornets' nest. 8)

Although, now it matters not as I've gathered sufficient info from this channel to draw some conclusions. Small karma hit, but man... it was certainly enlightening. 8)

Re:The reason for the upgrade

[Secret Agent X23]

It really helped to highlight some of the issue with the freeIpod stuff; people can ignore marketing, but are absolutely infuriated if they feel "deceived" somehow.

And you didn't see that coming because... ???

Re:The reason for the upgrade

[Anonymous Coward]

> But a lot of them don't. I would say most state employees work their asses off doing pointless things, rather than screwing off. The problem is more with upper management than with the rank and file...Someone is jealous they never had the goods to get promoted.

Boohoo. The low level lackies are the ones everyone sees every day, and they are the one slacking off. Where do you think these rumors start? Self-righteous employees like yourself? Think again.

The underlings don't have reason to revolt because you get your benefits and you can almost not ever be fired, when you work for the U.S. government anyway. That is the key difference between a government job and a job like mine. I actually have to create results.

Sure, it doesn't help to have a PoS bureaucratic system at the top, but at the bottom you guys are doing absolutely nothing to change the situation. Also, when one of you finally does get promoted, you simply become what you once cried about, a bureaucrat.

Re:The reason for the upgrade

[skids]

"Someone is jealous they never had the goods to get promoted."

I did get promoted, and was in danger of being recruited into management. I say danger because I wouldn't have wanted my manager's job. He was given short shrift as far as the pull he needed to do it right. (They even asked me if I wanted the promotion before promoting a new cooworker twice my age.)

So you were saying?

"The underlings don't have reason to revolt because you get your benefits and you can almost not ever be fired"

Here you are right. In order to hide themselves from being routed for their gross incompetance, the upper management actively seeks to promote the fat lazy clock-watcher environment. The strategy is simple -- take away any claim your employees have to acheivement, so they have no opportunity to distinguish themselves, and they will fall into the mentality that they are lucky to have their job, and they should ride along passively.

"That is the key difference between a government job and a job like mine."

Again, here you are right (I just exempt myself because my unit created results even though we really didn't have to.) But who's fault, again, would it be for a workplace where results are not monitored? What's that I hear? Management is responsible for monitoring results? Oh good, then you get my point.

"at the bottom you guys are doing absolutely nothing to change the situation"

Correct again! I am currently doing absolutely nothing to change the situation. The last thing I and a few other cooworkers did to change it was quit. Now I do nothing at all. I figured giving 110% effort to the state for seven years for no good reason was enough of a contribution for me.

However, may we look at the chain of command here? Since when did a typical organization or a company run on a populist structure? The vast majority do not. The responsibility for creating change in the system is with... the people that run it, duh!

Like I said, if you want your state government employees to work for you like your tax dollars mean something, take it to the top. Don't take your grievances out on a bunch of wage slaves, it gets you nowhere. Noone stages a populist coup within an organization, that's a political phenominon, not a business one. Blaming disenchanted lowlevel state workers doesn't help the situation any.

Re:The reason for the upgrade

[symbolic]

Now, unlike most of the rest of my friends that got laid off and sucked the government unemployement insurance tit, I am fending for myself with the money I saved by not buying useless crap.

Very good. This is exactly the type of discipline and self-reliance that has become foreign to many Americans.

Re:The reason for the upgrade

[uhlume]

"LOL! Someone, please mod this up :-D"

Re:The reason for the upgrade

[mikefe]

The problem is that if you attack the ones at the top (Don't vote for their tax increases or bond measures for more money or etc) they'll take the money from the low level workers you mention so they will go on strike in front of the media because their department had funding cuts. The end result is a guilt trip "think of the children!" for the people who don't believe in giving more money to beaurocrats.

So how do you go about getting the bad managers out at the top without them fucking everyone over in the process?

Re:The reason for the upgrade

[eskayp]

One thought on taking it to the top:
As we say in the wastewater trade,
'Shit runs downhill'.
Putting pressure on the legislators, bureau heads, or departmental directors only makes life more miserable for the people actually trying to do the work at lower levels.
Career bureacrats and politicians (at least the ones with enough brains to survive) have a variety of means for passing the buck on down the line.
A large part of the problem is the 'don't make waves' corporate culture.
No ripples, no trouble, and especially no negative publicity.
The same malady also exists in the private sector if the link between productivity and survival isn't strong enough.

Re:The reason for the upgrade

[BLAG-blast]

I'm still trying to figure out if the uptick in aggressively negative response was because I switched it from the obvious URL to the "tinyurl.com" one or if people's patience is just wearing thin over this whole marketing thing.

I'm not sure about the URL, but this free-ipod thing is annoying, when playing online games people are asking others to help them get ipods. In some ways it worse than popups and flashy banner ads, because it 'infects' a person who then becomes the annoying advertisement themselves. And they'll "popup" and advertise at any opertunity that they think they can get a lead (it's all about lead generation anyway).

There is a next level, when people are using false pretences to get the lead quota they need for to qualify for an ipod. I think this what the original person who commented about it was taking to task.

I fear if successful, then we will see this a lot more. It could be a big win for the companies, the more people to try to get ipod, the less leads that they are to go around, meaning many people won't fill their quota, meaning less ipod the adverising company has to give out (but they still get the same amount or more leads, with the less cost!).

What's the world comming to....

Re:The reason for the upgrade

[Not_Wiggins]

I fear if successful, then we will see this a lot more. It could be a big win for the companies, the more people to try to get ipod, the less leads that they are to go around, meaning many people won't fill their quota, meaning less ipod the adverising company has to give out (but they still get the same amount or more leads, with the less cost!).

As part of my research, I did happen upon someone who had explained the math of it pretty well here. (Please note, this isn't me, and I'm not affiliated with that site at all).

But, to comment on some of your other thoughts:

...when playing online games people are asking others to help them get ipods.

You're absolutely correct that this is an abuse of the original concept of viral marketing, and moreso, part of the effect of "annonimity" on the internet. I believe the original intent of viral marketing was to get people to ask their friends and family, really people that they know, to try offers. It would be that bond that makes the sale. However, people have taken the megaphone-internet and started shouting their message to anyone/everyone that they can reach... and with the internet, that's a whole mess of people.

This is the same thing that ticks people off about SPAM. I mean, if it was just one jerk sending email, it'd be easy enough to ignore him. But once thousands and thousands start doing it, it becomes more than just a frustrating endevour; it becomes a real hinderance to the use and enjoyment of the network.

So... what's the solution to this (these) problems? I think at this stage people are still upset over having to deal with anything (basically, the whole "why should it be MY problem when it shouldn't be a problem at all?!?" view). So, they tend to just get angry about it. Once we're past that, what can we do about it? Education? Regulation?

Assuming the collective attitude is that regulation is "taking away freedom," we must turn to education. So the question then becomes: what are you doing to help educate those around you without falling into the same "spam-trap" that gets one irritated in the first place?

Uh-oh...

[Dynamoo]

You know that sinking feeling when you've just pressed the wrong button...

..of course, it seems to be our friends EDS behind it, who are just great at making a mess of government contracts.. and then, the government just gives them another one.

Re:Uh-oh...

[Coryoth]

Everyone needs a second chance. And a third. And a fourth. And a fifth. And a ...

I'm sure the government has perfectly good reasons for continuing to hand contracts to EDS. It's just probably not a reason they want to tell you because it involves (bribery|nepotism|stupidity|all of the above)

Jedidiah.

Re:Uh-oh...

[ceeam]

You know that sinking feeling when you've just pressed the wrong button...

Well, actually, since I've read and understood BOFH writings, no. :)

Re:Uh-oh...

[Cally]

I'm signing on at the moment and noticed the 'EDS' badge on the side of the Jobpoint machines. These aer standalone appliances (literally, they're large freestanding devices with no visible interface apart from a flatpanel touchscreen; the UI looks like Flash to me. You occasionally see a flicker of a scrollbar on one side of the screen, it looks like Winders underneath.) These are actually pretty good, probably cos they've got one simple job to do, allow people to browse the jobs database and print off details of anything interesting (there's a thermal paper printer, rather like a shop till.)

However the rest of their systems look terrible. in particular, they're now forced to do a search for the jobs yuo've said yuo'er looking for, in front of you, before you sign. (For ex-UK people, the way it works is that you turn up once a fortnight, sign on the dotted line, and recieve a cheque / fnuds transfer to your bank a/c a fwe days later. I get GBP53 a week which I think is the standard rate.) The desktops run a revolting system that appears to have a Visual Basic front end - it has that awful "Wow, look how many colours I can use on one form!" look UI that I thought died out in 1995/96 - probably after this system was implemented. Watching them search is *painful*; they have to enter multiple arbitrary codes (for the job types, and for the geographical area to search) that are apparently random strings of numbers and letters. Watching staff using the interface it's obvious no proper HCI clue went into it - they've been taught how to use it in a 'monkey see, monkey do' style. "Click here, then pull down this menu, enter that code from the other dialog, click OK, and again, then CANCEL, then stand on one leg with fingers crossed UNLESS IT'S A WEDNESDAY..." horrible.

Of couse I'm looking for particular kinds of job - pentesting, security consultant, Linux system or network admin, Perl / Apache development work. This all comes under one category (of thousands) CM297 or some such. Naturally the best fit they've *ever* found for me was "Sales assistent at PC World" :(((

Naturally I would rather starve than sell people Windows for a living; as I've been signing for 13 weeks, they have now told me I have to accept the first thing offered - I'm stuck in the countryside where there are few proper IT jobs, so that would mean working as a cleaner or 'carer' (wiping arses of OAPs in care homes... :((( ) or they'll cut off my dole. I told them to sign me off and walked out - so hearing about this catastrophe this morning brought a sick warm glow of schadenfreude to me.

Fortunately Jobserve and OSTG have some reasonable things in the area & I have a couple of promising interviews lined up for the next couple of weeks. Fingers crossed... it's gonna be a pretty crappy christmas if I don't get something, the insurance payments on my car are about to lapse, "result: misery" as Dickens said.

Re:Uh-oh...

[Peil]

I know that feeling mate, couple of years ago I got bumped out of my consultancy job and the visit to the job centre was horrible. The complete lack of understanding that you don't want to p'ss away 8 years as a project manager to take a development job at the other end of the country for peanuts was very annoying. The how little would you work for was a particular gripe - you say £20k, thinking 5 below the national avg wage is indicitive of being willing to look for work and some bint says your being unrealistic why not £15k? I could go on at length, but keep the faith, Jobserve came through for me and I'm now with a big 4 financials company doing a dream job (for me)

Re:Uh-oh...

[WillerZ]

Local government (and possibly for most central departments) is obliged to award the contract to the lowest bidder. This is usually EDS. EDS than cock it up, get paid extra to fix it cos the contracts are badly written, rinse and repeat.

EDS should be dissolved for bringing the industry into disrepute. Why is it only lawyers and footballers that works on?

Phil

Re:Uh-oh...

[CmdrGravy]

I am out of work at the moment but luckily still have enough redundancy money left not to have to put myself through the signing on hell.

Last time I was out of work looking for IT work the jobcentre couldn't suggest a single suitable job in something like 3 months of trying. If I had wanted a job in a burger van, hotel, hairdressing or as a receptionist or secretary I'd have been laughing. Anything other than that and you have no hope of them ever offering you a suitable job so it just turns into a farce you have to go through to get your cash.

"What have you been doing to look for work this week then ?"
"I have looked on Jobserve quite a lot"
"Jobserve, what's that ? I've never heard of it, have you tried looking in the local free papers ?"
"Grrrrr, look just gimme my money !"

Re:Uh-oh...

[Anonymous Coward]

As a current EDSer, I'd ask: How about giving some of us a break? EDS is a huge company, and despite some recently publicized problems, the majority of us are competent, hard working folks trying to do a good job for our customers. Dick Brown did some spectacular damage to the company before he pulled the ripcord on his golden parachute.

Re:Uh-oh...

[Cally]

cheers geezer :) Yep I know something will turn up when I need to it.. which is about now, actually! To be fair I have to say that the pleasure in waking up at 7am every day & remembering I have nothing more important to do but walk around the local woods picking up litter (IO'm in the Forest of Dean on Welsh border, beautiful countryside) and then do a fe pages proofreading for Project Guteneburg, then over to /. for some metamod is still considerable.

This is typical of our government.

[bairy]

If something is actually working right (and it's rare), change it!
I can imagine it now
Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"

The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.

</cynical>

Re:This is typical of our government.

[Anonymous Coward]

I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.

That's the problem, though, none of these 60,000 systems would have been critical. After all, they'll lose a few every day, just with natural breakage. They'd have had only minimal support and backup, and any truly important data should have been kept on the network anyway.

People who like these automated update systems don't seem to realise just how much damage can be caused by losing 80-100% of your non-critical systems. That a single mistake can do this is simply unacceptable; people make mistakes - even sysadmins.

We're going to see a lot more of this over the next few years.

Re:This is typical of our government.

[Anonymous Coward]

"all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do..."

I'd be suprised if anywhere backed up 64,000 PCs in their entirety on a daily basis. The whole point of network storage and image servers is that you have a standard desktop install using some imaging technology or other and all your data is held in, er, the data centre.

Re:This is typical of our government.

[Anonymous Coward]

Backup 60,000 desktop PCs?

right...

Re:This is typical of our government.

[cannon fodder 0109]

Upgrading from Win2000 to WinXP SP2 needn't be painful.

The company I work for (Wessex Water - regional water and sewerage company in the southwest UK) is doing the same rollout and it's going quite smoothly. On the other hand we haven't outsourced it and are doing it stepwise - noncritical systems first - so our IT guys can squash bugs as they become apparant.

Heaven forbid that such a sensible and useful idea could occur to a government depatment.

Re:This is typical of our government.

[bairy]

I meant restore the 7 pcs they'd upgraded. I would assume the 60,000 died either because those 7 were feeding them, or because or an incompatibility with the upgrade. If you're upgrading ANYTHING that might affect anything else you should do a back up. So in both cases I would have thought a simple restore would have made everything happy again..

or have I missed something?

Re:This is typical of our government.

[bairy]

Oh, nevermind, I just got it. They somehow pushed the upgrade to 60,000. My mistake.. and theirs too

Re:This is typical of our government.

[blowdart]

Only theregister appears to talk about Win2k and XP, so lets see what they're saying.

According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

So if this is true then EDS pushed out a partial upgrade. Now come on, if you installed 75% of a new distro over an old one then rebooted would you blame Redhat because it didn't work?

Or there's the other version

DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.

So again EDS pushed out XP patches, overwriting Win2k files and the machines crashed

Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.

Of course theregister could be wrong. It might happen. Heh.

Re:This is typical of our government.

[Build6]

I meant restore the 7 pcs they'd upgraded. I would assume the 60,000 died either because those 7 were feeding them, or because or an incompatibility with the upgrade.

hypothetical scenario - they "upgrade" the 7 PCs. the 60,000 other PCs start going down left and right. the guys say "oh shit!" and they restore these 7 PCs... and the other 60,000 PCs *do not come back up*. what do you do then?

it's entirely possible that whatever the upgrade broke that the other 60,000 PCs depended on, resulted in things getting munged up on those PCs in *permanent* way (registry damage?).

in which case one wonders about whether/how all those 60,000 others were backed up (or, more likely, not).

Re:This is typical of our government.

[clickster]

I wouldn't consider daily system images of 80,000 PCs to be feasible or cost effective. Let's say there's an average of 5 GB per PC. You're looking at 400 terabytes of data storage just for the images, not to mention 400 terabytes PER NIGHT of bandwidth. 100 Mb/s throughout the organization or not, that's a lot of bandwidth.

Re:This is typical of our government.

[Anonymous Coward]

They should have disk images or something that would allow them to rebuild the 64000 back to the baseline W2K config. However, it's still going to be a royal bitch to do so.

Re:This is typical of our government.

[bairy]

hypothetical scenario - they "upgrade" the 7 PCs. the 60,000 other PCs start going down left and right. the guys say "oh shit!" and they restore these 7 PCs... and the other 60,000 PCs *do not come back up*. what do you do then?

TA is certainly lacking on info so we don't know a lot about these PCs.
My guess would be that the 60k would actually just be drones. I mean, it seems daft that each would contain individual information. On that basis, a) the 7 may have been feeding them and a simple reinstall from one backup (as we've both said, as if) should be able to fix one office at a time and b) just reboot the 60k and they'll reconnect to the leaders... if they had been restored that is.

Dunno, not enough info given, but I find it too unlikely that 60000 down means 60000 seperate repair jobs.

Re:This is typical of our government.

[niks42]

But bless them all! Every time something like this happens, I feel just that bit more secure in my job, since there will always be a call for people who can, to help clear up disasters after they happen. I am looking forward to the NHS computer system re-vamp, with BT and others running the program. It should guarantee me employment from now till retirement.

Re:This is typical of our government.

[swv3752]

If you are pushing out packages to Linux then almost assuredly you are using RPM or dpkg. Either has version checking that gacefully disallows fatal partial upgrades.

Re:This is typical of our government.

[FurryFeet]

or have I missed something?

Yes. Namely, TFA.
In a nutshell: As they upgraded those 7 machines, the patch was applied to al 60K PCs, seriously breaking them. Even if you revert the 7 original ones, the 60K ones will remain broken.

Re:This is typical of our government.

[gozar]

I wouldn't consider daily system images of 80,000 PCs to be feasible or cost effective.

You don't image every machine at night. You have a standard image (and with 60,000 PCs they probably would require several) that has the required software. All user files should be stored on a file server, which is backed up.

That's how I manage ~600 Macs. If I can't fix a problem with a machine in 15 minutes, it gets re-imaged. Quick and easy way to deploy and fix machines.

Re:This is typical of our government.

[clickster]

I imagine they have some basic images. But that still wipes out any stored e-mail (servers can't always handle your archive) that users may not have backed up recently, any software that doesn't need to be part of the "base image", and software configurations specific to the user, and you'll still have to do things like go around to 60,000 PCs to set up their individual e-mail, etc. Remember, you can only have so many base images. They usually contain the bare essentials such as the OS, office suite, and maybe a few utils that a very large group of people use. But in an organization that size, it'll be hell dealing with all of the small details that individual users will have. Imagine trying to deal with verifying licensing while you're doing that. The base images will only get them to the point that there is a PC that is booting and perhaps has some of the essentials. Nothing beyond that. Not to mention that they will not have been patched since the last round of images.

Re:This is typical of our government.

[man_of_mr_e]

And if you're pushing out updates to Windows, you're almost assuredly using a Windows update package that also checks OS versions.

Guess what? EDS chose to do it themselves using a third party product rather than use the much more mature and safe existing update tools.

Now who's fault is that?

Re:This is typical of our government.

[gozar]

I imagine they have some basic images. But that still wipes out any stored e-mail (servers can't always handle your archive) that users may not have backed up recently, any software that doesn't need to be part of the "base image", and software configurations specific to the user, and you'll still have to do things like go around to 60,000 PCs to set up their individual e-mail, etc.

E-mail should either be stored on the server and accessed through imap or in their documents folder which should be stored on the server and backed up with the server. Their settings should also be stored in their file server account, not on the individual machines. I'm assuming they would be using roaming profiles, so that all configuration for the user is stored with their documents. The only thing you should have to do after re-imaging is re-install any software that is unique for the user. But even with this software their configuration should be stored with their files.

Under OS X (and I'm assuming XP/2000) everything unique to the user is stored in their home directory. All settings, email, everything. I can wipe out machines at will, and not worry about losing files or settings.

Another nail?

[Justin205]

Hopefully just another nail in Microsoft's coffin...

When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.

On another note, How did upgrading seven machines to XP BSOD 60000?

Re:Another nail?

[Anonymous Coward]

From the Fing Article

"EDS were going to apply a patch to these [7 machines], unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops."

I've heard of this happening several times in UK local government (friends work for a small software house that supplies local government offices). Always mount a scratch monkey!

Re:Another nail?

[Anonymous Coward]

You don't work for a large organization, do you? ;)

In short, any large org (gov, corp, edu, whatever) will sweep such incedents under the rug and continue to use whatever they had that caused the problem (thereby insuring they'll have similar problems in the future). Genius, of course, but business as usuall.

[/cynicism]

Re:Another nail?

[Firehawke]

It's in the article. Quoting from the Register article:

"According to staff reports the outage occurred on Monday afternoon, disconnecting staff email, benefits processing and Internet and intranet connectivity. According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'"

More fun with patches.. puts any trouble you're having at work today into perspective, doesn't it?

Re:Another nail?

[turgid]

Hopefully just another nail in Microsoft's coffin...

No it won't :-(

They will be impressed my Microsoft's "comittment to customer service" by having flown out high-ranking people from all over Europe to attend to it.

They are already to stupid and lazy to see through it all. That's why they chose Microsoft and EDS in the first place.

BTW I once had an interview with EDS to work on one of these government jobs. They wanted me to cut off all of my hair and go to their corporate boot camp to learn COBOL and IBM S/360 assembler. This was in 1996.

Re:Another nail?

[Amiga Trombone]

When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.

No, but that doesn't necessarily mean it's bad for the rest of us!

Let's hope Congress plans to upgrade soon!

See? Even Microsoft is good for something!

Re:Another nail?

[gbjbaanb]

and you missed out big time. 4 years later you could have been naming your own price for Y2k fixes.

You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.

Re:Another nail?

[DrSkwid]

lol, you *must* work for a govt. agency, perhaps education ?

incedents : incidents

insuring : ensuring

usuall : usual

Re:Another nail?

[WebCrapper]

What gets me is the idiots decided to upgrade all of the machines at once. Who, in their right mind, upgrades these types of systems all at once?

Re:Another nail?

[turgid]

Everyone else was doing Y2K fixes. I very much doubt I'd have made much money at it at all.

You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.

I get to do cool stuff with UNIX nowadays. 40 years of cool stuff is better than becoming an EDS pointy-hair for 4 years and having to learn IBM JCL.

Re:Another nail?

[Stalus]

Other people have mentioned that the upgrade to the other computers was accidental. However, it's not just that.. the articles imply that some moron sent out XP patches to 2k machines.

"...unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops..."

Gee.. applying a patch to the wrong OS broke them. Really? Yes, in their code, Microsoft could have probably added some safeguards against stupidity, but this failure wasn't quite due to Microsoft. They do plenty of things that are their fault.. don't bash them when it's not - you just lose credibility.

Re:Another nail?

[djfray]

'Hopefully just another nail in Microsoft's coffin....' Buddy, you need to wake up and realize Microsoft aint going no where. Just like when faulty parts of cars kills tens of thousands of people a year(if not more), GM isn't going out of business. They are a force to be reckoned with. Microsoft is the same, and everyone needs to learn how to live with it, instead of constantly bitching about what wouldve happened to any leading OS, or software company, were Microsoft not in their place. Sheesh

Re:Another nail?

[DarkMantle]

On another note, How did upgrading seven machines to XP BSOD 60000?

Simple, Windows problems are too big to fit on the computers they start on. ;)

Re:Another nail?

[Anonymous Coward]

They only tried to update seven (7) of them. Its just they probbably were thick enough to press, "do you want to extend this insulation to the rest of the network?" yes, no, DOH!

Re:Another nail?

[Zangief]

Easy, a dialog like this appeared:

"Do you want to update the machines on your network now?"

[Accept]

No cancel button.
--
Wiki de Ciencia Ficcion y Fantasia, un cuento por Fly.

Re:Another nail?

[ultranova]

What gets me is the idiots decided to upgrade all of the machines at once. Who, in their right mind, upgrades these types of systems all at once?

No, they didn't try to upgrade all the machines at once. They tried to upgrade seven Windows XP machines, but somehow screwed up and ended up pushing this upgrade to 60 000 machines, which were running Windows 2000, and since the patch wasn't meant for them, they broke.

Of course, one might wonder why the patch couldn't detect the version of the operating system to prevent this kind of trouble...

Re:Another nail?

[blowdart]

Of course, one might wonder why the patch couldn't detect the version of the operating system to prevent this kind of trouble...

The usual patches from WindowsUpdate do detect operating systems. If that was the case it looks like someone rolled their own patches (easy to do, you can extract the patches from the windowsupdate MSIs, then bundle them into 1 file) and didn't do an OS check.

Re:Another nail?

[ThousandStars]

having to learn IBM JCL.

I can't argue refusing the Dark Side (EDS), but what's wrong with learning something new?

Re:Another nail?

[turgid]

At the time, Linux/UNIX was what I was interested in, and still am. Back in those days I could see that Linux was going to be big some day and I was right. It's far more useful both professionally and personally. IBM JCL and all it's other proprietary nonsense may be good for an employment niche, but that's about it. Even the evil IBM is talking about Linux on the mainframe. (OK it's only talk, they'll still try to upsell you but that's another story).

Re:Another nail?

[mikechant]

Even the evil IBM is talking about Linux on the mainframe. (OK it's only talk, they'll still try to upsell you but that's another story).

It's not 'talk', they have a number of customers running 100's of Linux virtual servers on one zSeries mainframe. Apparently ALL their sales growth for last year was attributable to Linux.

Re:Another nail?

[mikechant]

Question - could this happen with (say) RedHat? Or is there a check that would prevent this?

If it can happen in RedHat etc. as well as Windows, fair enough. If it can't, then MS *do* deserve bashing.

Re:Another nail?

[turgid]

Yes, but when those customers find out how slowly the Linux virtual servers run, they go right back out and look for proper UNIX or Linux servers. It is talk.

Re:Another nail?

[Fulcrum of Evil]

what's wrong with learning something new?

JCL is known to cause brain damage.

Re:Another nail?

[swv3752]

If people are actually using RPM and not using custom rolled *.rpms or just plain tarballs? Then no, it couldn't hapen on Red Hat or any other RPM-based distro, nor a Debian based one.

If one is creating custom packaged RPM's then it is just barely possible, if one ignores all testing practices.

Re:Another nail?

[lysium]

Can I buy your mercenary soul for thirty pieces of silver? I can't imagine you having much of a personality. Loser.

Re:Another nail?

[the original m0nk]

sorry, afraid i'm not terribly familiar with the acronym.

but since you mention it, does in fact EDS stand for Emperors' Dark Side?

oh hey what's going on with this upgrade

[Anonymous Coward]

OH SHI-

Re:oh hey what's going on with this upgrade

[Anonymous Coward]

:laffo:

Too slow.

[Lostie]

If only they had reached the conclusion hinted at in this BBC News article a year or two ago, this would not have happened.

It's certainly bad PR for Microsoft though, perhaps this will serve as a wake-up call to other governments that "other options" are out there.

Re:Too slow.

[Apathetic1]

It was EDS that screwed it up. I can't say I'm surprised. For once I find it hard to blame Microsoft - rolling an XP patch out onto a Windows 2000 machine (or 60000) will have the predictable effect of hosing the system. Given what I know about EDS (I worked there for two summers) I don't think running Linux would have helped.

Re:Too slow.

[Zocalo]

Actually, I tend to see this as potentially an opportunity for Microsoft to gain some excellent, and totally undeserved, *good* PR. The root cause of the problem seems to be that EDS erroneously pushed a Windows XP update out to Windows 2K desktops - hardly Microsoft's fault. Having got completely out of their depth (which isn't especially far out of the shallows given EDS' track record to date) EDS decided that it couldn't fix the problem and called in Microsoft.

Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.

Re:Too slow.

[Rick Zeman]

Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.

And me without mod points (so mod parent up!). I think you're exactly right...sadly.

Re:Too slow.

[Progman3K]

No doubt they were aware of the report, but probably got "incentives" from Microsoft to just upgrade.

All I can say is that now they will have to pay the piper for disregarding that report.

A lot of world governments are saying "The data we are safeguarding is The People's data, and as such is far too valuable to put in the hands of one corporation/vendor, so we are going with open-source"

Those who do not are failing their citizens.

Re:Too slow.

[Tony Hoyle]

My question is what kind of moronic update system do that have that would even *try* an XP-only update on a Windows 2000 system? Shouldn't it have performed the most basic of sanity checks first?

Secondly, since these are critical systems, why couldn't they be told to rollback automatically on boot?

Re:Too slow.

[Sxooter]

Really, I'd expect Microsoft to have designed the two very different operating systems to NOT take each other's patches. It couldn't have been that hard to do, just toss some identifier in there somewhere in the file and if XP sees a win2k id or vice versa, refude to install the update.

It's still Microsoft's fault, because they designed a system that accepts updates for the wrong system, and after that update is installed, it's damned near impossible to back it out. EDS has fault here too, but let's face it, they couldn't have screwed the pooch nearly as well with a non-MS based system.

Re:Too slow.

[Asic Eng]

Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

They could have called Novell or IBM.

Apart from that though - any setup can be screwed-up by an admin, no currently available OS can protect you from that. So for a TCO estimate at least we would have to look at the total loss due to screw-ups like this, and weigh them with the number of installations. Using a single data point can't be valid. That said, my gut feeling is that Linux provides considerably better TCO.

Re:Too slow.

[Apathetic1]

Microsoft can't help it if you force an XP update onto a Win2K machine using an automated tool. Trying to manually install a patch onto the wrong operating system WILL fail as it should.

Somebody else in the thread mentioned this - if you overwrite your Linux kernel with a botched version, your system's hosed. If you didn't keep a backup, it's damned near impossible to back it out.

Nobody can protect an incompetent admin from him / herself.

Re:Too slow.

[Wudbaer]

If you use a third-party upgrade tool or cook your own solution it can happen. The same if you just copy over the files by hand or some script. The service packs themselves do not install on the wrong OS, e.g. they politely remind you that they are intended for W2K, but that you currently are running the installer on XP (tried this myself and assume it also behaves the same the other way round which would have applied in this case). But in the end there is no final cure to utter stupidity any vendor or Open Source group can apply.

Re:Too slow.

[ibeleo]

Since the problem was that some person fat fingered the automated update of 7 test machines and accidently pushed an untested patch live to 60,000 production boxes, I have one question:

Which version of Linux is self-aware enough to say "No Dave, I'm afraid I can't do that - you didn't test this patch enough" ?

I need to know so I can welcome our new digital overlords!

Re:Too slow.

[GoofyBoy]

Will Novell or IBM support my RedHat with loads of modifications/customizations or Slackware?

Re:Too slow.

[penguinbrat]

And a situation arises where the endless amount of dependcies, that everyone bitches about on linux - are a GOOD thing!

If the same thing occured on a linux network - pushing out a patch for a system 2+ years newer onto an older system, there would be numerous, if not countless, dependency errors and likewise completely fail - that is the PATCHING/UPGRADE fail, not what unfortunately happened here...

Re:Too slow.

[Zocalo]

They could have called Novell or IBM.

Agreed, were this situation for real, the solution provider involved would certainly have a relationship with either directly with a major distro provider like Red Hat/Novell, and/or with a hardware provider with Linux expertise like HP/IBM. Extremely competent help would be just a phone call away, exactly as with the situation with EDS and Microsoft.

The problem is, that's not how Microsoft PR is likely to want to try and spin things, is it? I think it far more likely that they would paint a picture where the solution provider had simply downloaded a community driven distro from the 'Net and installed that. You and I know that there is still sufficient help available, even in that scenario, to get things fixed, but that's not the point. The big question is, will the PHBs that authorise these contracts see through the FUD, or will they just stick with the Devil they know?

Frankly, I'd be very suprised if EDS gets much more than a slap on the wrist for this. A few of their senior execs will get grilled by the House of Commons, then it's all sins forgiven and their stranglehold over UK Government IT contracts continues apace. Microsoft on the otherhand must be thinking that Christmas has arrived a month early this year; it's going to be one hell of a Thanksgiving Weekend in Redmond this year.

Re:Too slow.

[Random_Goblin]

I think if you had 80,000 seats they might think about it ;)

Re:Too slow.

[Skuld-Chan]

Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

This is actually a really good question. One thing I've found in Linux support is much of the software (the new software raid as an example) isn't clearly documented and when you do run into serious problems beyond a few simple things to try people generally seem clueless - even very experienced people. I blame a lot of this on constantly moving support targets (the day you document one issue, and its solution there have been 2-3 more kernels/patches with either a fix or a new issue). Plus there's no clear support message - who do I call - who can I blaim when everything blows up? Are the answers I get consistent (ie one support group will say/setup the system one way, and another a different way).

"the drives seem totally wiped - why that would happen after taking the raid offline and back online is beyond me" was quite literally the answer I got when I did have some serious problems with a 4 drive software raid setup.

Re:Too slow.

[drinkypoo]

Welcome to Slashdot, you must be new here :) If Microsoft protects you from shooting yourself in the foot Unix nerds will say it's not powerful enough, Unix would not only let you shoot yourself in the foot, but give you a choice of weapons and calibers. If Microsoft allows you to blow your foot off, Unix nerds will tell you that it should have only allowed you to shoot yourself in the foot with a crossbow without a target or hunting head, or a firearm of .38 caliber or less, non-magnum. While Microsoft certainly does not deserve to "win" the OS wars, even if their product had enough merit to defeat all other operating systems, it wouldn't have a chance around here.

Re:Too slow.

[iainl]

"let's face it, they couldn't have screwed the pooch nearly as well with a non-MS based system."

You've clearly not had much experience of EDS. I dread to think how much damage they could cause let loose on Linux boxes.

Re:Too slow.

[Bert64]

But the patch was designed to update the 2000 machines to xp.. Patches specifically designed for xp should just refuse to run on 2000 anyway.

Re:Too slow.

[yamla]

Actually, it shouldn't be at all hard to back out. All you need to do is recopy the correct kernel to the hard drive. You could do this easily enough with a restore CD, it should take no more than a couple of minutes per computer, almost all of which is bootup time.

No, you couldn't push a fix like this out to each computer because they are, as you say, hosed. Unless they boot over the network, of course, in which case you just fix a single central server (or 7, or however many).

Re:Too slow.

[carldot67]

I absolutely agree.
Im not a Windows expert but surely it was not beyond the wit of man for the "upgrade script" (or however it works) to begin:

if [ `uname` = "XP" ]
then
echo "I just saved your XP network"
echo "You can thank me later"
exit 1
fi

Re:Too slow.

[shippo]

Third-party update tool fails by ignoring the safeguards the OS manufacturer has put in place. I've lost count of the number of times I've seen that happen over the years.

I've vowed never to touch them, even since seeing the mess such a tool made back in 1992. That one was designed to speed up an OS upgrade by replacing a stack of floppies with just two floppies and a tape. A disaster, but luckily spotted prior to an upgrade of 20 fileservers. It was thankfully withdrawn from sale not long after.

Re:Too slow.

[ehb]

Why is being able to roll out an XP update to a windows 2000 PC a good thing? Microsoft should have made sure that was never possible! Imvho the flaw is also in Microsofts update mechanism.

Re:Too slow.

[mpe]

Really, I'd expect Microsoft to have designed the two very different operating systems to NOT take each other's patches. It couldn't have been that hard to do, just toss some identifier in there somewhere in the file and if XP sees a win2k id or vice versa, refude to install the update.

With many patches you can download from Microsoft's websites this is exactly what happens. Maybe SUS was developed by a different team/contractor. Microsoft had better fix this before the "black hats" work out how to make use of this "feature".

Re:Too slow.

[LnxAddct]

Point 1: An update should check the systems's versioning info against it's own before being applied...this is a basic principal.
Point 2: EDS would have just called Red Hat or Novell or whoever.
Point 3: If they were running linux, they could have been booting off the network from one or more central locations so that a) They are definitly all running the same versions of software, and b) Updates are only applied to a few machines (rather than 60,000) and fixing and problems only requires focusing on a few manageable machines.
Regards,
Steve

Re:Too slow.

[beowulfcluster]

Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

They could have called Slashdot and gotten help from a thousand people for the low low price of getting ridiculed by geeks with superiority complex for the duration of the project. *yes*

Re:Too slow.

[macemoneta]

if you overwrite your Linux kernel with a botched version, your system's hosed

Actually, that depends on the distribution. For example, when I "yum update" the kernels on my Fedora Core 2/3 machines, the grub configuration does NOT mark the new kernel as the default. When the machine is rebooted, if the new kernel doesn't work, just reboot again and the old kernel is used, automatically. In the case of these 60,000 PCs, when they malfunctioned the employees could have just rebooted (which they probably tried anyway), and they would be back in service. It never would have made the news.

Once you are satisfied that the new kernel functions as it should, you can just update the default kernel in the grub configuration. And, to answer your next question, this can be done via the GUI by using the "Applications->System Settings->Bootloader" menu entry. Of course, it can be scripted and done remotely as well (it's how I upgrade my machines).

That's one of the reasons there are many different Linux distributions; different people/organizations consider different things important or critical. Pick the distribution that does what you want.

Re:Too slow.

[Anonymous Coward]

Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

Microsoft products are "closed". That means some technical expertise is only available only through Microsoft. So-called "Trade Secrets", a form of intellectual property, prohibits EDS from learning everything it needs to know to diagnose a problem with Microsoft Windows.

On the flip-side, Linux is "open". There are no trade secrets when it comes to Linux, and therefore many people and organizations have the deep expertise necessary to address even the most difficult of issues.

Evidently, EDS had to call Microsoft because it wasn't legally able to learn more about the products they were using (via reverse engineering, etc). If EDS used Linux, they likely could have fixed the problem themselves. And if not, they could have gone to one of a hundred vendors to help them solve the problem quickly and effectively.

Re:Too slow.

[Anonymous Coward]

> Of course, it can be scripted and done remotely as well

And when that's done incorrectly, you have 60K dead machines that require a technician visit. Sorry I don't see a bit of difference here.

Re:Too slow.

[Apathetic1]

I don't disagree that Linux might have been a better choice for a government agency but what really would have saved them here was telling EDS where to shove their Junk Bond status and hiring some competent technicians.

The process you've described does not involve overwriting your old kernel. There's the right way to do things and there's the wrong way. Doing things the wrong way on any OS will screw you over.

Re:Too slow.

[Apathetic1]

In any case, you'd still have to walk around and fix 60000 computers. I'd hate to be the co-op student on that site.

Re:Too slow.

[killjoe]

I was once upgrading a debian system when I got a message saying that the upgrade I was attempting was probably going to break my system. It then said "Are you sure you want to do this?". I of course answered yes. The next screen said somthing like.. This is a really bad idea and I am not going to do it unless you type in the phrase "I know this is a really bad idea but I am going to do it anyway". So being the stubborn sort that I am I typed in that phrase and of course it broke the system.

The point is that the author of the update knew my system was going to break and made me type in that phrase before I installed. Why couldn't the author of the XP patch check to see if it was trying to be installed on a XP machine in the first place?

Re:Too slow.

[Apathetic1]

The patch itself won't install on a non-XP system unless you use some sort of third-party automated patch tool, e.g. Tivoli. The author of the patch installer script should have written a check that it was the correct operating system. Ideally, since the patch was to be deployed to seven test machines, the script should have checked that the node name was one of the seven correct node names. Somebody at EDS overrode the builtin failsafes and the shit hit the proverbial fan.

Re:Too slow.

[man_of_mr_e]

EDS wasn't using SUS, they were using a third party product called Radia, which bypasses the standard update protocols and just copies files.

Re:Too slow.

[man_of_mr_e]

The author of the XP patch script, in this case, was EDS themselves. They broke it, they screwed up.

Re:Too slow.

[Tony-A]

Microsoft can't help it if you force an XP update onto a Win2K machine using an automated tool. Trying to manually install a patch onto the wrong operating system WILL fail as it should.

Seems like Microsoft and/or the automated tool makers have something very backwards. If trying to manually install a patch will fail, anything automated should certainly fail.
Most likely both at fault.
Microsoft for not clearly labeling and identifying the patches.
The automated tool for subverting the sanity logic that would be used if the patch were manually installed.

if you overwrite your Linux kernel with a botched version, your system's hosed. If you didn't keep a backup, it's damned near impossible to back it out.
Rubbish. Boot from the install disks. Aways in, Alt-F2, look around a bit and put the original kernel on top op the botched one. Newbie-level skill are adequate.

Nobody can protect an incompetent admin from him / herself.
True, but there seem to be plenty of ways to make competent admins behave incompetently.

Re:Too slow.

[macemoneta]

I don't disagree that Linux might have been a better choice for a government agency but what really would have saved them here was telling EDS where to shove their Junk Bond status and hiring some competent technicians.

The process you've described does not involve overwriting your old kernel. There's the right way to do things and there's the wrong way. Doing things the wrong way on any OS will screw you over.

Well it's been a while since I used Win2K/WinXP, but as far as I remember, there was no way to switch between kernels. The architected limitations of the OS they chose forced a situation where an error could lead to a fatal situation.

I agree that EDS did not use due diligence in planning their upgrade considering the risk with Windows' maintenance limitations in mind.

I was simply commenting that for Linux, overwriting the kernel as opposed to installing an additional kernel is an aberation (there's no reason to do it, and most (all?) package managers will prevent it). For Windows, it's your only choice.

Re:Too slow.

[Apathetic1]

From what I read, there was never any intent to upgrade from Win2K to WinXP - there was an accident with an XP patch escaping the test environment and being rolled out on the production network.

Re:Too slow.

[Joey Vegetables]

"We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?"

They could have called me, or any other competent Linux admin, who would have built, in the space of about an hour, a customized Knoppix-like live CD which upon boot would fix the boot sector, kernel, glibc, or whatever had gotten b0rked.

Of course it wouldn't have been likely to happen in the first place, even with Windows, if you had competent admins.

The point is though that with free software you don't NEED the original vendor; ANY competent admin or developer can help solve problems.

Take back Bill Gate's knighhood

[Anonymous Coward]

Such wonderful software

I think they should take back Bills knighthood

I wouldn't want to be in their shoes now

[Lispy]

But still I have to say it: "HAHA!"

EDS again

[Hieronymus Howard]

Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?

Re:EDS again

[I confirm I'm not a]

Yet the government keep awarding them [EDS] contracts. Why?

I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky - you could spend millions only to not get the tender, and IBM felt that the chance of getting the tender awarded to IBM was too small. So... I'd suggest either it's too costly to play so players are dropping out (the reasonably answer), or someone in government really loves EDS, and IBM know it (the tinfoil hat answer).

Living in the UK, I'm minded to go for option 2.

Re:EDS again

[Leonig Mig]

being a british taxapyer I sent them a message to cease and desist:

It seems your firm is costing the British tax payer enormous quantities of cash, predominently through incompetence. Please take any drastic steps necessary to prevent any further IT disasters and consider if your firm deserves the billions of consulting dollars it has already banked.

Yours,

Jim.

anyone else can too:

https://www.eds.com/ssl/forms/contact/index.aspx/

Re:EDS again

[supersnail]

Because Accenture is the other choice!

This sort of cockup would have been impossable with the ex Arther Anderson crowd. They would still be struggling to get the shrink wrap off the CDs without wrinkling thier suits.

Seriously the problem is government procurement procedures. The contract goes to the lowest bidder and a record of past f****ups is not taken into account.

Re:EDS again

[justanyone]

EDS is one of very few companies that will accept government contracts. US Gov'mt accounting requirements are onerous (hard to comply with) by any standard, so in order to compete for the contract, you have to have a huge team of accountants that know how to produce the kind of records and reports that the Government accounting office(s) expect.

There is a huge hue and cry (outrageous exclamation of disgust and anger) over mismanagment and eggregious spending in government contracts. Having worked in the sector, I'm somewhat familiar. The contractor I worked for made sure there was no waste, fraud, or abuse. However, it spent 10 times as much as the job required, just to do this. The obvious choice for our firm was it would have been far cheaper to run things by GAAP (Generally Accepted Accounting Practices)(the private sector accounting standards), and have both a nice large internal audit division and "internal affairs" watchdog enforcement. Alas, most governments are not run this way, and if they are, they devolve into the current format due to political expediency.

I have friends that work for EDS and they comment on the kinds of hoops they have to jump through just to do simple stuff. They've built up a rather large experience pool in doing this hoop-jumping, so they can do contracts cheaper than some other companies.

EDS also tends to run things according to CMM levels whenever they're developing things, so at least if there's a mess-up (as there obviously was here), there will be some kind of follow-through to improve the process of doing this kind of work. EDS's management doesn't want the black eye any more than the government or Microsoft do, but they'll spend the money to make sure it doesn't happen the same way again. There is, after all, no way to prevent all errors, but I give them credit for trying most of the time.

Re:EDS again

[Anonymous Coward]

Because, like an old Dilbert strip, it's all down to experience. The government puts the jobs out to tender, they get the tenders in and only EDS have any experience of large goverment IT projects and therefore are the only qualifying tender.

Re:EDS again

[turgid]

I used to work for a UK-government-owned company. They're all the same. They are completely stupid and suicidal when it comes to dealing with contractors of any sort, but especially IT ones. It's a combination of laziness, stupidity, ignorance and herd mentality. It if ain't M$ they don't want to know about it, unless they can use it to negotiate lower prices from M$.

Re:EDS again

[t_allardyce]

Brilliant! hopefully we can sue these people and recover some of the money they wasted then put it into a project that actually has a clue?

Re:EDS again

[phaln]

Two words: Lowest Bidder.

Re:EDS again

[Ibix]

Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?

I suspect that it's something along the lines of "better the devil you know". Also, ObBlackadder, they've screwed up in exactly the same way seventeen times before, they can't possibly be stupid enough to make the same mistake again. The words "old boy network" also spring to mind, although that's unsupported cynicism.

I

Re:EDS again

[mikerich]

Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?

If you listen to the companies when they're trying to worm their way out of responsibility, it's because only a very few big companies have the resources to develop, test and run these enormous computer systems.

If you think about it for more than a couple of seconds, their track record seems to show that they don't have the resources to develop, test and run these enormous computer systems. So it must be down to judicious lobbying.

Still here's hoping that the levels of expertise they've shown in a kaleidoscopic range of networking, computer security and database disasters - combined with the complete crapshoot that is biometrics will make for one hugely entertaining ID card cockup.

Best wishes,
Mike.

Re:EDS again

[Anonymous Coward]

It's because the government persists with the 'preferred bidder' system.

From this weeks Private Eye:-

"The government seeks outline bids with no detailed specifications for most PFI or PPP projects and then chooses its "preferred bidder". It is only then that the detailed contract is worked out. But with a "preferred bidder" identified most of the other contractors decide its not worth wasting any more money and drop out. The preferred bidder can then hold a gun to the government's head ... so that by the time the contract is finalised, the cost is often considerably higher than first envisaged... As the EU recognised some time ago, this is because there is no longer any effective competition at the crucial stage of contract negotiation. Thanks to an EU directive which will come into force next year, the practice of appointing a preferred bidder will be outlawed as anti-competitive.

(Subscription to Private Eye is only £21)

The most disturbing thing about EDS is that if David Blunkett ever gets ID cards through parliament EDS will be one of the front runners to run the scheme. Be afraid, be very afraid.

Re:EDS again

[Akhen3sir]

Nah - SchlumbergerSema.

Re:EDS again

[Anonymous Coward]

You also forget that public sector contracts even when run are very difficult to manage. Public sector IT project governance is very different to the private sector governance. Far more paperwork, far more gates to jump through.

Even when a contract is won, there is still significant risk of it going wrong as the administration makes it very difficult to fullfil.

Also the publicity exposure risk is far, far higher in the public sector than private. Take for instance the Child Support Agency problems. When that goes wrong the IT problems are made public on a world wide basis. Who wants to take those sort of risks on board?

Re:EDS again

[SonicBurst]

I don't know anything about EDS, but I do know this: They could have done 100,000 projects/upgrades flawlessly and you'd never know it, but let them screw up once (or however many times this company has) and you'll never hear the end of it.

Please realize that I'm not defending them. I'm just pointing out that, as someone who works in IT, management never sees it when things go flawlessly, but they will not hesitate to throw your ass to the wolves should something go wrong.

Re:EDS again

[WIAKywbfatw]

Accenture is not the ex-Arthur Andersen crowd, it's the ex-ex-Arthur Andersen crowd: Accenture is what Andersen Consulting changed its name to (at the start of 2000, if I remember correctly) to differentiate itself from Arthur Andersen after it broke away from it.

Good thing too for those working at Accenture, given what lay ahead with Arthur Andersen's dealings with Enron. If the company had still been called Andersen Consulting when the shit hit that fan then it probably would have been dragged down too.

I can't speak from years of experience but, from the brief exposure that I had to them, Accenture's people seemed competent enough to me.

Re:EDS again

[Cally]

The excellent satire / investigative journalism magazine, Private Eye, has been digging up stuff related to EDS and government contracts for many years now - before the Blair government IIRC. Google doesn't find anything on their website, though, last time I looked it just had a few cartoons up rather than any article archives, which is a shame... this is the only reference I've turned up. I recommend the paper version of Private Eye, I've been reading it since 1983 and it's never less than amusing, sometimes 'piss yourself in public' funny, and the investigative journalism has turned up a lot of stories before anyone else was looking at them.

Re:EDS again

[mikechant]

Get with the program. SchlumbergerSema no longer exists. This January most of SchlumbergerSema was sold to Atos Origin and the Sema name dissappeared.
Schlumberger has gone back to being mainly an oil services company, and Atos Origin has moved closer in size to the big boys like EDS.
The good thing about Atos is the fish logo.

Re:EDS again

[DrPizza]

IBM do tender for UK government contracts.

Re:EDS again

[keesh]

No they don't.

Re:EDS again

[Anonymous Coward]

all these IT consultancy firms are about as competant as Bush is intelligent. They care too much about showing off their compliance to bullshit business practices (iso9001 etc.) instead of caring about doing the job well and for the best possible price for their customer.

Re:EDS again

[Anonymous Coward]

You seem to like using big vocabulary, so I'll just mention that it's spelled "egregious". Also, "mismanagement", though that might just be a typo.

Re:EDS again

[Builder]

That's flat out unfair - Siemens fucked up the passport office upgrade all by themselves and you do them a disservice by lumping that fuckup in with all of EDS'

Re:EDS again

[mpcooke3]

I love the goverment tendering process it fills me with much amusement.

The idea that you pay one company to come up with a box of requirements then send it out to tender, and get several boxes back from a few large companies like EDS. Then these get send off to the company contracted to deal with the subcontracting/tendering process. A haggling process commences between bunches of lawyers on both sides resulting in usually only one or two possibilities the cheapest one is then selected and fucks it up. Now a days most reputable companies don't even tender a bid cos of the cost and the fact they know it will be wasted money cos some company renowned for their failures like EDS will just undercut them.

My particular favourite was penalty clauses against downtime for an NHS system were introduced due to the fact that the system was so critical. But the company involved rather than implementing a backup system decided it would be more cost effective to ensure against a system failure.

Perhaps one day the out-sourcing-sub-contracting-legal-wrangling craziness will stop.

Re:EDS again

[Anonymous Coward]

It's the illusion of competency that's visible at Accenture.

Re:EDS again

[upsidedown_duck]

I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky

So the conclusion is that EDS is a complete whore. The worst kind of contractor: the guy that says "I can do that" even before the customer finishes a sentence. He ends up working more than you, makes more money than you, is able to sweet talk himself into even more work and money, and the customer doesn't realize the con-job until it's way too late...so they bring contractor-whore man in to do just a little bit more...ugh, what an ugly cycle, not unlike a pimp and his desperate and impoverished bitches.

Re:EDS again

[julesh]

Seriously the problem is government procurement procedures. The contract goes to the lowest bidder and a record of past f****ups is not taken into account.

That's actually not true. The criteria that are _supposed_ to be used to judge a tender are "best value" (which is not the same as lowest bidder -- other potential costs and benefits than the immediate bid value are supposed to be taken into account) and "proven ability to provide similar solutions". Perhaps it gets ignored, but that's the fault of the procurement departments, not the procedure they're supposed to follow.

Re:EDS again

[DrPizza]

They most certainly do.

*sigh*

[turgid]

The thing is, this sort of thing is expected and accepted by the UK public sector. They'll just find a scapegoat and keep on buying Microsoft. The sad thing is, that's my tax money.

Re:*sigh*

[Anonymous Coward]

Here, in Canada, we don't have that kind of problems, as they spend the money on importing cocaine, censoring radio stations, funding lefty newspaper that help them spread the haterd of jews and americans, spreading cash around healthcare but not making the personel wash their hands (resulting in hundreds of deaths from infection), hiring lawyers to keep the mouths of people who had infected blood transfered to them shut and helping pedophiles reintegrate society instead of buying Microsoft stuff

Re:*sigh*

[OwlWhacker]

The thing is, this sort of thing is expected and accepted by the UK public sector.

Just like the virus/security problem is expected and accepted.

Well, everybody trusts Microsoft, what with being a big company, and having all that cash behind it.

that's my tax money

Mine too.

But am I bitter?

HELL YEAH!

Re:*sigh*

[phaln]

It's just the same on the other side of the pond, I assure you that.

Come on now

[Mick Ohrberg]

Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.

Re:Come on now

[Anonymous Coward]

How come an XP update can be pushed into a win2k box in the first place? Shouldnt there be some basic checks in place?

Re:Come on now

[Anonymous Coward]

Incompetent Microsoft programmers can turn any service pack to a catastrophic failure. Don't blame the government for blue screens of death unless there are irrefutable proof that Microsoft did everything by the numbers.

And repeat after me, it is "in-com-pe-tent".

Re:Come on now

[_bug_]

Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.

The articles are a bit skimpy on details, so don't blame, or clear of blame, anybody.

Although I'd like to know why the XP upgrades went, apparently, half way? If the upgrade starts, why not either let it complete even after figuring out your mistake and at least keep the systems in a bootable state (admin at fault) or why isn't there a way to back out of the upgrade if you realize you don't want it to be performed (microsoft's fault)?

Re:Come on now

[KarmaMB84]

EDS pushed an update meant for only 7 machines to the entire network. I'd say they were the ones that fked up.

Re:Come on now

[Anonymous Coward]

If this happened at my job the SA(s) responsible would have been fired on the spot.

What should be done first...

[bogaboga]

...is to fire the IT manager, or whoever sanctioned this method of implementation. How do you fall into such a trap?

Next, they should setup a similar Linux environment and see how a similar upgrade would have gone. This should be done before the Linux zealots declare that theirs would have been any better.

In the meantime, let M$ and its cronnies be prepared for the wrath of Linux promoters.

Cb..

Re:What should be done first...

[DoctorMO]

Don't worry knowing Linux and the IT of the public sector they'd have chmoded root to 777 long before any upgrade.

Re:What should be done first...

[Asic Eng]

I think setting up a similar Linux environment just for a comparison would be prohibitively expensive. It will be very interesting though to see how the Linux project in Munich will go (the city administration is switching almost completely to Linux). If it works there (and saves money) then it will probably be useful for other administrations, too.

Re:What should be done first...

[supalexie]

If you had a *nix environment, the sysadmin would know wtf he is doing and he would have made he had a rocking testplan.
Once again this is a case of bad engineering. I wonder why that it always happens to large networks running MS products or so called It specialist experts?

EDS now stands for...

[BlackHawk-666]

Every Desktop Shutdown.

Re:EDS now stands for...

[Anonymous Coward]

We prefer to refer to them as "End of Dependable Service", but yours works too :)

We're stuck with the stupid bastards (I work for GM) and they fuck up on a regular basis and on an impressive scale.

We had our own IT department that worked extremely well (for the most part), when ISS (GM's in-house IT group -idiots-) came in and dissolved the group and put EDS in its place (while offering the ex-GMer's a position in EDS to try to keep the experience there).

Now we can't get upgrades (which it seems can be A Good Thing), our old software doesn't work, and support is nearly non-exsistent.

Ahhh, I feel better now.

Re:EDS now stands for...

[BJH]

"Extraneous Disaster Salesmen"?

Re:EDS now stands for...

[neko9]

Every Desktop Screw-up

Re:EDS now stands for...

[Sneakabout]

All Your Update Are Belong To Us!

Shutdown Every Desktop!

umm..

[REBloomfield]

I've been reading about this all day, and not once have I found a concrete description of what was actually being upgraded. Where exactly does it say that it was an XP upgrade, or are /. just jumping on the M$ bashing wagon....

If this was a Linux/Oracle/$flavourOfTheMonth upgrade, would you be just as scathing?

Re:umm..

[gbjbaanb]

I was surprised when this appeared on /. as nowhere I've read has explained what the issue was.

If it was just an upgrade from W2k to XP, they should spend some time rolling back the PCs. No big deal, might take some time, but still - no big deal. I imagine that they have images (or some procedure) prepared for new employees anyway.

The fact that the problem is continuing suggests a major cock up - someone deleted the database kind of thing, and put the backup tapes through the shredder (assuming the scheduled backups were running).

And to answer your other question - no, if it was anyone but MS it wouldn't be so scathing, but occasionally Sun and Oracle are bashed on /. (never Linux though).

Re:umm..

[REBloomfield]

I found it on the register, it's one of the linked articles - and before anyone claims i didn't RTFA, this has been updated since I read it this morning. The Reg still gives two possible explanations from unknown source, and while i'm not trying to move blame away from anyone, it would be nice if the slashbots didn't just start to feverishly type about how the admin should be sacked for not sandboxing etc. because they habe no idea of what went on nor the procedures in place. No one critisises their cockups, because they aren't as public, but for god sake, people make mistakes, and these things happen with computers.

Re:umm..

[Anonymous Coward]

Register article in original story describes te XP upgrade

Re:umm..

[Anonymous Coward]

If it affected 60,000 machines, then... YES.

somehow I doubt that anyone doing either of those on that sort of scale would have those problems though. As hideous as the Ora installer is its not as bad as that.

Re:umm..

[REBloomfield]

yeah, i saw that, but they're the only people that reference it, and it's still not quite was /. reported either was it?

Re:umm..

[Zocalo]

Reading between the lines I think EDS was trying to install a Windows XP patch onto a small testbed of PCs actually running XP (which is hardly what was implied by the Slashdot story). Instead, they managed to deploy the patch the the general pool of 60,000 PCs still running Windows 2K, essentially trashing the install and preventing the systems from booting. It could have been as simple as someone at EDS choosing the wrong deployment target from their network management tool (Tivoli, IIRC).

The question about all of this that I am left with is, how did the patch even install? Microsoft has had sanity checking on their patches for ages, checking not only the Windows version, but even service pack levels and any other prerequisites. Ever tried installing a patch intended for IE6-SP1 over plain IE6 for example? I'm assuming that this is some custom patch rolled by EDS, rather than an official Microsoft one downloadable by all and sundry. Still, the story appears to have made it onto UK prime time news, so no doubt more details will emerge...

Re:umm..

[JohnFluxx]

"If it was just an upgrade from W2k to XP, they should spend some time rolling back the PCs."

Not if the machines are stuck unbootable.

Re:umm..

[NFNNMIDATA]

They probably used Radia and it probably just forces the upgrade, not using MS' installer but by putting the files on the machine and setting registry flags for the ones that need to be updated on reboot.

UK agency made the correct choice!

[glomph]

Hey, they may have shite for brains, but the money was worth it! No trouble with the dreaded 228 patents that Linux supposedly infringes!

Re:UK agency made the correct choice!

[shadowsurfr1]

Does anybody even have a list of those 228 patents Linux is apparently infringing?

Re:UK agency made the correct choice!

[ajs318]

Does anybody even have a list of those 228 patents Linux is apparently infringing?

1. No, because that information is protected Intellectual Property.
2. Software patents are illegal in the UK anyway

Re:UK agency made the correct choice!

[jimicus]

Software patents are illegal in the UK anyway

Unfortunately, this hasn't stopped the UK patent office granting them.

http://swpat.ffii.org/players/uk/index.en.html

Testimony to power of insufficient testing

[justanyone]

Seems like the Ministry should have tested the rollout in a sandbox that didn't leak.

Of course, MS could have tested their product better.

Of course, this could all be the work of the NSA working with the little green men (Kodos and Frodos) and the Legion of Inappropriately Named White Guys Named Bruce All With The First Name Of Bruce.

Nooo!

[Turn-X Alphonse]

It's like a thousand solitaire players suddenly cried out in frustration and then silence...

Whose fault is it?

[rveno1]

I wouldnt rush to blame Microsoft so quickly, I find fault with the person who attempted to do the Update LIVE on the network.

(dont all big failures occur on monday?)

Re:Whose fault is it?

[INeededALogin]

Please... I would fault the IT Staff to a limit. No Operating System should kernel fault and entirely lock the system up. It is just poor design.

Similar things happen on Red Hat... You get a power outage, remote machine comes back up, gets an error doing an fsck and prompts the user at the terminal for the next action. Seriously, You have no idea how much it bothers me having to drive down to the warehouse because a Red Hat machine won't boot up to a remote accessible level(and power outages can happen in any environment)

Back to Microsoft, A non-working Operating System is useless, but I expect a machine to have problems, it is the options that the OS leaves you in the event of failure, that makes it good. Leaving me having to visit each machine individual is unacceptable.

Re:Whose fault is it?

[Anonymous Coward]

Write a better startup script.

Re:Whose fault is it?

[loconet]

Blame the person who attempted the live update? That's like placing the blame on the driver of a car that crashed while speeding and couldn't stop on time because of a faulty break system in the car.

It is partly the careless driver's fault but the bigger fault is that of the car's manufacturer for selling a faulty break system that should have worked regardless!

If this was in the private sector...

[The Dodger]

..someone (probably several people, in fact) would be getting fired and senior Microsoft staff would be getting carpeted.

But given that i's the public sector, I bet nothing happens...

D.
is for DOH!

Re:If this was in the private sector...

[Schreckgestalt]

someone (probably several people, in fact) would be getting fired

Believe me, they will. I know this sort of thing (not from myself actually doing it), and whether or not they are in the 'public sector', someone will have to leave.

and senior Microsoft staff would be getting carpeted

Um? What world do you live in? If you're really drunk and drive around and happen to kill 10 people, would anyone at Daimler Chrysler get carpeted?

Re:If this was in the private sector...

[davesag]

dude have you never heard of "The Peter Principle". That bloke is probably being promoted right now.

Re:If this was in the private sector...

[The Dodger]

Um? What world do you live in? If you're really drunk and drive around and happen to kill 10 people, would anyone at Daimler Chrysler get carpeted?

You obviously underestimate:

1. my capacity for getting pissed off when things go wrong,
2. my readiness to use any excuse, however tenuous, to vent my frustration on individuals who have come to my attention in ways which haven't made me happy,
3. the amount I worry about whether the person I'm ranting at bears any responsibility whatsoever for what I'm ranting about, and
4. my readiness to issue instructions along the lines of "Find someone who's surplus to requirements and fire them!"

Gotta keep the troops motivated, y'know?

D.
..is for "Don't..*smack*..Do..*smack*..That..*smack*..Again !*smack*"

Re:If this was in the private sector...

[FurryFeet]

dude have you never heard of "The Peter Principle"

Obviously, you have heard about it. And obviously you don't understand it.
Follow your own link and read about the Peter Principle. At most, it would mean that the bloke remains in charge, but certainly not promoted.

why did they decide to downgrade to XP?

[Anonymous Coward]

almost everyone who ran 2000 knows that XP
is less stable than it.

i need to laugh now. hahahahahahah

This has got to be the funniest story in weeks

[empaler]

I almost had to go buy a new keyboard...

*chuckles*

(Don't get me wrong, I'm not a MS-flamer... It's just fucking funny...)

Re:This has got to be the funniest story in weeks

[sbennett]

It's funny as long as you're not paying for it. A lot of us are.

Re:This has got to be the funniest story in weeks

[pew]

It is. You're not alone.

Re:This has got to be the funniest story in weeks

[Anonymous Coward]

And ex-employees and subcontractors saw this years ago when we set them up the bomb

Free Tivoli updates for everyone!

What the heck did they do?

[eltoyoboyo]

From the Guardian article: "At this point there is no known solution or ETA"

I RTFA and all I see is a money discussion, not a technical discussion. I would speculate that an SMS or Zenworks push or somthing similar which was supposed to be restriced to the 7 PC's went almost everywhere. It might be a fair bet that the remaining 20,000 might have been upgraded too if those people had been at work and turned on their computers. IT Computer management tools give the department much power, which could do plenty of damage in the wrong hands.

Re:What the heck did they do?

[PatrickThomson]

I used to work in one of the departments in the summer, and sometimes I'd come in to work in the morning and find my machine on, when I coulda sworn I'd turned it off overnight.
Eventually I realised that all the machines in the office woke up by themselves and applied whatever patches were being rolled out. To be honest, I'm surprised that any machines at all were left standing.

Re:What the heck did they do?

[Anonymous Coward]

I would speculate that an SMS or Zenworks push or somthing similar which was supposed to be restriced to the 7 PC's went almost everywhere.

I think this hits the nail. They wanted to test the update, but accidentally applied it to all the machines on their network instead. The update caused the machines to lock up so they couldn't roll-back the old system automatically.

It is not a question of Microsoft or having backups. Its a question of using zero-effort networking tools to destroy the network with zero-effort. There is a lesson here.

The real question is how did they recover from this on 60,000 machines? If it involved booting from CD then I'm impressed they got things working again so quickly.

Re:What the heck did they do?

[duffbeer703]

Some IBM desktops (probably HP & Dells too) can be booted against a network device remotely using wake on lan.

They probably had the capability to do that with newer machines... but the older machines need to be touched by hand.

Couldn't they just upgrade _ONE_ of their comps?

[Spy der Mann]

And then test it for a week to see if there weren't problems?

Gee.

For those not reading the article...

[JKR]

Apparently EDS attempted to do a test upgrade on a small network of 7 machines, but accidentally deployed it to all 80,000 machines instead. It's not clear that they'd tried it on any target machine, so it's entirely possible that EDS is to blame here...

Jon.

Re:For those not reading the article...

[Otter]

It's not clear that they'd tried it on any target machine, so it's entirely possible that EDS is to blame here.

If I understand the story correctly, it's roughly analogous to forcibly pushing a Mandrake Cooker update to XFree onto a network of mostly Red Hat machines and then having everything break. EDS certainly seems to be to blame and this has essentially nothing to do with Microsoft or Windows.

Re:For those not reading the article...

[stratjakt]

Shut up, it's Microsofts fault, it's always Microsoft's fault.

If you fuck up upgrading your system, or format your harddrive, or delete that SYSTEM32 folder that wastes so much space on your hard drive, it's Microsoft's fault, not yours.

Here's an article idea. "emerge -u samba" broke every machine on my network when it automatically upgraded from 2.x to 3, and now it cant parse the smb.conf files. The same thing happened last time I tried to upgrade to KDE, etc.

Oh, wait, that's my fault for not planning ahead, right. Because in no way could OSS developers be held to task for needlessly changing the format of configuration files and providing no migration tools whatsoever.

Re:For those not reading the article...

[DeathByDuke]

if it went to all 80,000 i assume the surviving 20,000 are now XP boxes?

Re:For those not reading the article...

[markus_baertschi]

Microsofts problem ist that they provide an operating system with no secure way to apply patches securely on a large scale.

If you apply whatever patch on a usable os (AIX, Solaris, Linux) the package manager will attempt to apply and then back ou saying there is a prerequisite missing. The result of the human error would have been 70'000 error messages instead of 70'000 broken systems.

Markus

Re:For those not reading the article...

[FuzzieNorn]

It's your problem for using fucking Gentoo and not a distribution which actually manages upgrades properly, to be quite honest. If you're using Gentoo, you've made the decision to make such things your own responsibility.

Re:For those not reading the article...

[Roguelazer]

Perhaps you should read the ewarns that beep loudly when played?

Re:For those not reading the article...

[Qrlx]

Apparently EDS attempted to do a test upgrade on a small network of 7 machines, but accidentally deployed it to all 80,000 machines instead.

It should be pointed out that this mistake is very, very easy to do. Why? by default when you create new GPO policies (such as one that would start the installation of XP) they apply to the entire domain by default. You have to create the secuirty container first, such as "Test XP Upgrade", and as soon as you start creating your GPO entity, point it to the security object. (Sorry if my terminology is a bit rough, it's been 2 years since I was bitten by this bug, which is of course a feature and not a bug at all.)

It's a boneheaded design by Microsoft to have new GPO entries apply by default to the entire domain. That idiocy was compounded by the ignorance of EDS. (And of course I could be completely wrong, since I'm Assuming that they were using Active Directory to roll out this upgrade, which seems at least plausible considering that there's 80,000 machines.)

Finally, let me say, my God it's nice to be working in Tech Support and not have to admin Windows networks any more. "I'm so awfully happy to be a Beta."

Re:For those not reading the article...

[iminplaya]

Probably the only thing that was deployed on to those 80,000 machines was nCASE.

Re:For those not reading the article...

[JKR]

Microsofts problem ist that they provide an operating system with no secure way to apply patches securely on a large scale.

And what do you think SUS Server (now called Windows Update Server, apparently) is for then? More blinkenlights for the collection?

Jon.

Re:For those not reading the article...

[markus_baertschi]

The Windows Update Server is perfect for the SOHO environment where a large number of individuals care for their own PC and install directly from the server. The *server* makes sure that the wrong patch don't get to the PC.

However, there is no such tool for a large IT organisation who wants to control what patches are installed where. This requires a distribution mechanism (there are plenty available on the market) and an tool to apply the patch. This is where Microsoft ought to provide a usable mechanism, designed to do pre- and co-requisite checking and to work unattended.

The Windows OS doesn't even have a decent software inventory which allows to list the level of the individual components. Windows and its updates is still managed at an individual file level, there is no notion of package grouping related items together.

Here an example (from AIX, have the box handy): First a list of the state and version of the networking components, second the history of the tcp server package. You can get similar listings on Solaris, HP-UX, Digital OS and Linux.

Show me something remotely similar on Windows.

maba:/home/maba > lslpp -L "bos.net.*"
Fileset Level State Description
bos.net.nfs.client 4.3.3.28 C Network File System Client
bos.net.nfs.server 4.3.3.10 C Network File System Server
bos.net.ppp 4.3.3.25 C Async Point to Point Protocol
bos.net.tcp.adt 4.3.3.26 C TCP/IP Application Toolkit
bos.net.tcp.client 4.3.3.28 C TCP/IP Client Support
bos.net.tcp.server 4.3.3.27 C TCP/IP Server
bos.net.tcp.smit 4.3.3.25 C TCP/IP SMIT Support
bos.net.uucp 4.3.3.10 C Unix to Unix Copy Program

maba:/home/maba > lslpp -ah "bos.net.tcp.server"
Fileset Level Action Status Date Time
bos.net.tcp.server
4.3.3.0 COMMIT COMPLETE 03/25/01 13:55:11
4.3.3.0 APPLY COMPLETE 03/25/01 13:55:11
4.3.3.27 COMMIT COMPLETE 03/25/01 13:57:26
4.3.3.27 APPLY COMPLETE 03/25/01 13:56:16

Markus

Sorry for the broken formatting, the slashdot replacement for is broken

Go on, Slashdot, you know you want to.

[mr breakfast]

This makes me feel so much better about the working mistakes I have made. I would love to see an interview with whoever clicked on "OK" to trigger this one off...

Local Government

[Bloke in a box]

I work in a local government authority myself.

Although we have several xp boxes (mainly used by my development team (along with Windows 2k Pro ones)), there is no way this IT department is going to roll out XP across the entire authority (approximately 400 machines) until at least Mid quarter 2005, there are far far too many problems to even contemplate it.

Heck, half the staff haven't even figured out the difference between a wallpaper and screensaver yet, yet alone giving them more fancy gadgets.

Why were they networked?

[Turn-X Alphonse]

Why didn't they cut them off from the network for a few minutes to do the upgrade... it's common sense.

Re:Why were they networked?

[subterfuge]

if they were not connected to the network then they could not have demonstrated the new [and very expensive] remote installation software to the PHBs...

Re:Why were they networked?

[empaler]

It's common sense?

EDS managed upgrade--Altiris?

[willith]

The BBC article mentions that EDS is responsible for the ugprade. They're partnered with Altiris, so I'd be willing to bet that the upgrade was carried out using the Altiris Client Management Suite.

It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.

Re:EDS managed upgrade--Altiris?

[Apathetic1]

Admittedly I don't know about any other sites but they used Tivoli at the EDS site where I worked this summer. That may have been dictated by the client.

Re:EDS managed upgrade--Altiris?

[earthforce_1]

It sounds like this software is dangerous enough to be considered a potential terrorist cyber-weapon, given how effectively it was able to totally knock out a huge chunk of government infrastructure for a very long time.

Who was it who said computers allow you to make mistakes far faster than you could ever do it manually?

Re:EDS managed upgrade--Altiris?

[Anonymous Coward]

It was tivoli at every site I installed my share of those 80,000 PCs in 4 years ago.

Re:EDS managed upgrade--Altiris?

[Anonymous Coward]

Tools used to deploy software is Novadigm Radia. Tivoli is used for remote control to machines.
You -cannot- accidentally send software to all the machines on the network (by the way there are 160,000 plus, not 80,000. Article inaccuracies ..), but you can tell someone that knows what they are doing to send something out without realising it won't work.

And no, I will not be posting under my usual /. account :)

RTFA!

[DaHat]

From the article: Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess." EDS is also thought to be flying in fire brigades."

Brilliant work on the part of EDS, trying to patch the wrong systems, lord only knows what can happen then.

You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it? That seems to be the case here, EDS screwed up, and of course it's Microsoft's fault in the eyes of /.

Re:RTFA!

[REBloomfield]

what gets me is "another source says". el reg is the only place that has mentioned this fact so far.

Re:RTFA!

[Richard_at_work]

Thats precisely what I thought. It says it right there in the article, the wrong OS version files were rolled out to the systems, resulting in obvious failure of that system. It can happen to any OS, it was the fault of the System Administrators, not the OS vendor, imagine the chaos if the wrong version of libc was rolled out to a linux system! But then, Slashdot loves to slant any MS related news so it can take a swipe at it.

Re:RTFA!

[Anonymous Coward]

You can still blame microsoft.. if they were smart theyd make their patches capabale of determining if the OS they're installing on is actually compatible

GetVersionEx and OSVERSIONINFO exist for a reason.

Re:RTFA!

[Anonymous Coward]

Its the deployment tool (non-MS) that should have checked it.

Its the extra stuff that the deployment tool does that messed things up.

Re:RTFA!

[Tony Hoyle]

You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it?

Umm...yes.

If it is possible to do that then XPSP2 is fundamentally broken. It should check that it is at least on the right OS before upgrading.

Re:RTFA!

[Anonymous Coward]

imagine the chaos if the wrong version of libc was rolled out to a linux system!

One of two things would happen:

1. Nothing. Glibc since libc6 is forward and backward compatable.
2. You would have to reboot the machines in single-user mode and, using all those shiny static binaries in /sbin, you'd simply recover the correct Glibc.

I get your drift though. Just wanted to argue the details.

Re:RTFA!

[DaHat]

I didn't say you could use the installer to force it... I'm talking about an old school xcopy style deployment. You know, manually copying over lots and lots of files.

I could force the same sort of thing on a Linux system, upgrade a Red Hat 6.2 machine by simply copying over a # of files from a Fedora Core 3 system... Shame that Fedora Core 3 is fundamentally broken where it's files can be forced on a legacy system.

Yes, my argument is stupid, but so is yours.

Just about every operating system out there can be hosed by throwing the wrong files on it, don't try to blame Windows because in this case that's what happened.

Re:RTFA!

[subterfuge]

It is 100% the fault of the OS vendor - the patch should include a check to ensure it is not installing on the wrong OS. Patches for GAMES do this, patches for applications do this [at work two days ago I tried to apply a Lotus Notes 5.0.12 patch to a 5.0.9 client and was curteously informed that it was the incorrect version - not that diffucult to code, apparently...] so why can't a [no doubt] Critical patch for an operating system make this simple check?

Sure, some paper MCSE used the wrong target scope when applying the patch - he/she totally screwed the pooch. However, the patch installer should have detected that is was trying to 'upgrade' an incorrect OS and terminated on the other 59,993 attempts rather than just slamming itself in - that is NOT the admins fault.

ANY patch that allows itself to be installed on an alien OS is flawed - I don't care what your pet OS is. Lame code is lame code no matter what is tastes like.

Re:RTFA!

[DaHat]

glibc since version 6 may be compatible... but that's like saying everything since Windows 95 is plug and play... when you end up dealing with a system that doesn't fall into your filter... there is hell to pay and problems up the wazoo.

Saying that as of a given version, things work is nice for future systems, but does nothing about legacy systems.

Re:RTFA!

[CptKron]

It take note at "I gather that MS consultants have been flown in from the US to clear up the mess."

Sure, Bill Gates won't do YOUR tech support, but having the real pros called in if you've screwed up and you're a big enough customer is probably something very appealing to governments looking for an OS.

Re:RTFA!

[subterfuge]

it looks something like this [translate into your favorite language]: ...code... read registry for OS version and set = x Does version x fit into the predefined range that this patch was written for? if No then bail and give err code if yes then run installer ...more code... I see no evidence of Brain Surgery and/or Rocket Science here.

Re:RTFA!

[duffbeer703]

The Microsoft tools do detect whether proper versions of Windows are present.

What probably happened is that the genius who developed the upgrade package with Tivoli/Altris/Unicenter/etc did something before executing the Microsoft installer.

I've worked pretty extensively with Tivoli tools. If you allow someone who is dumb, incapable or too disgruntled to work with any management tool like that... you have a problem.

Re:RTFA!

[DaHat]

We need to get some terminology correct here... the 'patch' is nothing more than the changes made to the system, it is the 'installer' that applies the patch. Yes, the installer needs some logic to make sure it's not putting a patch where it doesn't belong... but you can bypass the installer quite easily (even under Linux) and take matters into your own hands when applying the patch.

Every single MS patch I've ever seen does verify that you are running it under the correct OS... this is not mean that all admins do it that way.

I fear that it is not uncommon, for an IT person even today to take a snapshot of a given system, install the patch and take a another snapshot and end up comparing the two for the differences... and then applying those differences to the target systems... this can be far easier for the end users who may have to deal with dozens of patches, instead they've got one quick big one (albeit it does take a while to build such a diffgram for the admin)

The patch is dumb, it has no clue as to where it is, it's up to the installer to verify everything is proper, and if you bypass the installer, I guess it's still the fault of the patch.

Tell ya what... I'm going to go grab some binaries off of an old Red Hat 6.2 machine and dump them onto your system alright? When it breaks, be sure to blame your system because it did not expel these foreign files right off the bat... that is if your system is still working.

Re:RTFA!

[DaHat]

Honestly, you are an idiot.

I never said anything about the installer to begin with, I said that again above, that I am not talking about the installer. Why don't you go read my other post to you regarding the difference between a patch and an installer and then maybe we will talk when you have a clue as to what you are talking about... let alone what I've already said and that you've conveniently ignored.

What's the line? Oh yes... "I reject your reality and substitute my own." Your motto perhaps?

Re:RTFA!

[Richard_at_work]

The vast majority of tools that do these rollouts dont roll out the patches as supplied by the vendor. The patches are applied to a machine in a known state, and then that machine is scanned by the tool to see whats changed. This changeset is whats rolled out. And yes, jsut tried it, XP SP2 does indeed refuse to install on a Windows2000 system.

Re:RTFA!

[ad0gg]

On slashdot everything is microsoft's fault, including human error.

Re:RTFA!

[Richard_at_work]

Whats more likely is that the update tool didnt distribute the vendor patch, but rather the changeset files from an applied patch. This would jsut replace the files on the disk.

Re:RTFA!

[Anonymous Coward]

On slashdot everything is microsoft's fault, including human error.

Don't forget global warming.

Re:RTFA!

[subterfuge]

"...Tell ya what..."

Your final point supports the argument that the problem lies with the vendor of the OS - whether we are talking about a dumb patch being installed by bypassing the installer OS checks or if we are discussing why the OS allows systems files to be blindly overidden in this manner the point is really the same.

The argument that other systems have this flaw does not make the point invalid - it only included them in the general blame. If the problem exists in all operatings systems it makes more sense to correct the problem in all of them rather than ignore the problem because others have it as well.

The fact that you could do as you suggest only tells me that the entire file management system and patch process needs to be overhauled on all operating systems.

my email to EDS

[Leonig Mig]

It seems your firm is costing the British tax payer enormous quantities of cash, predominently through incompetence. Please take any drastic steps necessary to prevent any further IT disasters and consider if your firm deserves the billions of consulting dollars it has already banked.

Yours,

Jim.

Not a nail for Microsoft.

[alistair]

"On another note, How did upgrading seven machines to XP BSOD 60000"

If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.

I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.

I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.

It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.

Re:Not a nail for Microsoft.

[Leonig Mig]

agreed this is more and EDS issue than a microsoft one.

Re:Not a nail for Microsoft.

[Apathetic1]

I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.

At the EDS site where I worked they use Tivoli to push out patches and applications.

Re:Not a nail for Microsoft.

[bentcd]

If anything, this is the world crying out "what were you _thinking_ having a 60,000 unit network all running the same system???" and perhaps the world will wake up and realize that it _might_ be a good idea to mix systems a bit so that whatever happens to one system, you still have some significant percentage of the network still running.
Doesn't anyone do risk analysis anymore?

Re:Not a nail for Microsoft.

[DrSkwid]

if it was Linux / Unix the sensible thing would be boot via PXE

rolling back would then be a relatively simple process

Re:Not a nail for Microsoft.

[MHobbit]

Sounds about right, alistair.

Re:Not a nail for Microsoft.

[ilyaa1]

OK, so let's see. Someone installs WinXP dll's on Win2K machines. Hmm, I wonder, how come those don't boot now?..

It's true that Microsoft's robustness is rather mirage-like, but there's a thing called human error, and that can bring down any system. All the software did was follow human instructions, after all: that's why we need IT people with brains to decide who is doing what.

However, PXE boot and a server with HDD images ready would've been helpful...

Re:Not a nail for Microsoft.

[Anonymous Coward]

This parent is totally right on.

If you are going to have an organization with more than 1,000 pcs that do not have to have more than 256 colors to word process, mainframe, and data input then they should all be thin client and server side either with Terminal Server or Citrix. That way you only have one machine (or rack of machines)) and one upgrade.

And yes you should very well damn mirror that machine on a running box before you upgrade so you can just switch the patch cables in case the upgrade blew up in your face. (of course one would then to apply the patch to the mirror first and not the live box)

But thats quite less hassle than trying to put to more than 1,000 boxes with each it's variance of hardware, software, and reg key malfuntions.

This won't work for companies who need client side heavy apps like Photoshop or CAD programs, but usually those are with companies that soly deal in those types of business and don't have more than 1,000 machines (and if they do those desktops tend to be cordoned off from the rest of the vanilla ones).

Re:Not a nail for Microsoft.

[archen]

it _might_ be a good idea to mix systems a bit so that whatever happens to one system, you still have some significant percentage of the network still running

Or at least segment your network. I think the loss of 'maintainability' from segmenting the network into 10000 PC clusters might be better than risking something like say, an update that goes wrong and f's up 60000 PC's.

Although it doesn't look as impressive for those full page Microsoft e-week ads like:

Mr. I fucked up 60000 PC's in a couple minues on accident

Re:Not a nail for Microsoft.

[Donny Smith]

> if it was Linux / Unix the sensible thing would be boot via PXE
> rolling back would then be a relatively simple process

Windows can boot from PXE just as good as any UNIX.

To boot 60K computers via PXE you'd need about 1000 boot servers (at least - if one boots 60 systems from each boot server which is probably too high a ratio - 20:1 sounds more reasonable unless you never shut down those 60K systems i.e. they reboot only ocasionally and at random intervals).

Also, deploying to 60K systems wasn't the idea anyway - someone chose "Deploy to All" instead of "Deploy to Selected Systems" which were only seven.

Re:Not a nail for Microsoft.

[wdd1040]

and this was Microsoft's fault how?

I mean, this is like trying to do a live push of Red Hat Enterprise 3 to Fedora Core 2. It just wouldn't work.

Did they honestly expect an upgrade like to work as planned? Wouldn't it have been cheaper to higher some $8 an hour techs and have a planned migration schedule? Seems like that would have been MUCH cheaper.

Re:Not a nail for Microsoft.

[rikkards]

And chances are that was how they got them back online. 60k workstations and a multitude of ghostgast sessions. Properly sysprepped generic workstation image, you could probably do a 1000 easily within an hour.

Re:Not a nail for Microsoft.

[MalikChen]

perhaps the world will wake up and realize that it _might_ be a good idea to mix systems a bit so that whatever happens to one system, you still have some significant percentage of the network still running.

So we should make three different highway systems just in case there is a wreck on one of them? We should build three smaller power plants instead of one in case one of them crashes?

I can see your point, but the problem is initial cost. For each seperate system, you've gotta have additional servers, licenses, staff, etc. to maintain it.

You can't just run Apache, IIS, [whatever Apple uses], Netware, and Solaris on the same server. You also can't expect a single tech department to be able to know the ins and outs of each system. There will be a "Linux guru" and a "Windows guru", and a "Mac guru," who will all want things done their way.

I work in an IT dept. We've got four different types systems (P4 1.5, 2.0, 2.4 and XP2500+), all running XP. If there is a software problem that will take more than five minutes, we re-Ghost the computer with an image on the network, and it's up in a flash. Employees keep their data on the network, not locally. Uniformity is cheap and easy.

The benefit to having multiple systems just isn't great enough to outweigh the cost.

Re:Not a nail for Microsoft.

[bentcd]

For reasonably small networks (up to a couple hundred units), I agree. And also for networks that don't really matter all that much (100,000 units running SETI@home), ditto. When you get a really large network that is critical to your operation, however, then the cost is well worth considering. The cost will tend to be large, but it is mostly dependant upon the number of different systems you wish to support. It is not all that dependant upon the number of units in your network. If your network is really large, therefore, the cost will tend to be very small compared to overall maintenance expenses. You main problem, of course, will be applications. You will need to find and use applications that can interoperate between different operating systems. For a system of this size, it is plausible that it might be possible to sink some money into finding cross-platform solutions that cover your needs. Smaller organisations might find this economically infeasible.
As for your highway example, building multiple redundant highways is generally unnecessary because in the case of a major accident clogging the highway, there will always be smaller roads in the area through which traffic can be rerouted while the highway is cleared. If this redundance weren't already present, one would be well advised to consider explicit development of a redundant system.
The same goes for power plants. The power system in your country will generally be so interconnected that if one single plant goes down, other plants will cover for it while it recovers. The redundancy is therefore already there. Were it not, then, yes, most civilian planners would consider the risk and I believe many of them would go for the redundant solution.

Someone must deeply regret

[Lakedemon]

throwing Windows 2000 away for Win xp...
or not using
Linux
or
Macs
or
Solaris and sun hardware....

Re:Someone must deeply regret

[REBloomfield]

yeah, because they never break, and would enable the entire UK Pensions department to function more easily, because their wouldn't be any training required for the thousands of people that use the system, and aren't experts, and shouldn't need to be....

Re:Someone must deeply regret

[Anonymous Coward]

That's right. Windows is just popular. It's not broken.

Dumb, gullible fucker.

Re:Someone must deeply regret

[REBloomfield]

erm.. what?

a sudden switch to macs/linux/whatever is going to cost a hell of damn lot more in the long wrong, and will break more in the interim. guarenteed.

Re:Someone must deeply regret

[Lakedemon]

What I meant is that they could have sticked to their Windows 2000 machine and avoided this problem. It is know as the most stable version of windows. Besides, why repair (upgrade/take risks to break) something that isn't broken and work well.

Contractor

[HogGeek]

I know we all like to blame Microsoft when these types of things happen, bu this appears to be a major fubar by the EDS people.

The installation and update of operating systems is so easy any more, a blind one armed monkey masturbating could do it.

I've worked with EDS people, and the one armed monkey would be a godsend compared to most of them that I've had the "fortune" of working with...

Re:Contractor

[Phantom69]

I've sadly had the 'experience' of working with EDS (they've been paid £60 million to manage the IT systems at Thorntons PLC for five years). Utterly incompetent was the nicest compliment I ever managed.

Re:Contractor

[Jumpin' Jon]

I too have had the un-pleasure of working alongside some EDS employees, and they were utter, utter morons. You get that awful sinking feeling when you realise that their qualifications comprise a MSCE alone, but they just had no concept of life in the Real World of IT at all.

They actually did have their Linux zealot with them, and he was the worst of the lot -- a vile, unwashed grease-blob with the same mentality (brainwashed; blinkered; whatever) of most of the zealots here... v. annoying.

JJ

Re:Contractor

[upsidedown_duck]

As a blind one-armed monkey breaking from masterbating only to type this reply, I must say that using my kind as a representation of the lowest-level of competence is quite offensive.

Re:Contractor

[TheLink]

Well that says a lot about their customers (including the UK Gov) who seem to keep handing them money.

oh yea

[Llevar]

Hopefully just another nail in Microsoft's coffin...

When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.

Yea, I can just see them going bankrupt over this. Their coffin was half closed before, but now they're bound to be pennystock.

Re:oh yea

[CmdrGravy]

It's certainly not a Feather In The Cap.

Avoidable blunder

[Waffle Iron]

Obviously these sysadmins were incompetent. Everybody knows that a BSOD is impossible under Windows XP. If they had simply upgraded the other 60,000 machines to XP first, and then updated these 7 problem systems, this whole problem would easily have been avoided.

Re:Avoidable blunder

[rts008]

"Everybody knows that a BSOD is impossible under Windows XP" Apparently my version of XP Pro SP1 was not informed that it could not Blue Screen! It BSOD'd 7 (seven!) times in 2 hours when I was trying to download Mandrake 10, giving "Bad_Pool_Header" error message. Go back to uninformed fantasy land where you came from and leave the rest of us alone!

Re:Avoidable blunder

[ScrewMaster]

Well, sure. The XP firewall's stateful packet inspection routine determined that dangerous "viral code" was being downloaded to your system from the Mandrake servers and immediately halted the system to protect it.

TCO costs rise scarily with Windows XP failures?

[hattig]

So ... 5 working days, 60,000 PCs (= 60,000 employees?)

Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.

=> TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.

Re:Couldn't they just upgrade _ONE_ of their comps

[Wudbaer]

Apparently they were trying just to do that and accidentially "tested" it on all 60000. OOPS.

At least their automatic upgrade system seems to work both reliably and fast. :-)

Wrong!

[Mr_Silver]

Bad Slashdot reporting again? Quote Slashdot:

the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.

In actual fact, the Register quotes:

According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

and then below it:

Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.

So, by merging them you get the following story:

There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.

Somewhat slightly different to the Slashdot version wouldn't you say?

In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.

Re:Wrong!

[Anonymous Coward]

No, you got it wrong. This is Slashdot, and Microsoft is involved in this at some level, so the appropriate story would be:

After personally slaughtering 1,000 orphaned puppies to satisfy his blood lust, Emperor Gates and his side-kick Commandant Ballmer flew their invisible jet of doom to Jolly Olde Englande where they forced 60,000 innocent fuzzy penguin Linux users to overwrite all of their partitions by installing Windows Evil Edition by exploiting the breathing holes in their tinfoil hats. When the users started to realize what was going on Emperor Gates stole Thor's hammer, by offering Thor $50,000 which he accepted, and wrought terrible installation woes upon all mankind. Richard of Nazareth, the great Stallman, in disgust has vowed to help the poor fuzzy penguins by ranting rhetoric at some shack of a university in an unnamed part of the third-world.

Re:Wrong!

[Anonymous Coward]

LMFAO!

Mod that up %^_^%

Re:Wrong!

[prisoner-of-enigma]

Somewhat slightly different to the Slashdot version wouldn't you say?

How dare you remove the anti-MS slant from this article by stating truthful facts! Have you no bias, sir? Don't you know that it's blasphemy to not try and take the most negative possible stance on anything regarding Windows whilst simultaneously praising how Linux is completely immune to these things? And how dare you suggest that a botched kernel or buggy driver could cause a kernel panic! By revealing such truths, you destabilize the endless war on
Slashdot to make Microsoft responsible for cancer, hunger, and the lack of world peace!

You're new here, aren't you?

Re:Wrong!

[Anonymous Coward]

In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.

Well. I *did* that to a computer system a few days ago. What to do? Insert the boot cd, fix grub.conf. reboot. works.

Ofcource, it it had been a 60 000 systems system, i would have made a script... Burn some cd's, and send junior admins out to boot'em.

But then NTFS is a chore to acess from a "dead" system...

Re:Wrong!

[julesh]

But then NTFS is a chore to acess from a "dead" system...

Not in the slightest.

The fools deserve it.

[NoMoreNicksLeft]

On a much smaller project, I was supposed to do a minor softare upgrade on 95/98 machines in a single school system. By the time the first lab was finished, it was obvious that I had ruined the computers for the most part. I did not continue, I called up the project manager, stupid bitch that she was, and pleaded the case that I simply couldn't continue.

Of course, I waws removed from the project, which at the time amounted to going back on unemployment. Some day, I'll learn to ignore my conscience.

Re:The fools deserve it.

[Anonymous Coward]

what kind of retard cant upgrade windows? what the fuck are you doing on slashdot, shouldnt you be watching nascar or or playing checkers with your wife/sister

how the fuck do you even have a computer?!?

Re:The fools deserve it.

[KarmaMB84]

In my personal experience, installing updates to Windows 9x system (or even installing Win9x) was a hit or miss affair. One small problem with the system (school computers, OH GOD NO) + attempt at upgrading = hosed back in the 9x days :(

Re:The fools deserve it.

[Anonymous Coward]

Shit happens. But, you still should have posted AC.

what the hell went on???

[swordgeek]

They upgraded seven machines and 80,000 died? That sounds weird, but maybe they were the AD servers. Why then, on a small number of such critical boxes, didn't they just restore from backups?

Re:what the hell went on???

[Anonymous Coward]

The story is that a small group of machines on a small network were the target of the upgrade, but that the MCSE in charge clicked the wrong shiny button and upgraded a much larger network. A very much larger network.

Its a monumental EDS fuckup: no change there then.

Re:what the hell went on???

[PatrickThomson]

Nope. Every single desk-stuffers office terminal was rendered broken to the point where it won't boot to the automated patch applicator.

Fixing the hopeless...

[CoolSilver]

At least they might get some overtime in UK for fixing thing. Here in Us you have to suck it up and take what pay you get out of 40 hours even if you worked 60.

What you want to bet they are using some kind of linux or knoppix to recover the files.

You guys are amazing!

[Lee_in_KC]

I wish I could take one of you Linux "experts" up on your idea. "Here, upgrade these 2000 PCs, all of which are from different manufacturers and different configurations, to Linux. I need it done in the off hours and I need everything to work like it did before.".

*crickets*

Of course someone will reply and say "ok!" knowing it won't happen. It's not because I don't have the ability to make that decision but it's because I know better than to get real information/insight about IT from most /. posters.

It's painfully obvious that a scant few here actually have a clue about running a business that relies on IT. It's more than ripping CDs and DVDs kids. Sure, the company that did the mistake is at fault but the problem is not in the chosen OS, it's in the chosen technicians and management.

Re:You guys are amazing!

[rpozz]

different manufacturers and different configurations

You know that (re)installing Windows on a large number of systems of different types, for example when an upgrade fails, is a total fucking nightmare, yes?

At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).

Re:You guys are amazing!

[stratjakt]

I want you to upgrade 60,000 machines from samba 2.x to samba 3.0 and I want it done tomorrow, and they all better work.

Oh, the machines should auth against and become members of an AD Domain, too.

Whatever, the parent was completely right.

The Windows install ships with VASTLY more hardware support than any linux install, and everyone knows that.

Re:You guys are amazing!

[rpozz]

I want you to upgrade 60,000 machines from samba 2.x to samba 3.0 and I want it done tomorrow, and they all better work.

That can be done through a script put out on the network. Depends on the distro though. Either way, it's a hell of a lot easier than with Windows.

The Windows install ships with VASTLY more hardware support than any linux install, and everyone knows that.

Actually, a standard XP/2K CD doesn't have that many drivers. Let's pick an example - sound cards. If you use SuSE (or whatever) with ALSA, it will pick up practically any PCI sound card with no extra software. Windows, however insists you provide it with a driver CD. Another good example is with video cards (with the exception of ATI/nVidia-based cards).

Re:You guys are amazing!

[Lee_in_KC]

"At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?)."

Absolutely right .... because it takes more than a group of gamers wanting a particular driver to make the grade.

Besides, you really only serve to make my point for me. Businesses don't really care too much about sound cards in the office.

Re:You guys are amazing!

[stratjakt]

That can be done through a script put out on the network. Depends on the distro though. Either way, it's a hell of a lot easier than with Windows.

Think so? Ever use alteris or any big management suites? If anything it easier, but both ways require you to TEST THE FUCK OUT OF IT.

I mean, imagine your linux script is deploying a new kernel, and has to modify lilo/grub on every machine and reboot. Imagine you fuck up your edits to lilo.conf or grub.conf. And now, instead of pushing it out to the seven machines you want to test, you push it out to all of them because you messed up typing in the netmask

Voila, 60,000 borked boxes - and according to slashdot groupthink, it's all Linus' fault.

Re:You guys are amazing!

[Anonymous Coward]

You're changing the subject from the original point about hardware support.

Running a network script on any OS can fuck it up. Hence you need to hire someone with a brain to do it.

Voila, 60,000 borked boxes - and according to slashdot groupthink, it's all Linus' fault.

Slashdot groupthink would instantly blame anything other than Linux.

Re:You guys are amazing!

[stratjakt]

The orignal point was ridiculous. What dope actually thinks Linux has wider hardware support than Windows on x86 PCs?

No wonder they're having a hard time finding jobs in IT if they honestly believe that.

Re:You guys are amazing!

[Anonymous Coward]

Wider hardware support....... OUT OF THE BOX. Jeez.

The original point was that Linux takes less time to set up on a set of different systems than Windows does, because Windows insists on using seperately bundled drivers.

Re:You guys are amazing!

[Seeker_350]

>>>>

You little snot - you know nothing - and I mean NOTHING - about upgrading vast amounts of PCs.

Now...go back to whacking-off in your parent's basement and hurling smears at MS.

Asshole.

Re:You guys are amazing!

[Khuffie]

At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).

Wow. Have you tried using Windows recently? It works like a charm: install, and make sure things like printers are on, and everything is available as soon as you boot up: the internet, the printer, the sound, etc. Whereas on my Linux laptop everytime I install on it I have to spend hours getting my bloody soundcard to work. Half the time I have to manually configure the network card too.

Re:You guys are amazing!

[Craig Ringer]

Frankly, I think "a scant few" is pushing it ... despite the number of clueless morons, many here do have at least some idea what's going on and how to sensibly address some IT problems.

As for managing large networks of desktops, that's another very different matter. Not many people have high-level experience doing that.

My network, for example, is only thirty machines. Hardly huge. In fact, it gives me the opposite perspective on a lot of issues, because I find many of the large-site friendly features of Windows networks utterly useless for a small site, and no small-site friendly managability features to compensate.

Personally, I've trialled XP at work as a possible upgrade for our 9x machines, and come to the conclusion that it's not worth the pain. It might be good if you have the management tools, a dedicated test network, and an admin team dedicated to designing and rolling out updates. For small sites, however, it's pure hell. Even controlling how the clients update themselves is hard without an extra server to do the job. I also found accessible information for small-site management to be very thin on the ground.

We're now using thin clients for some of our network, and seeing very good results. Yes, they're Linux based - MS looked good until we figured in the CALs and the isssues with NT-based terminal server security. I'm far from floored by the results with Linux - the bugs, oh, the bugs, I'm drowning in stupid f***ing bugs. There's also more than a little totally retarded design, and the classic issues with no two apps having the same open/save dialog.

That said, for our basic users the results have been very good. They need little support, hardware and software costs are both low, and things generally run very smoothly. Trials with more demanding users aren't going as well (see above rant about bugs and bad design), but current development in the OS is addressing most of the issues I've run into and I expect to be able to move the 9x users across to the thin clients mid-late next year.

I do agree with you that managing a large collection of Linux desktops would probably be pure hell. It's awful to even think about, frankly, especially upgrades. *shudder*. My solution would be to simply not use desktops, but instead move most users to department level thin client services hanging off a redundant set of beefy servers. I'd use LDAP to store user and sytem information (yes, much like AD) as I currently do on my network. For many users, such a setup can be expected to work very well, and dramatically reduces the admin nightmare compared to Linux desktops. I also wouldn't even try to migrate all users to Linux - only basic users for whom it would work well, such as those who only need email, a browser, a word processor, and access to a couple of specific in-house apps.

As for migration - I can't possibly imagine how it could be done in a sane way. I suspect a lot of custom tools would have to be written, the migration would need to be a rolling one, and there would need to be a lot of staff on hand to handle glitches. That doesn't sound like fun to me.

The worst part of moving my users over to the thin clients was migrating their data and settings. That despite the fact that almost all of it was already on the servers, and their systems were pretty basic and very uniform. Doing it in a large company wouldn't be nice.

Re:You guys are amazing!

[Anonymous Coward]

I speak as someone who works there so can speak from the horses mouth. I post anonymously as I don't work for them and also don't work for EDS, but I would prefer to be off-the-record.

There are not lots of combinations of PC's there. Most, if not all PC's, are Compaqs or NEC's. I've never seen any other desktops, but they use a lot of Fujitsu laptops. They are all locked down tight, they all have exactly the same base image, they all work exactly the same way, they all have the same hardware spec. There was a major roll out of around 100,000 PC's over an 18 month period to a very, very standard desktop machine.

Much as I dislike EDS the PC systems do tend to work, slowly, but they do tend to work.

Re:You guys are amazing!

[Craig Ringer]

If the number of different types of systems is reasonable, it shouldn't be too bad. The PXE-based remote install tools and image generation options for recent Windows versions have been pretty impressive. You can roll a bunch of drivers, your own software, and all sorts of other things into the install process.

Of course, in action it's not as smooth as you'd hope, but really is anything? I haven't done it myself, but have seen others doing auto-install rollouts, and if they're well enough tested things seem to go fairly decently.

As for the NIC driver issue - buy better hardware, dude (the eepro/100 "just works") or slipstream install the NIC driver on your install media. Anyway, you're going to be doing a network install with a custom image on a large network.

I don't have a large number of nice things to say about Windows, but the job MS has done making installs of large bunches of machines less painful is pretty good.

Re:You guys are amazing!

[Anonymous Coward]

With comments like this, it would be useful to know some details like
- the Linux distribution you're installing
- the soundcard you have problems with
- your network setup.

Not that I don't believe you.

Re:You guys are amazing!

[Anonymous Coward]

but it's because I know better than to get real information/insight about IT from most /. posters.

Tell me about it. I've been flamed before for suggesting that in the Real World, business systems require a somewhat different approach to the single PC in your bedroom. Generally by someone who then claims to have spent 20 years in enterprise computing.

Re:You guys are amazing!

[gruhnj]

At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).

Apparentyl you have never heard of Remote Installation Services. Assuming that you have your systems all coming from the same vendor or similar vendors, its actually not as hard as one might think. Most corrporate deployments are like this. Turn on DHCP and have them boot off of it. RIS can take care of that if you have done it right.

Re:You guys are amazing!

[Anonymous Coward]

Dave? Is that you? Stop reading Slashdot, we've still got 59,473 machines to fix.

Re:You guys are amazing!

[randyflood]

You're kidding right?

Why aren't the machines already running the latest version of Samba already?

Did my automatic update script stop working for some reason?

And If I wanted them to authenticate to Windows, are they already doing so? If so, then, I don't even have to change my configuration file, now do I? I have have to do is let my cron job run that automatically updates them to to the latest version of the software that I have approved for them on my server that I am using to manage their updates. I would just Drop the latest version of SAMBA on their and then I would be done. How tough is that?

Re:You guys are amazing!

[Anonymous Coward]

Most system have on board sound now days

Re:You guys are amazing!

[Anonymous Coward]

I would venture to say that Knoppix is a _little_ bit easier to install/operate than WinXP. Its driver support is on par with XP and its auto-configuration of hardware is better. Combine Knoppix (deb based) with BIOSs that can boot PXE and use rdesktop to connect to your windows apps. Yeah, the OS reconfiguring itself isn't the most efficient way, but at least if hardware is changed it can adapt. If its a small office.. save some trouble and just have EVERYONE use your pre-configured OS off the CD. If they have to use Office, buy Windows terminal server or shell out the bucks for Citrix and use the built in citrix or rdesktop clients for linux.

Want to upgrade the system or install new programs? Run your own deb server and use one of the many distributed ssh scripts.

Re:You guys are amazing!

[pclminion]

Ahh, the classic Slashdot phenomenon: Hallucinate that people are saying certain things, when they are not, and then bitch at these imaginary people.

Who is suggesting that all those systems should be upgraded to Linux? Seriously. Give us a link to the comment where you read that.

Re:You guys are amazing!

[codermarc]

At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).

Actually, Windows 2000 includes drivers for most NICs, as well as most video cards. Windows XP contains drivers for most all of your hardware...

Re:You guys are amazing!

[phyy-nx]

At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).

This was certainly the case with Windows 98, NT, and often with 2k. However, with over 3 years of using XP at home and in a professional software engineering environment, I have never had to look for drivers. Either the hardware came with a disk with the drivers, or XP's mega-huge drivers.cab file was searched for the driver and plug-and-play worked. That drivers.cab knows a lot of hardware out there. 99%? Perhaps not. But I'd like to see you cite your source about Linux with 99% of all software drivers ever created being pre-installed. :) I'll eat my words w/ salt, pepper and respect if thats the case.

Re:You guys are amazing!

[Bigbutt]

Ok :-)

Just like EDS did, you'd have to start off with an idea of what you're doing and why.

Step one, evaluate the machines to see what special requirements are necessary for the systems to continue to operate in the same mode. That they have different hardware isn't really much of a requirement since virtually all distros handle hardware requirements pretty well.

Step two, break the systems in to catagories. A majority will probably be simple with office and web browsing requirements.

Step three, set up 50 laptops (WAG) with the distro chosen set up for "jumpstart" (or whatever it's called on the linux side) to install the distro including office and firefox.

Step four, set up a department by department schedule. Have a team stop by each of the systems, to ensure the data that needs to be moved has been backed up to a dedicated file server.

Step five, the night of the move, burn a backup image of the target system to a second dedicated server properly identified. Jumpstart the box. Move the data back, converting bookmarks and e-mail, moving documents back to a document folder. Leave a 1 or 2 page document instructing the user on how to get started. Have the browser start up to a more inclusive set of documentation showing how to start the new tools.

Step six, once the majority of the users are converted, set up a second schedule to address the specialized users who might need access to a mainframe or specialized application. They may need a windows box to continue to function, at least until their specialized needs are met.

Step seven, profit

For 20,000 users, this'd be a year long project with several levels of management, lots of interaction and training for the support staff. You can't just say, "here's 20,000 machines. Fix them."

There'd be weekly meetings charting the progress including the problem folks that need upper management motivation and where we were in the schedule.

By the way, these are the same steps you'd need to convert 20,000 users on Windows 98SE to Windows 2000.

And if you need something like that, drop me a line. I'd be happy to get further in to this. Sounds like fun :-)

[John]

Re:You guys are amazing!

[Anonymous Coward]

"I mean, imagine your linux script is deploying a new kernel, and has to modify lilo/grub on every machine and reboot. Imagine you fuck up your edits to lilo.conf or grub.conf."

Linux.

Which means we have proper package management tools. Not DOS batch scripts masquerading as package management.

When I upgrade the kernel on a dozen machines I don't write manual scripts to alter the boot loader configuration. The package manager can do that stuff for me, and it was debugged by lots more people over a longer period than any script I could throw together.

"instead of pushing it out to the seven machines you want to test, you push it out to all of them because you messed up typing in the netmask"

More stone age sysadmin talk from a Windows users. Why are you typing in netmasks? Did it make you hexedit a binary file to change the version number while you were at it?

When doing a real roll-out you select the groups of machines to receive the upgrade from a nice list, and then schedule their updates, eg. so that 60 000 machines would apply the update in groups of 600 during a Monday afternoon. Progress information shows you waiting/ applying/ success/ fail information that you can drill down to see if anything is unexpectedly going wrong.

When testing a half dozen machines you just manually apply the update remotely, two button clicks and you see the succeed / fail immediately and can then test it.

But of course the techs doing this will have been taught that there's only One Uniform Way To Do It and they'll use the same procedure to initiate a real 60 000 user roll-out as for a test machine, thereby maximising the risk of making a terrible mistake. Why were they taught One and Only One Way? Because Microsoft Certified Generic Know-Nothings don't have _understanding_ of the problems they're solving, and therefore divergence from the approved path causes problems. Problems they've never been taught how to solve.

Lessons:

1. Use good package management software, and insist on good packages. Great software that doesn't come in a package? Figure the price of packaging it yourself into the costs. Configuration changes needed? Package them, don't fudge it.

This makes upgrades more reliable, and rollbacks less painful. The phrase "wipes out" in this article title should put a shiver down your spine.

2. Hire competent staff. Make sure you hire people who can tackle problems they haven't seen before, so that they don't treat everything as a nail just because they happen to have a hammer.

No substitute for the right people.

Re:You guys are amazing!

[Elladan]

That's bound to do amazing things with no working NIC driver.

Re:You guys are amazing!

[yamla]

Yes, PROVIDED THAT THE NECESSARY DRIVERS ARE INCLUDED IN WINDOWS BY DEFAULT. My printer driver isn't. My NIC is, thankfully. But I've installed Windows on plenty of computers with NICs whose drivers were not included on the Windows XP install CD. Or, say, computers whose SATA controller driver was not included.

Now, perhaps your specific computer's components all had the necessary drivers included in Windows, but this is certainly not the norm these days.

Re:You guys are amazing!

[optimus2861]

Have you tried using Windows recently? It works like a charm: install, and make sure things like printers are on, and everything is available as soon as you boot up: the internet, the printer, the sound, etc. Whereas on my Linux laptop everytime I install on it I have to spend hours getting my bloody soundcard to work.

It must depend on the hardware you have. My experiences with WinXP Home and Mandrake 9.2 are exactly the opposite of yours, when I did a complete reinstall of everything earlier this year. WinXP did not recognize my sound card (no-name piece of junk, I admit), my printer (HP), or my modem (AOpen -- don't use it any more anyway), and put in crappy drivers for the video card (ATI). Mandrake got the first three all right, and also put in crappy drivers for the video card. Both had some trouble with the ethernet card, too.

What surprised me was how many of the Windows drivers still required reboots. With those added in, I figured I was finished with Mandrake in about 1/3 the time that it took me to finish with Windows.

Re:You guys are amazing!

[EnglishTim]

Windows insists on using seperately bundled drivers.

But it doesn't. Windows tends to include drivers for most common (and many uncommon) devices with the installation.

The main difference is that when you download a Linux distro, it's probably been put together in the last six months, whereas a Windows XP CD will have been put together four years ago, so the driver set on an XP CD will likely be a bit more out of date than a Linux distro.

Re:You guys are amazing!

[Anonymous Coward]

A /. poster, thats who. Ever heard of postulation? Theorizing?

Quit being obtuse, and assuming no one's allowed to propose a linux solution over a microsoft one.

6+ figure /. ids should be ignored at best.

Re:You guys are amazing!

[dn15]

It really depends on the hardware. There are just too many possible combinations. My experience differs from yours...

In my experience you can install Win2K or XP on a given computer and typically need at least one driver, whether it's video, audio, network or something else. But boot a Knoppix liveCD on the same system and everything works immediately.

Re:You guys are amazing!

[angulion]

In this light..
EDS might be the one to get the blame, but despite this I think that if they get 60000 machines up and running in one week, it is quite impressive. Mind, w/o network upgrades (?).

Nice to think - I takes *one* person and one day to bring down 60000 computers, but it takes probably 200-300 persons and a week to get them back..

(feel like looking for a new job, one?)

Re:You guys are amazing!

[Tough Love]

I do agree with you that managing a large collection of Linux desktops would probably be pure hell. It's awful to even think about, frankly, especially upgrades. *shudder*.

It's perfectly easy with any Debian-based distribution or Gentoo.

Re:You guys are amazing!

[Craig Ringer]

I'm afraid I can't agree.

If you think a package management system with remote repositories and automatic dependency resolution is the answer, basically any distro will do.

Unfortunately, that's not the answer. System management is much more than installed software managemnet - you must manage system and user configuration, user data, possibly additional software on top of the packaged stuff, etc. It's hardly impossible, but it'd be a real pain to do.

eds stint

[hostylocal]

i worked for eds about a year ago and i can tell you now that the 'about seven' win xp boxes is a blatant lie. the room that i worked in had over seven desktops with it on. and they were all legal ones too. i had to laugh at the local office staff using the fax to communicate with one anoother tho - 'do you see the white thing on your desk? the thing with all the numbered buttons and the thing that looks like a handle? well, that's called a telephone and it's used to talking to people. remotely. no really. give it a go'.

Hope they all loose their jobs tomorrow

[t_allardyce]

Im pretty embarrassed for my country right now. How the fuck did we go from technological pioneers to this? And its only the tip of the iceberg, what with Ken Livingstone's numerous stupid ideas, David Blunkett's insanity and the incompetence of 100's of 'IT' projects (hint: if its called an IT project it means its run by incompetent MCSEs and it will fail catastrophically leaving millions of people without a service or having planes crashing into the ground, time and time again) with tax money falling out of their pockets, fuck them! Why do these idiots get the contracts? What happened to all the competent people??

Re:Hope they all loose their jobs tomorrow

[REBloomfield]

let me guess: you're one of the compentent ones?

ffs, most IT projects go over budget and over time scale. The reason you hear about these is because it's fun to slag the government and fun to slag microsoft. apparently, i appear to have missed that one....

Re:Hope they all loose their jobs tomorrow

[0123456]

"What happened to all the competent people??"

They emigrated, most likely. One of the problems with incompetence is that it's self-reinforcing, the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time.

Re:Hope they all loose their jobs tomorrow

[KontinMonet]

They go over budget because when a project is accurately costed, some idiot manager somewhere goes beserk and says it must be done in time-(large chunk of time) and for cost-(managers' & directors' bonuses). Knowing this most s/w projects are unrealistically timed and funded. Anyway, EDS has right royally screwed up on all the big govt. projects yet the govt. continues to use them. Is that as a result of competence?

Re:Hope they all loose their jobs tomorrow

[Anonymous Coward]

That would help... Most (if not all) gov IT projects are out sourced.
This cock up is in fact down to a US co :D

Re:Hope they all loose their jobs tomorrow

[t_allardyce]

Its not so much the budget or the time, having spent years missing deadlines I can perfectly understand that. What it is, is stupid ideas, stupid projects that don't address the real problems. It seems like the people in charge of dishing out these projects don't want to listen to the people who actually have to use them. And then there's the rip-off prices of these things that make no sense, suddenly a simple machine will cost £100,000 when you absolutely know that all the parts can be bought at retail price for a fraction of that and the whole thing is going to be mass produced anyway and the development costs could be covered in one unit! Things are managed so badly its insane and not on. I want the government to be chock full of experts in all fields like it used to be, I want these projects to have _exactly_ the funding they need, no more, no less and I want them to be managed with integrity and decency without cutting corners and with good solid workmanship. Tax money should not be something to be given to anyone it should be wisely spent on the things we need and quite frankly fuck peoples selfish profit margins - what happened to doing things for your country instead of ripping it off? its not like people are being asked to do things for free.

Re:Hope they all loose their jobs tomorrow

[upsidedown_duck]

the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time.

This is true, at least in my case. After working several years with people who never ever learn anything new even after being told repeatedly, I just couldn't keep going. Right now, I'm proud and happy to say I no longer work in IT or government contracting at all. I wouldn't touch it again for even a gigantic pay raise, because my remaining sanity is too valuable. Even now, I can visualize my former co-workers making the same mistakes, learning nothing from them, all while wondering how they can manage to make very nice SMP servers crawl under any hint of a real-world workload, and, if I were to visit them, my vision would be proven true. I firmly believe that people who stick it out in IT are either into non-sexual S&M or are just so stupid that they can't know any better. I wonder how many MCSEs also frequently buy the extended warranties at Best Buy. They certainly bought (insert software company's name) bait-and-switch schemes. What a bunch of losers.

This just shouldn't happen

[RealProgrammer]

This is the kind of result that you should have to be trying to achieve for it to happen.

No, this is the kind of result that you should not be able to achieve, no matter how bad a sysadmin you are and how little you try, or no matter how good you are and no matter how hard you try.

Re:This just shouldn't happen

[dinivin]

So would it be Slackware's fault if I did a "rpm -Uvh --force --nodeps" on a glibc update on my Slackware system?

Re:This just shouldn't happen

[RealProgrammer]

• So would it be Slackware's fault if I did a "rpm -Uvh --force --nodeps" on a glibc update on my Slackware system?

First, Slackware doesn't normally use RPMs. If you want to use rpm, use RH/Fedora or Mandrake. I'll assume you meant pkgtool.

Second, Patrick Volkerding and the Slackware community don't tell you to push out massive upgrades, using pkgtool or otherwise.

Third, you still would have to try to configure a network so badly to cause an attempted upgrade to 7 machines to crash 60,000.

Re:This just shouldn't happen

[dinivin]

First, Slackware doesn't normally use RPMs. If you want to use rpm, use RH/Fedora or Mandrake. I'll assume you meant pkgtool.

I meant what I said. What I desribed isn't any different than some idiot network admin under Windows pushing a Windows XP update out to a W2K machine. In both cases, it's the network administrator's mistake, not Windows.

Dinivin

The funniest thing..

[WindSword]

was the government spokesperson. After the intro to this piece on Radio 4 this morning, her opening sentence was "Let me correct you, 20% of our workstations are functioning". Talk about a positive spin.

Re:The funniest thing..

[tigertiger]

20% of our workstations are functioning

These were the ones where the remote update didn't work...

Re:The funniest thing..

[Anonymous Coward]

I bet it was this person or this persons boss who signed the contract with MS in the first place.

BSOD is quite possible under WinXP

[Polarism]

It's just a tad more rare.

Seems to me....

[Raven42rac]

It seems to me that there is just a lack of importance/trust placed in computing in general, due to the bad rap that MS has being applied across the board. Call me a troll if you like, but this would not have happened in a *nix/Apple shop. There is far too much at stake for willy nilly mistakes like this. Also, with 500 million quid, could they have not built an isolated test subnet?

I call BS...

[Blue-Footed Boobie]

Sounds to me like someone in IT fucked up seriously, and is using Microsoft as a scapegoat.

Ofcourse, this is /. so this post will sink fast...

EDS

[Anonymous Coward]

EDS have screwed up time and time again, and yet the UK government keeps using them. Please look here: Wow, 188 results, most of which appear to contain the words "EDS" and "Cock up"! I appears as if though a lot of people have just failed to read the article or just see this as a good chance to bash microsoft.

What a big surprise

[joshsnow]

from the reg article;

"This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess."

So, even more of the money I pay in tax is being diverted to M$ then...

Overcharged, as usual...

[Anonymous Coward]

I feel sorry for the goverment sometimes, they keep getting taken advantage of...

"WHAT?, How much for a single piece of glass?!?"

"History lesson"

"Wastage"

EDS

[Anonymous Coward]

EDS have screwed up time and time again, and yet the UK government keeps using them. Please look here: http://forms.theregister.co.uk/search/?q=EDS Wow, 188 results, most of which appear to contain the words "EDS" and "Cock up"! I appears as if though a lot of people have just failed to read the article or just see this as a good chance to bash microsoft.

Re:Testimony to power of insufficient testing

[DrSkwid]

you do know that it was a 7 PC sandbox test that went wrong yes ?

Re:TCO costs rise scarily with Windows XP failures

[turgid]

The bean-counters will find a way of "writing off" this debacle so it doesn't show up in TCO. Not that I'm bitter and cynical or anything....

I once knew a bean-counter (quite senior) on nearly 3 times my engineer's salary. He was sat there in front of a spreadsheet adding up a column of numbers on a pocket calculator.

Welcome to the UK Public Sector. That was your tax money.

Corruption

[Anonymous Coward]

UK government IT contracts are awarded to a select group of companies. Senior executives of these companies sit on various committees and steering groups that determine government IT policy. There is often no tender process.

The companies that get contracts (e.g. EDS and Capita) produce poor software and have a dismal record when it comes to requirements elicitation. This latter failing is a good cash cow. When inadequacies in a system are exposed the tax-payer has to fork out more.

Exactly what government ministers and senior civil servants are getting out of this mess is not clear. It's worth noting that board-level jobs after retirement is a favourite in the defence sector.

Re:Corruption

[REBloomfield]

As a Capita user in UK Government, i can categorically deny that they produce poor software. They may have poor track record in keeping stuff up to date, which is why we've used DOS modules for the past few years, but this is to ensure that stuff actually *works* when we do upgrade, which we recently have, and it's an amazing suite of apps.

Re:Corruption

[Anonymous Coward]

Ooo I'm **sooo** impressed ahh an amazing suite of apps which cost the tax payer a fortune...it's crap - a court system that doesn't work - an NHS system heading toward the iceberg and the DWP farce....It is about time that the UK govt. purchased sensible n-tier software, produced by people that actually understand how to write this stuff either using J2EE or .NET......
The UK govt. needs to look to companies that have a record of good delivery in the UK private sector to deliver the public sector apps. Rather than throwing good money after bad on companies such as EDS.....

Re:Corruption

[REBloomfield]

i take it, of course, that you *use* the apps in question?

We need to educate the decision makers

[mishmash]

We've got to stop this happening again, we've got to educate the people spending our money on huge computer systems which are prone to failure.

I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.

There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme. This is one we could lobby our representatives on to ensure they do it right..

Where to have the debate where it might be read by those who mater:
Free service to fax your MP

Boris
Richard Allan
Tom Watson
Shaun Woodward
Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure. as well as the NHS computer system UK

Re:We need to educate the decision makers

[mikerich]

There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme. This is one we could lobby our representatives on to ensure they do it right..

No, no, no, this is the one we lobby them to employ EDS and Microsoft on!

If MPs are stupid enough to implement Blunkettcards we should at least get some entertainment out of it.

Best wishes,
Mike.

Re:We need to educate the decision makers

[mikerich]

Where to have the debate where it might be read by those who mater:
Free service to fax your MP

Can I take the opportunity to point out that faxyourmp is for UK citizens ONLY and should only be used to fax your own MP. It is not for international write-ins or mass lobbying.

Best wishes,
Mike.

Re:We need to educate the decision makers

[bigboard]

This is one we could lobby our representatives on to ensure they do it right

We need to be lobbying our representatives to ensure they don't do it at all.

Re:We need to educate the decision makers

[brad3378]

What is an MP ?

Re:We need to educate the decision makers

[lxdbxr]

EDS are preparing themselves to manage the UK national identity database and identity card scheme. This is one we could lobby our representatives on to ensure they do it right.

This is one we should lobby our representatives on to ensure they don't do it at all. The fact that they will piss away several billion quid of taxpayers money is by-the-by when there is no reason other than sheer control-freakery to want this database in the first place.

Re:We need to educate the decision makers

[KontinMonet]

Unfortunately faxyourmp does not work for all MPs - it certainly does not work for my MP (one of Blair's babes). But then again, she has never acknowledged a single email or letter either (probably doesn't have a clue what I'm on about).

Anyway, I rather suspect EDS has our govt. in it's back pocket. Just about everything they touch get totally fcuk'd up - but they continue to be given £billions.

Re:We need to educate the decision makers

[chary]

"Where to have the debate where it might be read by those who mater:"

And you lead with Boris?!

Re:We need to educate the decision makers

[REBloomfield]

Dude, Borris just got sacked from the cabinet. I'm sure they're going to give a damn what he thinks....

Re:We need to educate the decision makers

[chary]

"Hmmm? Oh, what? Oh, yes old bean, jolly bad show. Yes. Those bounders at the NHS need a...oh, just a moment, my phone's ringing, give me a minute. What? No, I'm giving a quote. Yes, thank you. Right, right, right. Yes. Microsoft. Very naughty blighters indeed. I shall immediately...I'm sorry, what was the question? Don't we do the caption round at this point? Gosh."

Re:We need to educate the decision makers

[Brown]

What is an MP ?

A Member of Parliament, i.e. a member of the UK's primary legistlative body. Each represents an individual constituancy (area), and the government is formed by senior MPs of the party which has a majorty in Parliament (usually).

-Chris

Re:We need to educate the decision makers

[foxprorawks]

Being a member of the opposition, Boris was never in the cabinet to begin with.

Re:We need to educate the decision makers

[ChrisLambrou]

It's short for Member of Parliament.

Re:We need to educate the decision makers

[REBloomfield]

i love Boris on "have i got news for you". he's great.... floppy haired twit....

Re:We need to educate the decision makers

[chary]

Certainly a character...

Re:We need to educate the decision makers

[Anonymous Coward]

With EDS's current track record I think they'd be perfect for running an Identity Card Scheme....

It will be overbudget, later than planed and quite likely more unworkable than can be hoped for.

Re:We need to educate the decision makers

[guidemaker]

If Fax Your MP doesn't work out, how about a more traditional way to influence your government? Fix Your MP

Re:We need to educate the decision makers

[mishmash]

"Where to have the debate where it might be read by those who mater:" And you lead with Boris?!

Yes. He's taking a stance on the ID card issue with his column in today's telegraph, entitled Ask to see my ID card and I'll eat it and has a discussion on his blog on the ID card issue

Is there another MP who's taken a clearer anti-ID card stance, and is prepared to discuss their positon so openly?

Re:We need to educate the decision makers

[ErroneousBee]

EDS were also the ones responsible for Sainsburys and their IT debacle. £500 million and the shelves were empty.

Re:We need to educate the decision makers

[trewornan]

A lot of people see Borris as a potential party leader (including members of his party) and recently sacked or not, MPs are likely to listen to what he has to say.

Re:We need to educate the decision makers

[deaddrunk]

This of course is never mentioned, our media would rather kick up a storm of hate against poor people doing anything they can to get into our country.

Re:We need to educate the decision makers

[Anonymous Coward]

Anti-id card stance or not, Boris Johnson still pond scum.

Re:We need to educate the decision makers

[pjt33]

If you ask a question, then she is required to reply to you within a certain period of time - I can't recall offhand whether it's 2 or 4 weeks.

Re:We need to educate the decision makers

[gumpish]

Right... because we all know Linux, *BSD, Plan 9, BeOS, Mac OS X, Unix, et al never have problems.

Re:We need to educate the decision makers

[mpe]

With EDS's current track record I think they'd be perfect for running an Identity Card Scheme....

For EDS, paid for by EDS, though.

Re:We need to educate the decision makers

[Anonymous Coward]

No, we know they have problems.

For you to make this kind of statement shows that you have no experience maintaining a number of Windows desktops in a production environment.There are always problems upgrading to the next version of Windows! And most of them involve applications (in my humble experience).

On the other hand, I run several versions of Linux at home and there have been very few problems with applications when upgrading to a new version of Linux. Drivers and libraries, YES; applications, no. Some of the most ancient of applications run just fine under the latest version of Linux.

Unless and until we have more information on this failure that is about all that can be said about this particular problem.

Re:We need to educate the decision makers

[legirons]

"EDS are preparing themselves to manage the UK national identity database and identity card scheme"

I trust they added the "300% overbudget and 10 years late" into their cost-estimates, as is traditional with EDS projects?

Re:We need to educate the decision makers

[Peaceful_Patriot]

Windows 2000 is the only decent OS that MS has ever made. XP is not an 'upgrade'. It is an unstable, tempremental, buggy POS. W2K, however is surprisingly stable and everything works with it. What is the compelling reason to go from W2K to XP? I know some people just have to have the 'latest', but I really don't understand the infatuation many have with XP. I believe it will be considered the WinMe of its time.

Re:We need to educate the decision makers

[Anonymous Coward]

"Can I take the opportunity to point out that faxyourmp is for UK citizens ONLY and should only be used to fax your own MP. It is not for international write-ins or mass lobbying."

You're a spoilsport, you are.

Re:We need to educate the decision makers

[Anonymous Coward]

can't yous guyz talk dis over at da pub?

What? ... Iraq? Troop placement? ... yes, .. yes... I see... yes.... okay... er, sorry.

Re:We need to educate the decision makers

[Martin Blank]

Wanna swap media? Ours would be all over this kind of thing there, and we need some media who are willing to point out the illegal immigration problem here.

Re:We need to educate the decision makers

[fucksl4shd0t]

Damn, I thought it was short for Military Police.

Oh wait, what's the difference?

God save the queen! She ain't no human being...

Re:We need to educate the decision makers

[Martin Blank]

Security, actually, presuming they're moving to XP SP2. It's been interesting to watch the last few months of patches from Microsoft, and to see where they go in terms of operating systems. In brief:

• Windows 2000 SP4: 5
  
• Windows XP SP1: 7
  
• Windows XP SP2: 1

The one affecting XP SP2 was a cumulative IE update. Generally speaking, XP SP2 is a bit better than the older versions in terms of security. I found the report a couple of days ago that Microsoft will not be making Win2000 SP5 to be rather disappointing, though not entirely unexpected. I was looking forward to seeing all of the major supported projects updated with the new anti-buffer-overflow code.

Now, I will add that I know about the flaws that Secunia, et al, have reported in all versions of Windows, and I'm not saying Microsoft is better on security, and their patch management frankly sucks. However, there has to be something else going on here if more than a handful of PCs -- even as a fraction -- were damaged by this. My guess is that some driver or other universal customized aspect of the system doesn't play nicely with XP and caused the problem. If you had 60,000 Linux machines and sent out an update that caused a critical driver to crash and that caused a kernel panic, you'd have a similar situation. (Of course, you could also have a generic kernel in place to choose instead of the default just in case, but that's not in Microsoft's nature to provide.)

Re:We need to educate the decision makers

[Lozzer]

Borrris has just been sacked from the shadow cabinet. The ruling party didn't give a damn about what he thought while he was in it.

Re:We need to educate the decision makers

[waynelorentz]

Can I take the opportunity to point out that faxyourmp is for UK citizens ONLY and should only be used to fax your own MP. It is not for international write-ins or mass lobbying.

Why? If British people can be encouraged to interfere with the American political process, then why can't Americans do the same to the Brits?

Re:We need to educate the decision makers

[Randy Wang]

Because the Brits weren't encouraged to do so by their own parliament? Because the Brits got back what they probably deserved, in terms of a year's supply of red-hot spam and a dirty look? Because the Brits themselves actually had a reason to voice an opinion?

Re:We need to educate the decision makers

[Adrian.Challlinor]

> And so another buzz word is born....

Re:We need to educate the decision makers

[mikerich]

And so another buzz word is born....

I do hope so! Blunkettcards can also be called 'Your inflexible friend' and should marketed with the slogan 'I know what's in your pocket.'

Best wishes,
Mike.

Re:We need to educate the decision makers

[anum]

Oh, so close! And then you had to go and spoil it with that last one. Shame really.

Re:We need to educate the decision makers

[satans_advocate]

God save the queen! She ain't no human being...

There is no future ah, and England's dreaming ...

Re:We need to educate the decision makers

[satans_advocate]

This is one we should lobby our representatives on to ensure they don't do it at all.

Lucky for you, EDS won the contract, so you can be almost certain that the project will fail.

In AU, EDS have FUBARed every company/government department they have won contracts in.

On the downside, you can expect the project so spend twice as much as EDS says it will cost, so more taxpayer pounds down the crapper.

Re:We need to educate the decision makers

[julesh]

We don't mind American citizens making requests of our MPs, or even of our voters. This is reasonable, and those MPs and voters are entitled to put whatever significance they think appropriate on those requests.

However, faxyourmp.org.uk is a web site run with donated bandwidth and telephone call charges. It is unfair to the people who have donated these to use them for a purpose it was not originally intended for.

Clear?

Upgrade to XP from 2000, LOL!

[bluntos]

..and they thought this would work?
They must be double stupid, with knobs on.
I can't believe it!! They Upgraded, rofl..

Are you sure it's not

[FusionDragon2099]

a £500 million contract ON Microsoft?

Re:Are you sure it's not

[Anonymous Coward]

Why? We have the SAS to do that for free.

Re:TCO costs rise scarily with Windows XP failures

[blastedtokyo]

Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software. It's like saying that a prison guard intending to open one gate to let someone out accidentally opened all of the gates and then they blamed the door manufacturer.

EDS screwup

[ebcdic]

It appears to have been a mistake by EDS (formerly Ross perot's company), so presumably they will be paying the bills. But no doubt they will price future contracts to compensate for their incompetence.

Ah. I've got the picture now...

[erroneus]

I'm guessing that they were attempting to use some fancy MS too to perform some automatic upgrades/updates and while they intended for a limited number of machines to recieve the updates, they went and installed on all the machines that used the service... just a guess mind you but it would fit the circumstances as layed out in the articles. And I would suspect when you add WinXP components to a Win2k installation that bad things would happen.

As much as I would like it to be, it doesn't seem like a "Microsoft" problem exactly and were a parallel Linux situation have happened I'm not sure anything less would have happened... well I guess it would have to depend on a number of things -- for example, if it were an RPM-using distro on the desktops and the wrong RPMs were sent to ALL machines instead of the select few, the machines for which the upgrades were unsuitable would have simply failed due to dependencies unless the --force option were used... okay I'm rambling now but basically, I don't see it as a Microsoft problem as much as I see it as a misuse of tools.

The TCO of their MS installation just went up though... and they shouldn't exclude the cost of firing, hiring and retraining either.

Knoppix

[Anonymous Coward]

No problem, Just get all the individual users to stick in a Knoppix cd. Remake the network, and let the main root look at all the individual cruft. Then burn a complete backup of all the individual important info. With Windows utilities they would need to fly in 200 WinIT guys to do the same thing. They have not got a hope in hell of doing the same thing on line with a Windows boot CD!

Shouldn't the patch installer stop this?

[januschr]

Of course EDS might just have copied out the relevant DLL's, registry settings and whatnot, but if the rollout happened through a preconfigured installer, shouldn't the patch installer detect that it was erroneusly being installed on the wrong OS?

Last heard at EDS...

[phaln]

"Someone set us up the bomb!" (...5 minutes later...) "You have no chance to survive! Make your time."

I wonder if theyre regretting that 500 pound c....

[djfray]

No, I don't think they are. I think they're regretting not updating those windows boxes. The exact same situation could have happened were they outdated macs or linux machines.

Not NECESSARILY Microsoft's Fault

[Staplerh]

Come now, people, if there was an inherent flaw of this magnitude in the Windows XP upgrade, this would be more widespread.

Windows XP could be a contributing factor, but NOT necessarily the causation. It's hard to speculate on a matter like this, but if I had to put money down I'd put it on shoddy IT work. Microsoft's an easy excuse.

This opinion might be unpopular, but the anti-Microsoft groupthink should be challenged from time to time. Did anybody else entertain this idea or is it just me - I'm not in the IT industry, so this could be more widespread than the article would lead me to believe.

Re:Not NECESSARILY Microsoft's Fault

[stratjakt]

If you automatically roll out an update that kills 60,000 machines, it seems to me you should be fired.

This is slashdot though, and the submitter and editors didn't really read the article. They're just keen on blaming MS for something.

I'd love to see 60,000 linux machines be automatically upgraded to a new major kernel version , or a new major samba version, or OpenLDAP, etc, without issue.

Re:Not NECESSARILY Microsoft's Fault

[fuzzy12345]

Lots of folks are saying it's unfair to blame the OS vendor if it's a combination of human error and a (possibly third party) upgrade manager.

I disagree. A reasonably competent UNIX sysadmin can himself write scripts that will automate an upgrade task across n machines. So he understands the theory, is less likely to be snowed by the desktop management system vendor, and feels more in control during the upgrade process.

Microsoft Windows machines have never had integrated scriptable administration systems, nor has it been easy to script one up, leaving IT departments at the mercy of SMS or a similar product. Mass upgrades typically look nothing like individual upgrades, as the latter is usually accomplished via the GUI, so the sysadmin doesn't even have a feel for what the product is doing.

Re:Not NECESSARILY Microsoft's Fault

[Anonymous Coward]

hahaha, you're blaming this on the lack of scripting in windows??

You're probably the gayest and most moronic geek sys admin, ever.

Imagine if it allowed scripting, do you REALLY think morons like you will be more clued than when using a simple GUI to perform an upgrade?!
And yes, a complete log of the upgrade process is kept if you wanted to know exactly what happened and what failed, etc.

Re:Not NECESSARILY Microsoft's Fault

[mattyrobinson69]

well i know that a new kernel version would be easy enough as long as all the machines were x86, just compile a custom kernel with generic optimisations, compile everything which could possibliy be used and use udev/coldplug on all the machines, that way the device nodes would be made on boot, from modules loaded as needed.

*im not claiming to work in IT, or have knowledge of networks bigger than 5 computers (home) but i do know this is possible with a kernel.

Plus, using debian (im not a debian nut, i use gentoo myself), an upgrade wouldn't be neccesary, ever - just use apt-get as a cron-job to keep all the desktops nice and secure.

Hey! let's be fair here, ok?

[orasio]

Microsoft sells itself as easy to administer, what in management terms means that the systems are so /user friendly/ that any moron can administer them.
So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS.

Aside from that, a point-and-click update cannot fail so miserably. A script made by the admin, of course should, because you can assume that someone smart (and bold) enoguh to make a little script should be responsible for their decisions. Some guy clicking checkboxes shouldn't be allowed by those means to break 60000 computers, through a /user friendly/ GUI program.
GUIs for dummies should have enough checks to prevent such underiable effects, they have a sufficiently constrained domain to be able to do so. If the guy wanted to do a legal task that the tools dosnt' allow, he could always write some Visual Basic Script, and then he would be on his own. Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.

Re:Hey! let's be fair here, ok?

[Mick Ohrberg]

Hold up.

Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.

So is it GM's fault if someone crashes and kills themselves (and others) by pushing the gas instead of the brake pedal by mistake?

Re:Hey! let's be fair here, ok?

[OwlWhacker]

So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS

It's interesting that you mention TCO.

There are so many companies that support/administer Windows systems, and Microsoft uses this as a positive reason to use its solutions.

The trouble is, many of these companies are extremely incompetent. I know, I deal with some of them. They charge 100% more than their services are actually worth.

Isn't the liklihood that you'll place your reliance on an incompetent company higher if you use Windows?

Surely this should be taken into consideration in the TCO studies?

Re:Hey! let's be fair here, ok?

[Anonymous Coward]

They charge 100% more than their services are actually worth.

They just double charge?? Not more?

Re:Hey! let's be fair here, ok?

[Skuld-Chan]

Are you saying that Linux is so much easier that it would have prevented me from rolling out a patch that was made for Yggdrasil Linux back in the early 90's over a network running Redhat 7? It doesn't care - it will let me do anything I like because there is no one true standard for deploying patches on linux over a network (I can think of 3-4 methods alone).

In my experience when you try to run XP patches on any other OS other than XP it complains - my guess is someone scripted it in such a way that it ignored all the safeguards so the user wouldn't know if there were problems.

If you incist on patching 60,000 machines at a time you test, retest, test again, and then deploy on a small subset of public machines and then go for it - no matter what OS you are running. 60,000 machines is a lot of machines to fix no matter what so it only makes sense you would make 100% sure that it would work on most peoples machines to keep support calls to an absolute minimum - that should be the #1 goal no matter what you're implimenting on any software package/os.

Re:Hey! let's be fair here, ok?

[Anonymous Coward]

^ Standard Slashbot justification. However, in order to buy this, you basically have to conceed the TCO argument to Microsoft and their partners. At least in the 99% case when everything works.

(Where are the system management tools that allow one to push patches out to 60K Linux desktops with a button? No, bash doesn't count.)

Re:Hey! let's be fair here, ok?

[cbiltcliffe]

Of course not...that's Ford's fault! :)

In all seriousness, though, no car manufacturer makes statements claiming their cars are so easy to use, a trained monkey with a bald head could do it.

Microsoft has a tendency to say they're cheaper, for this very reason.

Re:Hey! let's be fair here, ok?

[cbiltcliffe]

(Where are the system management tools that allow one to push patches out to 60K Linux desktops with a button? No, bash doesn't count.)

Novell Zenworks.

Re:Hey! let's be fair here, ok?

[orasio]

I was talking about how dumb admin problems are caused by management buying the TCO argument. I didn't say whether it was valid.

I don't know that 99% case when everything works, because back in the day, when I found that 1%, it was too much for me, so I got rid of MS software. GNU rulz.

Re:Hey! let's be fair here, ok?

[roemcke]

(Where are the system management tools that allow one to push patches out to 60K Linux desktops with a button? No, bash doesn't count.)

mount -t nfs my.file.server.com:/client-root/ /

Re:Hey! let's be fair here, ok?

[zx75]

And if the application was coded with multiple layers of passwords and a physical key lock and the guy still screwed up? How far are you going to take it before its no longer the application's fault? You know the saying about God inventing a better idiot.

The simple fact is that there is ALWAYS some way to do something that you didn't want, because there are times when you DO want to do that. For example, as soon as the patch checked out fine on the 7 computers maybe they were going to do EXACTLY what happened to them.

There is a level of common sence of the application not fking things up on purpose, but at some point you need to trust the user will make the correct choice that they do want to do something. In this case you cannot blame the 'guy who provided the checkboxes', the admins very simply completely fked it up.

Re:Hey! let's be fair here, ok?

[Anonymous Coward]

Interesting, but its only been out for a couple months, so I wonder how many large shops are using it (or even if there is such a thing as a 60K desktop Linux deploy).

Re:Hey! let's be fair here, ok?

[TheLink]

I believe they OEM from PatchLink. And PatchLink also does Windows software. The Linux version costs more, and I don't know if it's as advanced or as useful - there really isn't a popular "standardized linux desktop", so mostly Linux in big corps are on servers and I dunno if most would want to (or should) automatically roll out patches onto their servers (most sites aren't like google).

I wonder what EDS used to screw up so badly. You'd have to take extra effort to screw up so badly on PatchLink Update for windows. It's possible to screw up and put _compatible_ patches on 60K machines prematurely - e.g. deploy on the wrong group, but EDS seems to have put _incompatible_ patches on 60K machines. Big diff between "oops boss, I updated all the machines too early, but 99% are working fine after the update - the rest have viruses or something interacting with the update" vs "oops boss, I screwed up 60K machines".

Still, if I were doing 60K Linux desktops I'd do things a lot differently - it'd be something like booting off knoppix terminal server from servers at various sites - there will still be different sites (otherwise the network load would be crazy, and each site has different needs).

So update the client image on a site, and the clients will get the new image.

My poor Gran,

[Anonymous Coward]

I wonder how she is going to feel when she goes to pick up her pension next and be told "computer says no".

HA-HA!

[Anonymous Coward]

HA-HA!

who's fault

[cshah 1]

Well they shouldnt have even thought about upgrading to windows xp. The only person in trouble is the retarded IT guy. Or microsoft for providing support for OS upgrades.

Real men don't make backups..

[klang]

..real men cries a lot!

Seriously, you never "upgrade" a computer (let alone a lot) without knowing what will happen ..

At my company, they started upgrating on a limited number of machines, corrected a few problems and continued in small steps to upgrade the rest of the (12.000) machines during the next few weeks..

but, different strokes for different folks :-)

Re:Testimony to power of insufficient testing

[Tony Hoyle]

If it had the potential to 'go wrong' on that scale then it was *not* a sandbox.

why would w2k accept an XP patch at all?

[oogoody]

> rolling an XP patch out onto a Windows
> 2000 machine

Wouldn't it notice?

Re:why would w2k accept an XP patch at all?

[Ex-MislTech]

This was my question as well, they are so big on signed drivers,
and certificates, and they cannot tell patch version from OS version .

Sounds a little short sighted to me .

I have seen lots of software detect OS version and declare it
incompatible, I think they could have done like wise .

Granted EDS should know what they are doing, but M$ should not
ASSuME ppl will know what they are doing and put in at least a cursory
safety measure .

When you ASSuME you make an ASS out of U and ME .

Ex-MislTech

Re:why would w2k accept an XP patch at all?

[LanceUppercut]

What they probably menat is that they were tryng to install the upgrade version of XP over 2000. Although neither of the articles say that explicitly. The tone and the format of the Slashdot synopsys (like the reference to the BSOD) suggest that it was written by some MS-hater-lamer who simply forgot to have his juice in the moring and was pulling the whole thing out of his a$$.

Re:TCO costs rise scarily with Windows XP failures

[speed-sf]

Something that makes me curious, you hear Ballmer lament about the lower TCO of windows. You hear the linux community shriek about it's lower TCO. The bottom line is really this, if your sysAdmins are less than competent and bugger up something like this which system would have a lower cost to recover? This is a really good thing to know when you are considering any enterprise system. Call it, TCCR (total cost of catastrophic recovery). Ballmer, Linux communities answer me this!

Not a nail for Microsoft-Obese clients.

[Anonymous Coward]

*Pats DrSkwid on the head*

Now you know why there's such a push for thin clients and rich interfaces. Although all it would have bought them in this case is the ability to do a quick fix.

Re:Not a nail for Microsoft-Obese clients.

[Anonymous Coward]

Think Client is the way to go if your 60,000 desktops all do similar things and don't need a wealth of different applications. The DWP is a bit like 1000 companies all doing different things.

BUT you do need something for Laptops - Terminals and Web browsers don't work very well in planes, trains and automobiles.

Adam

The Blame Game

[Anonymous Coward]

This seems more like incompetent IT management versus anything else. Can't really blame Microsoft if someone else presses the wrong buttons.

This reminds me of the IT fiasco at Beth-Israel hospital in Boston. Although they lost their network for 4 days, the problem was pinned a researcher in a research lab and/or a poorly implemented Cisco router algorithm.

Regardless of the truth of that explanation, the fact remained that the IT management should have had safeguards in place so that no major cascading failure can happen. Failures should be expected, but isolated.

When the seat back tray of a 747 fails, the plane does not crash.

possibly intentional?

[Larmal]

considering 40,000 jobs were going to be axed thanks to the saving grace of IT, is it possible that this enormous crunch was intentional? Prove that if IT craps out those 40,000 jobs become invaluable?

Re:possibly intentional?

[LanceUppercut]

It could be intentional in a different way. SOme pro-Linux admin could have sabotaged the whole thing. After the death of desktop Linux things like this are no so uncommon. I often run into similar things at the place where I work.

Re:Testimony to power of insufficient testing

[Anonymous Coward]

Simple, it was the sandbox they were roling out too. Its just they hit the rong button and tried to update all the win 2000 mchines instead of the win XP ones in the sandbox.
Now then, it dosnt surprise me a bit.

Why do goverments allways go for the cheepest contracts even if there trach wecord is 0% sucsess rate?

Pr0n

[Anonymous Coward]

At least it will stop them accessing pr0n.

http://www.itsecurity.com/soapbox/corbelli1.htm

All systems are prone to failure

[slashhax0r]

Problem isn't the platform as much as the implementation. I'd say that someone bollocksed the whole thing up, which could be just as tragic rolling out a linux upgrade or whatever.

We've got to educate the people spending our money on large computer systems to spend part of that money on more testing!

Re:All systems are prone to failure

[jacksonj04]

I was just thinking the same thing. This isn't Microsoft that ballsed it up, nor is it inherently the fault of DWP. Chances are it's an underpaid sysadmin somewhere who hit the wrong checkbox when rolling out the patch.

Re:All systems are prone to failure

[mikechant]

One question that arises though is whether if they'd been running non-MS software is it so likely that the machines would have ended up so fucked up that they couldn't boot at all? Because that's what has made this into a real disaster - if the machines had been bootable with network access (even if applications etc were broken) they could apparently have undone the rollout centrally. As it is, it looks like they had to restore each PC manually one at a time.

Re:All systems are prone to failure

[LO0G]

That's because they had the equivilant of a linux 2.4 kernel running on a 1.7 distro.

You can bollux up ANY operating system so it can't boot if you work hard enough.

Re:All systems are prone to failure

[jacksonj04]

Chances are - yes. The bits in linux designed to keep it working are of the same idea as in MS software, and whilst it would be difficult to make them BSOD it would be quite possible to roll out an incompatible patch and knock out the networking.

Re:All systems are prone to failure

[Anonymous Coward]

Why do you think it is "difficult" to kernel panic a Linux box? Just force load an incompatible module and there you are.

Re:All systems are prone to failure

[mpe]

Problem isn't the platform as much as the implementation.

It looks like a problem with SUS.

I'd say that someone bollocksed the whole thing up, which could be just as tragic rolling out a linux upgrade or whatever.

If you implimented something similar on Linux. Except that it's a lot harder to put Linux into an unbootable state which is difficult to recover from and less likely that a Linux workstation will vital locally stored data which means you can't simply wipe and reimage.

We've got to educate the people spending our money on large computer systems to spend part of that money on more testing!

There were "testing" (best keep those involved away from any Ukranian nuclear reactors). I suspect that the problem is more with a lack of understanding of the basics.

Re:All systems are prone to failure

[mpe]

This isn't Microsoft that ballsed it up, nor is it inherently the fault of DWP. Chances are it's an underpaid sysadmin somewhere who hit the wrong checkbox when rolling out the patch.

If someone can manage this by selecting the "wrong checkbox" then the system is broken by design.
Microsoft sell a complex system with the claim idiots can administer it. The DWP employ/contract idiots to administer a complex, but vital, system. Niether of these are "innocent parties".

Re:All systems are prone to failure

[jacksonj04]

difficult to make them BSOD as in they don't actually go blue... not get the kernel to fall over.

Good point showing how easy it is though.

Re:All systems are prone to failure

[killjoe]

Right. Whenever they calculate TCO MS always claims you don't need "expensive sysadmins".

Re:All systems are prone to failure

[Sebastian Jansson]

Microsoft sell a complex system with the claim idiots can administer it. The DWP employ/contract idiots to administer a complex, but vital, system. Niether of these are "innocent parties".
Oh, I must have missed just that commercial...
In all *nix systems I've seen, the root user has the ability to run "rm -rf /*" and by that completly destroy their system. You can't expect an system to be completly idiot-proof at administration level imo.
Although I agree on that the design of the particular infrastructure seems a bit unsecure when you that easily can break 60000 systems. What if some bored hacker gained access to that main account...

Re:All systems are prone to failure

[Total_Wimp]

This is bullshit. You can wipe out a server hard drive with less than 10 characters and an "enter" on most Unix/Linux systems. But you want to blame Microsoft for the UK government not properly running their IT staff enough to make sure they're pushing the patch to a test group instead the production group?

Microsoft builds very powerful software. Their Active Directory and Group Policy can do amazing things over huge numbers of computers. But if the guy in charge gives admin rights over the directory to some monkey that doesn't know how it works it's no different than giving root to the same monkey on your big Unix file server.

I'm quite familiar with the Microsoft recommendations for big projects like this. Claiming Microsoft would "sell a complex system [of this magnitude] with the claim idiots can administer it" is flat out false. They recommend highly trained proffesionals, testing in an offline environment and strict change control procedures. Don't let your personal prejudice cloud the facts: MS sells big systems to big customers with the understanding and recommendation that they'll have tallented staff to run them.... Just like every Unix house does.

TW

Re:All systems are prone to failure

[Anonymous Coward]

Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.

The only way I could see this being in any way Microsoft's fault is that the patch should have checked the current version of Windows and denied installation on non-XP PCs.

Still, it seems like one of the sysadmins fucked up and accidentally deployed an XP patch to the entire network, including all the 2000 workstations. That is either stupidity on the part of the local systems admin staff or EDS.

Either way, they should have had a backup system...obviously they didn't have a contingency plan for quickly recovering from a disaster.

Shit happens. When it happens, you just deal with it.

Re:All systems are prone to failure

[mpe]

You can wipe out a server hard drive with less than 10 characters and an "enter" on most Unix/Linux systems.

One machine vs tens of thousands.

I'm quite familiar with the Microsoft recommendations for big projects like this. Claiming Microsoft would "sell a complex system [of this magnitude] with the claim idiots can administer it" is flat out false. They recommend highly trained proffesionals, testing in an offline environment and strict change control procedures.

"Recommend" means just that. Also what definition is Microsoft using for "highly trained professionals"? Do they mean MSCE (or similar).

Re:All systems are prone to failure

[Anonymous Coward]

"This is bullshit. You can wipe out a server hard drive with less than 10 characters and an "enter" on most Unix/Linux systems. But you want to blame Microsoft for the UK government not properly running their IT staff enough to make sure they're pushing the patch to a test group instead the production group?"

9 characters to do the same thing in windows. So what's your fucking point? Have you heard any stories recently of 60000 linux/unix workstations being blown away in a similar manner?

Don't be a total wimp.

Re:All systems are prone to failure

[jacksonj04]

Actually, they do. The emergency payments system made sure the really, really essential ones got through regardless and the backlog is being dealt with, albeit slowly.

Re:All systems are prone to failure

[jacksonj04]

Don't be a total wimp.

Says he of the AC posting.

Re:TCO costs rise scarily with Windows XP failures

[ultranova]

Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software.

Yes. It's not like the upgrade could detect the version of the program it's being applied to, and only install if the version matches the version it is intended for. That is completely unheard of, and would be impossible technically.

This was sarcasm, FYI.

It's like saying that a prison guard intending to open one gate to let someone out accidentally opened all of the gates and then they blamed the door manufacturer.

This situation is more analogous to a wrong signal causing the door to open and then jam. And yes, such a door manufacturer deserves to be blamed.

Re:TCO costs rise scarily with Windows XP failures

[hattig]

Well, maybe the XP update when pushed to a machine that it isn't suited to shouldn't do its stuff anyway?

It is a software issue, if the software was written correctly to handle the assumption that there are a lot of incompetant sysadmins out there, this wouldn't have happened! Sanity checks are a standard feature of software design.

Maybe it was the update software that EDS use that caused the problem. It clearly doesn't have a dumbuser compliant interface. In that case, it is the TCO of that software that has gone up horribly!

A day without your desktop computer - fine. A week isn't. This smells like EDS fucking up, and I think us UK geeks should write to our MPs and ask them why EDS keep getting contracts with their track record of fuck-ups and cost overruns. That doesn't mean I don't think that an update that is meant for a different version of the OS should be able to run on non-suitable systems.

They forgot the sticker

[sydtsai]

Did they change the Design for win2k sticker to the xp one?
This could be the problem.

Bill Gates says ....

[Ex-MislTech]

Whose your daddy ????

LOL !

[Anonymous Coward]

LOL Serves 'em right for using M$ - monoculture anyone?

"Blown out of proportion"

[DogDude]

The only real news article here, from the BBC, says that this was "blown out of proportion". I agree. And Slashdot is part of the problem. This is a non-story.

ID

[Anonymous Coward]

Hopefully this will happen to the UK's national ID card system as well.

Really minor quibble with article title

[Anonymous Coward]

The Department for Work and Pensions is a Department, not an Agency. "Agency" in the UK generally means an Executive Agency (eg. Highways Agency, Vehicle Operator and Services Agency) or Non-Departmental Public Body, rather than a fully-fledged ministerial Department, like DWP. I'm probably the only person reading this thread who is ever going to care about this distinction, but I'm going to speak up anyway.

A little history...

[ixpro]

With a service history like this:

http://www.theregister.co.uk/2004/04/26/child_supp ort_agency_it_failure/
http://www.theregister.co.uk/2003/12/11/inland_rev enue_sacks_eds/
http://www.theregister.co.uk/2004/05/04/eds_mulls_ abbey_offshore/

How do these guys win new contracts from major companies? Amazing, truly amazing... I interviewed there once, got an offer, but that very night when I was thinking about taking the job, I had a pentagram stigmata burns appear on my back! It took 3 months of holy water baths to get it off...

Re:Local Government Mod parent up!

[Anonymous Coward]

Here Here....My wife works in a similar environment, to be precise a Hospital. To "UPGRADE" to a newer working environment requires gobs and gobs of retraining.

It has the entire staff (that has been there the longest and knows networking) wishing they could just trash the MS button powered cruft and go back to a good old simple and faster vax unix system. The simple user friendly Unix guis are still yearned for by those who are in the know! Where do you want to go today Has become a slogan for suckers and foolish IT admins.

From the article

[pekoe]

all the clients connected to the network received a partial, but fatal, 'upgrade.'

Is it possible that someone noticed that the updates were going to 60,000 machines instead of just 7, said 'oh shit', and pulled the plug without thinking?

EDS is also thought to be flying in fire brigades.

Yeah, to put out the fires from their smouldering backsides.

EDS Acronym

[Ex-MislTech]

Extremely Dumb Solutions

HAHAHAHAHA...

[greatscot]

... that's what you get for using Windows. When will people learn?

Re:TCO costs rise scarily with Windows XP failures

[vrai]

The public sector in the UK is nothing more than unemployment benefit for the middle classes.

Are you too stupid to be allowed near a productive company? But not pikey enough to go on the dole?

If so then we've got the job for you!

Work in an industry whose sole measure of productivity is how much of someone else's money you've spent! Get at least 30 days paid holiday year, plus all the paid sick leave you can lie about! Use state of the art technology (subject to availability/EDS profits)! Do absolutely nothing of value for forty years and get a pension that most CEOs would envy!

Join the British Public Sector! The largest, fastest growing employer in the Western world!

Member of Parliament

[dominux]

you can call them senators if it makes you feel better.

Re:Member of Parliament

[DebtAngel]

Don't do that, it'll only confuse the Canadians.

Re:Member of Parliament

[Anonymous Coward]

News Flash: We learned what a US senator was in grade 10. We also learned about the governments of many other countries in the world. It was a mandatory class.

Re:Member of Parliament

[The Original Yama]

Actually, you shouldn't. In Westminster-style governments, a senator is specifically a member of the Upper House of parliament. In the UK the House of Lords serves this function (nobody is called "senator" in the UK). Members of Parliament (MPs) can be either from the Upper or Lower House.

Installation

[Anonymous Coward]

If it is so easy to install patches on the wrong computer, how come so many games say "you have the wrong version of Windows"? Why is it when using a dos 7 bootdisk to load Windows 95 comes back with "Wrong version of dos. Need version >=6.1"?

Who's Fuzzy Math?

[Morbid_Angel666]

"attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines." [Emphasis mine] Is this /. math where 7 = 60,000 ? Or if this was what came from the UK as to what happened ... well although I don't have any MS logos tatooed on me but maybe the problem in their case wasn't really MS but with the implimentors ...

Re:Who's Fuzzy Math?

[mlk]

Some admin tools let you roll out upgrades, so you say "PCs workstations_test1...7, install this", but some silly admin said "PCs install this". And they did, and it fucked up.

Come on!

[Jugalator]

Jeez, sometimes Slashdot readers are blind and zealous like headless chickens...

1. The patch they tried to update with wasn't a complete one for an OS upgrade.
2. Then they deployed it to their entire network by mistake.

This interesting piece of information can be gathered by RTFA.

I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch?
Maybe some... thing... would panic?

Re:Come on!

[MikeBabcock]

... and they'd reboot and select the last-known good kernel from grub and be all better again, because a kernel upgrade doesn't kill the system.

My main complaint about Microsoft patches is that I can't (easily) convince the system to store each patched version seperately.

I have /usr/local/openssl-0.9.7a, /usr/local/openssl-0.9.7c, /usr/local/openssl-0.9.7d, etc. on my system. Why? If a new version kills things, I revert easily by changing a symlink and retest.

Yay for versions.

Re:Come on!

[charlie_vernacular]

Shouldn't the computers running Win2K have simply rejected the upgrade?

Re:Come on!

[Cyno]

Or maybe Linux would verify the patch before applying..

Re:Come on!

[Anonymous Coward]

No. You ever did a Fedora kernel upgrade? You download and apply the kernel RPM, reboot, and ***WHOA*** - there's a new kernel in the GRUB list.

You'd have to screw with yum a lot to get it to do something like that. Or fuck up a roll-your-own RPM (did that myself once - thank God for backups)

Re:Come on!

[mattyrobinson69]

you'd botch the source code, which isn't a problem as it isn't used unless its being compiled (if it was even on the machine in the first place). The bzImage for a broken kernel (unless it was broken in the sense of no ext2 support for example) would not compile, so there would be no botched image to roll out.

apt-get?

Re:Come on!

[Tough Love]

I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch? Maybe some... thing... would panic?

No, the patch would fail. And by the way, that's not how it's done, the new kernel is compiled (and tested!) in a controlled environment, and only the binary is downloaded. Plus, the old kernel stays around so that if there's a problem, you can go straight back to the kernel that works.

Next question?

Re:Come on!

[TheLink]

Don't you remember the good old days of "oops forgot to run lilo"?

That's why I switched to grub for Linux systems.

I use FreeBSD at home though, and it sure seems a bit simpler to manage. Maybe I should try Gentoo one of these days, but looks like RedHat and Suse are the ones to use from the "officially supported" by 3rd party software point of view.

Perhaps I'm just missing something here....

[emrysk]

...but is there any actual evidence is was a Microsoft error? I like bashing Windows as much as the next guy, but it seems this is at least as likely to be a huge fumble by the admins.

Re:Perhaps I'm just missing something here....

[LanceUppercut]

Well, for any a typical lamerish pro-Linux BS site, like Slashdot, for example, everything is a Microsoft error. Learn to live with it of stop reading.

Re:Perhaps I'm just missing something here....

[zx75]

I don't see how it could be an MS error, perhaps the interface for the app they were using to patch could have been clearer (?) but what they essentially did was overwrite a bunch of OS files on tens of thousands of computers that had no real connection to the OS that was running.

If I were tech support, this is one of those times that I say PIBKAC, a clueless (l)User. Its like taking a hex editor and changing bytes in an executable, and expecting it to still work because the programmers ought to have accounted for it.

Re:Perhaps I'm just missing something here....

[zx75]

Yes, but it wasn't a patch that they were running. They were running an application that pushed 'patched' files out to networked computers, and they pushed those patched files out to computers that weren't even running the right operating system, which was caused the crash.

If it were truely running a patch on each system, an SP2 Windows XP patch will recognize the fact that a computer is running Win2K or NT and refuse to execute.

Re:TCO costs rise scarily with Windows XP failures

[jimicus]

This is an interesting question. Most large companies have at least a few gloriously incompetent people who really should have gone long ago but for whatever reason haven't.

However, I don't know any reports which consider Total Cost of Ownership Assuming Your IT Department Is A Bunch of Blathering Idiots. Most seem to assume a certain degree of competence.

NIC driver

[Ex-MislTech]

Ohhh when I read this I just died laughing remembering the
deployment for conoco where the damn Windows 2000 rollout could not
see some of the Intel NIC's by default .

I mean really, M$ not considering all Intel NIC's for default
drivers, how stupid can you get .

HAHAHAHAHAHA

Ex-MislTech

Wrong!-Witch way?

[Anonymous Coward]

"According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place"

"Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these"

The two DON't agree. The first is going from one OS version (W2K) to another (XP) (a significant step) The second is patching a preexisting XP installation (not as significant). Yes the slashdot summary is incorrect, but the above doesn't clear the issue up either. Either way there should have been a version check somewere. They were either doing one, or the other. Doubtfully BOTH.

"In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect."

It would be easier to recover from than say Windows.

Re:Wrong!-Witch way?

[tricops]

Uh, it makes sense to me. The first thing you'd want to do after upgrading to XP would be install the service packs. They probably upgraded those seven machines to XP then accidently rolled the SP2 update out to everything rather than just those seven. That's the way it reads to me....

Stupid Admin?

[Anonymous Coward]

When someone logs in to the console as root and reads all their mail real fast (rm /* -rf), we say "stupid admin." But when the same stupid admin does something similarly stupid on Windows, there's something wrong with Microsoft.

This is just your normal, everyday, "I hate Microsoft" propoganda...

Tell me about it.

[grundie]

I've had my mother moaning to me all week because her pension has ben delayed. She decided now was the time to have her pension paid in to her bank account instead of being paid by order book. She only decided to do this because the DSS were bugging her to do so as they are doing away with order books. I think EDS and the government should be apolgising to all the people who will undoubtably be in the same predicament as me ma.

Texas again

[Anonymous Coward]

EDS is from Texas. Enron too. Texas is full of the cult of the fast buck, the ethically challenged social darwinists who are good on selling and who you can never trust. Corrupt corporations, corrupt politicians (Tom DeLay the jerrymandering slimebucket, W the...well, you know). Any government trusting a Texas business gets what they deserve.

Re:TCO costs rise scarily with Windows XP failures

[gruhnj]

Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software.

This sounds like they were pushing out the upgrade via SMS. Checking that the upgrade was on an appropriate system here would not have mattered since the upgrade path from win2k to WinXP is legitimate. This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container. The real question here is why are so many systems under one system and even better why did the sysadmins who did this application not check to ensure the advertisement was sent to the proper container.

EDS takes the blame for this, not MS.

Keyboard Infantry since 2002

GPOs

[Anonymous Coward]

Well, assuming you've followed recommendations, testing a population of each of the flavors an organization is using, Microsoft makes this pretty convienet with GPOs. But because they let you smoothly blast all kinds of software, OS Updates, to applications, out to clients, with or without user intervention, they do advise a certain amount of double checking. Given how simple this kind of thing is to do, I can't understand how they made this mistake.

It's not bad, though I've never done 60,000 yikes, if you're starting where Microsofts assumptions meet reality. Perfect knowledge of your network inventory, and having checked that the package you're pushing out actually works.

World Domination.

[Anonymous Coward]

"They emigrated, most likely. One of the problems with incompetence is that it's self-reinforcing, the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time."

We're all moving to the Hurd. Care to join us?

TOO MANY LINKS!

[Anonymous Coward]

Why can't we just have the link to the story of interest, rather than click through 8 irrelevant pages looking for it?

What risk ?

[Znort]

PR is the key not analysis. A few weeks of PR damage control and EDS will be ready to shudown (upgrade) another part of the country.

Re:TCO costs rise scarily with Windows XP failures

[speed-sf]

Yeah, I highly doubt those Total Cost of Ownership Assuming Your IT Department Is A Bunch of Blathering Idiots type of reports exist. Maybe this is a case point that they should? I am not being negative to SAs, I have met some talented ones and some less than talented ones. I feel bad for them, they have a huge amount of responsibility, lots of stress and not enough respect. Let's be honest, mistakes happen, it would be nice to know what the bottom line is if you completely screw the pooch. I bet the execs wouldn't mind know it either. It will cost X dollars if we completely screw up, let's assume it will happen once a year.

ur right and as for 2000 machines

[Sai Babu]

surely you've a database with address-config info.

Re:TCO costs rise scarily with Windows XP failures

[malkavian]

They're probably using something like Novadigms's Radia. And instead of linking the correct 7 PCs, they linked to all of them (misconfigured group). In that case, it's not a case if installing a patch that is installed using the new mechanisms, the "Patch Manager" simply dumps the files to all the machines that connect up using it's client, and force an overwrite.

Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.

Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.

So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...

Parody

[Anonymous Coward]

Now, who is going to do a parody of one of those MS advertisements...

"Mr. Screwed-up over 60,000 desktops in a few hours"

Re:Parody

[Anonymous Coward]

It wasn't MS's fault.

It was down to the sysadmins (EDS) and their home-grown update rollout system.

If they'd used MS update route then the W2K boxes would have rejected the patch.

Still. Never let the facts get in the way of your blind bigotry eh?

Unit of time

[dackroyd]

= The ohno second - That minuscule fraction of time in which you realize that you've just made a BIG mistake.

Re:Unit of time

[Rosonowski]

Yeah, but the ohno second only comes right as you hit the "Commit" button. It's a real feeling of dread, like "Why the fuck did I do that?"

Re:Unit of time

[Anonymous Coward]

You mean ignisecond

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

They sysadmins of linux systems flat out don't have the option of managing 60,000 PC's from one location like this, competent or not. And, as in all things, the more powerful something is, the more damage someone who doesn't know what they're doing can do. And knowing about what they were trying to do, having done it. I cannot believe they managed to make a mistake so colossal.

The closest thing linux has is cron, and it is nowhere near as powerful, and way more of a hassle. (Maybe enterprise linux systems have something better.) And if one did use cron for push software they shouldn't on to the wrong computers it would fail just as spectacularly.

Any decent Windows Admin should know

[b00m3rang]

Upgrades NEVER work! Not for Windows 95, 98, ME, 2000, XP, Longhorn, whatever! It will never be a good idea to try and replace a MS OS without doing a clean install.

This is first day stuff.

Re:Any decent Windows Admin should know

[Anonymous Coward]

I do regular upgrade of my Mandrake Linux PC since version 9.0 (I'm actually running v10.1). Never got a problem so far...maybe they should switch to Linux... :)

Re:Any decent Windows Admin should know

[Anonymous Coward]

Spoken like a Win9x user. Upgrades in the NT line (NT4, 2K, XP) work pretty well. Besides, this was a patch, not a OS upgrade.

Re:Any decent Windows Admin should know

[quarkscat]

Amen!

The last successful Windows upgrade that I did
was from NT4SP6 to 2KProSP1. Copious verified
backups to the server, several Vallium washed
down with a single malt scotch, and I was ready.
BTW, all my "serious" work is now done on either
GNU/linux, Mac OS X, Solaris, or OpenBSD -- I
wouldn't trust any MSFT OS product ever again.

I would not want to be the IT manager trying to
explain this cockup, but he/she has my sympathy.

Re:Any decent Windows Admin should know

[Peaceful_Patriot]

No, they were 'upgrading' to XP from W2K. According to the article:

"...a limited network upgrade from Windows 2000 to Windows XP was taking place..."

Re:Any decent Windows Admin should know

[flibuste]

I always get drunk before upgrading my Windows, so I know I can't think of all the bad consequences when I click "OK", and will not care when I get a BSOD.

Re:Any decent Windows Admin should know

[Anonymous Coward]

> I do regular upgrade of my Mandrake Linux PC since version 9.0 (I'm actually running v10.1). Never got a problem so far...maybe they should switch to Linux... :)

Welcome to Slashdot, where you can get free deployment advice based on anecdotal evidence of trends traversing a single major version change of a product!

Re:Any decent Windows Admin should know

[Deliveranc3]

Day two you learn it's impossible to get joe shmoe user to back up his stuff intelligently.

BEWARE PYRAMID SCHEMES

[Anonymous Coward]

Whenever you see some scumball like "Lev13than" here promoting a "free ipod", go hide your money because it is a pyramid scheme and you will end up on the losing side, guaranteed.

Re:BEWARE PYRAMID SCHEMES

[Not_Wiggins]

Whenever you see some scumball like "Lev13than" here promoting a "free ipod"

Ugh... hate to point that out, but "Lev13than" was complaining about MY link. That user wasn't promoting free ipod, but was instead complaining about it.

Re:BEWARE PYRAMID SCHEMES

[Anonymous Coward]

No one else will ever see this at this point, but I just wanted to thank you for being a stand-up member of the community on this one. Well done, sir.

Re:BEWARE PYRAMID SCHEMES

[Anonymous Coward]

I saw it. Sir, I salute you!

Mod story as troll?

[Anonymous Coward]

I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now."

Why would they? Microsoft isn't at fault here. EDS f*ed up.. which you'd know if you read your own links.

About 4 years ago, I predicted right here

[Anonymous Coward]

that there would be a national BSOD day as a result of a Windows upgrade causing a calamity. This isn't far off.

Unfortunately, given that Slashdot has by far the lamest search capabilities on the planet (shame!), I can't find the post in order to provide a link.

Re:TCO costs rise scarily with Windows XP failures

[ranolen]

This was sarcasm, FYI

Maybe you should realise that on slashdot you can't always relise when sarcasm is being used, especially when MS is involved.

Just wondering

[Anonymous Coward]

Did the update work on the 7 machines they intended to test it on?

Windows? Or EDS?

[reverendslappy]

Without any specific details on the failure or what exactly happened, it seems like this is a huge admin error. My guess is they're using something like Altiris to do their builds, and if an admin were to accidentally "drop" the package meant for the the test group on to the production group, wham-o... every PC starts installing a build that probably isn't meant for them, and won't work. And you can imagine how that would go.

As much as I'm sure the zealots among us would like to make this seem like a Windows failure, it looks like it's more of an example of how outsourcing leads to disconnected, incompetent, and unmotivated IT staff. And that, of course, leads to mishaps like this.

Either way, if you work for a company that brings EDS in house in any way, drop your shit and run. And don't look back. The flash could be blinding.

We need some British Government over here

[Bun]

From the Guardian:"The disruption is the latest in a line of government technology failures and follows last week's resignation of the head of the Child Support Agency, part of Mr Johnson's empire, after the disastrous introduction of an EDS system contributed to only one in eight parents receiving the correct amount."

Wait a minute... a government official resiged over a major failure in his department? I wish that would happen here...

Re:We need some British Government over here

[nagora]

a government official resiged over a major failure in his department?

Don't be silly. The person in question was a civil servent, there's no way anyone in actual government would resign over anything up to and including murder. Plus, he was simply moved to another department and never admitted any substantial fault, failure or responsibility.

In other words: business as usual.

TWW

Re:We need some British Government over here

[Bun]

Don't be silly. The person in question was a civil servent, there's no way anyone in actual government would resign over anything up to and including murder. Plus, he was simply moved to another department and never admitted any substantial fault, failure or responsibility.

Thank goodness. I was worried I was going to have to re-think my world view.

It does make me feel much happier about ID cards

[rpjs]

Because I can't see them ever getting the scheme to work...

Why is it MS's fault?

[Anonymous Coward]

XP is a tool that if not used correctly can be dangerous, just like a poor implemetation of Linux, BSD, etc. That's why companies hire IT Pros. I think this should be chalked up to someone not doing their prep work. What happened in the sandbox testing I wonder?

Linux Live

[Anonymous Coward]

dont have much experience with linux etc, but have been shown a demonstration of linux off a CD - i think they call it a linux live disk??

well, anyway the point is where wasa a bunch of enthusiasts to rock up at the "warzone" with a case of these disks and put microsoft to shame?

I'm sure they could have tailor made a disk with apps and config settings etc etc. A bulk burner and a couple hours and they could have all those machines up and running and being used productively again.

Wont microsoft have had a bit of egg on the face then?

My company quoted for an NHS contract once...

[Phil John]

...and didn't get it because we were "too cheap"!

No wonder the NHS is in dire straights at the moment financially. They probably gave the gig to a "consultancy company" who would end up charging them 500k+, selling them stuff they don't need and all this time farming it out to the lowest bidder and therefore producing shoddy goods that don't work and further tarnish the reputation of IT in the NHS.

Re:TCO costs rise scarily with Windows XP failures

[mikechant]

The public sector in the UK is nothing more than unemployment benefit for the middle classes.

In my experience (having worked for both) in terms of inefficieny and stupidity, there's only one thing worse than the British Public sector and that's the British Private sector.
My company used to be part of a large public sector concern and was sold off. Since then we seem to spend nearly of our time/money:

Changing company logo and name every 6-12 months
Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others).
Paying huge bonuses to upper managent.
Paying huge car allowances to middle management including those who refuse to drive.
Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items.
Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later.
Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.
Repeatedly shuffling kit from datacenter to datacenter around the country at vast expense and disruption to our customers.
Ordering expensive buffets for management meetings , 95%+ of which get thrown away.
Managers having a schedule involving meetings all over the country which means that they spend about 25 hours out of 40 driving.
Managers refusing to use video-conferencing for meetings even in the light of the above.

How many of these things happened when I was in the public sector? Virtually none. We didn't have the money to throw around on such things. We were forced to be efficient.

Also if this private sector company I'm referring to was atypically inefficient, presumably it would do so badly it would collapse or be taken over. So this implies that many private sector companies are like this.

It's very easy to slag off the public sector if you use stereotypes, generalizations and distortions.

But It's Cheaper!!

[Anonymous Coward]

"But it's cheaper" scream IT heads everywhere in unison. Of course for them, TCO means iniTial Cost of Ownership. But, hey, you get what you paid for, or in this case, less than what you paid for.

But then again, what do they care? They still have a job the next day. The money doesn't come from their pocket, it's only taxpayers' money.

Maybe something like...

[Anonymous Coward]

Hi, I'm Clippy, your assistant to keeping your Microsoft products up-to-date!

It looks like you're trying to upgrade your Windows XP system. Do you want to:

x Update all the systems on the network to the latest version of Microsoft Windows, and enjoy enhanced features such as advanced security capabilities
o Do a custom install (For advanced Administrators only)

[OK] [Cancel]

How Stuff Works

[PingPongBoy]

She added that the emergency payments system was "working perfectly."

Jones agreed, "I still have plenty of blank cheques. My pen is at room temperature."

Re:TCO costs rise scarily with Windows XP failures

[Tony-A]

Call it, TCCR (total cost of catastrophic recovery).

Windows. Closed. Impossible to backup completely a running system without highly specialized software. Few effective recovery mechanisms.

Linux. Open. Almost trivial to backup completely a running system. Many effective recovery mechanisms. Live/Recovery disk of your own distro should work, although any other distro's may work even better.

Dunno about assigning costs, but it looks pretty lopsided to me.

FAT CLIENT

[carldot67]

Some interesting views here, but I would contend that this was a screw-up waiting to happen because screw-up potential was built in to the setup. A sysadmin has pressed the button here for sure but I wouldnt be too hasty to point the finger.

This is what happens when you have a fat client. There's a lot in a fat client. A lot to go wrong, a lot to be insecure. It therefore needs a lot of looking after. Many updates, many risks. Multiply by many desktops and it only becomes manageable by central updates. Central updates means lots of automation. Lots of automation means someone presses the wrong button and.. BANG.

But for the whole thing to go BSOD... now THAT is bad. It means you can't even back out. The reports I have seen imply that they had to nuke Windows or install stuff manually using some kind of recovery diskette... It's a disaster whichever way up you put it.

Would it have happened if they used Linux? Who knows. Linux is a complicated beastie too.

However, if they had used web apps or thin client for eveything then the issue might not have even come up.

It does make an interesting academic exercise to consider what would happen if the same screw-up hit other installations with many thousands of windows clients. Yes I am referring to the recently announced UK NHS (900,000 nodes) and US AirForce (500,000) Microsoft "wins".

I have seen NHS and DWP apps. Pretty basic stuff. Running these things on XP or W2000 is a bit of a hammer to crack a nut. The only earthly reason I can think of is the MS upgrade machine says they have to.

Re:FAT CLIENT

[SpaceLifeForm]

Would it have happened if they used Linux? Who knows. Linux is a complicated beastie too.

It would not have become a huge mess. There is no automagic update that could propagate like this.

Re:FAT CLIENT

[Anonymous Coward]

Really? A dumb-ass admin couldn't get an updated version of vmlinuz 2.6.8, Red Hat Edition, to be applied by simple copy by a shell script, and accidentally push that update onto all the computers, even the ones that expected a debian 2.4? Would the computers really come back up with just a warning after that?
This is what happened here: a brute-force update for XP specific files were propigated to all the 2000 machines too, due to the admin's extreme negligence.

Re:FAT CLIENT

[bogado]

Yes there is, yum, apt and etc. But those tools are mildly inteligent as not to update a system with the wrong version.

Re:FAT CLIENT

[adolfojp]

I personally like web services with rich clients. The business logic stays centralized and the people have a "better" experience on their desktops. I still don't understand the advantage that one would get by upgrading from W2K to WXP in a work environment.

Cheers,
Adolfo

Re:FAT CLIENT

[_xeno_]

I still don't understand the advantage that one would get by upgrading from W2K to WXP in a work environment.

Not that I'm usually one to bitch about Microsoft's monopoly, but, based on my experience: consistancy with new desktops.

Microsoft will no longer license Windows 2000 Professional on new desktops, so they instead come with Windows XP Professional. (Or at least Dell won't any more, or something. I don't know the details, just that all the new machines come with Windows XP like it or not.) Since it makes little sense just to toss those licenses, the company I work for just keeps them. They currently support both Windows 2000 and Windows XP but it's likely that all old Windows 2000 machines will be phased out shortly, since all machines are (supposedly) replaced after three years, meaning that all the existing Windows 2000 machines should be replaced soon.

Other than that, there's little reason to upgrade. But making it so that only one configuration (Windows XP + standard software) needs to be tested instead of two can be a reason to upgrade.

Re:FAT CLIENT

[julesh]

Microsoft will no longer license Windows 2000 Professional on new desktops, so they instead come with Windows XP Professional. (Or at least Dell won't any more, or something. I don't know the details, just that all the new machines come with Windows XP like it or not.) Since it makes little sense just to toss those licenses, the company I work for just keeps them.

Microsoft OS licenses include a 'downgrade' option. A WinXP Pro license will allow you to run Win2K pro instead, if you choose.

But I'd say there are a few inherent advantages of WinXP that would be useful for DWP. It is a much more responsive OS than Win2K, in my experience, and would probably improve staff productivity a little. Also, it seems that with SP2 many potential security problems are non-issues, therefore it becomes less necessary to constantly roll out patches to your systems.

Still Microsoft's fault

[dema]

Although numerous people have pointed out that this was the fault of EDS, and not the OS. We can still blame MS as the EDS sysadmins are probably all MCSE (:

Comment removed

[account_deleted]

Comment removed based on user account deletion

indemification !!!!

[budword]

How much has the UK goverment recieved from m$, after all, isn't m$ indemified ?

Ouch.

[Corellon Larethian]

Meanwhile staff were forced to communicate by fax because the e-mail system broke down.

That is most harsh, indeed. You'd think some guy could grab his laptop, shove a copy of Fedora and sendmail on there, and get back to work.

C'est la vie.

Remember the mail clients

[superfast-scooter]

which, after setting up your account and login details, asks if you want to download messages now? And if you said yes, it would download AND delete the messages on the server because the default setting is to "Delete messages from server on download"?
Seems to be like a case of the same. :)

Dependencies to save the day....

[penguinbrat]

I agree that it was the fault of EDS... What I see as the overall issue here is that the EDS admins (incompitant or not) were ABLE to do this. For anyone who has tried to install/upgrade an rpm from one version of a particular Linux distro to an earlier version (or even the same version), knows (and has bitched to the heavens about it) about the endless amount of dependency errors that result - and would likewise prevent anything like this from happening.

This is very unfortunate that it happened (I would not want to be one of the ones not getting their money because of it), but on the flip side I see the humorous side of this in that everyone who uses linux has bitched about the dependency problem at one time or another (I know I have), and ironically this type of situation is why the dependency tree was invented.

Now, don't get me wrong, this STILL could have happened with Linux with poorly built RPM's with obviously no dependencies. Note that I equate official rpm's from the linux community (companies or groups) as the same thing as a patch from M$ - in other words the patch it self should never had been able to be installed on an incompatible system (aka unsatisfied dependencies), whether it was attempted or not.

Re:Dependencies to save the day....

[DaHat]

Don't think that updating a linux system with properly built RPM's is immune from not enforcing dependencies... there is still the -i which does just that... install the RPM while ignoring any dependencies it may have.

As a former subcontractor ...

[Anonymous Coward]

to EDS(UK) working on the initial install of these 80,000 PCs I can't say I'm at all supprised this went tits up.

EDS (UK) is a true management culture, its all about statistics, statistics and damn statistics, regardless of the consequences.

To quote a former manager:
I know that if we replace these PCs now they won't work until we have the power cables next week, but we need to get them out

I guess as usual their development department had the 50% required sucess rate before launching something live.

It will be easy for them to roll back to the original install, I still have one of the ghost cd images around here somewhere if they need to borrow it.

text of article (in case servers slashdotted)

[Fortran6502]

Most of the desktop computers in the UK's Department for Work and Pensions were paralysed for four days on Monday, when a failed upgrade took them offline. The outage, covering 75-80 per cent of the DWP's 80,000 PCs, is one of the largest in the UK Government's not entirely impressive IT history.

And possibly one of the most costly. According to staff reports the outage occurred on Monday afternoon, disconnecting staff email, benefits processing and Internet and intranet connectivity. According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess." EDS is also thought to be flying in fire brigades.

If these claims are true the DWP could face grave difficulties in rolling all of its machines back to their previous, working state. Staff from Microsoft and EDS are reported to have been working around the clock to dig the Department out of the pit, while speaking on the Today programme this morning a spokeswoman amusingly insisted that the Department's systems had not in fact fallen over. They were working, it was merely the case that "80 per cent of desktop computers are not connecting through to the mainframe systems."

So that's cleared that up then. She added that the emergency payments system was "working perfectly." The emergency system appears to have kicked in on Wednesday, and the Department was preparing a press release on the matter yesterday. No sign of it this morning, yet.

Reports coming in this morning however suggest that at least some of the DWP's systems are coming back online. ®

mmmmm...karma

Get Bill and Steve over here working the overtime!

[pandrijeczko]

I'm sure Microsoft's "Chief Programmer" can jump in his Learjet and come over here and do a few hours with the EDS guys sorting out the problems, can't he? Have a pizza with the guys as they crowd round a few dead Windows servers scratching their heads?

Bring Steve along in a clown suit, have him dance outside the Social Security offices with sweaty armpits, amusing the angry hordes with his continual chanting of "De benefits, de benefits".

Smug mode engage...

Devil's Advocate

[graveyardduckx]

This could've happened on any platform, not just Windows. I know I've accidentally done some *stupid* things while trying to upgrade Linux and *BSD boxes that have rendered them near useless.

why this IS microsofts gaffe

[emptybody]

they applied a WinXP patch to Win2K boxes.

when you are selling a complex tool to simple folks to maintain ( PC = Personal Computer ) enough care must be made to prevent mixed OS Rev patches from installing.

Solaris Patches are OS Rev specific.
RPMs, if properly pakaged, check for proper versions.

Why is it not reasonable to expect that a WinXP specific patch would not verify the OS onto which it is being installed?

but 60,000 stupid things?

[JonathanF]

All the same, most OS reinstall mistakes don't wipe out 60,000 computers at the same time. I contend that this is the kind of goof-up you tend to associate with Microsoft. ;)

Fundamental Architectural Issue Here

[dunstan]

Interesting set of threads ... "it's not Microsoft's fault that EDS pushed the update out wrongly".

The fundamental error here is deep seated and architectural - they have 80,000 user interface devices which are stateful. By putting the wrong device on the desktop they have set this situation up.

In the olden days when clerks in government agencies used green screens this problem wouldn't happen. If a green screen failed, it would be replaced as a FRU. Today's equivalent is something like a SunRay - the user interface device holds only enough configuration to bootstrap itself and, again, is a FRU.

The situation at the DWP is different: the user interface device is a stateful device which holds configuration itself, and requires this configuration to be consistent before it gets enough connecticity to be remotely managed. The toolkits discussed, which are used to push config around these UI devices, are probably most excellent, but there should be no need for this sort of mularky.

So while I don't necessarily blame Microsoft for this incident, I do blame them for creating a monoculture where this sort of architecture is deployed. I expect the trials underway in government using SunRay devices as the user interface will be watched with more interest after this debacle.

A final question - how on earth do DWP recover 60,000 unbootable PCs?

Re:Fundamental Architectural Issue Here

[SuiteSisterMary]

Why do you assume that the desktops are stateful? If they're set up properly, everything except the executables is server side, and the executables are installed as part of an install image.

If some idiot working on the mains shorted something out, and fried an entire building full of dumb terminals, you're in the same spot.

A final question - how on earth do DWP recover 60,000 unbootable PCs?

The same way you replace 60,000 blown FRUs; you go to each one, pop in either a) a boot floppy or b) an install CD, and reboot from your base image. Done.

Re:Fundamental Architectural Issue Here

[Anonymous Coward]

You may have a point. Much of the work done in the DWP is still processed through mainframe systems so most of these PCs probably run some sort of 3270 emulator software and act as little more than dumb terminals. I suspect that apart from e-mail and word processing a lot of the MS desktop is not required. Unfortunately, politicians and civil servants have not yet realise that what the British public want is not the latest software release or the flashiest front end but a system that calculates their pension correctly and pays it on time.

Re:Fundamental Architectural Issue Here

[mattyrobinson69]

A final question - how on earth do DWP recover 60,000 unbootable PCs?

if boot from floppy is enabled in the bios, just make a custom bootdisk to recover from a central backup (presuming they use images to create new machines), make plenty of copies and walk round putting them in every machine, turning them on and taking the disk out when its finished (and pray).

if boot from floppy is disabled, it would take fscking ages

Re:Fundamental Architectural Issue Here

[upsidedown_duck]

the user interface device is a stateful device which holds configuration itself

Not to mention an unknowable number of personal files that employees put there in the course of their work, none of which are backed up, all of which may have been lost on those 60,000 PCs. Given that this new TCO component of such size it would make God hesitate is unknowable, I'm sure EDS and Microsoft will simply bury it somewhere with all the other bodies.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

They do use Novadigm Radia, and it's not at all simple to send software to the wrong machines. I called up someone I worked with there and got more information about it. The articles have a certain amount of FUD about them as you would expect.

Something was missed during testing, and got pushed out on the nightly updates. Simple as that, yet the consequences were a little more disastrous than someone losing a desktop icon.

Yes, shit happens, and it happened in a very large way here. It doesn't reflect on EDS as a company, it doesn't reflect on Microsot products either. Localised error. That's all. Nothing to see here. Except some faulty machines.

How many MCSE's does it take to do a patch?

[Anonymous Coward]

I guess one to do it and a hundred to undo the damage?

LOL

[The MESMERIC]

HaHAhaHAhaHAHA
Well deserved - you bafoons!

See I would blame the Americans, the poor gullible brits keep listening to your libellous flash add and buying into it.

The Brits just get EVERYTHING wrong: UK Passport Software: failed - they hired Siemens which in turned used VB5 !! .. Airport Check-in Software failed (talk about a software which is over 30 years old).. DHSS Payment scheme also failed - bet they hired cowboys like "Anderson Consulting" .. oh pardon me "Accenture" (how damn queer)..

In America do you guys have so many deployment blunders one after the other?

Even the outsourcing fever - which crippled many companies here - these poor Brits copycats got it from you guys.

Next thing you want to teach them is how the IP law is ever so beneficial.

Our problem lies with the "Charlies" - you know those cocky arrogant/ignorant IT managers, that know sweet FA - bluffing crap - and the technophobe dumb-arses (instead of learning something for themselves) just take the bull.

Sadly the real good stuff and principles you've created (Posix/W3C/ANSI/Open Source ...)- they don't want to know (I guess its beyond Charlie's IQ).

I seriously pity this Island I was condemned to live in.

A good case for dumb terminals

[iBod]

or at least, thin client devices.

It seems these desktops spend most of their time running terminal emulators to access mainframe sessions.

I've always thought that fat clients with big, complex OSs were a bad idea in this sort of massive, standardised environment.

Re:Come on! Bullcrap!

[Anonymous Coward]

Any doofuss sys admin that does not give the client a second kernel boot to run with LILO should be shot. Just try that under a Winbloze client! MS does not deploy this because the Win boot loader sucks and is a deliberate clone/competition blocking device.

Operator error?

[katorga]

"...wrong when an incompatible system was downloaded on to the whole network."

That sounds amazingly like operator error to me. Exactly how is that Microsoft's fault?

I get sort of tired of the rote "Its MS's fault" arguments. The majority of issues I see at MS sites are the result of operator error or companies not spending the money needed to manage their systems according to best practices. The same goes for Linux, Solaris, and AIX.

The catch is that unix engineers and admins are generally more experienced, higher skilled and higher salaried than most windows administrators. But where I find a highly skilled windows staff, what they are able to accomplish and the level of stability is very impressive.

I don't believe brake pedals are checkboxes.

[orasio]

-----
Hold up.

Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.

So is it GM's fault if someone crashes and kills themselves (and others) by pushing the gas instead of the brake pedal by mistake?

------

No.
I said "Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.
"

I mean that admin tools that are sold as easy to use do take the responsibility for the consequences. You cannot claim to do the work for the user, and then say you are not responsible.

Re:I don't believe brake pedals are checkboxes.

[Mick Ohrberg]

I said "Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too."

So, People killing themselves in the traffic is the responsibility of the guy that provided the cars, too?

Re:I don't believe brake pedals are checkboxes.

[orasio]

No

No choice for specific industries

[bravado2112]

Hate to say it...but depending upon what industry you're in you may have little or no choice on what operating system you can use. Case point...I'm an IT worker for a small architectural firm. Guess what! There is not one mainstream architechural program that is widely used that isn't on Windows! So in order to run Bentley Microstation or Autodesk AutoCAD, two of the top architectural programs out there, you have to have Windows. Sorry...no Microstation or AutoCAD for Mac or Linux! Might be an alternative out there...but is it as powerful and compatible as the top two that are widely used? Probably not. So...the problem isn't a question of just the operating system...it's a problem of whether the software companies are designing their software to run on operating systems other than Windows! Till Bentley get's their head our of their ass and starts distributing a version of Microstation for OSX or Linux...my company is stuck using Windows! Nuff said!

Did I speak about Linux?

[orasio]

(Checking...) No, I didn't say anything about Linux.
I didn't think so, Linux can't be compared to WindowsXP it's a kernel. It's like comparing a Turbo Diesel motor to a scooter. They don't match. I could have spoken about RedHat, or SuSE, but I didn't.

I was talking about admins, dumbed down admins, and whose fault it is.
I put the blame on people who buy dumber down software, and that promote dumbed down software culture, and hire dumbed down consulting firms that hire dumbed down admins.

Much of that blame is on Microsoft, for promoting that culture. Much is on stupid management, of course.

Re:Did I speak about Linux?

[Skuld-Chan]

You definately infered it. Microsoft does say its easy, but nowhere in any of their whitepapers do they actively encourage deploying things without testing.

Ease of use has nothing to do with this issue.

Upgrades

[Alioth]

Firstly, this looks like an administration cockup, not anything particular to Windows. It looks like from the circumstantial evidence, that the DWP don't have a separate test network.

We have a distributed system at work. This is why I have built a test network which is completely separate - there is no physical connections between test and live - (physical - not separation via a VLAN, but completely electrically separate) - so if you're in the test room there is absolutely no risk of trying to roll out a patch you're testing on the live system. We don't rely on software to keep tests within the test environment.

You have to pay extra...

[PhYrE2k2]

Normally you have to pay extra for the BSODs, but Microsoft wanted to be nice and threw them in absolutely free in their limited time offer.

Just one of the many features they offer.

-M

Re:You have to pay extra...

[Anonymous Coward]

> Normally you have to pay extra for the BSODs, but Microsoft wanted to be nice and threw them in absolutely free in their limited time offer.

Bullshit, BSODs are one of Microsoft's OS' core features. You never have to pay extra, they have always come bundled.

It's about processes

[stevelinton]

Yes, shit happens, and it happened in a very large way here. It doesn't reflect on EDS as a company, it doesn't reflect on Microsot products either. Localised error. That's all. Nothing to see here. Except some faulty machines.

It's this willingness to say "Localised error. That's all. Nothing to see here" that gives IT it's bad reputation. With properly designed processes and appropriate tools, localised error cannot have catastrophic consequences. In a system like this, I can see no excuse for pushing something out to 60K desktops in a nightly update without at least one, and probably both of:

a) Pushing it out to (say) 600 representative desktops a night or two before and monitoring

b) Having a cast-iron, regularly practiced and tested, process for pulling it back again.

Look at somewhere like SEI who make the Space shuttle flight control software. It cannot go wrong and it doesn't. Why, because they have processes! There are checks and testing and simulation and code walk-throughs and whatever, and if a problem NEARLY makes it through, and is caught in late testing or whatever, there are processes to look back and see how it got that far and make sure that the processes are improved so it doesn't happen again. The process writes the software and the people carry out the various roles prescribed by the process. There are processes for monitoring and improving the processes, etc.

Re:It's about processes

[Ayende Rahien]

You get what you pay for, if you want cast iron reliability, go get a 10,000,000 $ PC.

Re:It's about processes

[stevelinton]

But the DWP spends a lot MORE than $10M on their IT setup.

Re:It's about processes

[upsidedown_duck]

It's this willingness to say "Localised error. That's all. Nothing to see here" that gives IT it's bad reputation.

Excactly, and IT earned every bit of it. No one wants to pay for processes, no one wants to expend the extra effort for processes, and no one does. People in IT are more comfortable taking the intellectually lazy route and, because it works 80% of the time, they become quite comfortable doing it. For that other 20% or whatever, they figure out how to rationalize it as a "software glitch", even when it is their own fault, but the people they are explaining it to are so ignorant they will accept any explanation as the absolute truth. Management in IT must be the most ill-prepared and gullible bunch in any industry anywhere. The fact that accountants can't even match up trends in hardware and software costs with associated labor costs doesn't help (who here is still working on a 300MHz Pentium II with a buzzing hard drive and a 60Hz monitor, when the new hire in the next cube gets a 5GHz gold-plated dream machine who then wonders why you can't run their favorite dev tool of the week?).

Re:It's about processes

[grcumb]

"It's this willingness to say "Localised error. That's all. Nothing to see here" that gives IT it's bad reputation."

Hear hear. Any software solution that makes it possible to affect 60,000 machines in one go is simply, frighteningly wrong. It's a bit like mistakenly scheduling 100% of police staff to attend the same two days of training, then being suprised when the crime rate skyrockets. That's not an oopsie - that's felony stupid

Re:It's about processes

[ultranova]

Any software solution that makes it possible to affect 60,000 machines in one go is simply, frighteningly wrong.

The Internet allows one to affect far more than 60 000 machines with a single virus...

Re:It's about processes

[mysticwhiskey]

Fine, then get TWO machines!

Re:TCO costs rise scarily with Windows XP failures

[CmdrGravy]

Hey, you must work at the same place I did !

That buffet thing is bang on, every time management had a meeting on our floor it would take the starving call centre staff 30mins + to finish off all the food they had left and wanted to throw away.

No, I didn't

[orasio]

Microsoft imposes the culture of the point-and-click magic, through _marketing_, no whitepapers.
Management is sold (by MS) on the idea that a 4 year old can run their network, so they hire one to do it.
It's management fault, but Microsofts too.

Re:No, I didn't

[Anonymous Coward]

Yeah, these guys are paying EDS a crapload of money because they think "4 year olds" can manage their 80K seat network. Come back to reality, flameboy.

Spoken like a WebTV user

[b00m3rang]

Your experience is unique to anyone I've ever talked to about the subject, apparently. And, an upgrade from Windows 2000 to XP is not a patch. Just because most machines didn't finish the upgrade process doesn't mean the process was not an upgrade.

If they had done fresh CD installs, or maybe even used the OEM Preinstallation Kit to automate things and/or create a duplicatable drive image (which I'm so intimately familiar with due to my Win95-exclusive experience), this wouldn't have happened.

You can pray for your Windows upgrade to propogate over the network without incident in one hand, and take a crap in the other, and see which one fills up first.

Gateway Reviews

[CemeteryWall]

The UK Government's Office of Government Commerce has a heavyweight methodology for controlling the procurement of IT projects. This is meant to stop the endless IT cock-ups.

See www.gatewayreview.org.uk for a very sceptical view on this.

Can big projects ever be properly controlled? And do heavyweight procurement methods encourage the procurers to make projects even bigger so they don't have to do it again next year.

Re:Expected Behavior

[Anonymous Coward]

It's expected behaviour from moronic sys admins.

Imagine if it were running linux, the entire damned building would go up in flames instead of a simple error message, LOL!

And yes, the problem has been GREATLY exaggerated.

Lower TCO = Lower Cost of Labor

[eldapo]

The Holy Grail of IT is to reduce bugets by lowering payroll. Because Windows is really easy to install and maintain, the sales pitch goes, you can hire less skilled (expensive) people to do the work. Problem is, Windows isn't so easy to install and maintain anymore, if it ever was. Even before Active Directory, keeping 4,000 or more Windows systems up to date with the latest patches was a challenge. AD introduces even greater complexity, requiring the admins that ride herd over it have *at least* the same skill level as their brother (or sister) UNIX admins (I'd argue they actually need to be *more* skilled). Of course EDS and others have stubbornly refused to recognize this, and so you have foobars like the one reported in the original article here.

Re:TCO costs rise scarily with Windows XP failures

[CmdrGravy]

Radia is a bloomin' pain in the neck, the last place I worked used Radia and it was horrible ( Radia it's self is possibly very nice and useful, it's the way it was implemented that was annoying ).

Company polict stated that everyone should always turn off there PC's when they left for the day and you'd get moaned at if you didn't. The Radia team told everyone they must keep their PC's on at all times but this was never company policy.

Every morning it would take 20mins or so for Radia to install all the nights patches and reboot the PC's a couple of times. At random times during the day it would also reboot your PC automatically for you if you didn't notice what was happening quickly enough to stop it.

Various PC's were being used as servers but not offically classed as such ( due to the excessive hurt and pain involved in that process ) and they also would reboot themselves randomly cause outages on whatever they were doing.

Some PC's were still Windows 95 and Radia would never manage to install anything on them, just keep crashing, rebooting indefinitely.

In the end I managed to delete enough of it that it stopped working and gave me some peace of mind.

I think the lesson here is not to just deploy cool new tools willy nilly without assessing their place in your working practices.

I'll go back to Windows...

[namekuseijin]

... when all i need to do is something like

apt-get upgrade

and when it's as flexible, customizable, fast, free and secure as Linux.

Re:I'll go back to Windows...

[Anonymous Coward]

" ... when all i need to do is something like apt-get upgrade and when it's as flexible, customizable, fast, free and secure as Linux."

Yeah because going to the start menu and pressing Windows Update is really tough......

In terms of flexibility most people don't care as long as they can get their base system updates, which Windows Update allows them to do very easily. Windows Update is very fast, I can pull updates down at over 700kB/s and I have yet to use any apt/yum repository that can consistently give me that speed. In terms of free, most people really don't care about the small fee for an operating system; it is even included in the cost of most peoples PCs. It's called paying the developers for a product. However, from your comment you won't use it unless it's free. Do you pay for your Linux distributions? Do you contribute back to the community of developers at all or are you just a leech who wants everything for free? In terms of security apt is by far the least secure out of all the package management utilities on Linux, yum or portage/emerge are much better. Apts flexibility, for the average user, is also hindered by its weak security. Repositories must be added to give apt even the smallest functionality and unless you check GPG keys and md5 sums on every package your security argument is out the window. This is too complicated for the average PC user who is used to Windows. For the average PC user Windows Update is easier and it's what their familiar and comfortable with.

Re:I'll go back to Windows...

[Anonymous Coward]

let me get this straight:

typing in one line to the console and walking away...

is more difficult than this:

1) click start
2) click windows update
3) click scan for updates
4) click critical updates
5) click review/install
6) reboot
7) repeat from step 1 about 8 times until you don't have any more criticals.

wow. i don't even know what to say to that.

Re:I'll go back to Windows...

[Anonymous Coward]

The average user needs a simple GUI that is preinstalled for them. They cannot and will not remember any console commands. Once they have to open up a console it is to complex for most of them. They will not read man pages/instructions and posting topics like this on forums results in nothing but RTFM. I'm not saying their behavior is correct but it's just the way it is.

For example a user installs Linux and does not know any commands at all and is used to just locating a GUI(which is almost always located in Control Panel) to configure things (ie.MacOS X or Windows). What happens when he needs to update the system and cannot find any preinstalled GUI like MacOS X or Windows offers. The average user is not familiar with man pages, IRC or Internet forums. However, in Windows (or MacOS X) they can open up the main menu and the update option is right there and it takes them to a preinstalled GUI with clearly laid out simple options. No matter how many steps you lay it out in the GUI is easier for most users.

Re:I'll go back to Windows...

[Anonymous Coward]

So install Xandros, where there *is* a menu option for upgrading. It uses apt, but the user doesn't have to know that.

Re:I'll go back to Windows...

[Anonymous Coward]

The apt version:
1) Install apt since it's not installed on most base Linux systems. This involves another console command that the user is not familiar with.
2) Update the apt.source.list so that it has some decent repositories. This involves even more complex editing of text files that the average user is not familiar with.
3) Run another console command apt-get update
4) Run apt-get upgrade
5) Review/install
6) Depending on what you install you may have to reboot. If you updated the kernel you would have to reboot. If you were in X and installed video card drivers you would have to logout and modprobe the video card driver and edit the X configuration file. This to most users would be considered harder than a simple reboot. This argument is pretty much moot on either side of the fence as only few Windows Updates require a reboot and it really is not much of an inconvenience. If you were to keep up to date with Linux kernels you would also require frequent reboots.

I would agree with you that the Linux update system is far superior however it does not yet have the simple standard interface that users can rely on. This is what the average user needs. Most users are just not willing to have to learn console commands to use their computers. They aren't interested in computers and only use it as a tool to quickly accomplish some of life's tasks.

Re:I'll go back to Windows...

[Anonymous Coward]

Is it just me or is Xandros Linux for people that really want to be using MacOS X or Windows? I use Linux but wouldn't touch Xandros with a ten foot pole.

Re:I'll go back to Windows...

[Anonymous Coward]

I actually just went through installing Gentoo for the 1st time the other day. I have my beefs but apt-get isn't one of them. I believe portage autoconfigured itself so (at least in my case) I have not had to do steps (1), (2), (3), (5) or (6).

Re:I'll go back to Windows...

[Lord Bitman]

check it out, d00dz, he GOT THE POINT!

Re:TCO costs rise scarily with Windows XP failures

[CmdrGravy]

Doesn't Radia work on Linux boxes too? There is a tool very much like it which does I'm sure.

analogy?

[phyruxus]

Remind me again... what does the "T" in TCO stand for?

To err is human

[Doc Ruby]

... to really screw things up, you need a computer. - Anonymous (human)

Re:TCO costs rise scarily with Windows XP failures

[fucksl4shd0t]

This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container.

Dumbfuck: Hey boss, you want me to apply this patch to All Systems or Custom Systems?

Boss: All 6 custom systems.

Dumbfuck: So that's all systems, then, is it?

Boss: Yes, All 6 systems.

Dumbfuck: Ok, it's done.

*phones ring*

Not Microsofts fault, in this case

[nurb432]

It was the losers they had working with them.

EDS used to be a great company, with the top people.. Wouldn't make silly mistakes like this..

Doesn't appear they are like that anymore.....

Re:Not Microsofts fault, in this case

[MacDaffy]

If you give a chimp an Uzi with a defective trigger mechanism and a bunch of people get shot, whose fault is it: the chimp's or the Uzi's? My first networking experience was with AppleTalk; plug it in and you had a network. I was subsequently required--with co-worker--to learn everything we could about Windows networking so we could implement it in one of our products.

My co-worker and I spent the next period AMAZED that Windows networking even worked at all. The system of domain controllers and WINS servers and browse lists and host files... it's too byzantine to be believed. There is, without doubt, a corporate network somewhere that could be comopletely undone by someone opening a wireless laptop in the wrong place at the wrong time. Add Windows XP and the attendant SP2 fun they're having and you get chaos.

Yes, those delightful folks at EDS are the chimps in this scenario, but Microsoft's products are definitely the defective Uzi. And I note that the BBC News article studiously avoided mentioning either of them. Hmm... Microsoft wouldn't be doing everything it can to tamp down this PR disaster, would it?

Naaah!

Re:Not Microsofts fault, in this case

[Tough Love]

It was the losers they had working with them.

Not quite. Microsoft and EDS operate the system as a joint venture

f1rst ps0t

[Anonymous Coward]

yay

Re:TCO costs rise scarily with Windows XP failures

[Tony-A]

what do you consider highly specialized?
We have a computer we save all our CAD files on at work via windows networking, and it uploads a backup of itself to another server every week, but only uploads the changes. I believe it's one of Novell's nice little setups.

Highly specialized.
With the backups and nothing else restore all programs, configurations, registry setting, registrations, etc to a completely bare system. Like if a hard disk crashed.

Managing 60,000 computers.
With almost no admin or unix skills, dd from the network or a USB disk should cover a lot of ground. A few scripts and some skill should make it quite manageable.

Managing 60,000 computers smoothly.
No idea exactly what or how, but this should be very much in Novell's turf. Small minor details of just what is tweaked just which way will in total make a large difference. This is why corporations will gladly pay Novell for something that is not much worse that what they can get for free.

Of course they fucked up!

[orasio]

This is the fault of the admins, of course.

But also: management, that hires inderqualified admins.

Plus, Microsoft, that openly promotes that you don't need a real admin for your network.
Of course the guy who built the app is not responsible for that, but the guy who sold you this app so you didn't need a real sysadmin is!

Re:Spoken like a WebTV user

[Anonymous Coward]

According to the Register, they were not upgrading machines to WinXP.

Keep in mind that doing a 2K->XP upgrade is not really more complex than applying a service pack. Every file gets replaced, some reg scripts run, that's it. Of course, like a service pack, there's plenty of things that can go wrong.

Many shops time their OS upgrades along with upgrades of MSOffice, Notes, etc, so quite often the wipe/reinstall process is much simplier to plan for.

Re:TCO costs rise scarily with Windows XP failures

[swv3752]

Scripting. One can do it either client or server side through a number of ways.

How I would do it:
Central ftp server with clients running either apt or urpmi.update as a cron script. Combine with some scripts using expect and ssh to login into a few machines a night or two before uploading to the central ftp server.One might have more than one central server but that really doesn't matter for a simple explanation.

Educated Chimps

[nurb432]

Can do the job properly. If you look really close, thru all its faults, and thru ones own biases, a *properly* maintained Active Directory based/GPO managed windows domain is a piece of cake to support.

Doing piecemeal upgrades/patches is also simple.. Unless you are a total idiot, as it appears these people were.

So we aren't talking about a defective trigger that is fubar.... more like a Uzi that needs magazine lubrication to operate properly...

Re:Educated Chimps

[Anonymous Coward]

Argh - ADS is the worst - a magic black box that "just works", that does magical multimaster replication, relies on the DNS, which it is also used to manage. Uses mysterious vendor security protocols to lock others out....

Okay it is a big improvement on what went before in the Microsoft world, but these things are all way over the top. Heck NIS+ is way over the top for most admins and most networks, which is why many Unix sites stuck with NIS for distributed authentication for so long despite known major security flaws (its internal methods were a joke) - in practice it worked and was simple enough for admins to grasp fully what was going on. Not least we knew its limitation clearly and could design our networks to allow for its weaknesses.

Indeed it was possible to design variants that address the weaknesses, because the protocols were simple enough for admins to understand.

Sure Microsoft networking has great security features now, but it reminds me of that quote by Spafford about SSL (checks written in crayon on park benches, sent in security vans....). The whole things is way too complex for what 99% of the users wish to achieve, which is single sign-on, and delivering a small set of parameters.

Interestingly one of the people I worked with always pointed out the term "transparent" is used back to front in computing. The old systems were transparent because you could see the inner working, some modern systems are opaque because you can't.

A great example of what I mean is POP3 authentication. The old system used a login and password sent in plain text. It became popular, it worked (and works), and was easy for the admin to eavesdrop problems. Every admin knew the weakness - plain text passwords - and could address this in a variety of ways.

Some opted for POP3 over SSL (which Microsoft implements VERY badly BTW), this addressed the main weakness (plain text passwords, and the lesser one, no authentication of the server) and also gave some privacy to the messages, and there were still ways for the admin to intercept it server side and debug.

Others opted for complex authentication schemes that use zero information protocols, in general these merely managed to make the systems more difficult to "debug", and in one implementation the server side needs to keep a plain text copy of EVERY password, thus making compromise of such a system a gold mine for the cracker.

TRANSPARENCY matters - too much complexity can blind you to the obvious flaws in the approach - I'm afraid ADS is way too complex, and even where it isn't it is obfusicated by being a closed or proprietary technology.

Re:Educated Chimps

[Anonymous Coward]

Ahhh, but the educated chimp knows better than to pick up the Uzi in the first place.

Re:TCO costs rise scarily with Windows XP failures

[Apathetic1]

although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...

He or she has probably already been promoted to management. I find that's generally how EDS deals with people who screw up really bad.

CERT Guide to System Administration

[dexterpexter]

True, although if they had followed the CERT system guide to system administration (I know, this is more U.S.-centric), those 60,000 PCs should never be connected so intimately on the same network (segmentation on large networks is important for this very reason.) Secondly, a "testbed" should never be connected to the main network. It should be isolated for this very reason.

This was certainly an example of failure at the speed of light and exactly why networks shouldn't be administered in that way.

Of course, I am with you in waiting for more hard details.

Testbeds should be isolated

[dexterpexter]

A testbed should never be connected to a network in a way that such a large error could occur. One of the first rules to good system administration is to test on an isolated testbed, which this very apparently was not.

The grandparent post had it correct in calling this a "leaky" sandbox.

I for one welcome our NT chimera overlords.

[eobanb]

I for one welcome our NT chimera overlords.

Re:TCO costs rise scarily with Windows XP failures

[angulion]

Being a linux enthusiast and all..

This is no different from some admin writing an own update-script that (without checking) would just copy a new version of glibc to thousands of machines.. And the results could be very same.

Ever tried to rename /lib, or do an unsuccessfull attempt to switch from libc 5 to glibc (long ago), both cases ends in a less than fully functioning system.

Testing is everything.

Upgrades usually work

[Vacuous]

I disagree, although I am not talking about this in a server situation, 99% of the upgrades I've done with MS Operating Systems went flawlessly. The problem is, is that so many people do not do them properly. They don't uninstall anti-virus software (Disableing is not good enough, it still leaves filters and such in the registry), they try to upgrade from an unstable OS, they don't check application compatibility, they don't uninstall drivers where possible. Geeks do this as well.

I say this as someone who has done hundreds, if not thousands of windows upgrades and Windows installs.

Re:Upgrades usually work

[satans_advocate]

They don't uninstall anti-virus software (Disableing is not good enough, it still leaves filters and such in the registry), they try to upgrade from an unstable OS, they don't check application compatibility, they don't uninstall drivers where possible. Geeks do this as well.

How does uninstall == upgrade?

Re:Upgrades usually work

[Vacuous]

How does the anti-virus == the operating system?
How do the drivers == the operating system?

Re:Upgrades usually work

[satans_advocate]

How does the anti-virus == the operating system?
How do the drivers == the operating system?

Exactly! Are you saying that the operating system is so fragile that it can be destroyed by anti-virus software or drivers?

Doesn't having to uninstall software make it a little more difficult to roll out 60,000 upgrades?

Re:Upgrades usually work

[Vacuous]

But you forget the fact then when you are beginning the upgrade it ISN'T the operating system. When you are upgrading from say Windows 98 to Windows XP, Windows XP is NOT the operating system until the late stages of the install; it is just another application being installed, and has the same limitations of an application being installed. Most anti-virus related issues related to Windows XP occur before the first reboot. As for drivers being able to bring it down, well DUH, incompatibilitys with the software needed for the OS to communicate with the hardware causes problems, wow, that's so windows-centric (By the way, i'm being sarcastic).

As for having to uninstall software making 60k upgrades hard, I agree, but I never said upgrading was always the best solution to every situation, did I?

Re:TCO costs rise scarily with Windows XP failures

[MarkByers]

If Windows had some sort of easy way to back up before installing a patch so that you can revert to a known previous working version (like when you change your resolution or change the registry) then there wouldn't be a problem. Think about how grub works - if the kernel won't boot it uses the last known working kernel.

You would want a larger test set than one.

[dexterpexter]

Well, typically you would want a larger test set than that. But, as another poster pointed out, they were indeed trying to test this on seven PCs, but "infected" the other 60,000.

So, while they were heading down the right road, they still had the testbed connected to the same network as the production machines, which is a no-no. And they had scripts that were too wide-reaching.

But ideally, yes, you are correct in that you would want to set up a set of machines and test them first.

Oh..

[Anonymous Coward]

Man, imagine a Beowulf cluster of th- Oh, nevermind.

With Linux, it would be harder to do ...

[khasim]

and easier to fix.

Sure, I can script an update for a Linux box, but the version checking will mean that the update doesn't get applied if it doesn't have ALL of the available components.

So I'd have to break some of the packages BEFORE scripting the update.

Then, once the broken packages create the problems, it is a simple task to write another script to fix the broken packages.

So depending upon which packages your broke, the repair process could be accomplished in less than 5 minutes.

At the very, Very, VERY WORST, you'd have to distribute a bootable CD that would chroot the system and apply the correct patches.

Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.

No. This is an example of a weakness with Windows. This specific problem may have been brought about by EDS, but it only happened because Microsoft chose a flawed security model for their systems.

Re:With Linux, it would be harder to do ...

[man_of_mr_e]

First of all, that's precisely what happened here. EDS broke the update packages by bypassing them entirely using a third party product that did not do version checking.

Second, The machines wouldn't boot, therefore there is no way to run any kind of script to fix the problem, thus your third solution is likely what happened in this case as well. However, it takes some time to manually go to 60,000 machines and fix them, even if it only takes 5 minutes per machine.

The exact same thing would have happened with any OS, including Linux, had EDS decided to bypass the normal version checking tools and do things themselves, creating an unbootable system.

Re:With Linux, it would be harder to do ...

[ad0gg]

You can do the same with windows, you can have it run a script before installing the files but obviously this wasn't done. You can also create a CD to do unattended install of both patches or OS upgrade. Normal patch from microsoft's website does a version check, the automated patching system from MS can actually deploy the DLLs which they did in this case. This system can place any file on any win2k or winxp box that is on the domain(as long local machine has granted suficient permissions to domain admin group), it could also be used to place say a PDF corporate handbook on everyone's desktop. Think of it like trivial ftp with the ability to run remote commands(like rcmd in the admin pack). So please don't talk about windows unless you have a clue, you don't see me talking about linux. If you read my post I only make comments on apple and windows, two OSes I actually use. I don't take a stab in the dark about the features of linux.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

They sysadmins of linux systems flat out don't have the option of managing 60,000 PC's from one location like this, competent or not... The closest thing linux has is cron, and it is nowhere near as powerful, and way more of a hassle.

No, a closer system would be a combination of something like yum (using a yum server or cluster of servers to download package updates) and CFEngine (get the server version that -doesn't- segfault and you're fine!). I use this combination to reliably administer 1,000 Linux workstations at my company, but there's nothing there that would prevent it to scaling up to 60k. The load on single cfengine and/or yum server would be pretty high, but a clustering solution would fix this.

Re:The reason for the upgrade - TWO TIN-FOIL HATS

[Anonymous Coward]

Better put on TWO TIN-FOIL HATS for this one... the cock-up was intentional - they had to install spyware on government machines - as many as they could, as fast as they could. It required a BIOS re-flash... and the bios-based spyware needed a re-install of the OS to spoof some intall-time machine parameters. Had to be done immediately... no time to ease it in. Needed a cover story - EDS took one for the team.

They shouldn't have upgraded

[iminplaya]

It's much more reliable to back up your data and do a fresh install. I experimant with upgrades, but even(or especially) with linux, I prefer to clean the disk and start fresh. Apple on the other hand(before OS X anyway, don't know if it still is) was great. It would just create a clean new system folder. With the old one still there, I could just "bless" it if necessary. Oh, well...There's still nothing more trustworthy than pen and paper, and a good ol' mimeograph machine(the hand crank variety) for makin' copies...And they smell great.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

System Restore?

DMV

[iminplaya]

They should'vestarted with their version of the DMV. I bet nobody would have noticed any slowdown.

Huh??

[Anonymous Coward]

How do you upgrade 7 PCs that results in 60,000 BSODs?

"Upgrade"?

[Kris_J]

How is Windows 2000 to Windows XP an upgrade? Office 2000 to XP, sure, but what does Windows XP do that 2000 can't? Heck, we have one Windows 2000 desktop PC at work and I have (slightly) less problems with it than the XP PCs. 2000 to 2003 I understand, but why would anyone waste time going sideways from Win2000 to XP?

Re:"Upgrade"?

[TheLink]

XP "features":
1) Stops working if you change your hardware too often unless you "reactivate" it.
2) boots up faster
3) Can switch users without logging off and closing other apps (can be very useful)
4) has option to save credentials with their run as command, so it's much easier to run IE using a user account with less privileges and so help prevent worms etc trashing stuff created by your normal user account which you use for office work etc.
5) Has tons of eye candy stuff which gets in the way of the way I do things, which fortunately can be turned off - so that it looks and works more like win2K. My ex-colleague didn't know you could turn _off_ the group programs on taskbar thing - he was happy when I showed him it was possible.

The personalize menu's thingy on both O/S sucks. Hiding options and making them unpredictable does NOT make things easier IMO. Better if you make them different colours/shades gradually.

60,000???

[mslinux]

How did they screw-up so many machines? At what point did someone say, "Guys, this isn't working." 12,000, 40,000 or did they actually mess-up 60,000 systems before stopping to ask themselves if they should continue?

Also, what ever happened to TEST BEFORE DEPLOY? Are we to believe that these guys are that stupid?

Re:60,000???

[Anonymous Coward]

Of course they deployed all 60,000, they would have stopped if they'd known, even incompetence has its limits.

My guess is they have one of these systems that sets stuff to install when the machines are booted. I bet the first few users had a problem, and turned to a colleague who hadn't booted and said "is your PC working" and 60 seconds later the answer was "no". They are civil servants, the probably all turn up at 09:00 and switch their PCs on all at roughly the same time, and by the time someone realised what the issue was all were booted.....

There are some obvious problems with the design - rolling out something to 60,000 in one go is bad karma whatever - but without more details on the kind of "patch" it is hard to comment. Certainly reports on the radio said they were having problems with icons disappearing and such like, which suggests registry changes or a desktop management system, rather than core system software.

No idea on where the BSOD stuff came from, it wasn't discussed on the media reports I saw, but then with some Windows systems the GUI is an integral part of the system, rather than in the application layer, so it is possible to screw up the OS when fiddling with stuff that wouldn't break a real OS - like NT as the original designers intended.

Restoration?

[phorm]

Assuming that the machines nuked had the same config, couldn't they just reverse the changes by finding a non-nuked machine (perhaps one that hadn't been turned on), getting copies of all the files that were fuxored, and replacing them. I'd assume registry death as well, but you could export appropriate areas , regedit import them, and perhaps somehow delete any new uglies.

Samba upgrade

[phorm]

Yes, because "apt-get update && apt-get install samba" as SOOOO hard. At least that's how it would work here

Yes, there might be some questions from the installer, but that's not too hard to script fairly reliably.

Long long time ago

[Adrian.Challlinor]

I tried to sell some software to database stuff to the EDS group in charge of the tax system (based in Telford, England). I kid you not. We asked if they has an ERD of the database. "Whats an ERD?" we got back. You kn ow a database design. How do you design the database? "Well Dave here gets on the console and types SQL statements in to Oracle". On the test system, right? "no, direct in to live". We got up and left. There is no way I am going to be front page news for my software taking down the live tax system in the UK!

Why would you?

[phorm]

Since installation the times I've had to change grub.conf or lilo.conf have been few and far between indeed.

The best I could think of was moving from LILO to grub, or perhaps changing the grub splash screen - neither of which would be needed on any type of critical machines. As for the scripting of anything else (modifying the /vmlinuz symlinks, etc etc) - the kernel installer already does that automatically, and reliably enough that between hundreds of machines I've had no problems.


The problem is this: Qualified windows admins are far between. Yes, there are those that know enough to get by in many circumstances, but few and far between are there those that handle a major crisis well. In addition the crap ones make the whole lot look worse. In addition, MS encourages having less paid sysadmins.

Now why would MS encourage that? The answer is simple enough: look at those TCO comparison ad that MS is spitting out, /. has plenty. You know why the TCO can be lower, because the braindead sysadmins are a dime a dozen. Because you can get by in many circumstances with a braindead MS sysadmin, and when things bork like this because of said braindeadness, MS can count them out of the TCO as one of the "1 in 5" circumstances. Nevermind that the other 4/5 were saving 15%, and companies accounting 1/5% suffer a crippling downage due to the f-up.

There, is that a good enough answer for you? Rampant admin incompetence is because MS encourages it. Back in the day when server admins were 'leet and very well paid because of their scarcity often enough this situation wouldn't have happened.

Ahahahaaha

[Refrozen]

Bahahahahahahahahahaahhahahaha. Or, you could be like the Chinese Goverment, use Linux, not crash, but instead get sued. (read the OTHER slashdot article about MS suing Linux users) Stupid Microsoft. Stupid. Stupid. Stupid. (I need to waste 50 seconds to be able to post - it requires 2 minutes between post... 45 seconds........) Dot. Dot. Dot. Dot. Dot.

Failures are unavoidable.

[BaronSprite]

If/when linux becomes mainstream and problems of this type occur, do we then get to bash that distro? Or do we bash the OS as a whole? Where would fault be placed, the core technology or the flavor of the month distribution.

Re:Failures are unavoidable.

[The MESMERIC]

The problem with MS is that this sort of stuff happens WAY TOO OFTEN!!

But you have to admit...

[reality-bytes]

Boris Johnson is a lot more entertaining than David Blunkett.

Nope.

[khasim]

The simple way to solve this is to detail the steps that would have to be taken under both systems to achieve these same results.

Under Windows, that's already been done. That's in the article.

Under Linux, they'd have to knowingly break the package management system and overwrite the files manually. Why would they knowingly break the package management system? Hmmmm?

It isn't that you cannot do the same thing under Linux, it's that you have to take more steps and each step is stupid.

With Windows, since Microsoft doesn't have a decent package managment system, lots of companies sell products that attempt to fill that gap. EDS was using one.

With Linux, there is no need to bypass the package management system. Which is one of the reason you don't see very many 3rd party apps for that.

Re:Nope.

[man_of_mr_e]

You appear to be making some assumptions here. First, you assume there isn't a package management system in Windows. There is, and it's quite extensive. It's called the Windows Installer, and there are a ton of tools out there to utilize it correctly.

EDS choose the easy route, which is simply copying files, rather than implementing a full package deployment, something they would have had to do under Linux as well, and likewise would have probably failed to do so.

They did precisely what you are saying they would have had to have done on Linux. There was no reason for them to do what they did, but they did it anyways.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

Don't think you've really tried managing 60,000 desktops have you? From USB disks!! Can you really imagine your local benefits staff coping with that - and then there is the security issue of allowing local staff to boot machines from local media!

NDS for SUSE is coming but it's hardly in the spirit of open source and I don't see the benefits over M$ offerings. Truth is there are systems that you could use for last known good type recovery but like ap there are security issues if users can revert changes pushed out by admins.

A sys admin is a like a pilot of a plane - there may be any number of systems that will HELP to stop a catastrophe but it didn't stop the terrorists flying one into the twin towers.

Believe me there are no small and minor details in managing more than 500 desktops.

62 $ / per year / per pc

[Anonymous Coward]

MS contract is really cheap, based on 62 $ / per year / per pc. But if Redhat/Suse/Mandrake could form alliance probably they could out perform Microsoft. With Redhat sitting on 500 million USD investment and Size of contract it should be no brainer for open source community and would probably come up with better UI than MS.
I wish we could reverse the time and seeing this happening.

Failed Slashdot Colour Upgrade Wipes Out My Eyes

[Anonymous Coward]

http://shit.slashdot.org/article.pl?sid=04/11/26/1 40211

EDS is a joke at taxpayers expense

[horza]

It's well known that EDS are incompetant and unprofessional, costing UK taxpayers hundreds millions of pounds. Examples include tax, welfareand air safety. In fact they seem to be awarded contracts by default despite not a single success with projects running hundreds of millions over budget and those that aren't a couple of years late are junked as a massive write-off.

It's well known that the UK government are in the pocket of EDS and Microsoft. The worst thing is that it's not intentional. The people in charge of making these decisions are complete non-techies and haven't heard of any IT company that aren't a regular in the new headlines of the FT. It's not corruption, it's basically a lack of education.

Phillip.

Not all automatic upgrades are harmful

[Trogre]

apt-get update && apt-get dist-upgrade

has never let me down :)

Lowest Bidder..

[NekoXP]

This is what you get when you run businesses and governments on the contracts
of the lowest bidder.

Neko

Re:what company is this?

[theskeptic]

Unless you are under a NDA.. can you reveal which company is this? Never heard of such practices before.

Exactly

[b00m3rang]

That's why you redirect the users' home folder to a properly managed server, and/or implement an automatic backup scheme. That way, if the machine crashes, or if you want to perform a fresh OS install, you won't lose anything.

Re:Exactly

[Deliveranc3]

Really you can guarantee that all their apps and the settings for those apps will be properly backed up? I have seen nothing that can do this other than Ghost.

Re:Exactly

[b00m3rang]

Should have said, "wont' lose anything important." In most cases, users don't even know how to change any meaningful settings or care to, so resetting them to defaults is a small price to pay for being up and running again quickly. Just copy the disk image with the OS, drivers, and apps already installed, and go.

For the less likely instances where all settings and configuration are essential, a regimen of full/incremental backups will ensure you have relatively recent copies of all files and registry entries. With RAID and a decently administered system, data loss is quite unlikely.

Whatever.

[khasim]

You appear to be making some assumptions here. First, you assume there isn't a package management system in Windows. There is, and it's quite extensive. It's called the Windows Installer, and there are a ton of tools out there to utilize it correctly.

Hmmm, maybe you missed the part in my post where I specifically stated "With Windows, since Microsoft doesn't have a decent package managment system, lots of companies sell products that attempt to fill that gap."

EDS choose the easy route, which is simply copying files, rather than implementing a full package deployment, something they would have had to do under Linux as well, and likewise would have probably failed to do so.

Possibly. But if Windows Installer is so great and "there are a ton of tools out there to utilize it correctly", then why did EDS do what they did?

Windows Installer is NOT as developed or as robust as the package management systems on Linux. That is why so many programs leave so much crap behind them when you un-install them on Windows.

They did precisely what you are saying they would have had to have done on Linux. There was no reason for them to do what they did, but they did it nyways.

Fascinating. They're being paid a lot of money, yet they do really stupid things even when there are lots of tools out there to help them do it right.

You need to look into deborphaner and debfoster to see what functionality a REAL package management system provides.

Re:Whatever.

[AK Marc]

Hmmm, maybe you missed the part in my post where I specifically stated "With Windows, since Microsoft doesn't have a decent package managment system, lots of companies sell products that attempt to fill that gap."

Perhaps he read it and recognized it to be wrong? Microsoft does have a decent package management system.

But if Windows Installer is so great and "there are a ton of tools out there to utilize it correctly", then why did EDS do what they did?

They were incompetent enough to cock up thousands of systems trying to mess with seven. Are you really going to assume they made all the correct choices for all choices except for the installer? I've seen companies do something the hard way just to charge more. For all you know, EDS knew how to do it more quickly, more easily, and more cheaply with Windows built-in tools, but decided to use 3rd party applications and a more convoluted install system in order to be able to charge more. It is a common practice to up-sell from something adequate to something that sounds better, but doesn't work any better.

Fascinating. They're being paid a lot of money, yet they do really stupid things even when there are lots of tools out there to help them do it right.

Why yes. There are millions of accountants out there, and Enron spent millions on accountants, but still managed to have books that were wrong. I understand that you were trying to be sarcastic, but it is a simple business truth that throwing money at a problem will only make it an expensive problem. EDS screwed up. They went out of their way to screw it up. Somehow, you think that the OS is to blame for someone going in and screwing up the project. For all the logic you've demonstrated, Kenneth Lay shouldn't have been prosecuted for fraud. They should have gone after Hewlett-Packard for making the calculators that Anderson Consulting used.

Re:Whatever.

[man_of_mr_e]

Windows Installer is NOT as developed or as robust as the package management systems on Linux. That is why so many programs leave so much crap behind them when you un-install them on Windows.

BWahahahahahahahahahahahahah

Oh my lord.. I laughed forever.

No. You don't have even the slightest idea of why things are left behind on Windows. The first reason is laziness. Many programmers simply do not bother to write a decent uninstall script.

The second reason is ignorance. Many developers haven't taken the time understand all the issues associated with installs, and elect for the safe route of not uninstalling anything that might be used by anyone else. There are mechanisms in place to do reference counting, but many people just don't take the time to educate themselves.

The third reason is arrogance. They leave things behind in some failed attempt at copyright protection. If those files or registry entries or whatever still exist the next time the program is installed, don't give them a free trial. This is arrogance because they believe they have the right to litter peoples systems with their stuff for their own reasons.

Neither deborphan or debfoster are part of the package management system. In fact, one might argue that the fact they were written shows a weakness in the package manager, at least as far as the package manager is *supposed* to work.

Windows installer is very mature, and very extensive in what it can do. Here's a link to the SDK, so you can see the functionality it does that dpkg or apt do not, for example dpkg and apt cannot do transactioned installs and rollback a partially installed, or upgraded program to it's earlier state. Sure you can use other programs within your dpkg or apt script to accomplish these details, but they're not part of the installer itself.

In fact, these package managers have very little actual functionality. They list a number of files, where they're installed to, and what dependancies are needed, and they maintain a database of installed packages. That's about it, next to being able to execute scripts to do the configuration work. That's not to say that this is a bad thing. It works.

The problem is that you're making bold claims about both the package managers of Linux and the Windows installer, when you clearly have little idea what either of them really do or capable of.

MS Windows Installer SDK

Yes, It IS Microsoft's Fault!

[Master of Transhuman]

Bottom Line: NO XP patch issued over the network should have been accepted by the Windows 2000 workstations. The patch mechanism should have tested for OS release and version.

EDS triggered the stupid goof.

Microsoft created it.

Period.

End of story.

Now if you can prove that EDS created the patch system and was not using Microsoft's standard patch mechanism, you can have a point.

But as far as I can tell from the story, there is NO indication that EDS was doing anything other than applying a patch (probably Service Pack 2) to XP on a limited number of machines, and accidentally applied it to the entire network.

This doesn't surprise me. In my current Windows Support Technician class I'm taking (basically an XP how-to), the stupid lab server and XP workstations are repeatedly installing McAfee AV and Microsoft Office on every bootup. The teacher hasn't figured out how to stop this yet.

Not to mention that several XP lab machines have simply gone belly up for no apparent reason.

Just today my home machine's XP partition wouldn't boot because of "missing or corrupt hal.dll" - which in fact was nothing of the sort. The stupid boot loader was messed up and decided that it was too stupid to find the hal.dll, so it decided to send me on a wild goose-chase by telling me the dll was "missing or corrupt". Rebuilding the bootcfg fixed that.

Then it decided to bomb loading the AVG AV once or twice. That apparently went away on its own after another reboot.

Then the PPPoE connection to my ISP wouldn't work - had to delete and recreate it. Was working fine before.

Bottom line: Windows XP - like Windows 2000, Windows NT, Windows 98 and Windows 95 before it - is a bloated piece of crap that is unstable and unreliable.

Mod this troll, mod this flamebait, make me sit in the corner again. Is that all you got, huh? Are you nuts? Come at me!

Re:Yes, It IS Microsoft's Fault!

[Anonymous Coward]

I once worked for a Tier 1 Help desk run by EDS. The remote work was to be sent on to Tier 3 RDM. When they consolidated their IT and ran out patches remotely, suddenly users were taking 30 minutes to logon. It seems the profiles were suddenly growing to 120Mb or more. I started getting these calls, and figured it out in five or ten minutes. Telling the guys at RDM was useless, though, as they could do no wrong. The butthole who refuted the possibility of a corrupted patching process spent about half again as much time on his reply to me [copied to everyone] than I did in getting users back in operating condition. Therefore, I think the 'root cause' may be a systemic training error; they hire fully-inflated egos without the first shred of real-life experience.

Poor saps

[aurifex]

That really, really sucks. Poor bastards. Hope they can, err, recover their data? ;)

ever dreamt of it?

[Anonymous Coward]

clustered crashing at the tips of the finger.

Upgrade Not Required

[Anonymous Coward]

Leanna Paige (Wal-Mart Heriss of Bud Walton)
got her BS by paying a tutor to do all her
work while at UCLA - smart.

William B. Gates (nerdy no nothing but born
into a little bit of Seattle old money) threatened
Harvard Profs that if he did not receive a
grade of A on all submitted work that his dad
would have the Prof fired from Harvard;
end result, no degree - stupid.

Therefore, Mr. W. B. Gates, front and center.
Mr. W. B. Gates, stick you're head up the ass
of Ms. Paige.

Enjoy the life digital.

Toodles.

Promotion

[oliverthered]

The problem is that people get promoted until they can no longer do the job well enough to get promoted any more.
So your left with a lot of people in jobs they can't do.

They should be thankful

[Anonymous Coward]

They should not regret anything, they should be thankful.
They can launch a 1 billion dollar class-action law suit against Microsoft.

Efficient!

[PygmySurfer]

The DWP admitted 80,000 staff were not able to process new pensions and benefits claims for several days, but regular payments were unaffected.

Technical experts have worked around the clock to find the fault

It said it would have received 60,000 new claims during that time.

So does this mean it takes 80,000 government employees to process 60,000 claims? Kinda brings government inefficiency to a new level..

Re:TCO costs rise scarily with Windows XP failures

[zootman]

heh heh..... I think you've summed up most large companies..... I like the bit about managers banquets - I always refuse to eat managers leftovers... my co-workers dont seem to understnad why as they stuff themselves with old camembert thats been sitting out for a few hours.......

Good Troll!

[Vryl]

Just when I thought the art of the subtle troll on /. was dead ...

Easy

[MrYotsuya]

Americans have though nothing about interfering in democratic processes, even in their own hemisphere.

One example is Augusto Pinochet in Chile, who was installed after a left-leaning government was elected.

Easier

[MrYotsuya]

I forgot to mention that the States' intervention was much more than writing letter to voters.

You're pointed out a failure with most TCO metrics

[Nailer]

Microsoft (nor anyone else's) TCO metrics never include downtime costs.

Re:TCO costs rise scarily with Windows XP failures

[MarkByers]

System Restore only works if your computer can boot!

What if . . .

[swamysk]

. . . this was an upgrade to a Linux system? Would it have been this easy to bring down the network?

Doubt it was MS's fault at all.

[TheLink]

If you have admin rights to a Linux/MS PC, you can do almost anything including screw things up totally. Otherwise it'd be hard to replace system files that needed to be replaced don't you think?

I'd like to know what patch distribution and management software they used to screw up so bad.

There are various such patch distribution and management software around. My company used to sell Patchlink - seemed an OK software, with Patchlink I don't think you'd be able to screw up like this, unless you forced a custom update. The various Patchlink supplied patches will only apply if the relevant software is installed. So the way to screw up is you distribute custom software and run it without any checks - e.g. distribute an executable to ALL machines regardless of O/S and software, and ran it.

I doubt this was Microsoft's fault at all. This is either EDS's fault or the patch management software vendor's fault.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

If that spreadsheet program was Excel running on a buggy Pentium, it was probably a good idea to double-check the figures. Oh, and then there are all the macro viruses which could be messing with the numbers...

Never,

[robpoe]

Never Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever Ever test on the live system. period.

Magical multiplication ?

[Anonymous Coward]

Hold on. They ugraded 7 systems and ended up with 60 000 non-working PC's? Where did those 59993 other come from?

The prove it.

[khasim]

Perhaps he read it and recognized it to be wrong? Microsoft does have a decent package management system.

Go ahead, prove it. Show me how I can get the same functionality out of Windows Installer that I have with deborphan and debfoster.

You can make all the claims you want, but now it is time for you to substantiate them.

Re:The prove it.

[AK Marc]

You can make all the claims you want, but now it is time for you to substantiate them.

You first. You were the first to make claims. You claimed that Microsoft's installers available are inadequate. If you want the person making claims to back them up, start with your own claims. And no, it isn't a feature comparison with the tool you like best. It is evaluating whether Microsoft's tools would have been adequate for the simple task of pushing updates, as was botched in this case.

I've already done that.

[khasim]

You first. You were the first to make claims. You claimed that Microsoft's installers available are inadequate. If you want the person making claims to back them up, start with your own claims.

I've already done that. Look at deborphan and debfoster. That is the support for my statement.

And no, it isn't a feature comparison with the tool you like best. It is evaluating whether Microsoft's tools would have been adequate for the simple task of pushing updates, as was botched in this case.

You don't seem to understand package management.

It is evaluating whether Microsoft's tools would have been adequate for the simple task of pushing updates, as was botched in this case.

What was that again?

...as was botched in this case.

Yes, "...as was botched in this case." That's exactly right.

Now, I say that there is no Bigfoot (Microsoft's tools are not sufficient).

You demand proof that there is not Bigfoot (Microsoft's tools are not sufficient).

I laugh at your ignorance of basic logic.

It is up to YOU to provide evidence that Bigfoot exists. I say your evidence is not sufficient and I give an example of what sufficient evidence would be (the capabilities of a real package management system).

The ball is in your court.

Re:I've already done that.

[AK Marc]

It is up to YOU to provide evidence that Bigfoot exists.

The tools exist, so everyone is in agreement that bigfoot exists. The question is regarding the size of bigfoot's feet. You made the first assertion regarding size. I'm asking you to back up your assertion, without a comparison ("adequate" is not a comparison, "better" and "best" are). You can not. That is an answer all of its own.

So?

[jotaeleemeese]

IT people are paid to provide fault tolerant systems 100% of the time.

Nowadays there are no excuses for well funded projects to shut down like this.

Re:So?

[SonicBurst]

I challenge you to provide me an example of a large, highly utilized system that has never had any kind of failure, ie: 100% uptime. Even if there is an example, which I doubt, please realize that eventually it will fail. That's just the nature of the beast.

Re:TCO costs rise scarily with Windows XP failures

[Anonymous Coward]

Changing company logo and name every 6-12 months Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others). Paying huge bonuses to upper managent. Paying huge car allowances to middle management including those who refuse to drive. Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items. Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later. Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.

Eh .. I had heard that the UK was very Americanized, but now I really believe it.

My condolences..

And so you fail basic logic.

[khasim]

The tools exist, so everyone is in agreement that bigfoot exists.

ooooh. Looks like you've fallen for "Argumentum ad numerum" (or maybe "Argumentum ad populum" . Too bad.http://www.infidels.org/news/atheism/logic.htm l#numerum

Maybe you'd like to try again? This time, leave out the logical fallacies, okay?

I'm asking you to back up your assertion, without a comparison ("adequate" is not a comparison, "better" and "best" are). You can not. That is an answer all of its own.

No. I said that they do NOT exist. If they do, identify them. If you cannot (and your refusal to do so shows very clearly that you cannot), then you have failed to counter my claim that they do not exist.

Re:And so you fail basic logic.

[AK Marc]

ooooh. Looks like you've fallen for "Argumentum ad numerum" (or maybe "Argumentum ad populum" .

Are you saying that the Microsoft installers do not exist? Or are you saying that you agree that the Microsoft installers exist, but the manner in which I said it is not a rhetorically accepted manner for stating a fact?

No. I said that they do NOT exist. If they do, identify them. If you cannot (and your refusal to do so shows very clearly that you cannot), then you have failed to counter my claim that they do not exist.

Ah, so you are asserting that all the tools included in a basic Windows server install, like Group Policy Management Console (which lets you control software installs on computers, including non-Microsoft programs) and Microsoft Software Update Services (for updates to Microsoft products) don't exist, and that you bring up 3rd party tools, but are claiming that Systems Management Server or other add-on tools from Microsoft don't exist.

Well, there are numerous tools included with the cheapest Windows Server and add-ons designed for software and upgrade management. I didn't understand that you were so anti-Microsoft, even though you have absolutely no idea what tools were available. I expected that someone so emotionally tied to a topic would have at least have a basic understanding of that topic. But this is Slashdot. I should assume that all people that talk as if they hate Microsoft have no knowledge of any of their products other than what they hear from others that hate Microsoft.

So, now that I've named the products that are included with Server or available directly from Microsoft, do I need to provide some other "proof" that they exist? Perhaps a blury photo of a box? But, now that we've covered that there are tools in Microsoft to do it (and I didn't even cover the hands-on ways that others were saying worked better in Linux, which can also be done easily in Windows), the point goes back to EDS picking a tool that was massive overkill for upgrading 60,000 computers from 2k to XP, and managing to screw it up. Somehow, on Slashdot, a company screwing up configuration on a 3rd party tool is Microsoft's fault, even when there were Microsoft tools available for the same thing that were unused.

If you ask, does that means you cannot read?

[khasim]

Are you saying that the Microsoft installers do not exist? Or are you saying that you agree that the Microsoft installers exist, but the manner in which I said it is not a rhetorically accepted manner for stating a fact?

Why don't you just read what I posted? Hmmmm?

Now, I say that there is no Bigfoot (Microsoft's tools are not sufficient).

http://slashdot.org/comments.pl?sid=130875&cid=109 33321
Do you have a problem with reading?

Ah, so you are asserting that all the tools included in a basic Windows server install, like Group Policy Management Console (which lets you control software installs on computers, including non-Microsoft programs) and Microsoft Software Update Services (for updates to Microsoft products) don't exist, and that you bring up 3rd party tools, but are claiming that Systems Management Server or other add-on tools from Microsoft don't exist.

No. I have not claimed that. Nor have I claimed that 5cm open end wrenches do not exist.

What I claimed is that Microsoft does not have the tools sufficent to handle the situation.

Not that Microsoft does not have tools available.

What you are doing is called a "strawman" http://www.infidels.org/news/atheism/logic.html#st rawman

It looks like you've failed basic logic a SECOND time.

Well, there are numerous tools included with the cheapest Windows Server and add-ons designed for software and upgrade management.

Yes, yet like my wrench example, NONE of them are sufficient to the job. Again you attempt the strawman.

So, now that I've named the products that are included with Server or available directly from Microsoft, do I need to provide some other "proof" that they exist?

Again, the existance of tools from Microsoft was never in question. That is your strawman.

All you have to do is show that the tools you've mentioned are sufficient to the task described earlier.

Fuck it. I'll make it really easy for you.

Just post a script or whatever tool and flags I'd need to type in to or whatever buttons I'd have to click on to identify all of the .dll's that are not included in any dependencies of any programs installed on the machine.

Under Debian it is "orphaner -a".

Can you do that? Hmmmm?

So, now that I've named the products that are included with Server or available directly from Microsoft, do I need to provide some other "proof" that they exist? Perhaps a blury photo of a box?

Again with the strawman. http://shop.store.yahoo.com/tmt/crsowrandwr.html Look! Tools! Yet not one of those tools will correctly apply an update.

One last time. The existance of tools is not in question.

What is in question, and what you are required to provide proof of, is the existance of Microsoft tools that would perform the required task. Identify the tool and identify how it would be used.

Smoke me a kipper, can you do that?

Re:If you ask, does that means you cannot read?

[AK Marc]

Just post a script or whatever tool and flags I'd need to type in to or whatever buttons I'd have to click on to identify all of the .dll's that are not included in any dependencies of any programs installed on the machine.

You can't, and you don't need to. That isn't how the updates work. But I think you already knew that, and you are purposefully tailoring your questions in a way you know Microsoft will fail. They have the tools that will perform the tasks presented. I posted the names of the tools. You know what they are, you know they exist, and since you have asserted that the person that makes an assertion must back it up, and you've asserted Microsoft's tools are inadequate, they only way to prove your case and remain intellectually honest is to prove the tools I listed as inadequate for the task at hand.

However, I would expect to see another rhetorical deconstruction of my post with a complete lack of substance.

Under Debian it is "orphaner -a".

So, that will do it on some arbitrary list of networked PCs at the same time? That is what is being discussed.

identify how it would be used.

I've identified the tools. Requesting that I describe their use is a new requirement that you've added in this post and this post alone because you know I've adequately answered everything up to this point. Using the tools is deployment specific (since you seem so ignorant on Microsoft tools and features, I would have to explain Active Directory to you, as well as the dreaded "Next" button).

Re:what company is this?

[mikechant]

The company is very strict about employee comments - all have to be cleared with the press office on pain of dismissal. Perhaps unwisely, my slashdot id is quite similar to my real name. So the most I can say is that I work for a large (but not one of the very largest) European software consultancy which despite its size, few people seem to have heard of.

So you admit you cannot do that.

[khasim]

You can't, and you don't need to.

That's half right. YOU can't. But you do need to be able to do that.

That is the only way to show that your package management system is handling everything. Every single file on your system should either be identified by the package management system or be known to you as something you created.

But I think you already knew that, and you are purposefully tailoring your questions in a way you know Microsoft will fail.

That's called "knowledge". I know Microsoft products (probably better than you do). I know their functionality, I know their deficiencies.

But I did provide you with the opportunity to show that I was wrong and that you know more than I.

You only believe Microsoft has certain capabilities because you've never used anything else. The reason there are so many 3rd parties tools to facilitate pushing updates to workstations is because Microsoft's core system is fatally flawed.

All those other vendors are selling products that take different approaches to band-aid'ing those flaws. Which is why so many people have so many problems patching Windows systems.

They have the tools that will perform the tasks presented. I posted the names of the tools.

And that is the difference between your ignorance and my knowledge.

apt-get install deborphan
orphaner -a

I can easily and clearly post the instructions. You cannot. The reason you cannot is because you don't know how to. All you can do is recite the names of tools that you hope will perform the function. :)

You know what they are, you know they exist, and since you have asserted that the person that makes an assertion must back it up, and you've asserted Microsoft's tools are inadequate, they only way to prove your case and remain intellectually honest is to prove the tools I listed as inadequate for the task at hand.

And that is your THIRD failure of basic logic.

There is no way to prove a negative.

I state that Microsoft's tools are insufficient. You say they are sufficient.

My "proof" is that you are unable to provide instructions on how to accomplish that task while I have done so twice now.

Here's a third time:
apt-get install deborphan
orphaner -a

There, I've done it THREE times and you're still complaining. :D

I've identified the tools. Requesting that I describe their use is a new requirement that you've added in this post and this post alone because you know I've adequately answered everything up to this point.

Back to the Bigfoot example? :) Yes, you've identified what YOU claim is "proof". But you still haven't shown that it IS "proof".

To do that, you have to show how to use those tools to accomplish the task.

So far, you've been unable to do so.

apt-get install deborphan
orphaner -a

Yet you'll continue to claim that you don't have to show that the tools actually do what you claim they do. :)

Well, there's one reason I can think of that you'd refuse to do so. Particularly when I've shown how easy it is to show exactly how to do it with Debian.

Go ahead. Keep claiming that it is easy to do on Windows. I'm sure that someone will believe that's it's easy and you just don't have to show how to do it. :D

apt-get install deborphan
orphaner -a

Re:So you admit you cannot do that.

[AK Marc]

I've shown how easy it is to show exactly how to do it with Debian.

So, that will, over the network with no other intervention, check file versions over some arbitrary subset of the networked computers available? Since you claim it is all that is needed, how would one enter which of the computers to check and which not to check?

EDS

[orasio]

I was talking about EDS management.
In my country, IBM runs many critical government IT contracts, they sign the contract, and then start hiring junior java developers for that.
You don't always get what you pay for.

I'll limit this to your failings.

[khasim]

Since you claim it is all that is needed, how would one enter which of the computers to check and which not to check?

Awww, since I won't let you play with your strawman or other logical fallacies, now you have to resort to lies?

Where did I claim that that is all that is needed?

That is the evidence that Debian has a functioning package management system and that Windows does not.

With a functioning package management system as the base, I can do things you've never dreamed possible.

cd /lib
dpkg -S *

That will tell me the package that installed each file in the /lib directory. Every file.

Go ahead and show me how to do that with Windows.

Re:I'll limit this to your failings.

[AK Marc]

That will tell me the package that installed each file in the /lib directory. Every file.

Of an arbitrary number of computers over a network? The question has never been about local tools but has been about being able to do it for a subset of networked computers, multiple at a time, without having to touch the affected (or unaffected) computers.

So you admit that Microsoft failed?

[khasim]

Of an arbitrary number of computers over a network? The question has never been about local tools but has been about being able to do it for a subset of networked computers, multiple at a time, without having to touch the affected (or unaffected) computers.

So you admit that Microsoft does not have the tools?

Is that what you're saying? Before this goes anywhere else, that question will be answered.

Again ....

apt-get install deborphan
orphaner -a

That's all you have to do.

Re:So you admit that Microsoft failed?

[AK Marc]

That's all you have to do.

Please, for my edification, tell me in a yes or no whether the tools you mention are capable of performing the checks or upgrades on multiple remote computers at the same time without having to visit any of the remote computers affected or not affected.

Actually no.

[schon]

Glibc since libc6 is forward and backward compatable.

The functions may be compatable, but it's common for binaries compiled for newer versions to reject older versions.

You would have to reboot the machines in single-user mode and, using all those shiny static binaries in /sbin, you'd simply recover the correct Glibc.

While this is historically the purpose of /sbin, there is *no* Linux distribution I'm aware of that does this. The "s" in /sbin now apparently stands for "super-user", not "static".

If you accidentally install a version of glibc that's too old for your binaries, your system will refuse to boot; and even if you've got a static shell compiled, you won't be able to do anything.

The correct way to do it would be to boot the system with a rescue/live CD and then reinstall the glibc.

Stay focused, okay?

[khasim]

Please, for my edification, tell me in a yes or no whether the tools you mention are capable of performing the checks or upgrades on multiple remote computers at the same time without having to visit any of the remote computers affected or not affected.

We're still dealing with your claims about Microsoft's tools. Once that is settled, we can move on to other matters.

This reminds me of a funny story about Windows fans. It seems that they are rather, how to say it ...., they have limited mental capabilities. Given a problem, they try to push the buttons they've pushed before. When that fails, the try their magic words ("reboot" or "re-install" or "reload"). When that fails, they try to blame someone else in an attempt to distract attention from their problem.

Maybe you had to be there. I was amused. Anyway ... stay focused. This is still about Microsoft's tools. Once that is settled, we can talk about other subjects. Okay? Thanks. :)

Re:Stay focused, okay?

[AK Marc]

We're still dealing with your claims about Microsoft's tools.

No, we are dealing with your claims. You claimed they didn't have the tools necessary for the job. You said other OSs do have the tools. You gave an example. I'm trying to define the example you gave in order to name the comparable Microsoft tool.

Then you already have the information.

[khasim]

You gave an example. I'm trying to define the example you gave in order to name the comparable Microsoft tool.

Then I have already given you the information you are requesting. http://slashdot.org/comments.pl?sid=130875&cid=109 42806

What you are actually trying to do is to take the discussion off on a tangent in an attempt to find something that you can claim I am wrong about.

I'm not going to let you do that.

I said that Microsoft does not have the tools.
You said that Microsoft did.

It is up to you to identify the tools and how they are used to accomplish that. Otherwise, my statement stands.

apt-get install deborphan
orphaner -a

Once you that has been dealt with, then I will be willing to give you specifics on how to install/update software on a network of Debian machines via the network.

And that gets back the the story about Windows fans that I found so amusing. Once your magic words failed (you naming tools from Microsoft that I already knew do not have that functionality but that you did not know I knew and you hoped would "prove" me wrong), when pressed for specifics on how those magic words would work, you resort to attempts to blame others for your failings (demanding that I show the existance of functionality in Debian that you have not shown exists nor stated does not exist in Windows).

(Here's one example)
apt-get install deborphan
orphaner -a

(Here's another example)
dpkg -S /lib/*

There. I've provided TWO very specific examples (down to the exact keystrokes to type) of the functionality of a real package management system.

What have you provided? Nothing but empty claims.

So far, I'm up 2 to your 0.

Answer the yes/no question

[AK Marc]

Then I have already given you the information you are requesting.

No, you have not. You have not answered whether it is able to check multiple networked computers at the same time. It is a simple question I'm asking for explicit clarification which you are apparently refusing to answer.

This is called a "hyperlink".

[khasim]

No, you have not. You have not answered whether it is able to check multiple networked computers at the same time. It is a simple question I'm asking for explicit clarification which you are apparently refusing to answer.

Again, I'll deal with that once you've substantiated your original claim.

In case you're going to claim that you have, here is your first reply to me in this thread. It contains your original claim: http://slashdot.org/comments.pl?sid=130875&cid=109 30225

Go ahead, substantiate your claim:

Perhaps he read it and recognized it to be wrong? Microsoft does have a decent package management system.

Right now, all your attempting to do is to take the discussion off on a tangent that you believe you can win at.

That's not going to happen. You have to substantiate your original claim before any other tangents are examined.

Re:This is called a "hyperlink".

[AK Marc]

In case you're going to claim that you have, here is your first reply to me in this thread.

Your claim was first:

With Windows, since Microsoft doesn't have a decent package managment system,

You haven't backed up your initial claim that there isn't a decent package management system. You said it was impossible to prove a negative. So I provided the names of the package management systems that are decent.

So, can you now answer my yes/no question I've asked repeatedly that you refuse to answer?

Re:This is called a "hyperlink".

[AK Marc]

You have to substantiate your original claim before any other tangents are examined.

Why? If I were to agree with your initial assertion, made long before I came along, that they don't have a decent package, would you then answer my question regarding the operation of the tools you listed on networked computers?

You don't even know what one is, do you?

[khasim]

You haven't backed up your initial claim that there isn't a decent package management system. You said it was impossible to prove a negative. So I provided the names of the package management systems that are decent.

No you didn't. You listed a bunch of Microsoft tools.

You don't even know what a package managment system is, do you? Let me remind you of what you had claimed. http://slashdot.org/comments.pl?sid=130875&cid=109 37641

Ah, so you are asserting that all the tools included in a basic Windows server install, like Group Policy Management Console (which lets you control software installs on computers, including non-Microsoft programs) and Microsoft Software Update Services (for updates to Microsoft products) don't exist, and that you bring up 3rd party tools, but are claiming that Systems Management Server or other add-on tools from Microsoft don't exist.

I thought you were trying to confuse the issue by just listing whatever tools you could find.

Now I believe that you really do not know what a package management system is.

dpkg -S /lib/*
That will tell you which package installed which file in the /lib directory. That is a function of the package management system.

Windows' does not have that capability.

That capability is required for a decent package management system (which you claim Windows has). http://slashdot.org/comments.pl?sid=130875&cid=109 30225

Perhaps he read it and recognized it to be wrong? Microsoft does have a decent package management system.

So, you claim Windows "does have a decent package managment system", but I say it cannot even keep track of the most basic information (which package installed which file). So you demanded that I prove my point. I did so by giving a specific example of a task that you have so far been unable to duplicate. Too bad for you.

Go ahead and support your claim. Show how I can get the same functionality with Windows as:

#1
apt-get install deborphan
orphaner -a

#2
dpkg -S /lib/*

So, can you now answer my yes/no question I've asked repeatedly that you refuse to answer?

I've already answered that. You have to either admit you were wrong or support your claim by showing me the functionality I used as an example.

You think that just getting files onto a box is "decent package management", don't you. :)

Yep, you're definately a Windows fan.

Even when I give you specific examples you cannot admit that Windows is flawed. :D

Of course.

[khasim]

Why? If I were to agree with your initial assertion, made long before I came along, that they don't have a decent package, would you then answer my question regarding the operation of the tools you listed on networked computers?

Of course. But whether or not you'll understand it is another issue.

It's a very simple process once you understand the basics of package management.

#1. All files belong to packages.

#2. Packages have dependencies upon other packages.

#3. So to update a bunch of systems on a network, all you have to do is make the updated package available to the system. With Debian, this is controlled in /etc/apt/sources.list.

#4. Then you trigger the system to download and install the update. This is done via ssh or telnet or rcp whatever you have.

Because the system is downloading and upgrading/installing the package, it is much more difficult to make an error that gets installed.

If you are upgrading libfoo4, but the machines that you don't want to upgrade have libfoo3, that's not a problem. Both can exist on the same system. That's when orphaner comes into play so you can see what is installed that isn't referenced by other packages.

If you are upgrading something that would break the system (although I don't know what that could be), you'd still have to get the dependencies correct. The system would throw an error when it saw an upgrade that didn't match the dependencies of other packages. (if no dependency issues, then it wouldn't break the system, although that specific app may not work).

So, in order to get the same result as in the article, you'd have to:

#1. build an upgrade package with the bad file in it.

#2. build the package with broken dependencies

#3. build updates to the packages that depend upon that package

#4. put those packages in the incorrect directory.

In other words, too many things have to go wrong in very specific ways to cause these problems.

It's easy to check various items before installing the software. Kernel version, library versions, group existance, user existance, etc. All you have to do is parse the associated file/report. uname -a tells you the kernel package you have installed, for example.

So, for Debian, to do a controlled, automatic update of all machines on the network with kernel package 2.4.25-1-386, with packages libfoo4 and myapp4, the process is ....

ssh to first candidate
check kernel version
apt-get install libfoo4, myapp4
log success/errors
next machine
mail success/error log to myself

The key concept is having the stable, reliable package management system on the machines. Once you have that, each step above that is simple and logical and errors will cause the update/install process to fail rather than to break the machine (which is how it should be).

If you can, grab an old machine and install Debian testing on it. Play with it for a while and you'll see how easy a single machine is to administer. With just a bit of scripting, an entire network of them is only a bit more complicated.

Not to mention troubleshooting those workstations. I can quickly compare the exact package listing of two machines (one not working / one working) to see where they differ on packages, file versions, even file checksums. So troubleshooting a software package problem takes 5 minutes on matching workstations.

Re:Of course.

[AK Marc]

So, in order to get the same result as in the article, you'd have to:

#1. build an upgrade package with the bad file in it.

#2. build the package with broken dependencies

#3. build updates to the packages that depend upon that package

#4. put those packages in the incorrect directory.

In other words, too many things have to go wrong in very specific ways to cause these problems.

From how I read it in the article, they had a 3rd party tool that doesn't look for dependencies properly (something the built-in Microsoft tools do), it was otherwise perfect (presumably). They then had some error, whether software or human is undetermined. The software without proper checks copied files onto incompatible systems, overwriting valid files. There was one and only one failure. It was a 3rd party tool, not part of the OS, nor related to any Microsoft tool. It copied files without regard to what was already on there. That is one little error (improper version checking). It caused major problems ni affected systems.

It seems your bigest objection is that there isn't a command line version of the GUI tools. I can't give you a command to type in and do it. There isn't one. That doesn't mean that the tools don't exist. That doesn't mean that the tools aren't adequate. That just means that you have to use a GUI for functionality that is not available at a command line.

Command lines are easier for me to type here.

[khasim]

It seems your bigest objection is that there isn't a command line version of the GUI tools. I can't give you a command to type in and do it. There isn't one. That doesn't mean that the tools don't exist. That doesn't mean that the tools aren't adequate. That just means that you have to use a GUI for functionality that is not available at a command line.

Nope. My biggest objection is that the functionality does not exist for Windows, as I've said many, many times in this thread.

With a Windows system, the package contains the information, not the system itself.

I can build a package with SMS for SMS to distribute foo.dll and SMS will distribute that package.

I then install an update to foo.dll from Microsoft's website.

So, two different packages have installed foo.dll.

I can then have the SMS package remove foo.dll and it will be gone. Even though the file should now be owned by whatever package installed it from Microsoft's website.

From how I read it in the article, they had a 3rd party tool that doesn't look for dependencies properly (something the built-in Microsoft tools do), it was otherwise perfect (presumably). They then had some error, whether software or human is undetermined.

Again, such an error is easy with Windows (as demonstrated in the article). It is far more difficult with Debian. And that is because Debian has a working package management system.

There was one and only one failure. It was a 3rd party tool, not part of the OS, nor related to any Microsoft tool. It copied files without regard to what was already on there. That is one little error (improper version checking). It caused major problems ni affected systems.

Yet I have shown how having only ONE error will result in the update NOT being installed on a Debian system. Which is how it should be.

If there is an error, the update process should fail rather than corrupt the workstation.

Under Debian, it takes multiple, very specific errors to break the system.