Slashdot Mirror
White House CIO Describes His 'Worst Day' Ever
dcblogs writes "In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers."
All my machines have a floppy.
I don't understand how adding a peripheral can make the machine "worse"?
They inherited a system that "lost" months/years worth of emails during the Bush administration. Of course it all sucked, it was designed to.
I have to say I got a chuckle when I got to the part about "inheriting" their IT problems. Obama "inherited" all his problems after all!
I recently took over for a staff which had been interned in their positions for the better part of a decade. Out with the old in-house staff, in with the new outsourced IT 'team'.
I can easily see how this happens, outside procurement and ineptitude problems on the part of the previous WH IT staff. When you've got what amounts to 'institutional knowledge death', with the institution carrying on, you've got to over-staff for some time or things fall apart completely while you play catch up. With a situation where you don't understand it all, are under staffed or under skilled, you're faced with only a couple options when you come in behind the curve, with aging equipment and software: you either start replacing everything you can, as you are able, as quick as you can, or you start suffering outages. It's even worse if things are mismanaged and things are failing all around you.
As for the claims of the article? Meh. I'm actually not that impressed by his claims to the poitn where I think 'this is bad':
In 2008, "floppy drives" weren't all that uncommon. I remember servicing Core machines which had floppy drives, still. We're not talking biege boxes with ISA slots here, necessarily - with a 4 year replacement schedule for desktops, floppy drives don't speak of ineptitude.
The 80-hour-week thing means nothing. It might mean he was understaffed, or that he's a workaholic. To me, it sounds like the meaningless words of a political appointee.
"Over 82% of the White House technology had reached end of life" means nothing. If they were on a 3-year replacement schedule for desktops and they had 10/100 switching, I can easily see where you'd come to that number.
He had one "data center", with no redundancy. A bit of a contradiction, yeah? This is made somewhat less impressive by the fact that this administration, in particular, was a bunch of Nancys when they came in with "oh woes, look at this mess", quite obviously overstating things for dramatic media effect.
"Our email servers went down for 21 hours" isn't a statement of disaster, it's a statement of ineptitude. If they got the mail servers back up, with the data intact, the problem wasn't with the environment but the people involved (or the lack of staffing). His BB starting to have mail incoming suggests a reinstall wasn't required, so safe to say BES was OK, so who knows what the real 'problem' was which caused a day of outage...
Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%. Realize that a) this is a political appointee talking, b) it's a seemingly non-technical manager (he's up in his datacenter, lookin' for redundancy!), and c) this is the government we're talking about, after all. Anyone who's had any dealings with them on a technical level realizes that 'setbacks' and 'shortcomings' or 'difficult problems' or the like are (probably!) due to ineptitude. Yes, sadly, even amongst the elite (though not necessarily of their own doing - thank you bureaucratic bullshit).
Granted, this may not have been the case when BO came to the WH and took over. They may have had previous IT staffers who stayed through the transition, but I'm guessing they did not (due to political mistrust issues). It could've been a genuine clusterfuck. Sometimes it's nothing and people cry about the sky falling as they pull down the curtain; sometimes, it really is bad. (If you understand weather patterns, you may recognize a summer storm to not be the disaster that chicken little claims...)
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
23% down sounds about average for MSExchange servers.
Only on slashdot could such ignorance get modded up.
On a bad bad day as a consultant, I have to fix scenarios with Exchange where everything blew up and theyre down for a single day-- MAYBE 2-- out of several years uptime.
Thats with the clients who have no full time IT staff whatsoever and a shoestring budget.
Possibly if you have no idea what youre doing, or dont know anything about exchange, then yea 23% might be an OK guess.
Youre free to pretend we still live in a day where it is unnecessary to do group scheduling through email, but you would be wrong.
Group scheduling and email are different applications. Combining them in one backend is shortsighted.
Oh SHUT up.
The reason government can't get anything done, generally, is there's always some jackass out there questioning whether a thing is needed because it happens not to be exactly what they want, or why workers cost anything at all since their life is in the shitter so why should a government employee make money either?
There is a significant interest in this country in starving government, and then mocking it for under-performing. That's a combination of arguments only an imbecile would make.
From what I've read, there's only one firm that does White House transitions. I think it's Bechtel, but it's been so long that I've read anything about transitions that I have around a 15% confidence level in that piece of data.
Google "white house transition" and you'll see that it's a total mess. If you want to read about it, there's info here:
http://whitehousetransitionproject.org/
From what little I've read, you basically get a mostly empty building (the White House). It's up to the team to build/rebuild the infrastructure...but as any operations person knows, IT infrastructure is usually way behind everything else. The general executive branch IT has been a low priority for decades. What's more important, email or setting up the phone so the president can call someone (or someone can call the president)?
At that point, the team is probably so far behind that they're screwed continuously for the one or two terms.
Are the guys running the systems any good? I'd ask you: how many of you could pass a background and attitude check? You think the process etc at your workplace is bad, imagine how bad it is in the Executive branch.
That said, it might be fun...but it's probably a nightmare. "I can't print out this $15 billion dollar appropriation because the f*cking printer doesn't work!" "People in PA are starving because the email server ate all of our emails!"
Every minute is a crisis, with everyone breathing down your neck 24/7. Does that sound like something you'd want to do for 24/7/365/4 years?
If a system is end of life and dies can you say it was too bad or crappy without knowing how long it was kept running?
Everything has a shelf life and must be either upgraded or replaced eventually. Even the Large Hadron Collider is nothing more than a replacement of the Large Electron–Positron Collider before it which reached the end of its useful life. I had a similar discussion with an engineer at the industrial plant I work it. We have a vibration monitoring system which died and needed replacing, and he also called the system "crap." For some perspective the system was obsolete in 1995. The two subsequent models are now also obsolete yet this thing has been humming away just fine for 17 after the vendor stopped supporting it.
Yet someone called it crap.
I would deploy IBM Domino like in the days of Clinton, which Bush switched to Microsoft Exchange. Reliability went downhill with that decision.
Domino runs cheap and fast and reliable. And has always active clustering so you don't have to deal with downtime. IBM simply has a much longer track record of delivering reliable computing than Microsoft.
People are not replacing Domino with Exchange because it is more reliable. They do it because *everybody* hates Lotus Notes.
This being said, any user that has a complaint against Lotus Notes should be required to work with Groupwise for a week.
lucm, indeed.
You miss the fact that he INHERITED that system. That's how politics works. The lame duck guy in charge of the White House (which would be the former President DIRECTLY) let the thing rot...
Part can be attributed to the old staff being done with the position and the new guy will just buy new stuff anyway... Almost fair?
Part of all the downtime was a FEATURE that the pervious administration used to their full advantage... They were über controll freaks... Controlling information of their own "trusted" employees was part of D.C.'s daily routine. The rot was deliberate to stop communications from being added to the archives.
Group scheduling and email are different applications. Combining them in one backend is shortsighted.
True, but Exchange isn't an "email" application--it is a group productivity application that includes email, group calendaring and scheduling, tasks, and collaboration.
I understand there are MS haters who will bash Exchange relentlessly, with any label on it, but let's try to be even a tiny bit accurate. Exchange isn't an "email" service and hasn't been exclusively that for nearly 15 years: Time to come up with some new criticisms, the old ones don't apply.
Who did what now?
More to the point, businesses haven't wanted an email service for nearly 15 years.
They want the group productivity application. But they don't call it that because the most visible part - the part they really see - is the email.
in my experience, Zimbra is a bloated pig with its share of availability problems as well. I hate the whole "let's take a bunch of open source pieces but just throw them together as an inflexible blob of crap in /opt" approach. The installer leaves a lot to be desired as well, with key components around setting proper permissions resulting in an install that will never work until you manually fix it.