Slashdot Mirror
White House CIO Describes His 'Worst Day' Ever
dcblogs writes "In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers."
23% down sounds about average for MSExchange servers.
I'm sure they just wasted money buying more of the same crap.
All my machines have a floppy.
I don't understand how adding a peripheral can make the machine "worse"?
TFA indicates clearly not only that there's no urgent need for IT geeks to unionize, but also what havoc they could wreak if they ever did.
The Wolfpack Project: BitCoin + Crowdfunding = Political Accountability
They inherited a system that "lost" months/years worth of emails during the Bush administration. Of course it all sucked, it was designed to.
Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives ...
Considering the sort of people who are using these machines, it seems almost appropriate somehow.
Maybe keeping them technologically underpowered is actually a good thing. Those crafty, crafty White House IT gurus.
The Wolfpack Project: BitCoin + Crowdfunding = Political Accountability
The problem is the procurement process. It takes a hell of a long time to get IT resources ordered, and often by the time they are actually put into service half of their warranty life-time has expired. It has nothing to do with a lack of knowledge on the OMB IT front, it's got everything to do with the red tape they have to cut through to make anything happen.
Too bad we probably paid billions for such crappy infra.
The DOD still standardizes on IE 6 from what I am told.
It is not cutting edge at all.
Obama's staff at least did a WTF and quickly hired the first CIO to clear the red tape. True he was not a good CIO, but someone was needed. It is unacceptable to have email down PERIOD at such an important job. The president's job is the most important in the world and any loss of email or downtime when WW3 starts or something unrelated is unacceptable. In the private sector downtime is measured by costs with employees salary x time of outage.
As information is needed rapidly the whiteshouse should have the most advanced technology second only to the DOD and IT needs to be involved.
Maybe Bush was out to lunch which is not surprising but if I were president I would be flipping the second email would go down for more than a few seconds as I know my staff needs up to the second information to do their jobs.
http://saveie6.com/
That's madness! Everyone knows that the floppy drive dictates the speed, quality, and age of the computer!
I have to say I got a chuckle when I got to the part about "inheriting" their IT problems. Obama "inherited" all his problems after all!
Consider other people his age in that year. 2/3 of them didn't USE email. Many of us had parent that were that age at that time and know what it's like dealing with "old people that refuse to move out of the stone age".
It would not surprise me in the least to hear that Bush1 (or Bush2 for that matter) never opened email, ever, and got his information on dead tree and in meetings. "Email's down? is that bad?"
I work for the Department of Redundancy Department.
Perhaps also known as the last presidential Blackberry. Of course, parts of the government still favor Blackberry, but then apparently parts still like floppy drives too. With the recent /. posts on DOD Androids (not the kind that lead to Skynet comments) and the like, one wonders how much longer even this will last.
The equipment of some government agencies I have worked for is kind of prehistoric. I you are using Windows 95 as a file server, you are in serious trouble.
I used to work there, so I can't unload. Let's just say that the EOP has earned every bit of fail they've ever had. (Including the email system, which is what happens when you put brand new software on past EOL hardware against the advice of the people responsible for actually making it function.)
The DOD still standardizes on IE 6 from what I am told.
The different services have their own IT departments; they even have their own networks, NIPR and SIPR are just two of many. Not sure about the DOD proper, but even the Army is phasing out IE 6 and XP. I'm pretty sure the Air Force and Navy are mostly on Win 7, and the Marines got some new brightly colored rocks with sparkly beads.
Too bad they didn't teach George W Bush about IT stuff. With how often he went to Texas to clear brush, he'd have that exchange DB backed up in 5 places, compressed properly, and on SSD RAIDs :-P
Um, there were CIOs for the EOP before Brook Colangelo.
-- Two men say they're Jesus. One of them must be wrong. - Dire Straits
I recently took over for a staff which had been interned in their positions for the better part of a decade. Out with the old in-house staff, in with the new outsourced IT 'team'.
I can easily see how this happens, outside procurement and ineptitude problems on the part of the previous WH IT staff. When you've got what amounts to 'institutional knowledge death', with the institution carrying on, you've got to over-staff for some time or things fall apart completely while you play catch up. With a situation where you don't understand it all, are under staffed or under skilled, you're faced with only a couple options when you come in behind the curve, with aging equipment and software: you either start replacing everything you can, as you are able, as quick as you can, or you start suffering outages. It's even worse if things are mismanaged and things are failing all around you.
As for the claims of the article? Meh. I'm actually not that impressed by his claims to the poitn where I think 'this is bad':
In 2008, "floppy drives" weren't all that uncommon. I remember servicing Core machines which had floppy drives, still. We're not talking biege boxes with ISA slots here, necessarily - with a 4 year replacement schedule for desktops, floppy drives don't speak of ineptitude.
The 80-hour-week thing means nothing. It might mean he was understaffed, or that he's a workaholic. To me, it sounds like the meaningless words of a political appointee.
"Over 82% of the White House technology had reached end of life" means nothing. If they were on a 3-year replacement schedule for desktops and they had 10/100 switching, I can easily see where you'd come to that number.
He had one "data center", with no redundancy. A bit of a contradiction, yeah? This is made somewhat less impressive by the fact that this administration, in particular, was a bunch of Nancys when they came in with "oh woes, look at this mess", quite obviously overstating things for dramatic media effect.
"Our email servers went down for 21 hours" isn't a statement of disaster, it's a statement of ineptitude. If they got the mail servers back up, with the data intact, the problem wasn't with the environment but the people involved (or the lack of staffing). His BB starting to have mail incoming suggests a reinstall wasn't required, so safe to say BES was OK, so who knows what the real 'problem' was which caused a day of outage...
Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%. Realize that a) this is a political appointee talking, b) it's a seemingly non-technical manager (he's up in his datacenter, lookin' for redundancy!), and c) this is the government we're talking about, after all. Anyone who's had any dealings with them on a technical level realizes that 'setbacks' and 'shortcomings' or 'difficult problems' or the like are (probably!) due to ineptitude. Yes, sadly, even amongst the elite (though not necessarily of their own doing - thank you bureaucratic bullshit).
Granted, this may not have been the case when BO came to the WH and took over. They may have had previous IT staffers who stayed through the transition, but I'm guessing they did not (due to political mistrust issues). It could've been a genuine clusterfuck. Sometimes it's nothing and people cry about the sky falling as they pull down the curtain; sometimes, it really is bad. (If you understand weather patterns, you may recognize a summer storm to not be the disaster that chicken little claims...)
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
Yeah, and the rumor about "Yellow Cake" in Iraq was a fouled up email about what Saddam had with his friends on his Birthday.
The intern convinced Colangelo that there was a great need for automation.
WTF did he think computers were for before that little piece of enlightenment hit him?
I really hope that was just some random idiot fluff from author of TFA and not an actual sentiment from someone with "Information" in their job title.
.: Semper Absurda
he ever touched the 'real' computers the Bush admin used.
"Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago."
Rahm Emanuel has few redeeming virtues and deserved that fitting computer he got. He too had reached his end that the White House, no too soon enough, however. Floppy, Rahm? Memory? Nah...
The Navy is mostly on XP & IE7.
>> Over 82% of the White House's technology had reached its end of life. And all keyboards were missing the W.
Facts take all of the premium out of arm waving - T. Reynolds
At least all the keyboards still had the "W" key!
Ken
Apparently their systems are from the 1990s which was, in retrospect, the worst decade to buy any kind of IT equipment. Imagine they had some Unix system with VT100 terminals. This would have given them an easy upgrade path. They could have made simple and secure ways to remotely login, plus they could have simply replaced the system with a more modern Linux system, etc...
I do not believe the Oval Office has had a computer installed in it. Ever. If I'm not mistaken it is an even bigger deal than the Presidential Blackberry - it could be argued that every page POTUS surfed would be recorded and archived forever...
Do you really want your President to sit around wondering why his browser is frozen?
Ken
Oh SHUT up.
The reason government can't get anything done, generally, is there's always some jackass out there questioning whether a thing is needed because it happens not to be exactly what they want, or why workers cost anything at all since their life is in the shitter so why should a government employee make money either?
There is a significant interest in this country in starving government, and then mocking it for under-performing. That's a combination of arguments only an imbecile would make.
From what I've read, there's only one firm that does White House transitions. I think it's Bechtel, but it's been so long that I've read anything about transitions that I have around a 15% confidence level in that piece of data.
Google "white house transition" and you'll see that it's a total mess. If you want to read about it, there's info here:
http://whitehousetransitionproject.org/
From what little I've read, you basically get a mostly empty building (the White House). It's up to the team to build/rebuild the infrastructure...but as any operations person knows, IT infrastructure is usually way behind everything else. The general executive branch IT has been a low priority for decades. What's more important, email or setting up the phone so the president can call someone (or someone can call the president)?
At that point, the team is probably so far behind that they're screwed continuously for the one or two terms.
Are the guys running the systems any good? I'd ask you: how many of you could pass a background and attitude check? You think the process etc at your workplace is bad, imagine how bad it is in the Executive branch.
That said, it might be fun...but it's probably a nightmare. "I can't print out this $15 billion dollar appropriation because the f*cking printer doesn't work!" "People in PA are starving because the email server ate all of our emails!"
Every minute is a crisis, with everyone breathing down your neck 24/7. Does that sound like something you'd want to do for 24/7/365/4 years?
Well that was a very disorganized, poorly paced retelling of what was probably an interesting talk.
http://www.whitehouse.gov/blog/2011/02/11/
http://developmentseed.org/blog/2011/feb/14/white-house-using-open-atrium/
www.openatrium.com
For civilians trying to avoid another HBGary-type SQL-injection cascading breach, building a PHP website using the Drupal framework means benefiting from the eyeballs that watch sites like whitehouse.gov, and others. These same eyeballs, and many others contribute their security patches back to drupal.org. Although I imagine their OpenAtrium groupware is behind a firewall.
OpenAtrium is 100% free open-source server software, that reaches out really well to tables and other mobile devices too.
You can't be ahead of the curve, if you're stuck in a loop.
""In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers.""
So......I'm guessing no Crysis 2 then? Geez. I can run it with everything cranked.
I would deploy IBM Domino like in the days of Clinton, which Bush switched to Microsoft Exchange. Reliability went downhill with that decision.
Domino runs cheap and fast and reliable. And has always active clustering so you don't have to deal with downtime. IBM simply has a much longer track record of delivering reliable computing than Microsoft.
They should follow the example of Sarah Palin and use Yahoo Mail.
I can picture Kim Jong-un calling the helpdesk: "please add leaderofthefreeworld61@yahoo.com on my trusted recipients list, we almost initiated a nuclear holocaust because I missed an important message in my junk mail folder".
lucm, indeed.
... the day you find you're out of IPv4 addresses, and all your systems are not IPv6 ready, in spite of it being a government initiative to have all systems IPv6 ready by this year.
The Marine Corps is migrating to Windows 7 and Server 2008 R2. They are providing an in-house solution because the joint Navy Marine Corps Intranet (NMCI) wasn't working well for the MC part.
tightly regulated
Thanks for that, I needed a good laugh.
Then again, it's not actually hooked up. It's in there to block a bay in the front so there isn't a big hole. (The case on this PC build is more than a few years old and I didn't have the plastic panel to block that bay and I was too cheap to actually buy one online.)
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
My favorite part:
Screw the professionals, interns inherit the future even in White House.
You HAVE TO spend money on IT infrastructure and equipment yearly. I don't care what you think, Us in IT know a shitload more about IT than you even pretend to think you know. You hired us experts, why the hell don't you let us do our job?
Honestly, the White house is a small scale IT setup, why the hell dont they have a 2 year replacement on everything there? IT could be done for less than the floor cleaning expenses they have yearly.
Do not look at laser with remaining good eye.
Anyone who's spent significant time working around the federal government knows that they keep old equipment around much longer than anyone would expect. Simple example, In the '80s, I worked on an Air Force system that still had a punch card reader. It's not uncommon to find this kind of silliness even today. Reminds me of the cassette player on my new '03 Infiniti...needed that like tits on a bull.
Just another day in Paradise
Turn off the projector, son.
Nobody of any consequence in the Democratic Party has said that Capitalism has failed in general. Parts of the organizations that make up the model have certainly failed, but capitalism as a concept and economic model works fine provided everyone plays fairly.
This article is partially correct but leaves out the actual technical issues involved.
Someone *from* that Datacenter here at that time. Here's what really happened.
The old administration did not care about the existing IT infrastructure because they were on their way out. They wanted no changes made- just that things be left up. Yes the email system was old and past EOL, but the outages were really the perfect storm of everything that could hit the fan actually hitting the fan at the same time.
The facility was doing work on the power system- the UPS to be specific. Somewhere along the line they messed up, and cut the power. *All* of the power. Datacenter goes dark. They brought the power back up, but then tripped it again before bringing it up for good. This detail is what caused the weekend of hell.
The SAN that the clustered email servers (yes, clustered, they *were* redundant) had the stores on was an EMC Symmetrix. It has a built-in battery backup system so that if the SAN looses power it has enough stored to flush the cache to disk. The power going off started this process. The power going back on triggered the response to stop flushing the cache and start checking and rebuilding. Then the power went off again. This is the part where the specific details get hazy but in effect the SAN did not like this. I don't believe it had enough power to totally flush the cache and/or it did not have the logic built in to handle an outage while in recovery mode. The result was a downed SAN that *would not come back up*. Now all of the data was down and nothing could be done but wait for the vendor to show up and try to fix it.
At the same time we were dealing with *every* server being off and having to come back up. There were hundreds. Luckily most did. Some did not. Some were important, such as in the case of *both* the servers in a clustered system that would not boot- which just so happened to be the system that some of the say "more important" VIPs were on. These were old systems running Exchange 2000 on Windows 2000. Long past due, but kept up by the staff since the EOP would not approve a new email infrastructure.
Eventually the systems would be restored and everything would be back on-line. In the meantime though Brook thought it would be a good idea to spend untold amounts of money to bring in MS Engineers to look things. They cost a lot of money and made a bunch of reports but they didn't fix a damn thing. The staff that was already there found the issues with the servers and fixed them.
There were later headaches, such as when mentioned that the Sonnet was cut (thanks Verizon!) and further SAN maintenance but that was the weekend from hell.
Things to note:
Volume doesn't translate to quality. Between regulatory capture, dismantling Glass-Steagall, and companies "shopping" for the regulatory office least capable of keeping an eye on them, the financial industry has been dangerously unregulated for years. When your entire industry is failing and essentially bringing down the global economy, you are not "tightly regulated".
What's your point? Banking is a complicated industry, made more complicated every year by greedheads who want to separate the people who do actual work from their money as shadily as possible, so it can't come back to bite them. The current financial situation the country finds itself in is nearly directly attributable to a lack of oversight. I'm not in favor of regulation for its own sake, but for goodness' sake, let's not also have lack of regulation for its own sake.
Never underestimate the power of stupid people in large groups.
http://imgflip.com/memes/templates/futurama-fry.jpg
Not sure if modded funny because of the sad lack of technology --
Or there is a hidden joke in there somewhere that I'm missing
None of us know everything. Therefore we're all naïve.
Govt spend is over 40% of GDP and you think that is starving?
The average lifespan of a computer is 3 years.
6 months if you're a tech. 7 years if you're a technophobe.
I am John Hurt.
If the refers to the first 40 days of the Obama administration, isn't the pre-existing equipment and network setup a reflection on the Bush administration's blunt edge grasp of technology?
He was making a joke about how the Marines always have vastly inferior logistics compared to the rest of the US military. They have to make do with the brightly colored rocks.
Given how everything else that Bush left Obama with was a total disaster, no surprise that the White House email was a mess too.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
There is a significant interest in this country in starving government, and then mocking it for under-performing. That's a combination of arguments only an imbecile would make.
You're lucky. In Canada, those guys are the ones *running* the government.
Minister bans/forbids opposition MP (a Nobel prize winner!) from attending a climate conference. (Traditionally not only are they given the credentials to attend, but gov&opp go together as a group)
Shortly after the conference, the same minister mocks the MP for not attending the conference (that he basically banned her from).
"Doing one thing well" sounds great on paper until you realize what that really means in this context...
Either you'll have multiple address books/sets of contacts (you'll need one for each different application, one for email, one for calendaring, one for collaboration,) or you integrate that function between them all. Both are horrendous pains in the ass, unless the servers are written to work together in the first place. ...But if they are, you're suddenly "locked in" to using only products that are all compatible with each other. Of course, if any part of that integration breaks (likely, if the applications involved weren't written to work together in the first place, and you had to "improvise" a solution to make that happen) all the apps lose their contacts function at once. So we're back to multiple sets of contacts, and "one-tiny-failure" breaking all these functions.
And you'd have to have an account on each of these separate servers, permissions on all of them... Somebody would have to know how all five platforms work (instead of having just one Exchange admin,) and have monitoring capabilities configured correctly. That's another thing: Does XYZ Calendar App support your monitoring tool? Your backup platform? If not, better be prepared for some pissed off co-workers when it crashes overnight and you don't know that's happened until you arrive at the office the next day. Or better be prepared to find another job if your backup solution's support for it is limited, and you restore a "backup" that brings back the functionality while losing all the data.
I mean, if you really think that is "simple" then more power to you... I mean, it is your career, after all... But your co-workers and bosses will laugh you out of your job if you seriously propose this "model" which would turn the entire business' concept of productivity on its ear, since they almost certainly currently use Exchange and Outlook for productivity apps, and every single user would have to re-learn everything they currently know to make your "simple" solution work. My guess? You'd be fired before lunch the first day this "solution" went live, as angry people from the parts of your company that generate profit as opposed to just spending money on personal technology wish lists line up outside your boss' office to decry your insane, not-compatible-with-how-we-do-business decisions.
Not to mention the lynching you'd get for breaking everybody's Android/iToys integration to their work productivity apps, since those functions on both platforms rely on ActiveSync, which isn't available anywhere but Microsoft platforms.
Who did what now?
The 3-year lease cycle has to do with being able to deduct the full amount of a leased item over a three year period. If ypu purchase you can only deduct 20% per year. If you run your own business you probably know that.
Wearing pants should always be optional.
I'd lose my job if email was down for a day or two even if the data centre burnt to the ground. It appears that with MS Exchange I could instead blame the software and keep my job.
Still, it's far better than it used to be - you can actually do proper backups of all mailboxes in MS Exchange now without shutting half the thing down! Also virtual machines make it easier to keep the mail running when the thing crashes, which I've heard doesn't happen anywhere near as much anymore.
Yes, 23% was some sort of idiot troll, but the thing is still a shambolic heap that's very name is a warning to swap it for something else. Google gets a pile of business simply because they provide something that MS Exchange should have been over a decade ago - but it's the MTA part that really lets the side down with MS Exchange. It's as if it's abandonware from 1999.
Worst work day maybe, I can see that. But worst day ever? While some folks have sadly experienced their worst day while at work (think of your own examples, I have mine) I suspect the majority of us experienced the worst day ever in our personal lives. Of course, I don't view the WH tech infrastructure the most important thing in life. Which may explain why I don't work directly for the government anymore.
The only reason I draw this distinction is (not because I read TFA), I wonder if most political appointees (and politicians for that matter) increasingly come from well-connected, sheltered and privileged backgrounds where nothing bad ever really happens.
Perspective matters.
An IT department is specificly designed to suck when you put somebody that walked out on graduation day into the position of VP of a Bank and then didn't touch anything related to IT until given control of the White House IT department. Such a thing is not an accident. Incompetance perhaps, but very deliberate incompetance.
The person in question now works for a data recovery company after such an epic failure as the lost emails. The message that sends to me is that is you abolutely want inconvenient documentation to vanish you hire that company.
In my mind 7 years is kind of pushing it. After about 5 years it gets the point where I wouldn't trust the hard drives for anything important just due to old age. Sure, you can replace them but given the time investment it hardly seems worth it. Plus other components like power supplies and fans will start failing and have to be dealt with. Of course, it depends on how you use them too - if each machine just has a standard image on the drive you can easily swap out the entire machine when it fails it's a whole lot easier. Just keep around the ones that have been retired for parts, and you can keep the remaining pool going for a long time.
If you've seen the news, I'm not luckier over here.
Pipe down and let the adults talk.
A lot of agencies, even now STILL use XP. I know of at least one agency that STILL uses Lotus NOTES! 2001 called, they want their OS back.