Slashdot Mirror


White House CIO Describes His 'Worst Day' Ever

dcblogs writes "In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers."

72 of 333 comments (clear)

  1. No surprise by Anonymous Coward · · Score: 3, Insightful

    They inherited a system that "lost" months/years worth of emails during the Bush administration. Of course it all sucked, it was designed to.

    1. Re:No surprise by Vladius · · Score: 2

      Working as intended...

    2. Re:No surprise by TubeSteak · · Score: 5, Insightful

      Of course it all sucked, it was designed to.

      It wasn't originally designed to suck, but when you refuse to spend money on infrastructure improvements,
      you end up spending your time putting out fires instead of making improvements.

      This applies equally to computer hardware/networks as it does to our highway/bridge, electrical, and water infrastructures.
      FFS, there are critical metal pipes in DC's water distribution network that date back 150 years to Lincoln.

      --
      [Fuck Beta]
      o0t!
    3. Re:No surprise by Sir_Sri · · Score: 5, Insightful

      Well and as I have learned the hard way lately, if it's going to cost 500k per year to run IT for a couple of hundred employee outfit when it's government money, someone will complain. When I did private sector stuff the biggest issue was downtime, a million dollars, no problem if that means good uptime. I used to go into insurance companies and banks at 4pm, the regular staff left at 5 -5:30, if it wasn't ready to go the next day by 8 or 9am you were in seriously trouble. In government it's all about how much money they have to explain to some jackass who wants to make political hay out of it.

      The way I count it from http://www.washingtonpost.com/wp-srv/opinions/graphics/2006stafflistsalary.html the white house has about 400 employees. Figure 350k a year in desktop computers alone, for IT staff, another couple of hundred K in 'mobile' and accessory devices, ancillary office equipment you could easily be looking at 1.5 million or for just the non classified IT stuff. That isn't, in the grand scheme of things, a lot of money, but you have to know that whomever isn't in charge is going to want to curtail that spending, because it's 'wasteful'.

      (how you count IT spending can vary wildly. When you're up into that many people you have a lot of dedicated IT staff in various sub groups who may or may not count towards the total and so on). On top of the mess that would be trying to deal with 400 spoiled brats who want everything their way (I'm sorry, executives who want to maximize their productivity), you have to try and plug into everything else in government and have the secured computers/networks as well. That isn't cheap.

    4. Re:No surprise by LordLimecat · · Score: 5, Insightful

      Just a tip, if you ever want people (outside of a small echochamber) to take you seriously, you may want to grow up and stop referring to GW Bush as "Dubyah"-- its about as mature as calling Microsoft M$, or someone you dont like a doo-doo head.

    5. Re:No surprise by Anonymous Coward · · Score: 2, Funny

      You tell him Lord Lamecat.

    6. Re:No surprise by Darinbob · · Score: 2

      It's not that they refused, but it's a typical government problem. You think corporations have budget problems wait until you see the government. Everything is cut to bone. If it still works then there's no reason to upgrade, and if it doesn't work well maybe you can share with someone. Even if it might cost more to maintain, that's a person cost and not a capital expenditure.

    7. Re:No surprise by tburkhol · · Score: 2

      Well and as I have learned the hard way lately, if it's going to cost 500k per year to run IT for a couple of hundred employee outfit when it's government money, someone will complain.

      It's more general than that: When it's government money, no matter how much, someone will complain.

      There is always some better way to spend a government dollar, and if there isn't then government shouldn't have collected it in the first place. Government is the only activity where we all get to sit back and feel/act like the PHB, and most of us will happily cut any function that doesn't directly benefit us, personally. That means government salaries are too high, almost by definition. Meanwhile, no one worries if private industries, like phone or electric companies, offer million-dollar bonuses. You can't buy your city councilman a cup of coffee without someone thinking it's quid-pro-quo, but I can't imagine an industry trade show without at least one open-bar event.

      This is why government sucks. It is run by the cheapest, most arbitrary minds within the general public. People who would be rapidly identified as toxic employees can raise enough media furor to force policy. People who can not grasp the concept that you have to reward talented people to retain them are allowed to dominate budget and salary decisions.

    8. Re:No surprise by tomhath · · Score: 2, Informative

      he country was in the middle of a war and their party groomed no leader to continue it.

      Still the case

      Which side had 6 years of uncontested control?

      Neither

      Which side failed to run a ship that could endure the storm?

      Both

  2. Appropriate by sixtyeight · · Score: 3, Funny

    Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives ...

    Considering the sort of people who are using these machines, it seems almost appropriate somehow.

    Maybe keeping them technologically underpowered is actually a good thing. Those crafty, crafty White House IT gurus.

    --
    The Wolfpack Project: BitCoin + Crowdfunding = Political Accountability
    1. Re:Appropriate by wvmarle · · Score: 2

      Besides, in an environment like the White House, I think it's more than just a good idea to keep a few PC's with working floppy drives at hand. Preferably down to 8" drives. Just in case you need to read some long-archived file, that has never been put on a more modern medium.

  3. OMB IT has their hands tied. by gimmebeer · · Score: 5, Interesting

    The problem is the procurement process. It takes a hell of a long time to get IT resources ordered, and often by the time they are actually put into service half of their warranty life-time has expired. It has nothing to do with a lack of knowledge on the OMB IT front, it's got everything to do with the red tape they have to cut through to make anything happen.

    1. Re:OMB IT has their hands tied. by Billly+Gates · · Score: 2

      Well it sounds like a CIO to manage and streamline the procurement process is what is needed and well overdue. Obama did the right thing as past presidents were old and out to lunch in terms of technology.

      If they need to check for spyware crap and security then set a budget with interns providing the wiping of the hard drives 7 times with an IT department to provide the encryption and come up with procedures to retire and fix PCS and so on.

      I would think a job as important as the executive branch would be important enough. If I were president a 2% downtime would have someone's head on the block let alone a 23% downtime.

      This is coming from someone who is fairly conservative I may add too. This is one cost that is needed

  4. You get what you pay for by forgottenusername · · Score: 2

    Too bad we probably paid billions for such crappy infra.

    1. Re:You get what you pay for by thegarbz · · Score: 4, Insightful

      If a system is end of life and dies can you say it was too bad or crappy without knowing how long it was kept running?

      Everything has a shelf life and must be either upgraded or replaced eventually. Even the Large Hadron Collider is nothing more than a replacement of the Large Electron–Positron Collider before it which reached the end of its useful life. I had a similar discussion with an engineer at the industrial plant I work it. We have a vibration monitoring system which died and needed replacing, and he also called the system "crap." For some perspective the system was obsolete in 1995. The two subsequent models are now also obsolete yet this thing has been humming away just fine for 17 after the vendor stopped supporting it.

      Yet someone called it crap.

  5. Re:Indication of Government Ability? by Billly+Gates · · Score: 2

    The DOD still standardizes on IE 6 from what I am told.

    It is not cutting edge at all.

    Obama's staff at least did a WTF and quickly hired the first CIO to clear the red tape. True he was not a good CIO, but someone was needed. It is unacceptable to have email down PERIOD at such an important job. The president's job is the most important in the world and any loss of email or downtime when WW3 starts or something unrelated is unacceptable. In the private sector downtime is measured by costs with employees salary x time of outage.

      As information is needed rapidly the whiteshouse should have the most advanced technology second only to the DOD and IT needs to be involved.

    Maybe Bush was out to lunch which is not surprising but if I were president I would be flipping the second email would go down for more than a few seconds as I know my staff needs up to the second information to do their jobs.

  6. Re:Not a bad number by Sancho · · Score: 4, Informative

    Zimbra. The enterprise version also has ActiveSync support.

  7. Floppy Drives! by Anonymous Coward · · Score: 2, Funny

    That's madness! Everyone knows that the floppy drive dictates the speed, quality, and age of the computer!

    1. Re:Floppy Drives! by INeededALogin · · Score: 2

      Yeah, it seems to me most Windows computers up until very recently came with floppy drives by default.

      Whats a floppy drive?

    2. Re:Floppy Drives! by enemorales · · Score: 2

      Have you seen the "save" icon in some applications? A floppy drive would let you put some of those (the real thing) to read or write information into them. Amazing, ugh?

    3. Re:Floppy Drives! by kenh · · Score: 4, Interesting

      OK, perspective is called for - Obama took the White House in 2009, up until 2009 HP had floppy drives STANDARD on business desktops - so as Obama took the White House, HP was still shipping floppy drives as STANDARD.

      Yes, sitting in 2012 we can all agree that floppy drives have been obsolete for years, but in 2009 HP was still shipping them as standard.

      The note about Dell Dimensions is nice, but those are "home" computers, not "professional".

      And that 6 year-old software? I can guarantee you it was Office 2003 - sure, as Bush was preparing to leave office his staff certainly could have gone around and upgraded everyone to the latest/greatest version of office (Office 2007), but it is now 2012, and the latest version of Office on PCs is 2010 - does that have 100% market penetration, or are there a few stragglers on 2007 or even 2003?

      Maybe, like most office users at the time, the Bush White House wasn't a big fan of the ribbon interface introduced in Office 2007

      --
      Ken
  8. Re:Indication of Government Ability? by v1 · · Score: 2

    Maybe Bush was out to lunch which is not surprising

    Consider other people his age in that year. 2/3 of them didn't USE email. Many of us had parent that were that age at that time and know what it's like dealing with "old people that refuse to move out of the stone age".

    It would not surprise me in the least to hear that Bush1 (or Bush2 for that matter) never opened email, ever, and got his information on dead tree and in meetings. "Email's down? is that bad?"

    --
    I work for the Department of Redundancy Department.
  9. Re:Not a bad number by houstonbofh · · Score: 2, Informative

    Postfix. Yes, it is ONLY e-mail, but if you need other stuff, you can add it when e-mail is stable. It is also lightweight enough to handle the load on really old hardware.

  10. Re:Floppy... by Fluffeh · · Score: 4, Funny

    I don't understand how adding a peripheral can make the machine "worse"?

    The same way that a Mercedes with an attachment on the front to allow it to be pulled by horses isn't as good as one with a normal bumper bar.

    Car Analogy, Check. Snideness, Check. Condescension, Check. Now time to get that coffee I deserve...

    --
    Moved to http://soylentnews.org/. You are invited to join us too!
  11. As a former employee, I can only confirm by Anonymous Coward · · Score: 2, Funny

    The equipment of some government agencies I have worked for is kind of prehistoric. I you are using Windows 95 as a file server, you are in serious trouble.

  12. Re:Indication of Government Ability? by sco08y · · Score: 4, Funny

    The DOD still standardizes on IE 6 from what I am told.

    The different services have their own IT departments; they even have their own networks, NIPR and SIPR are just two of many. Not sure about the DOD proper, but even the Army is phasing out IE 6 and XP. I'm pretty sure the Air Force and Navy are mostly on Win 7, and the Marines got some new brightly colored rocks with sparkly beads.

  13. what if... by slashmydots · · Score: 2

    Too bad they didn't teach George W Bush about IT stuff. With how often he went to Texas to clear brush, he'd have that exchange DB backed up in 5 places, compressed properly, and on SSD RAIDs :-P

  14. Similar situation... by CAIMLAS · · Score: 5, Insightful

    I recently took over for a staff which had been interned in their positions for the better part of a decade. Out with the old in-house staff, in with the new outsourced IT 'team'.

    I can easily see how this happens, outside procurement and ineptitude problems on the part of the previous WH IT staff. When you've got what amounts to 'institutional knowledge death', with the institution carrying on, you've got to over-staff for some time or things fall apart completely while you play catch up. With a situation where you don't understand it all, are under staffed or under skilled, you're faced with only a couple options when you come in behind the curve, with aging equipment and software: you either start replacing everything you can, as you are able, as quick as you can, or you start suffering outages. It's even worse if things are mismanaged and things are failing all around you.

    As for the claims of the article? Meh. I'm actually not that impressed by his claims to the poitn where I think 'this is bad':

      In 2008, "floppy drives" weren't all that uncommon. I remember servicing Core machines which had floppy drives, still. We're not talking biege boxes with ISA slots here, necessarily - with a 4 year replacement schedule for desktops, floppy drives don't speak of ineptitude.

    The 80-hour-week thing means nothing. It might mean he was understaffed, or that he's a workaholic. To me, it sounds like the meaningless words of a political appointee.

    "Over 82% of the White House technology had reached end of life" means nothing. If they were on a 3-year replacement schedule for desktops and they had 10/100 switching, I can easily see where you'd come to that number.

    He had one "data center", with no redundancy. A bit of a contradiction, yeah? This is made somewhat less impressive by the fact that this administration, in particular, was a bunch of Nancys when they came in with "oh woes, look at this mess", quite obviously overstating things for dramatic media effect.

    "Our email servers went down for 21 hours" isn't a statement of disaster, it's a statement of ineptitude. If they got the mail servers back up, with the data intact, the problem wasn't with the environment but the people involved (or the lack of staffing). His BB starting to have mail incoming suggests a reinstall wasn't required, so safe to say BES was OK, so who knows what the real 'problem' was which caused a day of outage...

    Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%. Realize that a) this is a political appointee talking, b) it's a seemingly non-technical manager (he's up in his datacenter, lookin' for redundancy!), and c) this is the government we're talking about, after all. Anyone who's had any dealings with them on a technical level realizes that 'setbacks' and 'shortcomings' or 'difficult problems' or the like are (probably!) due to ineptitude. Yes, sadly, even amongst the elite (though not necessarily of their own doing - thank you bureaucratic bullshit).

    Granted, this may not have been the case when BO came to the WH and took over. They may have had previous IT staffers who stayed through the transition, but I'm guessing they did not (due to political mistrust issues). It could've been a genuine clusterfuck. Sometimes it's nothing and people cry about the sky falling as they pull down the curtain; sometimes, it really is bad. (If you understand weather patterns, you may recognize a summer storm to not be the disaster that chicken little claims...)

    --
    ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
    1. Re:Similar situation... by TubeSteak · · Score: 4, Informative

      Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%.

      This was the first relevant article kicked up by google:
      http://www.computerworld.com/s/article/9142268/_Lost_Bush_e_mail_settlement_requires_that_White_House_reveal_IT_practices_

      The e-mail problem began in 2002 and 2003 after the White House moved from Lotus Notes to Microsoft Exchange. As it moved to the new platform, the President's IT staff also discontinued use of legacy, circa 1994, electronic management and archiving system, called Automated Records Management Systems (ARMS.) Development began on a new archiving system that ran into its own issues and wasn't implemented.

      Without an automated archiving system, the White House relied on manual processes to archive e-mails, and that's when the problems evidently began. Files were mislabeled and commingled on back-up tapes containing all types of information.

      The public didn't find out about this for years until federal prosecutor Patrick Fitzgerald disclosed it in 2006 while investigating the outing of Valerie Plame.
      The Bush Jr. IT infrastructure was broken from the day they installed it and remained broken for the full 8 years he was in office.

      --
      [Fuck Beta]
      o0t!
  15. Re:Still blaming Bush? by INeededALogin · · Score: 2

    I have to say I got a chuckle when I got to the part about "inheriting" their IT problems. Obama "inherited" all his problems after all!

    On day 1, every problem is indeed inherited. This is a fact. A big difference for me is that Obama is actually fixing issues.... especially in the executive branch

    Bush's Whitehouse.gov
    Obama's Whitehouse.gov

    The Obama version is very nice IMHO.

  16. Re:Floppy... by Stormwatch · · Score: 2

    It does not per se, but it is a sign that it is an outdated machine.

  17. Re:Not a bad number by davester666 · · Score: 5, Funny

    This is completely out of the question. Unless the email server also includes file sharing, calendaring, a contact database, all supporting multiple group and individual access rights, it simply can't be used for email.

    And the product name must include "Windows" or "Live" in the title, preferably both. And if it can be configured to only support Windows machines, we'll pay double.

    --
    Sleep your way to a whiter smile...date a dentist!
  18. Re:Not a bad number by X0563511 · · Score: 5, Funny

    Microsoft: where "five nines" means 9.9999%.

    --
    For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
  19. Re:Not a bad number by LordLimecat · · Score: 5, Insightful

    23% down sounds about average for MSExchange servers.

    Only on slashdot could such ignorance get modded up.

    On a bad bad day as a consultant, I have to fix scenarios with Exchange where everything blew up and theyre down for a single day-- MAYBE 2-- out of several years uptime.

    Thats with the clients who have no full time IT staff whatsoever and a shoestring budget.

    Possibly if you have no idea what youre doing, or dont know anything about exchange, then yea 23% might be an OK guess.

  20. Re:Not a bad number by LordLimecat · · Score: 2, Insightful

    Youre free to pretend we still live in a day where it is unnecessary to do group scheduling through email, but you would be wrong.

  21. Re:Floppy... by peppepz · · Score: 2

    I don't understand either (nor I understand why you were modded "redundant"). Some desktop machines you can buy even now still have floppy drives. It's even more understandable for PA machines to have them, since they may have had special compatibility requirements until a couple years ago. And even if we assume that those PCs were not exactly new, since they're not supposed to run Crysis, as a taxpayer I'm happier if my administrators can take the most out of not-so-old machines instead of watching them spend tax money on the latest hip hardware. Windows 7 runs successfully on Pentium IV hardware. Linux does even better.

  22. Re:Not a bad number by jrumney · · Score: 4, Informative

    Other solutions allow you to use "groupware" functionality with IMAP (less so with POP, as that pulls all your mail in locally). Only Microsoft intentionally cripples their IMAP implementation so that you cannot see important fields (like Date/Time of meeting) when you try to browse the vcal messages that are stored in the calendar folder over IMAP.

  23. Re:Floppy... by jrumney · · Score: 3, Informative

    Not necessarily. Their purchasing contract may have adopted to the fact that Pentium II processors are no longer available, and now specifies a "required peripheral list", leaving the processor and RAM spec open to whatever is current at time of purchase. So the machines they are buying might be modern fast machines, but somewhere in the contract it says they need to have floppy drives, so they do (most motherboards are still coming with floppy controllers on them for some reason, so nothing has forced them to reevaluate whether they need one, and the supplier is probably happily collecting a premium for supplying them, so doesn't want to rock the boat).

  24. Re:Not a bad number by Errtu76 · · Score: 2

    If it's only send and retrieve mail, I'd suggest postfix & dovecot. If you want the 'fancy' stuff, give Zarafa a try. It's Dutch, so it must be good (vim, python, etc) :P and under the hood it still uses postfix.

  25. Re:Love it by ryanov · · Score: 2

    My union covers our IT personnel (I am chair of the union's IT committee).

  26. Re:Indication of Government Ability? by ryanov · · Score: 5, Insightful

    Oh SHUT up.

    The reason government can't get anything done, generally, is there's always some jackass out there questioning whether a thing is needed because it happens not to be exactly what they want, or why workers cost anything at all since their life is in the shitter so why should a government employee make money either?

    There is a significant interest in this country in starving government, and then mocking it for under-performing. That's a combination of arguments only an imbecile would make.

  27. White House IT: thumbs down by mveloso · · Score: 3, Insightful

    From what I've read, there's only one firm that does White House transitions. I think it's Bechtel, but it's been so long that I've read anything about transitions that I have around a 15% confidence level in that piece of data.

    Google "white house transition" and you'll see that it's a total mess. If you want to read about it, there's info here:

    http://whitehousetransitionproject.org/

    From what little I've read, you basically get a mostly empty building (the White House). It's up to the team to build/rebuild the infrastructure...but as any operations person knows, IT infrastructure is usually way behind everything else. The general executive branch IT has been a low priority for decades. What's more important, email or setting up the phone so the president can call someone (or someone can call the president)?

    At that point, the team is probably so far behind that they're screwed continuously for the one or two terms.

    Are the guys running the systems any good? I'd ask you: how many of you could pass a background and attitude check? You think the process etc at your workplace is bad, imagine how bad it is in the Executive branch.

    That said, it might be fun...but it's probably a nightmare. "I can't print out this $15 billion dollar appropriation because the f*cking printer doesn't work!" "People in PA are starving because the email server ate all of our emails!"

    Every minute is a crisis, with everyone breathing down your neck 24/7. Does that sound like something you'd want to do for 24/7/365/4 years?

  28. The current White House uses Drupal/OpenAtrium by SpzToid · · Score: 4, Informative

    http://www.whitehouse.gov/blog/2011/02/11/

    http://developmentseed.org/blog/2011/feb/14/white-house-using-open-atrium/

    www.openatrium.com

    For civilians trying to avoid another HBGary-type SQL-injection cascading breach, building a PHP website using the Drupal framework means benefiting from the eyeballs that watch sites like whitehouse.gov, and others. These same eyeballs, and many others contribute their security patches back to drupal.org. Although I imagine their OpenAtrium groupware is behind a firewall.

    OpenAtrium is 100% free open-source server software, that reaches out really well to tables and other mobile devices too.

    --
    You can't be ahead of the curve, if you're stuck in a loop.
  29. Re:Floppy... by Retron · · Score: 5, Informative

    most motherboards are still coming with floppy controllers on them for some reason,

    If only... None of the HP machines we've bought at work in the past couple of years have had them and we buy both the slimline desktop variety and mini-tower PCs. The few Dells I've seen likewise don't have any floppy ports on the motherboard.

    As for build-your-own PCs, or ones from companies that assemble generic parts into PCs, very few come with floppy ports on the motherboard. Indeed, the only non-industrial Intel motherboards I know of that have a floppy port are the ASRock Extreme boards - and that's powered by a SuperIO chip on the motherboard, as chipset support for floppies was dropped by Intel years ago.

    Note: the reason I mention all this is because I'm looking at getting a Z77 motherboard in the next few months with a floppy connector, so that I can hook up a 5.25" floppy drive I've acquired (purely for the heck of it, before anyone asks - I've a big box of old disks from the early 90s that I wouldn't mind rummaging through, the PC I used for those having been chucked out years back). ASRock are pretty much the only option nowadays and I have no doubts that when Haswell comes out next year the old 37-pin floppy connector will be well and truly extinct.

  30. Re:Floppy... by bedouin · · Score: 2

    Could be wrong here, but I'm thinking even if you get the drive hooked up you're going to have issues finding a modern BIOS that can support 5.25" disks.

  31. Re:Floppy... by jrumney · · Score: 2

    Indeed, the only non-industrial Intel motherboards I know of that have a floppy port are the ASRock Extreme boards - and that's powered by a SuperIO chip on the motherboard, as chipset support for floppies was dropped by Intel years ago.

    Maybe my experience is skewed by the fact that the last two PCs I've built have been AMD based. I stand corrected, I just assumed it was a standard part of every IDE controller, and though my current motherboard came with half a dozen SATA ports, I was surprised to see it still has the dual IDE port and floppy controller of old, and the BIOS even still lists Floppy as the first boot device by default.

  32. Re:Floppy... by TapeCutter · · Score: 5, Insightful

    It's a strong possibility. I've seen similar bizzare things based on what's written in a contract and in private industry not just goverment. My first encouter with contract crazyness was at a large telco in the mid 90's where I had I authorised putting 250MB disks into ~100 laptops with dead drives. However this upset a PHB somewhere in the money spending chain of command because the original maintenance contract stated 120MB disks (which were by this time out of production and as rare as hen's teeth). I tried explaining the supply problem and that the contrator was actually giving us twice the storage for no extra cost. In the end it was simpler to explain the situation to the contrator and (sheepishly) ask them to refomat the 250MB disks down to 120MB than it was to continue butting heads with a dick-swinging autocrat from the finance dept.

    Office politics is really no different to real politics, the vast number of people who work for large organisations be they private, public or charitable are for the most part reletively efficient at whatever it is they do, but one or two clowns in the wrong position can turn the whole thing into a circus. In an evolutionary sense large organisations exist because they can do what no man can do alone. However our tribal instincts are still evolving such that we can live with and within groups of more than ~150 that are required to produce what a single mind can imagine, large groups (civilization,cities) simply did not exist until we invented agriculture and yet our current civilization(s) cannot function without them.

    For example the multi-national I work for has about 175K people, a death in that "tribe" would happen quite frequently (say one a week), but it's only the handful of people I personally work with that I care (or even know) about. I think the fact that telecommunications have gone from simple morse code to their current star trek capabilities is part of that evolution, we are tool-makers, it's in our nature to invent tools to overcome the problems caused by inventing tools. So in a way that will probably upset bioligists it can be said that our tools and our instincts are co-evolving to accomplish greater feats, but our tools are evolving at a geometrical rate whereas our base instincts evolve at a glacial rate. So I'm betting our tools will evolve to the point where the size of the organisation is (almost) irrelevant to the effectiveness of its internal organisation long before our pumy minds can name, let alone care about, 175k individuals.

    --
    And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
  33. Re:Floppy... by c0mpliant · · Score: 2, Funny

    AMD based PC built in the last two years? Why?

    --
    There is no -1 disagree
  34. Re:I would deploy a Domino cluster by lucm · · Score: 3, Insightful

    I would deploy IBM Domino like in the days of Clinton, which Bush switched to Microsoft Exchange. Reliability went downhill with that decision.
    Domino runs cheap and fast and reliable. And has always active clustering so you don't have to deal with downtime. IBM simply has a much longer track record of delivering reliable computing than Microsoft.

    People are not replacing Domino with Exchange because it is more reliable. They do it because *everybody* hates Lotus Notes.

    This being said, any user that has a complaint against Lotus Notes should be required to work with Groupwise for a week.

    --
    lucm, indeed.
  35. Re:I would deploy a Domino cluster by maXXwell · · Score: 2

    I would deploy IBM Domino

    A company I worked at a few years ago used Domino. I thought it was a great proof-of-concept for some future groupware product, but not ready for real-world use. It was broken in so many ways! I saved my list of Domino issues, which I've included below. This is for Domino version 7, so some of these issues may be fixed in subsequent versions. But to be this broken as recently as five years ago (and after 16 years of development, too!) is unforgivable.

    So, check out my list of issues, and decide whether this is a product you would want to deploy in your organization!

    Domino Issues:

    - Slow.

    - Spell checker with mailer is lame. Better to have MS Word-style
        spell checker.

    - When using View -> Find in view, defaults in such a way that
        deletes all entries when the user thinks they are deleting a single
        entry. Virtually impossible to undo.

    - When using View -> Find in view, can't delete individual e-mails.
        (see previous). Messages that are de-selected disappear from the view.

    - Really crappy mailbox search algorithm

    - Very weak mailbox filtering capability (compared with procmail)

    - Hard to gauge where to wrap lines when using so-called
        `Internet-Style' messages. No automatic line wrap.

    - When replying using `Internet-Style History', quotes sender
        in message envelope rather than sender in `From:' field.

    - "Show source" on e-mail message does not show message envelope.

    - Won't display HTML content of messages . . . good that it doesn't
        happen by default, but wish it were an option.

    - View -> Show -> Source doesn't work for messages with no text in
        the message body, so no way to view headers of empty spam messages.

    - Message size bears little relationship to actual content.

    - Very slow over low-bandwidth connection. Much more overhead than
        IMAP.

    - No multiple levels of undo -- can only undo last change

    - When using find, it checkmarks all found messages. Then if you
        highlight one and attempt to delete it, it delete all checked messages
        *without prompting*. And no option to undo!

    - When clicking on links in e-mail messages, unclear whether browser
        has been launched. Mouse cursor doesn't change, as it does with
        most other mail clients, unless you move it outside of the Notes window.

    - Can't sort by date/title/etc in View -> Search this View in Tech Docs

    - No Day of Week in Message. Month is numeric only.

    - Mail Search is fucked. Try:
        "Author contains Sender/Organization AND outgoing".

    - When opening mail attachments, no option to select which application
        to use.

    - When opening mail attachments, cannot open an attachment with an
        unknown extension.

    - Crashes when reporting certain messages to Symantec

    - Cannot set different chimes for incoming mail. E.g. mail going to
        group folder due to mail rule makes same chime as mail going into
        mail inbox.

    - Can't cut-and-paste into mail rules.

    - No log to see when messages are deleted by mail rules.

    - Can't respond to a message in a "meeting accepted" / "meeting
        declined" without cut-and-paste to a new memo.

    - Copying a memo from a folder to a nested folder with the same name
        causes a duplicate of the memo to appear in the original folder.
        E.g. copy something from "Sent" to "Folders->Temp->Sent".

  36. Re:Not a bad number by Enter+the+Shoggoth · · Score: 4, Funny

    Microsoft: where "five nines" means 9.9999%.

    Nah, I think it was more like this:

    Gates to Balmer: our Enterprise products need to have 5x9 uptime.
    Balmer: ok Boss.

    Balmer to VP of Engineering: Bill wants all our products to only work between 5pm and 9am. ...

    --
    Andy Warhol got it right / Everybody gets the limelight
    Andy Warhol got it wrong / Fifteen minutes is too long.
  37. Re:Not a bad number by Mabhatter · · Score: 5, Insightful

    You miss the fact that he INHERITED that system. That's how politics works. The lame duck guy in charge of the White House (which would be the former President DIRECTLY) let the thing rot...

    Part can be attributed to the old staff being done with the position and the new guy will just buy new stuff anyway... Almost fair?

    Part of all the downtime was a FEATURE that the pervious administration used to their full advantage... They were über controll freaks... Controlling information of their own "trusted" employees was part of D.C.'s daily routine. The rot was deliberate to stop communications from being added to the archives.

  38. Re:Not a bad number by marcello_dl · · Score: 2

    I agree, I think that nothing prevented the former administration from getting decent hardware. If they lived with a broken system they wanted it. The new guy should have taken it all, put in a room sealed it and never thought about it again, IMHO, for the same reason.

    --
    ---- MISSING MISCELLANEOUS DATA SEGMENT --- [sigdash] trolololol
  39. Re:Not a bad number by Karl+Cocknozzle · · Score: 5, Insightful

    Group scheduling and email are different applications. Combining them in one backend is shortsighted.

    True, but Exchange isn't an "email" application--it is a group productivity application that includes email, group calendaring and scheduling, tasks, and collaboration.

    I understand there are MS haters who will bash Exchange relentlessly, with any label on it, but let's try to be even a tiny bit accurate. Exchange isn't an "email" service and hasn't been exclusively that for nearly 15 years: Time to come up with some new criticisms, the old ones don't apply.

    --
    Who did what now?
  40. Re:Not a bad number by Immerial · · Score: 2

    23% down sounds about average for MSExchange servers.

    Only on slashdot could such ignorance get modded up.

    And only on slashdot could someone WOOOOSH so bad... it was modded up FUNNY.

  41. Re:Not a bad number by nahdude812 · · Score: 3, Interesting

    Gates: We need five nines of uptime
    Ballmer: Engineering, we need 9 + 9 + 9 + 9 + 9 uptime.
    Engineering Manager: Guys, our uptime goal is 45%
    Engineering: We already deliver about 72%.
    Engineering Manager: Steve, we actually have 9 + 9 + 9 + 9 + 9 + 9 + 9 + 9 uptime!
    Ballmer: Bill, we're so stable we have 8 nines of uptime! Let's see our competitors beat that!
    Gates: Great Steve, let's add some more bloat and see if we can bring that number down some so we leave ourselves with room for improvement.

  42. Re:Indication of Government Ability? by Enry · · Score: 2

    Turn off the projector, son.

    Nobody of any consequence in the Democratic Party has said that Capitalism has failed in general. Parts of the organizations that make up the model have certainly failed, but capitalism as a concept and economic model works fine provided everyone plays fairly.

  43. Re:Not a bad number by Wild_dog! · · Score: 2

    Workers are workers.... government or private.
    Can't get more human than human and we are all the same animal.

  44. Here's the real story first hand by Anonymous Coward · · Score: 5, Interesting

    This article is partially correct but leaves out the actual technical issues involved.

    Someone *from* that Datacenter here at that time. Here's what really happened.

    The old administration did not care about the existing IT infrastructure because they were on their way out. They wanted no changes made- just that things be left up. Yes the email system was old and past EOL, but the outages were really the perfect storm of everything that could hit the fan actually hitting the fan at the same time.

    The facility was doing work on the power system- the UPS to be specific. Somewhere along the line they messed up, and cut the power. *All* of the power. Datacenter goes dark. They brought the power back up, but then tripped it again before bringing it up for good. This detail is what caused the weekend of hell.

    The SAN that the clustered email servers (yes, clustered, they *were* redundant) had the stores on was an EMC Symmetrix. It has a built-in battery backup system so that if the SAN looses power it has enough stored to flush the cache to disk. The power going off started this process. The power going back on triggered the response to stop flushing the cache and start checking and rebuilding. Then the power went off again. This is the part where the specific details get hazy but in effect the SAN did not like this. I don't believe it had enough power to totally flush the cache and/or it did not have the logic built in to handle an outage while in recovery mode. The result was a downed SAN that *would not come back up*. Now all of the data was down and nothing could be done but wait for the vendor to show up and try to fix it.

    At the same time we were dealing with *every* server being off and having to come back up. There were hundreds. Luckily most did. Some did not. Some were important, such as in the case of *both* the servers in a clustered system that would not boot- which just so happened to be the system that some of the say "more important" VIPs were on. These were old systems running Exchange 2000 on Windows 2000. Long past due, but kept up by the staff since the EOP would not approve a new email infrastructure.

    Eventually the systems would be restored and everything would be back on-line. In the meantime though Brook thought it would be a good idea to spend untold amounts of money to bring in MS Engineers to look things. They cost a lot of money and made a bunch of reports but they didn't fix a damn thing. The staff that was already there found the issues with the servers and fixed them.

    There were later headaches, such as when mentioned that the Sonnet was cut (thanks Verizon!) and further SAN maintenance but that was the weekend from hell.

    Things to note:

    • There has been 24x7 NOC (Network Operations Center) for the EOP data center long before the current administration.
    • There was a DR (disaster recovery) data center. It wasn't *great* but it was there. Due to the SAN outage and estimated time to fail over it was determined by those in charge that the best call was to repair instead of failing over.
    • The "some previous experience" listed for Brook was *all* of his previous experience.
    • The GOALIE position is a joke. They took perfectly good technical government people off of doing technical work and put them in a useless role basically overseeing time sheets. Unfortunately things were now slower because changes had to go through the GOALIE's who generally didn't have quite the overall end-to-end system expertise to make final decisions.
    • Brook really wanted to push a "mobile desktop initiative" that was a joke. He wanted the remote experience to be "just like working in the office" with requirements of the laptops being encrypted. Let's disregard that this can never happen if not simply for bandwidth constraints. But still, they tried. Vista SP1 would have been perfect for this (because of bitlocker), or hell even waiting until Windows 7. But no, Brook just said "no" to Vista because it was Vista and forced the engineers to
  45. Re:Not a bad number by jimicus · · Score: 5, Insightful

    More to the point, businesses haven't wanted an email service for nearly 15 years.

    They want the group productivity application. But they don't call it that because the most visible part - the part they really see - is the email.

  46. Re:Love it by BVis · · Score: 3, Informative

    Please stop perpetuating the lie that union workers are "unfireable". Unions do not protect workers from being fired for gross incompetence, theft, sabotage, and so forth. What they DO give you is the right to 'progressive discipline', where you can't be fired for wearing the wrong color shirt or being two minutes late for your shift, without a hearing with a union representative advocating for your interests.

    Until American workers enjoy some of the protections of their European counterparts (even if limited to being required to provide a REASON for a termination - employers in 'at will' states can fire you and say to your face "we're not going to tell you why"), then unions will be necessary in this country.

    --
    Never underestimate the power of stupid people in large groups.
  47. Re:Indication of Government Ability? by MimeticLie · · Score: 2

    Volume doesn't translate to quality. Between regulatory capture, dismantling Glass-Steagall, and companies "shopping" for the regulatory office least capable of keeping an eye on them, the financial industry has been dangerously unregulated for years. When your entire industry is failing and essentially bringing down the global economy, you are not "tightly regulated".

  48. Re:Not a bad number by Hatta · · Score: 2

    The "no true email client" fallacy I see. Whatever it is that Exchange does, I don't want it.

    --
    Give me Classic Slashdot or give me death!
  49. Re:Indication of Government Ability? by BVis · · Score: 2

    What's your point? Banking is a complicated industry, made more complicated every year by greedheads who want to separate the people who do actual work from their money as shadily as possible, so it can't come back to bite them. The current financial situation the country finds itself in is nearly directly attributable to a lack of oversight. I'm not in favor of regulation for its own sake, but for goodness' sake, let's not also have lack of regulation for its own sake.

    --
    Never underestimate the power of stupid people in large groups.
  50. Re:Not a bad number by Pumpkin+Tuna · · Score: 2, Informative

    You miss the fact that he INHERITED that system. That's how politics works.

    Oh, you mean like when Dubbya showed up, and all the W keys were missing from the keyboards...
    http://www.people.com/people/article/0,,619013,00.html

    Except that the GAO investigated and decided that there wasn't enough evidence to prove any of the "missing W' allegations. I did like the "office of strategery" sign though.

  51. Re:Not a bad number by Karl+Cocknozzle · · Score: 2

    Exchange isn't an "email" service and hasn't been exclusively that for nearly 15 years

    I think that's what most people are complaining about, why have one server doing multiple things? KISS. It should be that those other parts are broken off onto their own servers so that downtime is restricted to each individual part.

    First of all, you've proposed using five servers to provide the same services as one server in an effort to "Keep It Simple ..." Do you recognize how ridiculously not-simple such a plan is? You've gone from one server to 4-5 (depending on what features you're using.) How is that "simple?" ...and I have to train my users to utilize a different everything (calendar, email, tasks, contact management, and collaboration) too?

    Where do I sign up!

    Additionally, Exchange Server now comes with redundancy and fail-over capabilities which means that unplanned downtime from "failure" is now down to the efficacy of the individual engineer, and his employer's willingness to spend money to reduce downtime. How much downtime is "acceptable" is directly proportional to how much value your organization derives from a particular service. How much you actually have is directly proportional to how much you spend to prevent it (up to a certain point.)

    --
    Who did what now?
  52. Re:Not a bad number by SCHecklerX · · Score: 3, Insightful

    in my experience, Zimbra is a bloated pig with its share of availability problems as well. I hate the whole "let's take a bunch of open source pieces but just throw them together as an inflexible blob of crap in /opt" approach. The installer leaves a lot to be desired as well, with key components around setting proper permissions resulting in an install that will never work until you manually fix it.

  53. Re:Not a bad number by Myopic · · Score: 4, Funny

    I would like to say that I am absolutely shocked that George W. Bush didn't have a team of IT professionals able to expertly administer the White House technology infrastructure. Given his record of surrounding himself with the best of the best, it's almost impossible to believe.

  54. Re:Not a bad number by nine-times · · Score: 4, Informative

    Yes, unfortunately, Exchange works very well for many businesses, and it's providing something that they need. You may ask, why do you want to combine scheduling and email into the same application and back-end?

    The answer is obvious to people who've used Exchange. You send meeting invites through email. When you're sending the invite through email, your email application can also tell you whether the intended recipients are already busy. You can also schedule resources (e.g. a conference room or projector) and view availability while creating your email invitation.

    And why include contacts? Well that, I'd hope, is obvious. All email applications keep a contact list anyway, since they need to store email addresses. If you want to create a contact database that includes email addresses, you may as well include that in the email application.

    Tasks? Well, for many of us, our task list comes straight out of email. I get an email, and I need to create tasks for what I'm going to do in response. Plus, "tasks" and "calendars" are logically linked together as tools for effective task-management.

  55. Re:Not a bad number by obsess5 · · Score: 2

    Never spent hours on the phone with your health insurance company trying to get something straightened out?

  56. Re:Love it by ryanov · · Score: 2

    Thank you. BTW, that's not a slur. Why is it a badge of honor to some people to be as spineless as possible?