Slashdot Mirror


US Government Begins Largest IT Consolidation in History

miller60 writes "Saying 1,100 data centers is too many, the federal government has begun what looms as the largest IT consolidation in history. Federal CIO Vivek Kundra has directed federal agencies to inventory their assets by April 30 and prepare a plan to reduce the number of servers and data centers, with a focus on slashing energy costs (full memo). Kundra says some applications may be shifted to cloud computing platforms customized for government use."

31 of 283 comments (clear)

  1. ... if you can spell "Cloud Computing" by Zarf · · Score: 4, Insightful

    I predict a rash of job openings that you can get hired for provided you can spell "Cloud Computing"

    --
    [signature]
    1. Re:... if you can spell "Cloud Computing" by Anonymous Coward · · Score: 4, Funny

      Clud Crumpooting

    2. Re:... if you can spell "Cloud Computing" by dkleinsc · · Score: 3, Funny

      It depends entirely on the political clout of your congressional representation.

      --
      I am officially gone from /. Long live http://www.soylentnews.com/
    3. Re:... if you can spell "Cloud Computing" by cayenne8 · · Score: 3, Funny
      "Why? It doesn't seem to have been a job qualification to be able to spell or pronounce "nookuler"

      Nor to know how to pronounce corpsman (hint:not corpse-man)

      --
      Light travels faster than sound. This is why some people appear bright until you hear them speak.........
    4. Re:... if you can spell "Cloud Computing" by billcopc · · Score: 3, Insightful

      Pride in your origins ? You were born. Whoop-tee-doo! One person's birth doesn't make an arbitrary geopolitical territory automagically awesomer than the one next to it.

      If we're going to take the gloves off, I'll posit that the entirety of "American English" is ignorant, as it is an inferior, jocular, slurred dialect that only loosely resembles English syntax. It is to English what Afrikaans is to Dutch.

      --
      -Billco, Fnarg.com
  2. Dear Contractors... by fuzzyfuzzyfungus · · Score: 4, Funny

    Dinner is served! Please approach the money trough in an orderly line...

    1. Re:Dear Contractors... by hedwards · · Score: 3, Informative

      I know you're joking, but it could go either way. Trying to manage, secure, track and backup the huge number of servers that the various agencies and departments use costs a pretty considerable amount of money to do right. Of course they haven't been doing it right up until now. Consolidating into a smaller number of server farms that are somewhat spread through the US has definite potential in terms of dealing with those factors more efficiently. That being said, we won't know until it happens, there's still plenty of ways for pork and waste to creep into the equation.

    2. Re:Dear Contractors... by wealthychef · · Score: 5, Funny

      I for one have full confidence in the government that after reorganizing their data centers, they will have a lean, optimized, efficient operation. Who's with me?

      --
      Currently hooked on AMP
    3. Re:Dear Contractors... by BitHive · · Score: 4, Funny

      Sure, as long as they bring in the free market to do it. Nothing gets a job done on time and under cost like unfettered free enterprise and rugged individualism.

    4. Re:Dear Contractors... by ArcadeX · · Score: 4, Informative

      I work on the DoT network, and this thought scares me. Please remember the lowest bidder gets the job in most cases, we recently started putting VM servers in, and these guys can't even reboota a virtual server without screwing it up. As a regional subcontractor, I'm completely locked out, to the point that I had to spend 10 minutes on the phone with our official helpdesk explaining the runas command in windows to the guy on the other end so he could run a command I don't have access to...

      --
      An I.T. motto in the hands of an idiot is a dangerous thing...
    5. Re:Dear Contractors... by fuzzyfuzzyfungus · · Score: 3, Informative

      There are, absolutely undeniably, substantial economies of scale in IT at virtually every step of the game. Hardware gets cheaper as it is produced in greater volume, software costs serious cash to write but nothing to copy, small shops might have a 1/25 admin/server ratio because the minimum number of admins is 1(.5 admins just lie on the floor oozing organ goo) while large ones might have a 1/10,000 ratio plus a few screwdriver monkeys.

      And, honestly, I'd be delighted if the feds can realize some of those economies. I'm sure that there are plenty of grossly inefficient little fiefdoms out there, just waiting to be consolidated.

      My concern is twofold: one is that there are non-obvious potential diseconomies of scale(and not just because this is the evil gummin't with its waste and corruption, a lot of the good stories are private sector). Centralization and standardization are all well and good until you end up waiting three weeks and submitting petitions in triplicate just to get some software installed or setting changed, and don't even think about setting up a little wiki or git repo or something for your team with approval from a half dozen departments.

      The second issue is, of course, concern over the government contracting process, regulatory capture, revolving door incentives, outright corruption, and whatnot. The magical efficiency of the private sector isn't going to do us a whole lot of good if the project ends up as a cost-plus job for SAIC or one of the other byzantine contracting behemoths that specialize in landing(and on occasion even fulfilling) contracts.

    6. Re:Dear Contractors... by Anonymous Coward · · Score: 4, Insightful

      they will have a lean, optimized, efficient operation.

      I'm sure that it'll be as lean and efficient as most of the companies where I've worked. While those 'darn government bureaucrats' take a lot of heat for like every one who runs for office, incumbent or not. However, personally, I've seen plenty of waste, fraud and abuse in the private sector to know that those issues are just examples of human nature run amok in large organizations.

      Of course with government, special problems exist, in particular voters and politicians who instead of trying to improve government seem more willing to destroy it and 'start fresh'. Of course that that does is empower the status quo. Practical people who talk of incremental change and steady leadership are downed out by radicals who demand 'nothing', or 'everything'.

    7. Re:Dear Contractors... by wintercolby · · Score: 3, Interesting

      There in lies the problem. The fed will likely outsource to Raytheon or Northrup Grumman, who will again contract out to recruiting firms. Before I know it, I'll be getting calls and emails from people who clearly aren't even state-side asking if I'm willing to relocate for a 6 month contract. Of course it would be on my dime to move. That's why they can't get qualified people. Qualified people won't (don't) sub-contract for firms based in India to get paid the lowest going wage and relocate at the same time.

      I know this because I get 5 to 10 emails/week asking me to relocate to the DC area for jobs that I've already seen posted on Ratheon or NG's websites. I work with plenty of multinationals, and can easily tell English Grammar from an Indian perspective. Not all Indians speak or write poor English, just the recruiters that have contacted me regarding subcontracting for the Fed through NG and Raytheon.

      --
      Most ignorance is vincible ignorance. We don't know because we don't want to know. --Aldous Huxley
  3. Bout time by Monkeedude1212 · · Score: 3, Funny

    See, the last time we upgraded we put everything on eleven hundred windows 95 machines with 1 gig hard drives. That did pretty good for a spell, all things considered. Now we're thinking about one of them pointy computers... whaddya call em? Blade servers? Yeah, we hear good things about those.

  4. Prediction by maugle · · Score: 4, Insightful

    This being a government IT project, I predict it will take 5 years longer than planned, cost 10x the initial budget, and still never really work quite right.

    1. Re:Prediction by Blade · · Score: 5, Insightful

      This being an IT project, I predict it will take 5 years longer than planned, cost 10x the initial budget, and still never really work quite right.

      Fixed that for you.

    2. Re:Prediction by timeOday · · Score: 3, Interesting
      The high failure rate for large software projects is well known: "If Las Vegas sounds too tame for you, software might just be the right gamble. Software projects include a glut of risks that would give Vegas oddsmakers nightmares. The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991)."

      Here is another fun page: "Most IT experts agree that such failures occur far more often than they should. What's more, the failures are universally unprejudiced: they happen in every country; to large companies and small; in commercial, nonprofit, and governmental organizations; and without regard to status or reputation."

      I only question why, when large projects are almost universally over-budget or fail altogether, we persist in being surprised and outraged every time? The simple fact is, we don't know how to do it, any more than we know how to land on mars; that is, we can do it, sometimes, but you better know going in it is likely to end in tears.

      (In general, it seems to me that most of the problems in government have direct parallels in private industry because they flow from the same underlying cause; the unaffordability of medicare/medicaid corresponds to skyrocketing premiums in the private market; social security corresponds to slashing pensions and now even 401k matches in private industry. But private industry does hold a trump card - they can always cut their losses by tossing people aside and moving on, whereas government is the safety net.)

    3. Re:Prediction by Sandbags · · Score: 3, Informative

      I don't know, we started with a network of well over 3,000 servers, and in less than 2 years we've moved almost entirely off Win 2K, virtualized over 1,000 servers, moved to AIX 6 and VIO, delpoyed a VMWare infrastructure, deployed an Exchange architecture in place of a legacy e-mail system, and converted more than half our web apps into SOA and put it on IFLs in a mainframe. We cut from over 3,000 systems, nearly all physical, to under 2500 with near half virtual, and saved significant money in the process vs budgeted outlays. We've reduced our data center footprint by 60-70%, and our power draw is down dramatically.

      The forward looking TCO now that the bulk of the migrations are done is impressively smaller.

      We did all this while holding to DOD network standards.

      The problem the government will have, which we fought with a bit but was no where near as big of a deal, is getting hundreds of small business units to agree to consolidate to central systems, and to convert from "these servers are mine, see, here's where I paid for them" to a metered utilization budget system where hardly any smaller government agency owns it's own infrastructure.

      --
      There is no contest in life for which the unprepared have the advantage.
  5. Better not use Northrop Grumman by r_jensen11 · · Score: 3, Informative

    If Virginia's IT overhaul is any indication, this is going to be a slow-motion cluster of a mess for the next 10-20 years

    1. Re:Better not use Northrop Grumman by Anonymous Coward · · Score: 5, Informative

      If Virginia's IT overhaul is any indication, this is going to be a slow-motion cluster of a mess for the next 10-20 years

      Let's not forget that Vivek Kundra was Virginia's CIO when that fiasco took place. I predict that this will be at least as bad as the Virginia situation.

    2. Re:Better not use Northrop Grumman by mikefocke · · Score: 3, Interesting

      Of course it is

      consolidations are always a mess and ones full of job implications mean political interference (I want em in my district).

      But you have to do something as the growth of government IT gets out of hand and we can only afford so much.

      IIRC, the government consolidated all the payroll systems it had into about 4 pay centers back about 10 years ago. Went from maintaining hundreds to one s/w run 4 places for redundancy. Everybody screamed they needed theirs because it had unique features, they learned to do without or incorporated the features into the new s/w. Wasn't that fairly successful?

      While all govt computing is a bit more complex now than a single application was then, still if we are to afford the things we really need, consolidation and standardization makes sense.

      Now the contracting and execution...that will be a challenge. And so what if it takes 5 years, if we are going in the right direction and saving money in the long run. Because we can't sustain even the current government spending on what we are willing to vote as taxes.

  6. Consolidate by halcyon1234 · · Score: 4, Funny

    Every business I've ever worked for has had that one dusty 8086 off in a corner. It would run a single batch file every few hours. No one would touch it, because no one knew what it did-- just that whatever it did do was mission critical.

    Thus, the US government should just consolidate everything down to a single batch program run by a 8086. I'm sure there's a spare closet in the White House or something they can use as a server room.

  7. Re:IT as a commodity by clang_jangle · · Score: 4, Insightful

    And if you like the way your bank is not liable for identity theft, you'll just love the upcoming government data-filled Cloud!

    --
    Caveat Utilitor
  8. Vivek by bigmattana · · Score: 5, Interesting

    Oh, Vivek, what brilliant thing will you think of next? How much energy will it take to replace all of these server farms? How much energy will be required for the taxpayers to earn the money necessary to pay for it? What about security concerns of consolidating all of this data?

    I think Vivek wants to make himself look useful after being exposed as a fraud by John C. Dvorak. http://www.dvorak.org/blog/2009/08/12/special-report-is-us-chief-information-officer-cio-vivek-kundra-a-phony/

  9. Re:IT as a commodity by FuckingNickName · · Score: 5, Interesting

    I am not quite sure what you are talking about. Because everyone has access to yesteryear's supercomputer on their desktop, there is no reason whatever to go back to a 1960s outsourcing model. If you want to distribute load over your machines, go ahead! But why do it over someone else's?

    If you think this is going to reduce IT expenditure requirements, you have barely worked a minute in IT. When you outsource, you are simply paying someone else to do your job, plus profit, plus a gaggle of negotiators in middle management collecting their kickbacks, plus downtime costs because your business is less important to them than your business is to you (if you have enterprise e-mail and it has been down more than, say, GMail, you have done something very wrong)...

  10. What about "use it or lose it"? by joocemann · · Score: 5, Insightful

    Didn't read the article, but my experience with government entities is that they receive a specific value of funding each year to spend on gear, training, energy costs, etc.

    The nature of the funding goes that if you don't use all of it this year, you get a reduced amount next year. Now this may seem logical -- it may seem like a policy that governs spending. Instead what it is is a policy that drives UNNECESSARY SPENDING.

    The places I have been were frugal but appropriate in their spending throughout the year. As the funding for the year would approach a close (in October), all-of-a-sudden the leadership would start spending money like crazy because they had a large surplus. Money would be spent on things that were not actually necessary; if they were necessary, why not get them at any other time during the year?

    In several cases, seeing this strange frenzy of spending I would ask the leadership what was going on. They explained the 'use it or lose it' policy and that in order to maintain the funding they got this year, for next year, they *must* spend it all. I was in conflict because I was taught integrity/honesty and there is no integrity in spending up dads helpful money on worthless junk so as to appear that you still have 'need'.
    ------

    The reason I bring this up is because I am curious if the units that will save money via IT consolidation will actually save us money or if they will be (by obvious standing procedure) driven to spend it in pointless/needless ways.

    Discuss? Anyone else experience this?

  11. Nice idea in theory... by hrieke · · Score: 3, Interesting

    Now just wait for a data center to be scheduled to close in some Congressman's home district and see how big of a block is put into place.

    --
    III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIIIV IIVIIIIIIVIII...
  12. History to Repeat Itself? by lax-goalie · · Score: 4, Informative

    I have no problem with the CONCEPT of consolidation, but Virginia's IT outsourcing/consolidation project to Northrup Grumman happened on Kundra's watch. It is an unmitigated disaster.

    Years into it, there's not even a complete inventory of the systems that NG is supposed to be managing for the Commonwealth, and at least as of a few months ago, NG couldn't even produce an invoice for the Commonwealth to pay that had more than six or eight line items on it.

    I sat through a special meeting of the House Committee on Science and Technology on the issue a few months ago, and the legislature is NOT happy about the situation. Privately, you will hear from them words like "gross negligence" to "I'm convinced it's corruption". The Delegates who engineered the legislation enabling the IT outsourcing are especially pissed.

    No disrespect to Kundra, but I don't think he's the right guy to oversee it.

  13. Re:IT as a commodity by eln · · Score: 3, Informative

    I agree with the government's effort to consolidate, because you can take advantage of cheaper per-gigabyte costs and have more robust backup, recovery, disaster recovery, and redundancy solutions when you're using enterprise equipment in large data centers. I think the government has a lot to gain from consolidation in this manner.

    However, I don't see that they'd have much to gain by outsourcing. Government data, by nature, is quite a bit more sensitive than just about any private company's data. The kind of security the government needs is not going to come cheaply, and it's arguable that any private company is really capable of providing it (although they say they are). Even if they can provide it, it's doubtful they can do it cheaper than the government could. For people in need of true commodity services like web hosting, outsourcing makes sense because it can be done far cheaper that way. For people in need of large-scale custom solutions, like the government, keeping it in-house is going to tend to be both more secure and less expensive.

  14. Re:IT as a commodity by jimicus · · Score: 4, Insightful

    If you think this is going to reduce IT expenditure requirements, you have barely worked a minute in IT. When you outsource, you are simply paying someone else to do your job, plus profit, plus a gaggle of negotiators in middle management collecting their kickbacks, plus downtime costs because your business is less important to them than your business is to you (if you have enterprise e-mail and it has been down more than, say, GMail, you have done something very wrong)...

    I've worked in IT for... a few years. And I agree with the GP.

    See, the thing is that while huge organisations will continue to require significant IT infrastructure (either managed inhouse or managed by an outside firm), huge organisations do not provide the majority of jobs in this world. The great majority of jobs are provided by SMBs. The really small SMBs have been outsourcing their IT for years - though "outsourcing their IT" probably translates to "get Dave's son to do it, he knows about computers".

    Slightly larger SMBs have been outsourcing their IT to some little company who thought they could earn easy money doing installation and support. Look in the yellow pages, you'll find hundreds of little companies offering services like this. Few of these little outsourcing companies are making serious money - there's simply too much competition in the market.

    Larger still SMBs (think medium rather than small, 40-200 employees) may have historically had a full-time IT person. But today there are dozens of companies offering outsourced Exchange, or you can sign up for Google for Domains and the price is so cheap that there is no way a single full-time IT person (even if you ignore their salary) can compete economically - never mind offering four or five nines uptime and spam filtering which doesn't leave people crying. Meanwhile, the cost of a single desktop PC is now so low that it's cheaper to have a spares cupboard containing enough spare PCs to re-equip an entire team at a moment's notice than it is to keep someone on staff to maintain them. Sure, they won't be particularly elegantly managed (there may not be a domain, antivirus may be totally forgotten about, they certainly won't have a standardised build) but let's be honest here - how many non-techies ever display any sign of caring about any of that? And business-specific niche software is frequently sold with a support contract anyway.

    Seriously - while anyone who takes careers advice from a stranger on /. probably needs their brains looking at, I'd say if you want steady employment with minimal risk of finding that not only are you redundant from your current post, supply and demand has made you worth considerably less since you last were jobhunting - get yourself a job in the public sector or get the hell out of IT.

  15. Good idea, but huge problems are coming by ErichTheRed · · Score: 5, Interesting

    I've actually done a lot of smaller server consolidation projects. In most cases, the results are great...those lonely database and file servers that get hits 5 or 6 times a day are all combined into one big box that actually uses all the hardware capacity.

    The biggest problems I've seen with VMs are the project managers who treat it as magic, never-ending capacity. The new favorite phrase in IT project management circles seems to be, "Oh, we'll just build a VM for it." Problem is, unless someone else is hosting your data center, you can't just call up and order more capacity without paying for more hardware.

    Second-biggest with a consolidation like this is incomplete requirements. Lowest-bidder contractors are not going to do a good job of gathering every single requirement...even high-bidder contractors have problems with this. And the problem is that the more they miss, the worse the fallout. A certain large company I used to work for found this out the hard way moving their inhouse data center to one of the big IT services companies. I'm a systems guy, and had all my stuff well documented. Others were pissed off they were losing their jobs and intentionally withheld information...the contractors didn't follow up, and a lot of last minute scrambling had to be done to complete the migration.

    Third problem for a government IT consolidation? Some huge services company like Accenture or IBM is going to win the bid and staff the project with dumbasses they pulled off the street in order to maximize profits. (Yes, this happened in my case in point #2 above...the sales staff presented the A Squad and swapped them out as soon as the contract was signed.) Not that government employees are rockstars, but they at least have a vested interest in keeping the data safe. IBM will probably win the contract too, given their involvement with government systems already. IBM has been so India-happy over the last ten years that I wouldn't be surprised if a lot of the (non security critical) work ends up there.

    Just like PMs treat VMs as magic hardware, CIOs treat outsourcers as magic black boxes that flawlessly run their IT operations. Unfortunately, the reality is not as sunny beneath the surface!