Build a Data Center or Contract Hosting?

bbsguru asks: "Our Government agency has around 100 independent divisions that share a dozen national applications and a private WAN. We are working to consolidate some of these applications (e-mail, SQL databases, specialized web services), and are facing a familiar choice. One option is to contract out data hosting, e-mail server hosting, and so forth to various vendors (with negotiated SLA's and all the best guarantees, of course). We have already started doing this for our private WAN-to-World gateways, VPN management, and one major SQL application, each with a different vendor, so far. Others are advocating the creation of a national agency-owned facility, where employees would perform these functions instead of contractors. Network management, IDS, data replication and so forth, for all the consolidated applications under one umbrella. Is a series of contractors really the way to go, or are there real benefits to keeping it in house?" The costs are always a factor, but the one-way nature of the contractor choice is also weighing in this decision. Some are concerned that if the expertise to create and manage these highly custom databases and services is farmed out to contractors, there will be no other choice in the future. Trouble is, as we evaluate our options, the process of contracting out bits of the whole is already underway. With each new contract, one more service to be brought into a datacenter is lost, making the whole thing less practical. Are we swimming upstream here?"

Comments

We're doing both

[plover]

Our company has gone both ways. We contract with a vendor to host our mainframes, but we've also built our own secure data center to house the many PC and Unix servers.

Contracting is attractive because the lawyers have this idea that you can sue the hosting service for failing to deliver services as promised. Of course if they fail to deliver, you have roughly three days before your company is permanently crippled, and seven before you are out of business, so that ultimately means only the vultures and the lawyers will get paid; but it sounds like a good idea to management. As a government agency you'll get yelled at and fired, and a few members of Congress won't get re-elected, but you won't go out of business.

One advantage to hosting is that they keep us current with hardware. Our contract stipulates an upgrade schedule for both hardware and operating systems, so we're constantly shuffling in the latest and greatest technology.

I don't know what the price difference is, as I don't ever see those kinds of numbers. But a new data center is mind-bogglingly expensive after you factor in generators, fuel tanks, chillers, security, alarms, power, fork-lifts, flooring, racks, cubes, offices, operators, guards, etc. With a hosting service you're sharing some of that overhead with the other customers of the host.

Running your own data center is good if you have a good team that knows what they're doing, and enough depth to survive the inevitable turnover. We do. But knowing how to successfully run a data center is different than knowing how to build one from scratch -- you need both kinds of knowledge before embarking down this path.

Why put all your eggs in one basket?

[dubl-u]

If the choice is between screwing yourself by becoming dependent on a bunch of different companies or screwing yourself by turning everything over to one monstrous internal bureaucracy, I'd say go with the former. It might be hard to fire one vendor and turn a project over to another, but it will be completely impossible to fire the central organization.

Really, though, I suspect you've created a false dichotomy. Among the vast soup of tasks you're looking at, some are probably done best by vendors, some by distributed internal staff, and some by centralized internal staff. But even for the centralizable ones, there's no reason it has to be the same center for each one, any more than your phone provider and your electric provider need to be the same company.

I manage a financial data center

[Travoltus]

My boss's business was swirling around in the same toilet bowl that you are.

We found that for accountability reasons and, in related issues, reliability and reputation issues, we had to bite the bullet, build the data center, expire (domestically) outsourced (er, contractor) contracts, and take it all in-house.

If you're not a financial services company it might be a less dire necessity. If you're a public company of any type? Between you and me, I'd take the data center. For many reasons requiring about a megabyte sized post, SOX will inevitably bite you on the butt when your data is "elsewhere" - elsewhere being anywhere except right there in the data center where you can control its usage in a highly draconian matter. There was also a recent law that came into effect regarding keeping all internal emails.

Contractors don't necessarily screw up, but there's an old war term my pappy taught me that applies here... don't let your supply lines get too numerous or too thin. Too many pipes tend to spring one leak, and nowadays one leak is very bad news. Keep it all in-house and you're statistically guaranteed to have less drama.

Oh and before someone says it, yes, have two data centers. In case the first one becomes the real life setting for "Destroy All Humans" or something.

3rd Option

[wired_scribe]

I think there is a third option that you should consider. Use a co-lo facility. Instead of trying to build your own data center (which is outrageously expensive,) or have someone else manage everything (which is unreliable,) put the servers in an existing data center and manage them in house. I am part of a hosting initiative at my company (we host environments for some of our customers,) and we've either priced out or tried the first two options. We are in the process of spending millions to move from a managed hosting center to a co-lo facility. We have found a 3rd party organization that can handle the hardware portion (if a drive fails they change it out at the data center, they change tapes out during backups etc.) We decided how much of the system they manage, and we take care of the rest. That way I don't spend my time dealing with updating windows and creating users. I spend it managing the databases and applications which I specialize in.

First step - what do you really need?

[avronius]

100 independent divisions that share a dozen national applications and a private WAN

1. Identify your business, and what the basic requirements are.
   • 100 different divisions / private wan, virtual lans, vpns / disaster recover, failover
2. The next step is determine what services are required.
   1. backend - the infrastructure
      • database servers and various network components
   2. middleware - the application layer
      • application, e-mail, etc. servers
   3. frontend - the pieces that talk to the 'net
      • firewall, load balancers, content switches, intrusion detection servers, web servers or portals
3. Consolidate where possible - eliminate if not needed - improve if required
   • reduce complexity and duplication of services / standardize on a single database platform (if possible) / standardize on a single web architecture (if possible)
4. Determine what your SLA's *really* are.
   • 99.99% for network and SAN - less than 5 minutes of downtime each month
   • 99.9% for major services - less than 45 minutes of downtime each month
   • 99% for individual servers in redundant or failover groups - less than 8 hours of downtime each month

After you've taken the time to gather that information, and get a real understanding for scope, you should begin to look at your various options.

1. Partial outsource - retain everything internally except for the actual server room(a)
   • rent or lease rack space from a third party
   • provide your own gear
   • continue to do all administrative tasks - including OS and hardware support
2. Partial outsource - retain all administration, with all hardware support from vendor(b)
   • rent or lease server and rack space from a third party
   • continue to do all administrative tasks - including OS
3. Partial outsource - retain all administration except for OS - all hardware support from vendor(c)
   • rent or lease server and rack space from a third party
   • obtain OS support from third party
   • continue to do all administrative tasks - except OS
4. Piecemeal outsource - each service provided by different vendor - you become contract administrator
   • rent or lease server and rack space from a third party
   • contract out OS support to different third party
   • contract out application / database administrative tasks to other third parties
5. Inhouse - this options requires a large initial expense, but results in the greatest overall control
   • requires facilities administrator - to take care of power / cooling / space allocation
   • If the facility does not currently exist, or requires expansion...
     • purchase generators / air conditioners / humidifiers or dehumidifiers / racks / network wiring / fire supression components / wiring harnesses and conduits
     • construction - server rooms are considerably more expensive than normal office space
   • OS support and administration / application and database administration / network administration

Only once you have a thorough understanding of the current state, a more robust array of options, and an understanding of cost vs. control will you be able to make the right decision. This is not meant to be a complete template, but should allow you to see the steps that are required more clearly.