How Would You Build a Datacenter?

InOverOurHeads asks: "Some of my coworkers and I are building a new datacenter for our company. We're a growing startup, we have about 50 servers now and expect to have about twice that before too long so building to grow is key. Now that we're about $15,000 in to the project, it is looking and feeling more and more like we were way over our heads. We have 4 racks wired to a single 20amp circuit. Our UPS is at 90% load and we only have 10 machines on it. We have all of our cooling on one side of the server room where it is about 60 degrees, the other side of the room where the servers exhaust is about 30 degrees warmer, so it appears that we have some convection problems with only a handful of the machines on, right now. We're realizing that there is a lot more to building a datacenter than racking servers, what else have we missed?"

"On the positive note, we have a really nice overhead wire rack, that's looking good and all of our wiring is really tight looking; all the colors match, all the cables are labeled, they are all the right length, etc.

Are there any guides or how-tos on this? Since we're going to bite the bullet and tell the boss that we messed up we want to try to correctly measure the rest of the work involved in making it work. What happens when the UPS is at 100% load and how Dell servers react to being under powered?"

Comments

Buy an existing one.

[lostindenver]

I know of four for sale in the denver area going for 1/3 to 1/2 what it cast to build one. They include redundant backbone connections, Power backup and great locations.
The best thing they are oversized, sell the extra space as a COLOC

Department of Redundnacy Department

[adamy]

Have no single point of a failure.Multiple UPS,network connections inside and outside, routers, firewalls, switches, etc. If anythinggoes down, you need to be able to replace it as quick as possible.

Are you in an earthquake zone (The Bay ARea)? If so, make sure 1, the building is earthquake retrofit, 2) the racks are all bolted to the wals suck that a little shake up doesn't turn into a shake down.

Make sure you are getting enough power to the building. Have Generators in case power goes out. UPS should only keep things going long enough for the generators to kick in.

Off site backups, of course. It is hard to beat the bandwidth of a stationwagon full of storage. Daily backups should be moved out of the building, I'd suggest on firewire/SCSI hotswappable hard drives, but there are many ways to solve this problem. Longer term backups should be a geographically out of disater range (east coast to west coast ideally)

Did I mention redundancy? Make sure you havea duplicate of everything.

OK, you have it built? Now test it. Kill the power and see if the UPSs can hold it long enough for the generators kick in. Now do it again, but pull out one generator.

Get one of those devices that allows you to remotely power cycle your machines as well, incase it locks up.

Havea back door (IE a dial up) to get into your data center unless you are going to have it manned 24/7. THis will keep you from coming in at 2 AM when a router blows.

Thats all off the top of my head. If I am wrong, please point out where, as the alternative viewpoints will be wuite helpful.

Re:Department of Redundnacy Department

[FattMattP]

the racks are all bolted to the wals suck that a little shake up doesn't turn into a shake down.

Although bolting the racks to the wall or floor will prevent them from falling over, it'll transfer the shock of any movement of the building right into the rack and your equipment. You should look at installing some form of isolation platform instead.

Recipe for a computer room

[LunaticLeo]

I assume you are not going to build a "datacenter", but rather build out a computer room. Given that here is what I have to say.

Don't build a computer room or datacenter. Find a commercial hosting service. Rent some cages and contract for reserving contiguous cages.

If you don't like the commercial hosting service here are the things I did to build out a computer room.

Power: Contract with a commercial electrician to get many more 20amp drops. The electrical contractor will know how to deal with the owners of your building to arrange the additional circuts. For most two processor intel boxes you can estimate 3 amps per box.

You can calculate the required volt-amps of your UPSs with this approximation UPSs volt-amps = Volts * AMPs * .7 . Computer volt-amps is really less than the volts times amps, due to complex impedence. Disk arrays are closer to 1.0 scaling. Don't skimp on power for disk arrays.

Get rackmounted UPSs spec'ed out for the hardware connected to them. Don't skimp out here either.

Cooling: You can purchase "portable" air conditioners and put them in your computer room. They will drop the excess heat into your office ceilings; assuming you are in one of those buildings with popup ceiling tiles. Office buildings recycle heat this way so it is OK. Find out if your building turns off AC on the weekends and nights. I was at a place that did that, and it sucked working weekends and it sucked worse for our computers. If they do cut AC on the weekends, then you will need more BTU cooling from your portable air conditioning.

If you are really going to build a datacenter contract with an appropriate architecture firm. In my mind a "datacenter" is a basement or whole building with full on-site deisel power generators and raised floors or overhead wire guides. That is probably not something required for upto 100 hosts. Over 100 hosts is where that might be a good idea.

Did I mention that commercial hosting service? You may grow out of your office space with employees and want to move. A commercial hosting service provides far greater quality computer and network capacity, and the don't tie you down to much.

Server Room Design

[aaarrrgggh]

The first thing to do is hire someone that knows what they are doing. You are talking about a fairly small computer room-- they are often the ones with the most problems.

Some quick pointers:
-A single rack can be at about 2 kW with overhead air conditioning. Underfloor AC will get you closer to 3 kW. Much more than that, and you get in trouble. If you guess a real demand of 150W/server, you can fit about twelve servers in a single cabinet before you start to get into trouble. Plan for 5kW per rack on your UPS system and distribution - 2x20A outlets or 1x30A, 208V per cabinet (non-redundant)-- double for redundant cords.
-The back of the racks should be hot. That isn't a problem, in and of itself. Good data center design is based on hot and cold aisles for just this reason. To see if you have a problem with the air conditioning, check and see what the return air temperature is-- if that is too low (close to the cold aisle temperature), you are going to get stuck.
-If the backs of the racks are hot, make sure you have blank covers over all the open spaces on the racks. That keeps the hot air from mixing with the cold air on the front.
-If you have raised floors with AC, try putting a tile or two on the hot aisle to induce flow and make it more comfortable. That should help some of the hot air get back to the AC units.
-Have an engineer look at it. If you can, hire someone that specializes in data center design. Plan for at least $1,000/day of their time, $2k minimum-- just for looking at it and giving you a report. It's money well spent! You can bolt on a number of fixes for a problem, but it won't fix the root cause. Maybe that is good enough.
-Be careful of the breadbox UPS vendors. They want to bypass the engineers and the contractors. They don't always tell you what you need to know.

{shameless plug}I work for a company called Syska-- there's plenty of other companies that do this type of work though. {/shameless plug} Find someone close to you that can help.