Slashdot Mirror
How Would You Build a Datacenter?
InOverOurHeads asks: "Some of my coworkers and I are building a new datacenter for our company. We're a growing startup, we have about 50 servers now and expect to have about twice that before too long so building to grow is key. Now that we're about $15,000 in to the project, it is looking and feeling more and more like we were way over our heads. We have 4 racks wired to a single 20amp circuit. Our UPS is at 90% load and we only have 10 machines on it. We have all of our cooling on one side of the server room where it is about 60 degrees, the other side of the room where the servers exhaust is about 30 degrees warmer, so it appears that we have some convection problems with only a handful of the machines on, right now. We're realizing that there is a lot more to building a datacenter than racking servers, what else have we missed?"
"On the positive note, we have a really nice overhead wire rack, that's looking good and all of our wiring is really tight looking; all the colors match, all the cables are labeled, they are all the right length, etc.
Are there any guides or how-tos on this? Since we're going to bite the bullet and tell the boss that we messed up we want to try
to correctly measure the rest of the work involved in making it work. What happens when the UPS is at 100% load and how Dell servers
react to being under powered?"
You're definitely going to need a lot more AC and A/C.
You'll need at least one 20-amp circuit per rack in my opinion if you go with standard 110 battery backups. For that many servers though, you might be better off going with 220 service and the high voltage battery backups that APC and others offer.
Our old server room started small with a couple of servers and quickly outgrew the AC service and A/C. We heated our whole office in the winter with just the servers! Maintenance ran several new 20-amp circuits for us until we filled up the breaker box.
When we moved a couple of years ago, I made sure to get the new room right before we moved any equipment. We have central A/C fed by several outside units plus a very large auxiliary unit just for the server room. 20-amp circuits are ran every few feet on separate breakers. I don't know what type servers you are using, but large multi-processor, redundant fans, RAID, etc. boxes use LOTS of power. We use mostly Compaq DL380's, two of them will draw 50% off an APC 1400R battery backup. For extended runtimes, we made sure to not overload the battery backups, so only two servers per backup with no more than two backups per 20-amp circuit. It's slightly overkill, but I got very frustrated in our old location and resolved to never blow breakers or kill battery backups this time.
Since you're just getting started, it will pay off big time in the long run to get everything setup right before you start loading in servers. It makes things so much easier to just plug in without having to call maintenance or a contractor to upgrade services.
Jason
"FORMAT C:" - Kills bugs dead!
Have cool air coming up from the floor into each machine (and it'd be freezing)
Have a diesel generator with at least a few day's worth of fuel, and contingency plans for obtaining more fuel. It should be feasible to run on generator indefinately in case of a major power outage.
Redundant data centers. Have data mirrored between them for complete redundancy in case of any disaster striking one of the locations.
/. However, it would be worthwhile to specify the degree of importance and the budget of this project.
Obviously I am being facetious. If you had a budget and the necessity to do something on that scale, you wouldn't be asking
Ecce Europa - Web Design for Business
The datacenter at my last job was about two thousand square feet and had on the order of 100 servers.
Some things that stood out for me were (I'm just a programmer, so this might be obvious you):
One rack was dedicated to what amounted to a switchboard. All the networking stuff was there. The wires running into the datacenter terminated in one set of ports (one pc, one port). These were then individually connected to a switch or hub using standard cabling. The servers in the room were wired the same way. Each station was hardwired to its set of ports on the switchboard. The idea was that it made it really easy to change the subnet of each machine w/o having to physically move it (just go to the switchboard and change the wiring). The internal telephone network (PBX?) was set up the same way.
There were the lab coats near the door. It was freezing. The AC units (I think there were 4-6 of them in the last place I worked) covered an entire wall. These were giant commercial ones. There was about two vents for every three aisles of machines. At the end of each aisle you could still feel the AC blowing.
The more expensive the machine, the closer it was to the AC.
The UPS covered about 100sqft in the basement and there was a diesel generator. Think the generator could run 15 hours on a full tank. Battery power was good for 15 minutes or so. There are about 300 pcs and monitors plugged in in addition to everything in the datacenter.
Orange outlets for clean (UPS) power, normal looking outlets for dirty power. The generator fed everything.
There was a control panel of some sort in the room where all the sysadmins sat. It had alarms for temperature and power and some other things I can't remember.
Hope this gives you some ideas.
regarding the generators... I work for a company that must maintain a large LAN, a call center, and a DNCS (digital network control system). Our generators are hard-piped to natural gas; they'll also run on diesel fuel if needed. Natural gas may sound like a bit extreme, but consider this: in June we had a surprise storm -- not a tornado exactly, but a large 'team' of microbursts destroyed approx 200-thousand telephone poles (and it took the utilties with it, of course). Our main DNCS location was without power for 6 or 7 days; without natural-gas-powered generators, we would have been screwed. (No electricity in the entire city = gas stations can't pump gasoline ... natural gas is your friend!)