Slashdot Mirror
How Would You Build a Datacenter?
InOverOurHeads asks: "Some of my coworkers and I are building a new datacenter for our company. We're a growing startup, we have about 50 servers now and expect to have about twice that before too long so building to grow is key. Now that we're about $15,000 in to the project, it is looking and feeling more and more like we were way over our heads. We have 4 racks wired to a single 20amp circuit. Our UPS is at 90% load and we only have 10 machines on it. We have all of our cooling on one side of the server room where it is about 60 degrees, the other side of the room where the servers exhaust is about 30 degrees warmer, so it appears that we have some convection problems with only a handful of the machines on, right now. We're realizing that there is a lot more to building a datacenter than racking servers, what else have we missed?"
"On the positive note, we have a really nice overhead wire rack, that's looking good and all of our wiring is really tight looking; all the colors match, all the cables are labeled, they are all the right length, etc.
Are there any guides or how-tos on this? Since we're going to bite the bullet and tell the boss that we messed up we want to try
to correctly measure the rest of the work involved in making it work. What happens when the UPS is at 100% load and how Dell servers
react to being under powered?"
If the state of IT in this country is ay indication, your best bet is to fire everyone and outsource your needs to India.
But maybe I'm just bitter.
El riesgo vive siempre!
don't do what slashdot does. They're datacenter sucks like a Friday night gay orgy at the geek compound.
There must be somebody out there who does this stuff for a living.
The latest Slashdot meme.
I recommend sprays of superchilled saline solution or if available locally seawater.
I know of four for sale in the denver area going for 1/3 to 1/2 what it cast to build one. They include redundant backbone connections, Power backup and great locations.
The best thing they are oversized, sell the extra space as a COLOC
Have no single point of a failure.Multiple UPS,network connections inside and outside, routers, firewalls, switches, etc. If anythinggoes down, you need to be able to replace it as quick as possible.
Are you in an earthquake zone (The Bay ARea)? If so, make sure 1, the building is earthquake retrofit, 2) the racks are all bolted to the wals suck that a little shake up doesn't turn into a shake down.
Make sure you are getting enough power to the building. Have Generators in case power goes out. UPS should only keep things going long enough for the generators to kick in.
Off site backups, of course. It is hard to beat the bandwidth of a stationwagon full of storage. Daily backups should be moved out of the building, I'd suggest on firewire/SCSI hotswappable hard drives, but there are many ways to solve this problem. Longer term backups should be a geographically out of disater range (east coast to west coast ideally)
Did I mention redundancy? Make sure you havea duplicate of everything.
OK, you have it built? Now test it. Kill the power and see if the UPSs can hold it long enough for the generators kick in. Now do it again, but pull out one generator.
Get one of those devices that allows you to remotely power cycle your machines as well, incase it locks up.
Havea back door (IE a dial up) to get into your data center unless you are going to have it manned 24/7. THis will keep you from coming in at 2 AM when a router blows.
Thats all off the top of my head. If I am wrong, please point out where, as the alternative viewpoints will be wuite helpful.
Open Source Identity Management: FreeIPA.org
Dont run everything off 1 UPS. You already stated you have 1 UPS at 90% of the load as it is. 1 Point of failure and the money you saved just caused a horrible outage and pissed off customers.
Sounds like you already know this, plan on secondary backup, AC, and make sure you have terminal servers. (Using unix right?)
You're definitely going to need a lot more AC and A/C.
You'll need at least one 20-amp circuit per rack in my opinion if you go with standard 110 battery backups. For that many servers though, you might be better off going with 220 service and the high voltage battery backups that APC and others offer.
Our old server room started small with a couple of servers and quickly outgrew the AC service and A/C. We heated our whole office in the winter with just the servers! Maintenance ran several new 20-amp circuits for us until we filled up the breaker box.
When we moved a couple of years ago, I made sure to get the new room right before we moved any equipment. We have central A/C fed by several outside units plus a very large auxiliary unit just for the server room. 20-amp circuits are ran every few feet on separate breakers. I don't know what type servers you are using, but large multi-processor, redundant fans, RAID, etc. boxes use LOTS of power. We use mostly Compaq DL380's, two of them will draw 50% off an APC 1400R battery backup. For extended runtimes, we made sure to not overload the battery backups, so only two servers per backup with no more than two backups per 20-amp circuit. It's slightly overkill, but I got very frustrated in our old location and resolved to never blow breakers or kill battery backups this time.
Since you're just getting started, it will pay off big time in the long run to get everything setup right before you start loading in servers. It makes things so much easier to just plug in without having to call maintenance or a contractor to upgrade services.
Jason
"FORMAT C:" - Kills bugs dead!
When my company set up a server lab last year we had to have the room retro-fitted with a dedicated air conditioning system... high volume flow with multiple output points around the room and it's own inflow duct as well, including filtration, etc.
We supplement it with several large industrial size fans for increasing air circulation to the racks.
Make sure there is plenty of space behind your racks and enough access points to that rear area to allow air to circulate.
More UPS units... two per rack at least depending on their capability.
I'm sure there's more to say.. my 2 bucks though.
A fool throws a stone into a well and a thousand sages can not remove it.
Realize you are in over your head and use an existing datacenter.
They have already done all the things you have done, all the things you are realizing you forgot, and all the things you will not find out about until everything fails.
You can now move on with you core business, whatever it may be.
Christopher McCrory "The guy that keeps the servers running" chrismcc@gmail.com http://www.pricegrabber.com
Have cool air coming up from the floor into each machine (and it'd be freezing)
Have a diesel generator with at least a few day's worth of fuel, and contingency plans for obtaining more fuel. It should be feasible to run on generator indefinately in case of a major power outage.
Redundant data centers. Have data mirrored between them for complete redundancy in case of any disaster striking one of the locations.
/. However, it would be worthwhile to specify the degree of importance and the budget of this project.
Obviously I am being facetious. If you had a budget and the necessity to do something on that scale, you wouldn't be asking
Ecce Europa - Web Design for Business
On the positive note, we have a really nice overhead wire rack, that's looking good and all of our wiring is really tight looking; all the colors match, all the cables are labeled, they are all the right length, etc.
"I don't care whether it works boys, just make it look good for the investors, ok?"
*shudder*
-Adam
On the off chance that this isn't a troll ($15 grand for 50 name brand servers, plus racks & UPS? It don't smell good here), did you guys do any research ahead of time, or did you just start slapping shit together? If I were your boss you would be FIRED, because you obviously have no clue as to what you are doing.
You obviously didn't talk to an HVAC engineer, because they would have set you up properly from the start, getting accurate heat output ratings for all the present and planned equipent (3.413 BTU per Watt, they tell me). Then, looking at the placement of the racks, would have had the cold air pumped in at the right places. This may be correctable by tossing a box fan or two in the room to move the air. Not fired yet.
You also didn't consult with an electrician or electrical engineer, because they would have given you some very sound advice on your load and UPS needs. You are so woefully underpowered and under-UPSed, your servers will lose power long before they have a chance to properly shut down. By the way, do you mean to say that you have 50 servers and only 10 are on UPS? FIRED!
"But look on the bright side, Mr. Boss, the cabling is all neat and pretty!" FIRED for spending more time on the cabling than you did planning this disaster!
Do the tech industry a favor, and go get a job shoveling shit at the zoo.
I assume you are not going to build a "datacenter", but rather build out a computer room. Given that here is what I have to say.
.7 . Computer volt-amps is really less than the volts times amps, due to complex impedence. Disk arrays are closer to 1.0 scaling. Don't skimp on power for disk arrays.
Don't build a computer room or datacenter. Find a commercial hosting service. Rent some cages and contract for reserving contiguous cages.
If you don't like the commercial hosting service here are the things I did to build out a computer room.
Power: Contract with a commercial electrician to get many more 20amp drops. The electrical contractor will know how to deal with the owners of your building to arrange the additional circuts. For most two processor intel boxes you can estimate 3 amps per box.
You can calculate the required volt-amps of your UPSs with this approximation UPSs volt-amps = Volts * AMPs *
Get rackmounted UPSs spec'ed out for the hardware connected to them. Don't skimp out here either.
Cooling: You can purchase "portable" air conditioners and put them in your computer room. They will drop the excess heat into your office ceilings; assuming you are in one of those buildings with popup ceiling tiles. Office buildings recycle heat this way so it is OK. Find out if your building turns off AC on the weekends and nights. I was at a place that did that, and it sucked working weekends and it sucked worse for our computers. If they do cut AC on the weekends, then you will need more BTU cooling from your portable air conditioning.
If you are really going to build a datacenter contract with an appropriate architecture firm. In my mind a "datacenter" is a basement or whole building with full on-site deisel power generators and raised floors or overhead wire guides. That is probably not something required for upto 100 hosts. Over 100 hosts is where that might be a good idea.
Did I mention that commercial hosting service? You may grow out of your office space with employees and want to move. A commercial hosting service provides far greater quality computer and network capacity, and the don't tie you down to much.
-- I am not a fanatic, I am a true believer.
The first thing to do is hire someone that knows what they are doing. You are talking about a fairly small computer room-- they are often the ones with the most problems.
Some quick pointers:
-A single rack can be at about 2 kW with overhead air conditioning. Underfloor AC will get you closer to 3 kW. Much more than that, and you get in trouble. If you guess a real demand of 150W/server, you can fit about twelve servers in a single cabinet before you start to get into trouble. Plan for 5kW per rack on your UPS system and distribution - 2x20A outlets or 1x30A, 208V per cabinet (non-redundant)-- double for redundant cords.
-The back of the racks should be hot. That isn't a problem, in and of itself. Good data center design is based on hot and cold aisles for just this reason. To see if you have a problem with the air conditioning, check and see what the return air temperature is-- if that is too low (close to the cold aisle temperature), you are going to get stuck.
-If the backs of the racks are hot, make sure you have blank covers over all the open spaces on the racks. That keeps the hot air from mixing with the cold air on the front.
-If you have raised floors with AC, try putting a tile or two on the hot aisle to induce flow and make it more comfortable. That should help some of the hot air get back to the AC units.
-Have an engineer look at it. If you can, hire someone that specializes in data center design. Plan for at least $1,000/day of their time, $2k minimum-- just for looking at it and giving you a report. It's money well spent! You can bolt on a number of fixes for a problem, but it won't fix the root cause. Maybe that is good enough.
-Be careful of the breadbox UPS vendors. They want to bypass the engineers and the contractors. They don't always tell you what you need to know.
{shameless plug}I work for a company called Syska-- there's plenty of other companies that do this type of work though. {/shameless plug} Find someone close to you that can help.
The datacenter at my last job was about two thousand square feet and had on the order of 100 servers.
Some things that stood out for me were (I'm just a programmer, so this might be obvious you):
One rack was dedicated to what amounted to a switchboard. All the networking stuff was there. The wires running into the datacenter terminated in one set of ports (one pc, one port). These were then individually connected to a switch or hub using standard cabling. The servers in the room were wired the same way. Each station was hardwired to its set of ports on the switchboard. The idea was that it made it really easy to change the subnet of each machine w/o having to physically move it (just go to the switchboard and change the wiring). The internal telephone network (PBX?) was set up the same way.
There were the lab coats near the door. It was freezing. The AC units (I think there were 4-6 of them in the last place I worked) covered an entire wall. These were giant commercial ones. There was about two vents for every three aisles of machines. At the end of each aisle you could still feel the AC blowing.
The more expensive the machine, the closer it was to the AC.
The UPS covered about 100sqft in the basement and there was a diesel generator. Think the generator could run 15 hours on a full tank. Battery power was good for 15 minutes or so. There are about 300 pcs and monitors plugged in in addition to everything in the datacenter.
Orange outlets for clean (UPS) power, normal looking outlets for dirty power. The generator fed everything.
There was a control panel of some sort in the room where all the sysadmins sat. It had alarms for temperature and power and some other things I can't remember.
Hope this gives you some ideas.
Just a good KVM setup.
Dark Nexus
"Sanity is calming, but madness is more interesting."
[see parent post]
You need help. (With an architect, I used to do this stuff for the "largest router company in the world".)
From an architectural perspective, don't underestimate the complexity of space planning. Equipment access, emergency egress, and growth of all engineering and supporting systems may put you at a very different place than you might imagine if you consider only your direct server capacity. I'm sure every geek around here would like to think they can solve most engineering type problems with a little extra effort, but building design has more than a few gotchas you don't want to miss.
On the building engineering side, the general trend is for higher and higher densities. Ten years ago, one might have projected that data centers would be getting exponentially larger, but the increasing density of electronic components keeps that growth more reasonable. However, density of equipment has a nasty side effect in that it pushes HVAC, power, UPS, and structural limits far beyond what your average spec office building is designed for. I know from experience that increasing structural floor load capacities from 80psf to 150psf is eyebrow-raising expensive with an operative data center!
Don't make dangerous mistakes. Beyond the expense, embarrasment, and possible job loss, you could create a serious life safety problem for yourselves or those working around you. Obviously four servers isn't exactly a major data center, but if that triples in the middle of a low load floor bay, (or if they're already some mondo racks) you might be closer to floor capacity than you realize. Sounds like you're beyond UPS, power, and HVAC load now--hire an architect with an engineer in tow for a few hours ($400-ish) to advise you. (Or mail me with your geographical location if you need recommendations. ;)
There is no need to use a SlashDot sig for SEO...
... but there are some other things to be said.
... you've got noplace to stand without them.
First off, you don't tell us a lot about your business. Do you have well-stated service level agreements? If not, get the hell off slashdot and do SLAs
Second, once you know what your service level committments really are, ask yourself "do I have half a clue what the fuck I'm doing?" If the honest answer is "no", do what has been suggested and move your services off to a hosted-service firm. (I have heard good things about Rackspace but there are a bunh of these firms around.) Even if the honest answer is "yes", you should seriously consider a hosting firm, because a lot of the things you need to do have a big fixed cost up front; you get better payoff by buying a part of the up-front costs someone else alerady paid.
Third, one topic you've left off the list is disaster recover -- what they now call "business continuity planning" on the grounds that "disaster" sounds so scary. You need to make sure that you have (1) off-site backups sufficiently far offsite to be pretty certain that a local disaster won't compromise them. (I had several clients and acquaintances who were knocked off the air on 9/11/01. Some of them had backups outside Manhattan, and they got working again quickly. Some of them didn't, and some of those firms are gone now.) You also need (2) to know where you're going to go with those backups if the disaster comes. A number of firms will let you pay them a small fee that guarantees access to backup servers; the other end of the spectrum is to have a secondary site that's geographically well separated, eg, New York City and Princeton, NJ or southern Connecticut for Wall Street firms, or San Francisco and Salt Lake City for the west coast. Once you've set this up, test it. Cut over to the backup every so often.
How do you know what level of this stuff you need? That's why you need a well-stated SLA.
* redundancy and over-provisioning
for instance you should never have to worry about not having enough power because you never let the load on the UPS's go above 75%.
you should never have to worry about bandwith or cooling.
you should be able to do maintainance on one machine in a group (including routers, UPSs, etc) and not effect anything. nobody should notice at all if a machine dies. that helps you a LOT.
* simplicity
make each group of hardware do ONE thing if possible. I.e., web servers, firewall, etc. don't cut corners.
* expensive
it will cost a lot more than $15,000 when you're done if you do it right.
* planning
every change you make to the setup should be well documented and planned. you know how, at home or in your office, you set everything up beautifully with racks and color-coded cables and everything? then it gets all fucked up after about a month? well in your datacenter/computer-room, you should NEVER have things get fucked up. every machine should have a reason for being there. if you need to make a change it should be after a few days of discussion.
good luck , you'll probably find it's not worth it unless your "core competency" as a business is managing datacenters.
Rackmounted UPS systems are OK, but going overboard with them doesn't help things. They need maintenance, batteries only last a few years, and most importantly, they are fire hazards.
Using rule-of-thumb numbers, for 50 servers I would guess between a 10-15 kVA UPS. Generally, 15 minutes of battery is a good number, but if you don't have a generator backing it up, all it will give you is orderly shutdown. That may be good enough for your needs... much of it depends on how good the utility is.
15 kVA is small enough that you can add some batteries and go for 30 minutes, but a generator isn't really cost effective. If you have redundant power supplies, the money is better spent getting two systems each with 15 minutes. At half load, you will get almost 30 minutes on each. Going longer than 30 minutes is going to be a push on the AC; you start to need a generator. Generators don't make much sense smaller than 100kW, or 50 kVA of UPS load. If you are in a high rise, try to negotiate for generator backup power... a few 30A circuits can go a long way.
Also, make sure that your UPS has an EXTERNAL bypass.
The tech's were fired/offshored. This is the marketing guy trying to do this.
Chapter 17 of the ever popular Practice of System and Network Administration (ISBN 0-201-70271-1) by Limoncelli and Hogan gives a fairly broad overview of some of the things to consider when building a data center. Everything from chosing a site, to climate control, to wiring and cable management are considered.
Go find it at your local computer bookstore and read the aforementioned chapter.
Better yet, buy the book. If you're like many of us and have had system adminship thrust upon you, the rest of the book makes a good reference.
Co-locating your servers at an established datacenter will give you (assuming you choose your vendor carefully) a datacenter that is far nicer than you ever could have built yourself. They will have had experts design a good HVAC system and redundant networks to boot.
It sounds like a bit more research may have been in order before diving into this project. Your power requirement estimations seem WAY off to me. We generally have at least 2 20 amp circuits feeding each rack. The power coming to each rack is coming straight off of Liebert UPS systems. If you have just regular old AC power coming to each rack then you need to size your UPS appropriately as well. Having two smaller UPS's will make you happier than one beefy UPS in the long run. If one small one fails, you can hopefully at least limp along until you can fix and/or replace the faulty UPS. And running your UPS at 90% is bad as well. What if you have a power outage long enough to deplete your battery? Then, when power comes back on and all of the systems plugged into your UPS try to boot up at once, you'll spike the power to levels higher than your UPS will be willing to provide.
As far as cooling goes, having raised floors with A/C ducts underneath gives you the ability to shoot cool air up from the bottom, rather than trying to push the hot air down from the top. So, if your A/C unit pulls hot air from above, cools it and pushes it back down below to begin the cycle again, you'll find that your machines will run a lot cooler. Having cooling dedicated to each rack helps a lot too.
Re: your nice wiring job
It may look nice but if all of your machines burn up and/or are under-powered then what's the point? Priorities man, priorities.
Bottom line: I'd spend a bit of your $15K on some professional consultation before going any further. This is a pretty small data center, but if you want it to last and be somewhat scaleable, you should start out on the right foot.
How about our fancy machine room with the nice raised floor, plenty of power, genset, cooling, and all that jazz? It's on the half of the floor we had to sublet to a real company with 100 times our revenue.
Our servers? Piled in a reclaimed conference room with big box fans on the floor blowing into the room. And if you think that's the dumbest thing we could do, guess again. It's not even in the top ten.
Go some place that has done it already and talk with their engineers about it. Your ISP is a great starting place as you already have a relationship with them and they are usually happy to show off their setup to their customers.
Dell has enterprise consulting that can also help you setup your datacenter. Ain't free, though.
Good luck.
-m
http://www.invisik.com
I hate to say it, but for $15,000 you could have paid for 20+ months of full rack rental at a datacenter. At least that's the cost here in Austin, TX. Now of course, that doesn't include bandwidth, another $350/month for 2Mb/sec. So maybe we are talking 15 months. But I think it's going to cost you a lot more money to build a REAL datacenter (on a small scale) that can even match a pre-existing datacenter.
A bunch of random comments.
1. If you are doing this for internet based servers, you might well be better off colocating your servers in a commercial data center. The cost of doing this is a lot less than it used to be and is likely less than doing it yourself. This might not be the solution for all of your systems, but may be appropriate for many.
2. You will need a lot more power than you think. Most "single/dual" cpu servers are 3-4 amps each. If you load up a rack with 1U servers, this can easily require 4 20amp circuits. UPS ratings are also in VA which is less than watts, so a 3KVA UPS is required for a 20amp circuit.
2a. Professionally install your power. Conduit and "lots" of plugs are approrpiate. 20A plugs are usually not necessary (the ones with the sideways prongs).
3. If you overload your UPS, there are two side effects. If you are only close, you just lose backup run-time. If you actually go over, then the servers all shutdown, rudely. It does not really matter what kind of servers you have in this case as the UPS itself will usually take everything to zero.
3a. Do you have a generator. If not, then the UPS will become very expensive to get any type of real run-time.
4. It sounds like you have a bit of an airflow issue. If you are using enclosed racks and have no need for rack "security", take the doors and sides off completely. If you need to lock the racks, then get perforated fronts and mount several 8-inch fans thru the back doors to blow air. You also need to get the air moving around the room. Some high-quality fans up high might move things around enough.
4a. If you are in a "seismic" area (like I am in California), be sure to bolt everything down. Even if your datacenter goes down during a 6.5, you will be less likely to kill someone.
4b. Pay attention to fire control. Sprinklers are "bad", but sometimes required. Haylon is more desireable, but often water is required anyway.
5. Buy equipment used if you can. Things like racks etc. are plentiful from failed co-los.
6. Remember remote management. You might not need managed switches, but remote power controllers are really nice. We use stuff from BayTech because it does both AC and serial. Other solutions are available from APC, etc. A lot of this stuff is also available used (ebay) so shop around.
7. If you buy used, consider possible licensing gotcha's. Some vendors have steep software/firmware relicense fees.
I've seen all sorts of "datacenters"... from a boy's room of a convereted elementary school to two-story behemoths in high-rises.
There are plenty of problems to be had, even when there is a professional staff of electricians, carpenters and plumbers. (Yes, plumbers... one place had a water-cooled mainframe)
There is also the issue of floor load... at one building they discovered that placing three IBM Shark's in the middle of the floor caused the cantilevered floor to "jump"... we actually had to borrow 1/4" steel plates from the local DOT to roll the sharks to the edge of the building.
There is a glut of computer rooms in this country... take advantage of the glut and pay someone to handle this.
Conformity is the jailer of freedom and enemy of growth. -JFK
Ask your local utility if they can help. I know of a couple data centers (all on one utility so I don't if others do this, but they should) who pay a reduced rate for the power to their data center. In return on the high demand days the utility sends a command to their UPS (and in turn the generator) to switch off of mains power onto the backup. Not only do you get the benifit of lower power costs, but it tests your backup under a real situation.
Look at the power requirements of your servers, vs how much you use them. If some of your machines are a few years old you might be able to combine tasks on one new machine freeing up more space and useing less power. If you really need seperate machines, but not all the power look for low power alternatives (either mobile chipsets, or ARM based systems depending on your needs). I know of one company that replaced a bunch of old (and insecure) SUNs with a single x86 system and marveled at how much more comfortable the machine closet became.
Continue to re-evaluate everything continually. Something that doesn't work now may be good tommorow. Something best practice for good reason today may be obsolete tommorow. Sometimes the obsolete and ridiculed still gets the job done better than the fad replacements.
Keep in mind that 1 watt is 1 amp at 1 volt.
20 amp circuits at 120V will power about 5 or 6 400 watt power supplies reliably. if you have 50 machines (assuming 400 watt power supplies), I would go with no less than 10 20 amp circuits. Get a certified electrition to wire up power panels. He will know how many breakers you need and should be able to do the full job for around or under $5000.
Battery backup and generators are s imilar problem. Keep track of how many watts you are running and buy a surplus of battery backup and generators according to the rule of 1 watt is 1 amp at 1 volt.
Why give this retard a chance to live. He should be fired immediately. He is an incompetent idiotic asshole and helping him is not the way of capitalism, Darwinism or any other force that works to eliminate the retards and idiots that dilute the good things. We as humans can't even build our own habitats yet with the only requirement being power [e.g.; make a submarine of a spacecraft that would support life indefinitely with only fuel being the requirement], and people like this having any power whatsoever are the reason. Imagine this guy trying to make that LIFE-SUB or LIFE-SPACESHIP with moronic tendencies like this. Amazing.
This "data center" is way too bland. What the hell are they doing? Shoveling packets? Testing equipment? What kind? What kind of gear, what network infrastructure, how far to the next NOC, doing Telco/IT crap or doing simulation jobs? Need storage? Application Servers? Large backend farm? Lots of CPU? What? What is the goal? The various designs for various things are basically available in template form. You think it takes a room full of engineers to pull out a new NOC for a large company like former Exodus or Verio? This stuff is fairly standard. If common sense and maintainability are something you have to ask others about, you just don't have it. Maybe working for some place that has to deploy in a planned, procedural sensical manner might be in order. Let's get on thing straight. This guy has NO experience relevant to his responsibilities. Another probably-failed startup.
I mean, all this 'tard has to do is get some capacity estimations and build blocks of n number of units and lather rinse repeat.
50 servers. heh. to the Slashdot crowd that might seem like a lot. Those are also the same people who have no idea what process means.
Ridiculous fucking question. Another classic loser from Ask Slash Crap Puke Dot.
There's this little aluminum products manufacturing facility in rural Nebraska that makes great stuff for data centers, and the cost will pleasantly suprise you: www.mtpartners.com
Also, put the cables under your tiled floor, not in overhead racks. Typical overhead racks are open-air and human-accessible in a way in which tiled-in areas are not.
Finally, have someone skilled in the arts of physical security take a look at your building, entrance, etc. For example, I've seen people build really nice data centers about ten feet away from a busy road. They are unwittingly about one violent car wreck away from having their whole enterprise go down. Sure, there are concrete barriers for that sort of thing, but in both software and hardware, the best security is built in, and planned for from the beginning.
http://tinyurl.com/4ny52
Using KVMs is something indenmic to PC weeny faggots. Real servers have management processors.
A Little Windows or Lin-sux box with a little mouse and keyboard so one can mouse the user manager from afar!!!
Heh. Anything that is a PC and called a server is usually a long shot.
Make sure that you have a good offsite backup system. I would suggest on a professional outfit like Iron Mountain (formerly Arcus). Be sure to run through some sort of disaster recovery simulation at least annually. The worst thing is having to re-build a datacenter from nothing after a disaster. You may be able to re-build the center, but the data is the really hard part.
US Democracy:The best person for the job (among These pre-selected choices...)
Firewire backups? That is the metric by which you measure expense? Jesus Christ. You clearly are a bit "kiddie."
You would mention Ultrium Tape Libraries, maybe an EMC or FCAL SAN [or for the daring/bleeding edge, some new iSCSI crap from NetAPP] or some such for high end backups, but fucking FIRE-WIRE? Classic.
And yes, off-site backups are required.
And your advice in summation is:
Do things half-assed and don't assert yourself to your boss and pretend to be able to do the job properly with no resources and no budget because you would risk the company in order to keep your job because you have no scruples or one iota of professionalism.
The allusion to firewire backups makes me think you are some fool who runs those pathetic Apple X-Serve's and think lots of things that are wrong. Like IDE is okay in servers, no ECC memory in servers is fine, OS X doesn't suck, etc.
- Never, ever overload a UPS. Bear in mind, in fact, that as UPSen age, the alternate supplies (typically gel batteries) degrade in performance, and so "100%" when new becomes "110%" when about three to four years old. The rule of thumb we use here is no more than 60% load on any UPS.
- Make everything - everything critical to the mission, that is (power to building, UPSen, generators, network kit, servers, disks) - fully redundant. Much like this post, in fact.
:)
- Make sure that redundant PSUs in computers are truly redundant - that is, they are completely separate all the way down to the motherboard (some use a common input bus, which is, well, stupid). If a PSU blows, it's really bad if it takes out not just one whole circuit, but both...
- Use two separate phase supplies, if available, for your datacentre. Power here in Australia is three-phase; I'm sure it's similar over where you are. Often an upstream transformer blowing will knock out one of the three phases completely and leave the other two partially intact or completely intact; ensuring that you're on fairly separated circuits is a damn good thing.
- Never underestimate surge protection. UPSes are great, but make sure that they isolate input power from output, and only pass through when/if passthrough knife relays are available or used (i.e. when offlining the UPS for maintenance).
- Network redundancy means having separate cables, separate switches, separate NICs, separate, well, as much as the budget will allow. I've personally had to look after a system that had fully redundant switching and cabling, but then relied on single paired-outlet NICs - and the loss of one "half" of the NIC took out the whole system. This was an NT4 PDC too, and that's a bad situation to be in...
- Longer term, if you're really into reliability and money becomes not much of an object, you probably want to look at having geographically separated datacentres, and again, duplication of everything at each. Eh, I can dream.
:)
They're the main things. Redundancy is the key here. Did I mention that yet?Never take manufacturer's reliability figures with less than an entire salt mine full of salt. Systems always fail more often than quoted.
Never underestimate the power of full backups, and preferably a standing image in GHOST format or similar of each critical system. Being able to restore a machine without having to sit through an install will help your sanity infinitely.
On the positive note, we have a really nice overhead wire rack, that's looking good and all of our wiring is really tight looking; all the colors match, all the cables are labeled, they are all the right length, etc.
That's great, kid. No offense but you are way out of your depth. Call IBM and get them to build it for you, or at least send over a consultant to advise.
If you have to ask slashdot for advice on building a mission critical data center, I think that you should resign and hand over to someone else.
Raised floors are damn useful. By all means do all your cabling overhead, but if you can connect the A/C to pump air under the floor it flows nicely up the inside of the racks and out the top. This means that people can actually work in the server room without freezing, yet it still keeps the servers cool.
Environmental monitoring is a good idea, for when something stops working at 6:00pm on Christmas Eve.
Fire extinguishing is important. The sky is the limit for what you can spend, but at least make sure you have handheld extinguishers even if you can't afford the more elaborate stuff.
Patching per-rack is nice. When you set up it can be easy to just patch from point to point, but if you can put a panel in each rack, through to a central patch-panel, then re-patching etc. etc. will be much easier.
Minimise monitors. We run our entire server room (about 120 servers) on a single monitor and keyboard with cascaded KVM switches. Of course we ssh in for about 99.999% of the admin work, but every now and again you need to sit on the real console. A KVM generates less heat, uses less power and (per server) costs considerably less.
Redundancy. Or did someone say that already?
Windows. It's good to be able to see from outside what is going on in there. Not from the big blue room, of course, but from inside the rest of the office. This is particularly the case if you have to swipe a card or something to get in there just to go looking for someone who isn't there in the first place. Usually the place should be secure, in any case, so having it like a fishbowl tends to promote that.
Speak to a real estate agent. It may be that you can rent a server room that meets all your needs already. Lots of business have gone to smaller server rooms, or outsourced their servers to a hosting facility, and it is possible to walk right into something that's already got 90% of what you want.
Finally: Expect the unexpected. What happens when you have a server room full of mainframes, 1000's of litres of diesel, and honking great generators, all on the 14th floor? Well, about 4 hours into a major power cut you discover that the water cooling for the generators comes from water pumped to a reservoir on the top of the building.
And the pumps don't run off backup power.
And it's empty...
This isn't something you can just throw together.
You might find these guys helpful: http://www.lampertz.com/DuS.htm (not affiliated, just had one installed).
They put in secure datasafes and modular IT rooms that are literally blastproof, watertight and fireproof. Not cheap, but datacenters aren't - which you are going to learn as soon as you try and create one.
I would make sure you have a few flashlights and maybe one of those lights you can clip on things. And make sure the overhead lights (or at least some of them) are on redundent power as well. If the power goes it would be really nice to be able to move around the room and see things if you have too. Its always easier to work on stuff if you can see well.
Some of those light sticks that campers use might be good too.
Erlang Developer and podcaster
Speaking as someone with a degree in civil engineering, who has been doing system administration for the last 8 years, I'd say that first, you should be informed about what all of the issues and variables you're not even thinking about. [eg, chilled air and warm air return issues, proper sizing of power systems, etc]. So, pick up a copy of Enterprise Data Center Design and Methodology, and read it.
... you have to determine what the cost of the system is, and what the consequences are if you didn't have it. Then you have to compare the costs to the benefits, and see if it makes sense in your environment. [eg, if your SLAs for a system make no provisions about down time, then there's no reason to have them on a UPS or generator, as there would be cost without benefit. [you don't save money in the time of a power outage]]
Then, once you know what you need to be looking for, hire a professional -- odds are, there's stuff they'll know that's not in the book, and with years of practice, they'll have experience to fall back on that might fit your needs. [But with the info from the book, you'll at least be able to talk to them about your issues, and make sure that they're not trying to pull one over on you about your qualifications].
But let's look at this like sizing for a UPS
Oh...and if you're completely cheap, read the first chapter online -- Data Center Design Philosophy
Build it, and they will come^Hplain.
Download their Data Center Site Planning Guide (PDF) at the following URL: http://docs.sun.com/db/doc/805-5863-13?a=load
I've done several of these. For $185/hr, I'll help you do yours :)
That said, even on this small a scale, you should have an environmental and electrical engineer involved already. More than likely they're going to be pushing you towards a facility UPS, Dedicated AC, and diesel/natural gas generator.
My best advice, if you can't make these calculations yourself, is to LISTEN TO THEM!. Let them worry about power factors, UPS load, powerline harmonics, Thermal outputs, etc. That's what they get paid to do.
Alternatively, contract someone like me in to do it for you. Of course, I'll do the same things you could do, and probably use the same engineers you should, but give you the privilege of paying me to do the job so you can concentrate on things you know how to do and are important to your business.
Sig??? I don't need no stinkin Sig!
Ummm, ok. I hate it when people brag on slashdot about how much hardware they have but, so be it.
If it's a pissing contest you want, you've got it.
Yeah, as I said in another post, I probably have about $15,000 in hardware sitting at my desk right now. I stayed late at work today installing Windows 2003 on a system but it took a while to format the 22 attached hard drives.
Frankly I wouldn't know if IDE or non-ecc memory is ok in servers because I've never really had to bother with it. I've got at least six P4 2.8 Ghz processors sitting at my desk right now, that I haven't put in systems yet.
But, I also know that I've been totally satisfied with a Duron 600Mhz system at home running Gentoo (and Redhat before that) for about three years.
I'll be honest, I don't know how much firewire backups cost. I don't know how much data the OP wants to back up. Considering they're worried about maxing out the UPS, or not having enough AC maybe there are bigger fish to fry than how fancy their backup solution sounds.
The moral of the story is that sometimes people are required to "do the job properly with no resources and no budget" and standing up to the boss won't make money magically appear in the server budget. And if the company goes out of business because they spent all of their money on backups you won't have a job anyway, no matter how many scruples or iotas of professionalism you have.
And yes, off-site backups are required.
It totally depends on the situation and budget. Do you keep an offsite backup up of your entire hard drive(s) at home? Probably not, so I guess it's not required in every situation, is it?
Well, anyway whoever you are, Coward, you don't have much business sense because you'd rather piss away money on the hot new thing than spend wisely on just what you need. I hope we're never stranded in the desert together because you'll be brushing your teeth with the last of the water supply.
Yeah, and by the way, OS X doesn't suck. I've never used it as a server, but I don't see how you can really put it down as a desktop OS (who in their right mind buys Apple servers anyway). More user friendly and more powerfull than windows, for a Unix geek anyway.
If a server has a 500w power supply you are drawing 4amps of current.
500watts/120volts ~= 4amps/server
Now if you have 4 servers...
4amps/server * 4 = 16amps
Give a little extra to the UPS say 1-2amps for a total of 18amps, so yeah, I would say you are at 90%. Oops!
You should have a minimum of a 30amp circuit for EACH rack, or better yet, dual 30amp circuits, and a UPS for each rack.
$15k should have been able to cover all of that.
You also sound like you have serious heat issues, and as you probably know your silicon is going to break down a lot sooner than you thought.
First things first. Call an HVAC professional and get your cooling under control, ask for him an idea of what kind of circuits HE will need. Next, call an electrician and order more circuits.
While you are waiting for those guys to come, go buy some kick ass box fans and place them strategically to move cool air in, and hot air out. I have no idea what your physical security concerns are for the data center, but it sounds like you are going to need a door open.
In short, you have fucked up. Making cables all the same color or for particular purposes is nice, and it's damn cool looking, but at the end of the day, or at the end of the year, when shit needs to be moved or re-arranged or assumptions change, all of that goes to hell in a hand basket.
No.
Seattle has a good abandoned data center. Diesel generators, huge battery banks, lots of backbone, earthquake braced cabinets, biometric scanners, top of the line and mostly unused. The company I used to work for layed us all off and the whole thing is just sitting there. So you can hire a cheap crew - I know a few - and we will have you up and running in no time. Just don't expect to fill up the unused space with our old sales team and don't run the company like fools.