Slashdot Mirror
Building an Energy Efficient Datacenter?
asc4 asks: "The company I work for is a webhosting and colocation company. As our power utilization grows, we have begun searching for ways to make our datacenter more efficient. The biggest hit from the utility company comes in the peak usage charge, which penalizes (rather severely) for the highest sustained burst of usage during a billing period. Due to the nature of the colocation business, we can't control how much or when client devices use power, so I'm wondering: is there's something we can do at the datacenter level to help smooth out our power consumption, over the course of a given period of time?"
"In these days of hybrid cars, Energy Star devices, and in general more eco-friendly power consumption, it seems like there must be some products out there that can help make datacenters more efficient, as well. Could fuel cell technology be something to look into? Would flywheels or capacitors help? How about using more efficient AC units than what are available from the big names? What are others doing to reduce peak power consumption in high-drain datacenter environments?"
There are solar systems that allow you to only 'buy' from the electric Co when your batteries are low. This isn't a bad option, but on the scale of a Datacenter, the sheer volume of the batteries required to convert the DC Solar to AC, and the accompanying conversion losses might not make it worthwhile.
Another option would be to get a natural gas line into the building, purchase your own generator, and when you aren't using the excess capacity of your generator, sell it back to the electric grid, if your utility provider would allow such a thing. I have no idea if that is any cheaper though.
Once you get a feel for how the datacenter is 'breathing' (i.e. watch the usage graphs and become familiar with the pulse of workload, etc) you should be able to come up with good solutions to your problems (like starting your monthly billing processes 2 days early, so you can only run the batch processes at night when the power is cheaper).
Also, never underestimate the cost of lighting and A/C. Maybe you can get by with only turning on every 3rd fluorescent light. Maybe you can use exhaust fans instead of A/C in a colder climate.
The point is you'll never know what problems you need to address unless you monitor your DC.
In many states, you can save substantial amounts of money by agreeing to scale back energy utilization during critical times. In New York, NYSERDA (www.nyserda.org) is the agency that administers the peak load reduction program.
Conformity is the jailer of freedom and enemy of growth. -JFK
For smoothing out power usage, there are a number of different options -- aside from alternative energy, you could do rolling brownouts throughout your datacenter and rely on UPSes or generators to keep things going -- but you *will* take a hit in reliability. Every switchover -- one mains circuit to another, mains to battery, etc. -- carries some risk.
I've watched an entire datacenter go out on what was supposed to be a controlled switchover -- power company needed to do some work, pulled the plug (with the datacenter's consent), the backup generators start... and then die. The UPSes kicked in, but could only supply 15-20 minutes of power. Everything failed over to a backup datacenter, whose link then decided to go out to lunch.
Total cost of the outage was measured in tens of millions of dollars.
Just keep this in mind when doing the business justification calculation (cost savings from lower energy bills, minus upfront cost of equipment, minus risk of additional downtime times cost of downtime, minus cost of maintaining the equipment). Unless energy prices go *way* up -- like oil hitting $250/barrel -- I'd be surprised if this would pay for itself.
You may want to look into AMD based systems instead of Intel. We have reduced our power load considerably and gotten a boost in performance by using HP DL385 2 way servers with Dual-Core processors rather than Dell 4 way Intel servers. Don't underestimate how much this can impact power utilization when you have 100 servers.
1. Cool down the centre during the night when hydro is at its cheapest.
2. During the day raise the thermostat so the AC does not kick in too soon.
3. If you have windows use the blinds on the sunny side. Thermal load is a royal pain. Where I work it hit 27c inside even though it was -14c outside. The north side was running at about 21c.
4. Put all non-essential equipment on powerbars and turn off the bars. Most monitors and other electronics still draw a bit of current for 'instant on'. That takes hydro and dumps more heat for the AC to handle.
Panic now, beat the rush!
If you've got a datacenter large enough that energy efficiency is a problem, I recommend you move the whole shebang to a location where energy is more plentiful. Upstate NY, which has plenty of hydroelectric power, would be a good choice. Nowadays, thanks to the internet, you don't have to keep your datacenter next to part of your operation.
Solar panels kick out small voltage through out the day. True they will likely peek at the same time as your peek electricity. But the amount of power they put out at peek is not going to be much compared to your total consumption. So instead of using the power as it comes through out the day, where in the morning you may save 10kWh for say, 8 cents per kWh, you can instead store that juice in a battery for peek time and save 9kWh (due to loss) and 20 cents per kWh. Yes, it would cost extra for a battery array, which is why I listed it seperately, but with it, you could replace more of your most expencive power with the cheapest, instead of replacing a smaller amount of power through out the day.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
I built a data center power monitoring system about a year and a half ago for exaclty this purpose (I installed it in my house and posted a writeup to slashdot... the article is now here). This system monitors every branch circuit in the data center and allows you to assign circuits to customers so you can track usage by customer. The first data center it was installed in was a colocation facility and their intention was to start billing for power like they do bandwidth. That is, you purchase power in 5amp blocks and when you spike, you pay for another 5amp block, etc... the thought being that if customers start having to pay for power, they will optimize it just like any other expense.
:: While I built the system, I don't own it, sell it, or work for the company that does. Anyone looking for more info should visit TrendPoint Systems.
To this end, the system was designed to let customers login to their accounts and see their power usage (with one minute resolution for a year... the gui is a java applet with real-time graphs, etc..) as well as set alarm thresholds, notifications, etc. The customer I built this product for recently completed a new data board that gathers meter grade current, voltage, watts, power factor, and kwh readings so all this history is now available to the user (and colo). There are independent threshold alarms sets for both customer and admin across all data sets, including panel level, per-phase rollups, etc.. (which amounts to almost 1400 alarm points for an 84 circuit panel).
The folks that use the system have told me it's actually almost more useful for capacity planning and load balancing given the increasing power density of customer cabinets these days.
*disclaimer*
- If your server room is not enclosed on the roof of the room (just using plain false-roof tiles) make sure they are atleast insulated very well. The more A/C escapes, the more it has to work.
- Make sure there is enough air-flow through your server racks (best placements and setup ideals very from person to another), best not to have the rear right up to a wall. Middle of the room or offset (5 feet or so from the well) allows for good ventalation.
- Keep server room lights off unless needed with the exception of a low-heat emergency lighting.
- If you have raised flooring and the a/c comes through the bottom, place the racks behind vent openings (so the air is rising to the front of the rack, getting sucked in by the fans in the front) instead of having the rack on the vent itself.
- Upgrade older servers if possiable. Older servers (expecially the old HP NetServer series) are a lot less efficient as newer servers. Not componet (CPU, HD) but also overall engineering.
- Turn off monitors when not in use. LCDs are not as bad but better to be safe then sorry. If you do not need it running, just leave it off.
- Do not allow people to keep the server room door open, may sound simple but you wouldn't beleive how many times I've seen this. If the doors don't close automatically, get automatic closers for them!
- Make sure the doors are weatherstripped.
- Multiple airconditioners! I have a small server room that runs on three airconditioners. Two always run, one does not, this rotates weekly. Also great for redundancy.
I'm sure there are many more things you can do. Hiring outside consulants who have worked with issues such as this are always benifitial. Be sure to get second/third opionions.
Wow, spelling really sucks when you haven't slept for 72hrs. (I really, really hate Exchange. Expecially when custered.)
I've been saying this for years, any outfit that already has a DC infrastructure should be installing photovoltaics on the roof. In a traditional PV installation, inverters and output wiring are a big part of the expense, but if that work is already done, the payoff period is a lot shorter.
Plus, in the event of a grid failure, your generator doesn't have to work quite as hard, which translates to slightly longer runtimes on the same fuel tank.
The available solar resource depends largely on latitude and weather patterns, though. Do some research and talk with the PR and marketing people about advertising your facility as "greener". If I were in the market for colo services, I'd lean slightly towards an enviro-conscious outfit, especially if they had a clue about reliability.
Fourth, sub-teranian cooling. Once you get a little ways under the surface of the ground, the temperature becomes a pretty consistant mid/high 50's.
Actually, I think you'll find that the deep temperature of the earth is the average between the highs and lows on a yearly basis. In other words, if you live in a hot climate with temps of 120 in summer, 60 in winter, the deep earth temp would be 90. In the frozen arctic, the deep earth temp is below freezing ('permafrost'). Granted, for a lot of the continental US, that means a deep earth temp of around 50. But, remember, putting heat (or cold) into the earth at that depth will cause it's temp to rise (or fall). It is not an infinite heat dump.
CHeck out http://www.axwoodfarm.com/PAHS/UmbrellaHouse.html for more info.
As a data center manager myself, I can understand your pain. Unfortunately, I'm in charge of a corporate data center rather than a pure hosting arrangement; many of the tricks I've used to manage power consumption wouldn't work for you, but...
I'm able to play building load for the laptops/desktops off against data center consumption, and also able to relocate equipment to other sites to juggle the load. I have the option of passing the cost on to the customers because most of what I do is cost-plus contracts. I know this might cost you business, but it's something to consider. Other things that may help:
This may sound silly, but don't leave systems running with a failed component. A lot of servers run the cooling fans at higher RPMs if there's a power supply or fan failure.
If you're not already using SAN storage, consider switching to it. If you are, make full use of it by having your servers treat the SAN as the boot drive and removing all local drives. Better to have 20 servers accessing a 20-disk RAID on the SAN than those same 20 keeping 2-8 hard drives each spinning.
I'm going to assume that you're a high-availability setup, with UPS and generator coverage for a multi-day power outage. One of the simplest things you could do is set up a system where, if your power draw approaches your previous peak (or acceptable peak if there is a limiting factor), you switch to generator power. Whether this really helps you would depend on how far you need to limit your use, and how much refueling the generator costs you.
The other option may not be workable for you, but... if there's other office space in whatever building you're in, I'd look into renting it as a separate office of your company, and set up a second data center (shared staff, but separate electric service.) Dividing your current electrical load between two bills even 75%/25% would be a great way to limit peak load. This may even work with your current space if you occupy multiple suites in the building.
Pavlov's Dog ate the bell, and now he's barking at Schroedinger's cat all the time... -Me
Unless in specific circumstances, it's rather doubtful that solar panels will actaully lower your overall bill.
Solar power costs something like 18-22 cents/kwh once you amortize the cost of the panels over it's entire lifetime, etc. Commercial power is generally less than this, maxing out around 17cents/kwh in the pacific northwest. In the midwest commercial power costs like 7 cents/kwh.
Solar power is currently [i]extremely[/i] expensive compared to other energy sources. It's main penetration currently is in the residential sector as water heaters, because a) this doesn't require a lot of power, thus not a lot of space (ie on a roof), b) residential electricty rates are much higher than commercial, thus making it almost viable, and c) large tax breaks/credits makes it viable. You have none of these advantages when talking about a commercial data center (though depending on where you live, you *may* have a small tax credit).
I find it extremely doubtful that solar will be cost-effective.
Have all IT people work from home. :-)
No office space to cool or heat. No coffee machine or water cooler. No overhead. Just house the machines and an small maintance staff.
This has been another valuable and informative opinion from:
Catahoula!
No, I'm not the experts but I refer you to the: Rocky Mountains Insititute . They are a not for profit environmental think tank who work with corporations and governments (Ford, the US Military for example) to increase profits or reduce costs through more efficient environmental practices. They ran a Design Charrette around this specific question. This is where they take their staff members with general energy efficiency expertise and a whole bunch of industry types (data centre types, power company types etc. and worked at redesigning the entire data centre idea from scratch with energy efficiency in mind. There is a detailed report including return on investment figures and detailed financial breakdowns. The information available is extremely comprehensive and free (as in beer). These guys are excellent and slashdotters might also like to look at similar exercises they have done with cars and energy security.
- Just trying to survive until the nanobots make me immortal -