Slashdot Mirror
Building an Energy Efficient Datacenter?
asc4 asks: "The company I work for is a webhosting and colocation company. As our power utilization grows, we have begun searching for ways to make our datacenter more efficient. The biggest hit from the utility company comes in the peak usage charge, which penalizes (rather severely) for the highest sustained burst of usage during a billing period. Due to the nature of the colocation business, we can't control how much or when client devices use power, so I'm wondering: is there's something we can do at the datacenter level to help smooth out our power consumption, over the course of a given period of time?"
"In these days of hybrid cars, Energy Star devices, and in general more eco-friendly power consumption, it seems like there must be some products out there that can help make datacenters more efficient, as well. Could fuel cell technology be something to look into? Would flywheels or capacitors help? How about using more efficient AC units than what are available from the big names? What are others doing to reduce peak power consumption in high-drain datacenter environments?"
It's not necessarily cutting power consumption, but will reduce monthly bills and is eco-friendly. I'm thinking like solar or wind assist (depending on your geographical location)
During off-peak time, pump water uphill to a holding reservoir - a big swimming pool on the roof might do.
:)
Heat the water with the waste heat from the cooling units.
Sell access for swimming - nice warm water (well, here in Canada we like it warm
During peak hours, drain the pool back via generators to make electricity. (make sure you tell people first)
Use warm water to cool more - generate steam.
Run steam through turbines to generate electricity.
Use electricity to pump more water to pool on roof
continue as needed
Been there, done that, paid for the T-shirt
and didn't get it
If you can, install solar panels on your roof. It will smooth the peak a little, and also reduce your overall expenditure. If you are in a sunny location, the investment can often be recouped after only a couple years. Most utilities will even subsidize such ventures.
If that's not an option, server consolidation and virtualization for the people whom it is appropriate for are the only other options I can come up with...
I am going to guess you have 3Phase power perhaps through more than one primary link. Do they charge
based on the peak of one phase or the average of all. If you aren't balanced on your phase input into your building, you may be able to rebalance and see some benefit there. If you have one or two large UPS systems that are pulling equally across all three phases, make sure that the output of the UPS system is also balanced, that could end up bringing your input usage down.
This of course wouldn't help with your peak usage, but something to consider anyways.
Short of that, you would be looking for something that could store power and charge that at a regular rate. But then you could end up possibly shorting your demand on the output side based on the available power in that 'system' at peak times.
I am going to guess your best bet is to look at phase and load balancing through your power distribution network and make sure you have placed your clients. If I was in a similar situation, I would set up a collection of load coils across each hot lead in your power distribution network and graph the values on a tight schedule (in order to catch peaks) and determine what is responsible for your peaks.
Don't know if any of this would help, but it is discussion, mod accordingly.
In many states, you can save substantial amounts of money by agreeing to scale back energy utilization during critical times. In New York, NYSERDA (www.nyserda.org) is the agency that administers the peak load reduction program.
Conformity is the jailer of freedom and enemy of growth. -JFK
I used to work for a company where I was in charge of building automation and peak demand limiting. We used several strategies for this. 1. Use thermal storage where possible. The only real source you can control is the cooling/heating for the building, and you want to build uip as much of what you need during low periods of usage, like in the middle of the night. If you're in a cold climate, store heat, and if you're in a cold climate store cold. Use water large water tanks to achieve this. It will cost you to install them initially, but they will pay for themselves in a surprisingly short period of time. 2. Monitor the usage and trim where you can when you're hitting peak demand. Turn off lights, coooling units, etc., for the short time that it's required. Pre chill or heat the building ahead of time. 3. Run your backup generator to supplement existing power if you have seasons where usage is much greater than at other times of the year. If you have to run it every day of the year it won't help due to maintenance and fuel costs. But if you need it for short periods to chop the peak then it's well worth it. Again, it will more than pay for itself. The power company may even pay you to supplement them with it. 4. Look for alternative methods to heat or cool, or even generate power. You'd be surprised at what's available now for that.
If you're right on the edge of getting nailed for peak load, you could run the aircon aggressively before the peak load period and try to coast through it with the unit off. Chill the place to 60F, shutdown the aircon a few minutes before peak load and see how long you can go before turning it back on. Economizers can work well at reducing your aircon load. We pull in cold outside air at 5AM and cool the building down to 65f. This saves us about 2 hours of aircon running during summer days.
"Eve of Destruction", it's not just for old hippies anymore...
For smoothing out power usage, there are a number of different options -- aside from alternative energy, you could do rolling brownouts throughout your datacenter and rely on UPSes or generators to keep things going -- but you *will* take a hit in reliability. Every switchover -- one mains circuit to another, mains to battery, etc. -- carries some risk.
I've watched an entire datacenter go out on what was supposed to be a controlled switchover -- power company needed to do some work, pulled the plug (with the datacenter's consent), the backup generators start... and then die. The UPSes kicked in, but could only supply 15-20 minutes of power. Everything failed over to a backup datacenter, whose link then decided to go out to lunch.
Total cost of the outage was measured in tens of millions of dollars.
Just keep this in mind when doing the business justification calculation (cost savings from lower energy bills, minus upfront cost of equipment, minus risk of additional downtime times cost of downtime, minus cost of maintaining the equipment). Unless energy prices go *way* up -- like oil hitting $250/barrel -- I'd be surprised if this would pay for itself.
Your utility might charge you based on KVAR hours or apparent power. If you have a bunch of computers and UPS then your power factor may be bad.
Make sure there's no charge there for KVAR hours instead of Kilowatt hours, or no surcharge for power factor. If there's on there it would benefit you to get a consultant in to install PFC correction.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
"Could fuel cell technology be something to look into?"
No. Fuel cells are a way of transporting energy, not creating it. This is such an important concept to grasp that cannot be understated.
We are in deep trouble, energy wise. There is no immediate solution (within the next 30 years) that can help us. We need to get used to that concept, fast. Doing "your bit" for the environment is simply not enough.
Welcome, too, China and Inda. Welcome to the powerdown.
Invoicing, Time Tracking, Reporting
I take it you have quite a server farm.
Intel sells a lot of crap, so take some of it and use a methane generator to produce power.
I have one of these (1.2GHz) and with 1 large HDD, encoder card, network, DVD etc - it idles at less than 20W and maxes at about 60 (encoding, playback, DVD all going, CPU 100%). Burst power when switched on seems to be about 72. This is less than the processor alone on a high spec box.
This will only work with non-CPU intensive operations. However IO seems to be pretty good on these boxes, so an IO bound server would probably not suffer too greatly using a VIA mobo.
The trick is to use the heat from computers to drive a turbine that generates power. Then use that power to run the air conditioners.
You may want to look into AMD based systems instead of Intel. We have reduced our power load considerably and gotten a boost in performance by using HP DL385 2 way servers with Dual-Core processors rather than Dell 4 way Intel servers. Don't underestimate how much this can impact power utilization when you have 100 servers.
I smell a Nobel Prize in Physics here.
Some things are easier to do in the design phase. but something can be done now.
/. earlier this week, keeping the building cooler in the morning and warmer in the afternoon can drop your peek time costs.
First, pre-cool the room. There was a good article on
Second, install a solar power system. Kinda pricy, but if you have a large roof you can generate some solid power. And don't think that being in the north excludes you from solar power. Uni-Solar has a great sun index map showing what level of solar output and electrical output you can expect in any given area.
Third, going with solar, a battery array or some other type of power storage. By using the solar pannels to juice up the batteries, you can pull power from the batteries at peek time, but charge them all day.
Fourth, sub-teranian cooling. Once you get a little ways under the surface of the ground, the temperature becomes a pretty consistant mid/high 50's. Using sunken water tanks you can run 60 degree water through a radiator in your HVAC system. I know there are companies that can install these system but I can't recall any names off the top of my head.
Fith, solid state storage. If you can swing paying $50/gig as opposed to $1/gig for storage space you can dramaticly cut down on your both your cooling bill and your electric bill.But at $50,000 per ter vs $1,000 per ter, it's going to take a while to recoup the costs.
Sixth, custom server cases/cabinets. Traditional closets are great for cramming a lot of servers into a small area, but they about suck for heat management. You could fund a research project at any number of engineering schools to create a better storage solution.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
1. Cool down the centre during the night when hydro is at its cheapest.
2. During the day raise the thermostat so the AC does not kick in too soon.
3. If you have windows use the blinds on the sunny side. Thermal load is a royal pain. Where I work it hit 27c inside even though it was -14c outside. The north side was running at about 21c.
4. Put all non-essential equipment on powerbars and turn off the bars. Most monitors and other electronics still draw a bit of current for 'instant on'. That takes hydro and dumps more heat for the AC to handle.
Panic now, beat the rush!
Do you folks charge per amp as well? More and more places are doing this, and while customers may not like it, the reality is that all boxen are not created equal, and you may want to take a look at measurement systems for passing those peak costs back to your peak use custys.
CPUs with six or eight cores, with four threads per core. Sun says their new CoolThreads Servers offer significant power, cooling, and space savings.
I believe the servers are too new for anyone to have a solid opinion about, but I know Sun has been actively moving in this direction for a while.
Even as efficient as power supplies have gotten, you'll still save a lot by doing the AC to DC conversion once, rather than repeatedly. Get a good power distribution system (don't assume the average UPS can deal with a DC load) and hire an engineer who understands how to use DC power (they're not hard to come by, while it's somewhat new to the computer field, telcos have been doing this for years and years). You gain less energy loss in the conversion, plus less cooling required to deal with all that loss.
The only drawback is in convincing your customers to use appropriate systems for DC power input. You'll also need to work creatively to deal with all the little trinkets like firewalls and switches that run on odd voltages.
Plus as others have indicated, there's more to the datacenter than just the computers, like the rest of the building: lights, blowers, AC, security gear, etc. I'd suggest flourescent lighting keyed to your security system (lights are on when someone's access card is in the room). LED lighting could be put in on DC power, but these seem to not be as efficient per brightness as flourescents can be.
Whoa... what? If your peak isn't the same as solar peak, what's the point of this? It just sounds like much more expensive electricity to me.
"I assumed blithely that there were no elves out there in the darkness"
Have you thought about installing wind turbines on the roof of your building to generate electricity which you can then feed back into the grid?
... that's savings on your electric bill. Your ROI could be substantial over time.
You will probably never be able to generate enough power to completely power your data center, but even if you generate 1%, 5%, 10%
Buildings in Chicago are strongly thinking about this (for obvious reasons).
Other areas could probably benefit too.
Get a professional electrician in that knows about peak charges.
Older installations used to use giant flywheels, but not to limit peaks. They were used for power conditioning and limited power backup.
I'd do an extensive survey before trying anything else. Buy or rent a power meter that does logging and graphing. Check everything out for a month - each phase and the current draw on each phase, and current draw on each rack (each computer if possible).
Proper sequencing of cooling can drastically affect your power consumption. Never start your cooling motors when you're drawing a lot of power - motor startup is a huge peak. After doing a survey of your power needs you may be able to identify times when you can avoid turning the cooling system on which will lower your peak. For instance, before the daily peak, cool the data center down a few degrees more than usual. Then shut off one or more cooling system until after the daily peak. This can be tricky to correctly manage and implement, especially since it has to be automatic and failsafe.
Alternately, shop around for your power. check with a few competitive companies and see if they offer a better deal.
-Adam
Being a datacenter you would undoubtedly have a generator backup to your UPS solution.
Would it be cheaper/feasable during these peak times to "test" the generator... ie turn the mains power off and run on diesel?
A Tale of 2 idle hands
Check this out for relaxation since you guys are so angry it seems Check this out
If you've got a datacenter large enough that energy efficiency is a problem, I recommend you move the whole shebang to a location where energy is more plentiful. Upstate NY, which has plenty of hydroelectric power, would be a good choice. Nowadays, thanks to the internet, you don't have to keep your datacenter next to part of your operation.
Solar panels kick out small voltage through out the day. True they will likely peek at the same time as your peek electricity. But the amount of power they put out at peek is not going to be much compared to your total consumption. So instead of using the power as it comes through out the day, where in the morning you may save 10kWh for say, 8 cents per kWh, you can instead store that juice in a battery for peek time and save 9kWh (due to loss) and 20 cents per kWh. Yes, it would cost extra for a battery array, which is why I listed it seperately, but with it, you could replace more of your most expencive power with the cheapest, instead of replacing a smaller amount of power through out the day.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
I smell something alright ;)
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
Seriously. Try killing the flourescents and not allowing "lighted" maintenance during certain peak times.
On the other hand, that might be a dumb idea.
Don't think that a small group of dedicated individuals can't change the world. It's the only thing that ever has.
Switch to natural gas to run the air conditioners. Your peak electricity hit is in the middle of the day when the air conditioners work hardest, but the peak natural gas hit is in the middle of the night when the exterior temperature is coldest. Price wise that works to your advantage.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
You could use a giant capacitor to store electricity for a while. It produces some waste, but by electronics professor says that it is efficient enough that power companies use them to help ballence load.
Simon's Rock College
Some pics from google cache - limited time only.
You are being MICROattacked, from various angles, in a SOFT manner.
"is there's something we can do at the datacenter level"
Yeah use the Ultrasparc T1 CPUs, use lower power scsi disks including compactflash disks for boot and OS, keep all lights out when you dont need em, add heavy wall insulation unless youre living far north, add lots of ram in all machines so the disks can be powered down etc.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
youre funny.
If the machine is doing 100% cpu utilization, just replace it with a weaker CPU.
Maybe he can save more energy by running hundereds of vmware virtual machines on an Geode GX.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
Most med-large datacenters have a bank of batteries either in an UPS or as a DC power source that they just charge all the time. If you can use an alternate source to charge them, then you just saved some on your bill and if the peak is at the time when the solar cells are at it's peak, then that's a load off you peak charge.
I think he kinda meant that if they're not at 100%, they could replace them with lower usage processors.
What goes around, comes around.
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
I built a data center power monitoring system about a year and a half ago for exaclty this purpose (I installed it in my house and posted a writeup to slashdot... the article is now here). This system monitors every branch circuit in the data center and allows you to assign circuits to customers so you can track usage by customer. The first data center it was installed in was a colocation facility and their intention was to start billing for power like they do bandwidth. That is, you purchase power in 5amp blocks and when you spike, you pay for another 5amp block, etc... the thought being that if customers start having to pay for power, they will optimize it just like any other expense.
:: While I built the system, I don't own it, sell it, or work for the company that does. Anyone looking for more info should visit TrendPoint Systems.
To this end, the system was designed to let customers login to their accounts and see their power usage (with one minute resolution for a year... the gui is a java applet with real-time graphs, etc..) as well as set alarm thresholds, notifications, etc. The customer I built this product for recently completed a new data board that gathers meter grade current, voltage, watts, power factor, and kwh readings so all this history is now available to the user (and colo). There are independent threshold alarms sets for both customer and admin across all data sets, including panel level, per-phase rollups, etc.. (which amounts to almost 1400 alarm points for an 84 circuit panel).
The folks that use the system have told me it's actually almost more useful for capacity planning and load balancing given the increasing power density of customer cabinets these days.
*disclaimer*
Well, you could also have an automated cutoff for nonessential load (like 3 of 4 fluorescent lights or something). Or, you could use a battery UPS instead. But the flywheel is cooler...
random link
- If your server room is not enclosed on the roof of the room (just using plain false-roof tiles) make sure they are atleast insulated very well. The more A/C escapes, the more it has to work.
- Make sure there is enough air-flow through your server racks (best placements and setup ideals very from person to another), best not to have the rear right up to a wall. Middle of the room or offset (5 feet or so from the well) allows for good ventalation.
- Keep server room lights off unless needed with the exception of a low-heat emergency lighting.
- If you have raised flooring and the a/c comes through the bottom, place the racks behind vent openings (so the air is rising to the front of the rack, getting sucked in by the fans in the front) instead of having the rack on the vent itself.
- Upgrade older servers if possiable. Older servers (expecially the old HP NetServer series) are a lot less efficient as newer servers. Not componet (CPU, HD) but also overall engineering.
- Turn off monitors when not in use. LCDs are not as bad but better to be safe then sorry. If you do not need it running, just leave it off.
- Do not allow people to keep the server room door open, may sound simple but you wouldn't beleive how many times I've seen this. If the doors don't close automatically, get automatic closers for them!
- Make sure the doors are weatherstripped.
- Multiple airconditioners! I have a small server room that runs on three airconditioners. Two always run, one does not, this rotates weekly. Also great for redundancy.
I'm sure there are many more things you can do. Hiring outside consulants who have worked with issues such as this are always benifitial. Be sure to get second/third opionions.
Wow, spelling really sucks when you haven't slept for 72hrs. (I really, really hate Exchange. Expecially when custered.)
I've been saying this for years, any outfit that already has a DC infrastructure should be installing photovoltaics on the roof. In a traditional PV installation, inverters and output wiring are a big part of the expense, but if that work is already done, the payoff period is a lot shorter.
Plus, in the event of a grid failure, your generator doesn't have to work quite as hard, which translates to slightly longer runtimes on the same fuel tank.
The available solar resource depends largely on latitude and weather patterns, though. Do some research and talk with the PR and marketing people about advertising your facility as "greener". If I were in the market for colo services, I'd lean slightly towards an enviro-conscious outfit, especially if they had a clue about reliability.
Disks (and other mechanical parts) will consume a lot of energy, but you don't need to replicate every single physical disk - if the data is under two gigabytes, RAM disks should be fine. In the event of a hard drive failure, backing up off RAM disk is no different from backing up from physical disk, so what's the difference? A single SAN-based disk pack, copied into RAM on the servers, would be the least power-consuming design - especially if you powered the hard drive off except when syncing up.
It costs power to task swap, so the more active tasks there are, the more swapping (if the tasks are all being given fair time) and therefore the more CPU time is taken by kernel activity, therefore the more power is being used up on housekeeping. You should be able to reduce the power consumed by heavy kernel activity by load-balancing.
If you're going to load-balance, you don't need high-power server-rated or desktop-rated CPUs. Mobile CPUs will take less power, you'd just need a larger cluster to load-balance over. If using Linux, also look at CPUs other than Intel - many MIPS and MIPS64 implementations are pretty low-power.
Networks take power to run. There's no escaping that. Don't run more wire/fibre than you have to (that also includes not running longer cables than you need), and don't use more intermediate network devices than will get the job done properly. Oh, and don't overspec the network for a given technology. CAT6 is good stuff, but if your machines never exceed 10 mb/s on the network, you're going to lose efficiency. The "for a given technology" matters, as different technologies will consume different amounts of power for a given spec. Shop around.
Cooling systems are another mechanical system and so are necessarily power-hungry. You can't put those in RAM, however. Again, shop around. You want the best cooling power per unit of energy. This may turn out, for your system to involve having several fans on a single component. It might equally well work out that you can link ducting together such that a single fan can directly cool many components. Since the energy efficiency is what is important, go for the most energy efficient solution for your system.
Depenmding on the system, it MAY (this is not guaranteed) improve the efficiency to have a variable-speed fan, with the speed controllable by the CPU, and where all components cooled by this system have thermal sensors readable by the CPU. You can then vary the cooling as a function of both temperature and predicted load levels. (Varying according to temperature alone is useless, as the loads on the components will change faster than the sensor readings - but could change in either direction. Since the OS knows what tasks it is currently doing, it should be capable of predicting the likely loads for a much more reasonable timebase.)
Connectors are notorious for high resistance and therefore power loss. If there is something that you're unlikely to change for the productive lifetime of the computer, all power loss through all unnecessary connectors (whih are generally made from poor conductors anyway, just adding to the problem) is power you can conserve simply by improving the connection. If you insist on using connectors, make sure the wires that go to the connectors are soldered and not just held in place by pressure. Also, clean the connectors thoroughly, as buildups of oxide and dirt will increase the resistance. You WILL be better off by removing the connectors entirely and soldering anything that's not going to change in place.
Finally, the data center's power grid. You want very high voltage, very low current. (Power dissipation is proportional to voltage, but proportional to the square of the current.) The industrial powe
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Slow low power processors with multiple cores handling multiple processes:
n dex.jsp
http://www.sun.com/servers/coolthreads/overview/i
There's just one thing wrong with the "dependent on the Middle East" scenario. Everyone forget's Russia, and Canada. Both have large oil deposits. Especially the Canadian's and their oil sands. True that makes us dependent on Canada, but I'd rather be dependent on them, than the Middle East. Second, I'm not certain the oil situation is everything people is saying it is. Especially the oil companies, with their decommishining of oil refineries, while they claim we don't have enough capacity. Plus the sending of oil (and oil products) to foreign markets, while claiming domestic shortages.
The other side of the lake has more wind.
They have to be engineered for power efficieny to extend their battery lives, and they have a built-in ups/load balancer. Of course you's have to engineer your own DC system instead of those wall warts, but...
Someone had to do it.
As a data center manager myself, I can understand your pain. Unfortunately, I'm in charge of a corporate data center rather than a pure hosting arrangement; many of the tricks I've used to manage power consumption wouldn't work for you, but...
I'm able to play building load for the laptops/desktops off against data center consumption, and also able to relocate equipment to other sites to juggle the load. I have the option of passing the cost on to the customers because most of what I do is cost-plus contracts. I know this might cost you business, but it's something to consider. Other things that may help:
This may sound silly, but don't leave systems running with a failed component. A lot of servers run the cooling fans at higher RPMs if there's a power supply or fan failure.
If you're not already using SAN storage, consider switching to it. If you are, make full use of it by having your servers treat the SAN as the boot drive and removing all local drives. Better to have 20 servers accessing a 20-disk RAID on the SAN than those same 20 keeping 2-8 hard drives each spinning.
I'm going to assume that you're a high-availability setup, with UPS and generator coverage for a multi-day power outage. One of the simplest things you could do is set up a system where, if your power draw approaches your previous peak (or acceptable peak if there is a limiting factor), you switch to generator power. Whether this really helps you would depend on how far you need to limit your use, and how much refueling the generator costs you.
The other option may not be workable for you, but... if there's other office space in whatever building you're in, I'd look into renting it as a separate office of your company, and set up a second data center (shared staff, but separate electric service.) Dividing your current electrical load between two bills even 75%/25% would be a great way to limit peak load. This may even work with your current space if you occupy multiple suites in the building.
Pavlov's Dog ate the bell, and now he's barking at Schroedinger's cat all the time... -Me
You clearly aren't responsible for any significant server group. You *never* want to have your servers running at 100% for more than the occasional peak. 20 - 30% average utilization leaves you space to handle demand peaks, and many people will tell you I'm not being conservative enough here.
Additionally, nobody makes "enterprise-grade" servers based on Via parts.
That being said, using Opteron processors will save you some significant juice over Intel. HP has some nice models (DL385 and DL585).
What I meant was "How many of them are actually running at 100% CPU? Those that aren't can be replaced...".
If you re-read my comment, lower down you will see:
...and don't forget Sun, if you feel like paying $20,000 for an 8-core Ultrasparc T1 chip that uses less power than an Opteron. Windows not allowed though. But since the original question is probably talking about managing a datacenter full of customer machines (which you can't control, unless you lease them), I would imagine that just changing the way you do cooling would make a rather large difference. Have you considered piping heat from the racks to the outside through the roof, and using a high specific-heat fluid (say water/ethylene glycol mix) that could be prechilled to an extremely low temperature during off-peak times, then used in a radiator-type arrangement for cooling. I would say that sealing the sides of your rack and replacing the front and rear doors with liquid-cooled radiators/heat collectors should work quite well. If you can pre-chill the air entering the front of the machines, it will be cooler coming out, and if you absorb the excess heat coming out the back, less heat will enter the room, meaning your air conditioners won't have to work as hard. Other good option would be to duct the exhaust heat (if it is significantly above room temp) up and out of the building with a few smaller fans.
--That's the point of being root, you can do anything you want, even if it's stupid.
Make two rows of racks face each other. Place a roof on the lane between the racks and doors on either end. This is the cool lane. On the opposite sides of the racks, place no roofs. This is the warm side.
Only let cool air enter the room in the cold lanes and suck the hot air from the warm lanes. Use racks with perforated doors and use shields to completely cover unused space in the racks.
Now the cold air from the enclosed cool lane can only flow one way out: through the computers.
This will dramatically reduce the power needed for airconditioning.
This is your sig. There are thousands more, but this one is yours.
Not only are the fuels cheaper, kWh-for-kWh, than mains electricity, but you get to use the waste heat from the generator to heat the building at the same time. Doing both at once gives you huge savings.
Typically people tend to use I/C engines for the generators --- gas turbines would be more efficient, but I/C engines are cheap and reliable and will scale down far more effectively for gas turbines. A car engines would do fine. (Remember that one horsepower is .7 kW; even a small car engine has a huge power rating.)
Plus you get to sell power back to the grid if you're not using it.
Ok some simple things like running gycol AC that do not have to run a compressor when it's cold out (circulation pumps and fans are a lot cheaper)
:) air is cooler and does nt need significant humidity adjustments. You will go through a lot more air filters but it's cheaper. Depending on the building the basement is actualy a pretty massive heat sink to the ground this works great if it's mostly open bulk storage etc. It also has the added advantage of keeping the basement air circulating.
Ducting the return air to the outside when the outside (or basement
Get power monitoring per circut to fild your bigest issues so you put time and effort into the right places. If your mostly an internet facing colo investigate any trends the differ from your outbound MRTG. For example I once found that the data backup systems were causing a lot of power usage during the mid day peek and could be shifted.
15k SCSI is a power hog, if you need IO's look at solid state anything else can run on slower drives with good savings. As an added bonus anything that needs the IO's like DB's will scream on solid state.
Insulate interior walls aroun the DC th ceiling above etc it will cut down on noise and heat tranfer from your normaly much warmer spaces around. Ancilary rooms like power block rooms can be ducted to outside air all the transformers panels etc are designed to be outside and can generate a lot of heat load, UPS's not as much they need to avoid getting to hot but batterys dont like being cold. You would be surprised it was taking 10kw of cooling to take care of the power room at one DC I was at.
Check power in front of and after your UPS some models can cost you 20% or more due to there design. This is a trade off in switchover time though so it is something to be looked at vs your SLA's and desired public image. A lot of real server class gear does nto mind increased switchover time but cheap PC PSU's are not as tollerant (it's all about the Cap's in the PSU)
Fuel cells can be a good alteritive or primary power soruce depending on pricing of natural gas vs the power grid. They have to be one of the most reliable sourced of power generaly because the gas companies bury most of there infrastructure. On site backup can be via propane.
No sir I dont like it.
Have all IT people work from home. :-)
No office space to cool or heat. No coffee machine or water cooler. No overhead. Just house the machines and an small maintance staff.
This has been another valuable and informative opinion from:
Catahoula!
How about giving colo-customers certain rebates if they choose more
power-efficiant servers and CPUs like AMD over Intel, PowerPC over
x86, etc? You could get an el-cheapo power-meter from any electronics-
store and test all servers. Doesn't matter if the meter shows a few
watts wrong if you use the same meter for every new server.
0-50 Watts -
51-100 Watts -
101-200 Watts -
etc.
Can you honestly suggest that a shorter Ethernet link will consume less power than a longer one? Sure, there's a tiny difference in cable resistance. But the transmitting end is putting the same amount of energy into it either way, since it doesn't know the difference. Any that doesn't turn into heat in the cable will turn into heat in the receiving chipset. Hence, the same power draw.
You *may* have an argument on very long fiber links. If you can get away with a short-reach transceivers instead of long-haul, you might save a watt or two. But again, if it's the same equipment at either end, the cable length doesn't matter, because any energy that doesn't succumb to cable attenuation just gets dissipated in the receiver. Or in some cases, in an optical attenuator pad just before the receiver, used to weaken a very strong signal so it doesn't destroy a sensitive receiver.
In both cases, the energy actually transmitted through the cable is so infitesimal, resistive losses are negligible and they don't matter anyway, because the energy that makes it through isn't used by the receiver, it's interpreted, then dissipated.
Tell me again how using CAT6 cable on a 10Mb/s link is inefficient? Financially yes, the cable's expensive, but electrically, the signal doesn't care. Neither cable is going to heat up more than the other one.
That being said,
Power cables are another matter entirely! Since they carry lots of current, resistive losses, even in adequate power cables, are measurable and significant. Particularly in 48v DC environments, as compared to 120v or 240v AC systems. To offset this, the power conductors in DC distribution plants are usually appallingly thick. Still, with a thermal infrared viewer, you can find "warm spots" in your power system. Fuses and breakers will always be a little warmer than the cables that feed them, out of necessity. But your power cables should be as short and thick as possible, within reason.
I say within reason because you're still only looking at a few dozen watts throughout the entire datacenter being lost to resistance in power cables. Rewiring the mess to use shorter cables will cost you more in labor and downtime than it's worth, but designing it right from the ground up might be worthwhile. In the meantime, just turn off the lights when you're not using them.
The rule to follow would be "if it produces measurable heat, it's something to look at". Network cables don't. Power cables, power supplies, processors, chipsets, drives, and memory do.
If you're large enough, I'm going to assume that you have a decently sized backup diesel generator. Say one about 500kw. Talk to the power company about running it during peak hours and selling the electricity to them. I know one TV station that does this with their backup generator. When load gets high enough on the grid, they call up the transmitter site and they turn on the generator. The station gets paid enough from the power company to pay for maintainence and fuel for the generator.
As for more energy efficiency. I hope you don't bother with having a heating system in the building. The datacenter should put out enough heat to heat a large portion if not all of the rest of the building.
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
They are nice, but they have their limitations. On the positive side:
- Very low power consumption.
- Very high IO speed. In fact considerably higher than expected. I have been getting 2+ times faster IO than from an Intel Xeon from them.
- AES acceleration on the higher end models, high quality hardware RNG and RSA acceleration on the models coming up this year
On the negative side:- Very small cache. Much smaller than anything else on the market. As a result it is blazing fast to start and blazing fast on any small data set, but churning large data slows it down considerably. This also causes task switching under higher loads to be considerably slower than expected.
- The default factory fans on the higher end models are not temperature controlled. They have an MTBF of around a year and a half which is too low by modern Datacenter standards. On the positive side, the thermal throttle on them is good enough to allow them to run with the fan failed.
Overall they are good enough to run a mail relay, a DNS server a firewall, a QoS conditioner or an Odds-N-Sods. They are definitely slower then I would like to for a news server. CGI performance and MySQL are abissmal and computation or compilation are in the "you might as well forget it" area.The most painful part is actually the relatively low number of people providing good server class enclosures for them. This http://www.icp-epia.co.uk/ is one of the few sites. Even they do not have everything so you end up assembling most of them yourself which costs time and money.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
seriously
Biggest bang-for-the-buck: remove all your Intel Xeon and P4 machines and replace them with AMD Opteron dualcores, preferably the HE (High Efficiency 55W max) series. Each core will do more work than a Xeon and burn a fraction of the power doing it. Sun's new multicore CPU is interesting if you don't need x86 compatibility.
Power supply efficiency is important too. I switched to Seasonic high-efficiency power supplies for my desktops years ago. I'm not sure what you'd do about rackmount servers. There's been mention of making servers that plug directly into DC power so one power supply can feed several machines but I haven't followed that closely. Besides lowering power consumption you're also generating less waste heat, reducing the load on your A/C.
If you pipe all your heat from the racks, make sure to look for stirling engine.
Converts heat to electricity, which you can then re-use or resell.
In addition huge Vapo/chill could also be used, whith your heat exhaust as a cooling source....
You now have the problem of keeping your exhaust as hot as possible...which is much easier done than keeping it cool at all time.
Best of Luck.
It takes 40+ muscles to frown, but only four to extend your arm and bitchslap the motherfucker
Yes, it would cost extra for a battery array, which is why I listed it seperately, but with it, you could replace more of your most expencive power with the cheapest, instead of replacing a smaller amount of power through out the day.
Forget the battery array, I want to know what kind of monster inverter you are going to need to run an entire data center full of equipment.
Maybe a solution is to relocate your data center to a place where your peak usage will be at the middle of the night local time. This could potentially add up to a significant savings, unless the center is currently in the Midwestern US, then you'd have to move off-continent. But switching coasts seems like a plausible money-saver.
/. community is offering some trendy and cutting-edge ideas to solve this problem, but none of those ideas are practical unless your building is owned by your firm and is located in rural Vermont. I mean seriously guys, do you have any idea how unpopular solar and wind power structures are in the center of any US city?
/. is dropping the ball on this one.
Also, it appears the
Some of the ideas suggested would require massive structural modifications, which is almost certainly not the solution the seeker is looking for. The seeker appears to be looking for off-the-shelf, easy to implement methods, strategies, and hardware, and
Heat source managementA /id.300/.f efficiency %95A /id.417/.f efficiency %97
e _right_power/page3.html and http://www.tomshardware.com/2005/07/11/how_thg_tes ts_power_supplies/page3.html for more info.
Do not put large transformers in HVAC space.
Encourage the use of dc power supplies. (12 24 48 72)
http://www.mini-box.com/s.nl/sc.8/category.13/it.
http://www.mini-box.com/s.nl/sc.8/category.13/it.
this is only %70 at full load http://www.powerstream.com/DC-PC-48V.htm other units can be %30-%75 see http://www.tomshardware.com/2004/01/22/getting_th
Cooling
Take a look at heat pumps for local heat dumps.
Do not run your ac on UPS
Peak shaving
Run your generators during peak loads
Use the heat from your gen set to cool the datacenter (continuous-cycle absorption cooling) http://www.nh3tech.org/absorption.html
Use solar / wind to recharge the battery bank.
smart non data center power use
turn off elevators/automatic doors during peak usage
use efficient low level/low power (led) lighting.
Have your users pay a heat tax per 100wt
Look not only at the cost upfront but the total cost over the life of the data center. There are a whole bunch more, but it depends on your needs/design/issues.
How about something a bit more realistic. Pentium M based blade servers. Soon you will be able to get Core Duo based blade servers which should pack enough density for anyones tastes. The thing are designed for mobile applications but they beat the snot out of the P4 in most server tasks.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
No, I'm not the experts but I refer you to the: Rocky Mountains Insititute . They are a not for profit environmental think tank who work with corporations and governments (Ford, the US Military for example) to increase profits or reduce costs through more efficient environmental practices. They ran a Design Charrette around this specific question. This is where they take their staff members with general energy efficiency expertise and a whole bunch of industry types (data centre types, power company types etc. and worked at redesigning the entire data centre idea from scratch with energy efficiency in mind. There is a detailed report including return on investment figures and detailed financial breakdowns. The information available is extremely comprehensive and free (as in beer). These guys are excellent and slashdotters might also like to look at similar exercises they have done with cars and energy security.
- Just trying to survive until the nanobots make me immortal -
Dealing with number cruncher clusters (where Niagara is almost no help at all unless you recode everything in fixed point); I don't find the "North to the Future" datacenter idea as silly as some. In fact, since my workplace has underice instrumentation near the pole maybe it would even be a win :-).
http://sun.r.delivery.net/r?1.1.3J1.2U2.11FNWo.Bx3 E_i...CoF6.1PF8.37719z
1) Most new servers will be damaged if you attempt to run them without their covers on (warnings are usually written inside the case)
2) If a server runs cooler with it's cover off than when it does with it on.... THROW IT OUT, GET A NEW ONE. That indicates shit-poor design and I'd replace it if you value the data stored therein, let alone the power savings going with a newer, well engineered system.
However I do second the box fan notion. It's lowtech but it can be the perfect solution for a less than perfect server room layout or AC design (often this is out of your hands...)
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
None. Almost everything in a datacenter runs on DC power.
"I assumed blithely that there were no elves out there in the darkness"
If your A/C is centralised, you might consider using a ground-source heat pump to take part of the load. It would require some ground area for the pipework, which can be laid either horizontally or vertically.
The general principle is a heat exchanger with a compressor, much like an ordinary air-sink A/C system, except that it uses the ground which has a much larger specific heat capacity; this makes it considerably more efficient. I'm not sure whether this technology is directly applicable to data centres, as it's usually used with liquid systems in underfloor heating and cooling.
The cost for a GSHP capable of providing all the heating requirements for a UK home is currently in the 5-10 kGBP range.
Alternatively, there are energy storage systems which might allow you to transfer some (probably not all) of your peak load to off-peak hours. Economic constraints probably prevent you from using fuel cells, but deep-cycle lead-acid batteries may be worth investigating, along with compressed air.
None. Almost everything in a datacenter runs on DC power.
Sure it does, and I suppose all those plugs going into 120V AC outlets are just for show? Or are you suggesting he try to distribute DC directly to his various server components directly?
It's not such a dumb idea. Most things are probably on a UPS. With some creative design and purchasing, you could run 48 volts DC directly to the UPS. That's assuming solar panels or whatever are worthwhile to begin with, of course.
"I assumed blithely that there were no elves out there in the darkness"
Pre chill or heat the building ahead of time.
Rather than rolling-your-own off-peak pre-chilling system, check out the Ice Bear ice-based thermal energy storage system. This system was named a GreenSpec® Top-10 green building product of 2005. It's designed to work with standard central air-conditioning units, in buildings from 2,000 to 50,000 ft^2.
This system can reduce peak cooling load by up to 95%. Not only does it improve your peak load profile, it can reduce the overall cooling cost, since the A/C units run at night, when the ambient temperature is cooler. Condensation is not an issue, since the ice is stored in insulated containers near the A/C units (typically outside, often on the roof.)
Disclosure: I'm not affiliated with Ice Bear, but I do work for BuildingGreen, which published the articles I linked to above.
I have a better solution, switch all your Air conditioners etc, to Natural Gas models. then you're billed by the commodity rate they sell for, rather than 'when' peak consumption hours fall. of course the cost of the equipment, the cost of natural gas, the infrastructure changes etc could well add up to a big cost, and i don't know what the bottom line in savings would be (if any)
since a good data center should have back up generators anyways, you could run the cooling subsystem from those backup generators (during peak hours) and use diesel/brew your own biodiesel from waste veggie oil depending on whatever is cheaper.
https://www.gnu.org/philosophy/free-sw.html
If you are using Linux look at Zen so you can run multiple VMs on one machine. For windows VMWare.
Stick to AMD for now. They put out a lot less heat and burn fewer watts per mip than the current Intel Machines.
Look at the SunT1 line.
Few Hard drives. Use a storage server with a Raid instead of an HD per machine if you can.
Low heat blades?
There are lots of options to reduce power consumption the problem is will you save more than you spend? I would ban P4s right now. No reason to run a server with an Intel chip right now except for the rumored better stability. Intel's new chip line should put them back in the running but right now I say AMD, PentiumM, and Duo Core for X86 and Sun T1s are machines you should look at.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.