Slashdot Mirror
Intel Shows Data Centers Can Get By (Mostly) With Little AC
Ted Samson IW writes "InfoWorld reports on an experiment in air economization, aka 'free cooling,' conducted by Intel. For 10 months, the chipmaker had 500 production servers, working at 90 percent utilization, cooled almost exclusively by outside air at a facility in New Mexico. Only when the temperature exceeded 90 degrees Fahrenheit did they crank on some artificial air conditioning. Intel did very little to address air-born contaminants and dust, and nothing at all to deal with fluctuating humidity. The result: a slightly higher failure rate — around 0.6 percent more — among the air-cooled servers compared to those in the company's main datacenter — and a potential savings of $2.87 million per year in a 10MW datacenter using free cooling over traditional cooling."
If they're paying ten cents a kilowatt-hour, that 10MW data center is paying about $9M/yr for power.
Cooling systems move about 15 times the power than what they draw. So the savings for a 10MW datacenter would be around $600K. Wonder how they came up with $2.9M ?
This is just speculation, but isn't much of new mexico rather arid? So this study is not actually useful for people who need to build data centers in more humid places then new mexico which I think includes most of the places there are actually people.
But if you are going to allow for an arbitrarily re-locatable data center, what does it matter that it can handle 90 degree whether when you can move it somewhere cold enough that you can have a humidity controlled room that gets passive cooling from the exterior.
The fluctuating humidity probably wouldn't be a problem in New Mexico either. The rest of us might have a problem.
A company I used to work for (SeaChange International) would ship systems that, in some cases, were large enough to be considered their own datacenter. Some customers would order -48 volt DC power supplies. They'd do their own wiring at the site, having one big AC-DC converter to handle the entire system. They were certainly more expensive than the ATX supplies.
SIG: HUP
So true! Anyone with a background in unairconditioned manufacturing plants can tell you that new computers do just fine in rough conditions, but after a few years you will get power supply failure rates out the ass! Give them DC power inputs, standardized, please (but you KNOW intel won't do that - they don't even use standardized front panel connectors) and you might see the failure rate reduced even further.
Almost all data centers are designed with A/C in mind. This means that as long as A/C is pulling the load no one needs to worry about well designed buildings. As soon as you are challenged with having to design for reduced A/C usage that you end up thinking smarter and how passive systems can do the same thing. Another advantage of trying to design without A/C is that you won't find your servers frying because of an air conditioner failure.
Below are some links on passive solutions to cooling. Some of the techniques are surprisingly old, but effective:
- http://en.wikipedia.org/wiki/Passive_cooling
- http://en.wikipedia.org/wiki/Windcatcher
- http://www.arabrise.org/articles/A040105S.pdf
Jumpstart the tartan drive.
We have a monsoon season here, in mid summer. Gets pretty humid at times.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Isn't it more efficient to do the DC conversion as close to the last second as possible? Once the juice is DC it becomes much less efficient to move it around, no?
Sun is also running a comparable experiment with Belgacom and allows you to log in to a live interface to view stats on in- and outlet temperatures and more at http://wikis.sun.com/display/freeaircooling/Free+Air+Cooling+Proof+of+Concept For more details and analysis see http://www.datacenterknowledge.com/archives/2008/09/18/intel-servers-do-fine-with-outside-air/ or http://securityandthe.net/2008/09/18/intel-sees-the-future-of-datacenters-and-it-does-not-include-airconditioning/
DC Knowledge also has a nice video of this experiment at http://www.datacenterknowledge.com/archives/2008/09/18/video-intels-air-side-economization-test/
Having very large PSUs is a pain in the ass. Failures tend to be catastrophic and dangerous. They're more expensive to build and maintain. (think basic economy of scale problems) They also may not be any more efficient than distributed conversion. You also tend to distribute much lower voltages with DC than you do with AC. (240vac vs 48vdc) This gives very high amperages which requires much thicker wiring. Copper is EXPENSIVE right now, this makes it a big factor in the cap-ex of building a new DC.
This is why a lot of work is going into improving the efficiency of commodity power supplies. Groups like 80plus.org are doing great things.
Also some other links:
http://www.treehugger.com/files/2007/07/secret_efficien.php
http://services.google.com/blog_resources/PSU_white_paper.pdf
That's why a lot of places use aluminium. And the practice isn't new. CP&L (now Progress Energy) wired the feed for a data center (150KW) with aluminium "wire" (if you wanna call something 1" in diameter a wire.) And that was 15+ years ago.
A company I used to work for (SeaChange International) would ship systems that, in some cases, were large enough to be considered their own datacenter. Some customers would order -48 volt DC power supplies. They'd do their own wiring at the site, having one big AC-DC converter to handle the entire system. They were certainly more expensive than the ATX supplies.
-48V DC is nothing special in many telco applications. Sun equipment (which has been historically popular with telcos (they have lots of NEBS-certified hardware)) has DC power supplies as a standard option on a good portion of their servers.
Of course many other manufacturers also offer DC P/S options (and NEBS).
http://www.epanorama.net/wwwboard/messages/1142.html
The lower the DC voltage, the higher the current and line loss. And running 3-4 different voltages throughout the place leads to confusion and much higher costs (4 voltages == 4x the wire.) -48VDC systems have been common for decades... in the telco world. They just haven't been common for computer datacenters.
I can say with much certainty that most of the big vendors are starting to warm up to this and know that needless cooling is not going to stand up to scrutiny much longer. In fact, Intel is not the only one looking at this. These standards that we apply for acceptable heat and humidity levels were a) never designed for IT equipment and b) were never actually tested. The come from old telecom standards and they were primarily assumptions based on very old technology. Anyone looking at datacenter eff is looking real hard at these and asking themselves, what are the real acceptable ranges for modern equipment, under modern conditions. When this is all said and done, the answers are going to be much more heat tolerance and far greater humidity tolerance in both directions.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
We're talking less than a minute needed. In the end they couldn't use the several large and expensive flywheels because they could not provide power long enough.
If you're powering your whole data center 'for a while' with these... you must have very few servers (like a handful).
Experience teaches only the teachable. -AH
If you want to go straight DC, you need to use the economies of scale, not replace AC power supplies with some alternate power scheme that still uses AC on the rack and DC into the server.
Instead, use large, very efficient AC-DC transformers and wire the rack DC.
If you convert AC to DC in bulk with more expensive but highly efficient equipment you will save significant money on the power conversion PLUS you can put that transformer outside in its own enclosure with a big metal heat exchanger for a case.
DC can be stepped down very easily and efficiently so various voltages are available from the transformer or from a seperate step-down box that doesnt create much heat because it is pretty efficient.
Now, you dont have to worry about the heat from the power supply and dont have to cool for it. You gain savings in efficiency and less AC use.
also, the transformer can very easily be cooled but an extremely simple ground loop and small pump can handle that for a few bucks per month.
No. The reason AC was more convenient to move around is the ability to step it up and down with transformers. But in fact line losses are higher for a given voltage with AC than DC, for various reasons (e.g. peak voltage is higher, some of the power radiates). Nowadays, converting DC to DC is about as easy (it goes through a high frequency AC step on the way, however). A switching power supply actually converts AC (60Hz) to DC to AC (tens of kilohertz) to DC.
100% relative humidity is when the dewpoint is reached and water condenses out of the air (aka fog). The popular idea that 100% rel humidity = rain is not accurate.
You're correctly stating the conventional wisdom for properly managing air in a datacenter. However, the whole point was that Intel was doing their cooling with outside air, minimally filtered to see what the effects of disregarding the conventional wisdom might be. So, one way to improve the energy efficiency might be to use a chimney to avoid having to use fans.
That's because aluminum significantly contracts and expands with temperature changes. When it does so in a residential setting, it will cause shorts and sparks and such in outlets and switches. The 1" wire (probably more like a crossbar) was probably specifically designed for electrical use, and had appropriate connectors and so on so that it was NOT a danger (as noted in the physicsforums post you linked to. Given the price of copper any more, the special work needed for aluminum is possibly worth it.
My blog. Good stuff (when I remember to update it). Read it.
You're only half right. If you actually read any of the articles you linked to you'd know that.
Aluminum wire by itself is no hazard at all. It just doesn't do well when you connect it copper or other galvanically dissimilar materials that can cause corrosion. And there are some issues with dissimilar thermal expansion rates, but that's largely dependent on the terminal size and type.
You're right that the standard 14-10 AWG wiring used in homes is typically not aluminum, and that the wiring of that size that was aluminum and installed in the the 60s and 70s needs to be treated specially.
But aluminum was and still is commonly used in large-gauge wiring, starting around 8 AWG -- the ~2 AWG feed for many homes *is* aluminum. And it's entirely possible to safely wiring aluminum, even of smaller gauges, even of older alloy types, so long as you understand the limitations and use CO/ALR-rated devices.
Relative humidity can go above 100% as well in certain situations it is called Supersaturation.
An Education is the Font of All Liberty
Computers run hot enough to get rid of moisture and one assumes that these data centers run around the clock.
But dust can be lethal to computers and in particular to power supplies and CPU fans. I clean my PCs guts at least twice a year and what comes out is amazing. Fans are great at collecting dust and they don't pump much air when coated with dust either.