Slashdot Mirror
Intel Shows Data Centers Can Get By (Mostly) With Little AC
Ted Samson IW writes "InfoWorld reports on an experiment in air economization, aka 'free cooling,' conducted by Intel. For 10 months, the chipmaker had 500 production servers, working at 90 percent utilization, cooled almost exclusively by outside air at a facility in New Mexico. Only when the temperature exceeded 90 degrees Fahrenheit did they crank on some artificial air conditioning. Intel did very little to address air-born contaminants and dust, and nothing at all to deal with fluctuating humidity. The result: a slightly higher failure rate — around 0.6 percent more — among the air-cooled servers compared to those in the company's main datacenter — and a potential savings of $2.87 million per year in a 10MW datacenter using free cooling over traditional cooling."
How about reducing the need for AC POWER as well by cutting down on the number of AC TO DC PSU's.
I do wonder how things could be improved with a decently sized stack... the higher an exit chimney, the more draw you'll get from the temperature differential. If your computer rooms are near the base of a decent sized office building, and you have a 20 story stack, I'd expect you could get away without any intake or exhaust fans.
Anyone here that can confirm or deny this?
Makes sense to me. The most efficent places to store data centers is in the northern US or Canada where you have sub-zero temperatures from November - March and ranging between 0-15 in April/May and Sept/Oct and the rest of the year 20-30+ (Celcius of course) With these lower temperatures they could run a data center entirely off outside air from September - May each year. Put a heppa filter in between to scrub out dirt and dust and vola, o'natural cooling solutions
I leave my systems on the deck.
I asked the president of an engineering firm that I work for about this. He ships racks of boxes, each holding DSP boards on backplanes, each backplane has it's own PSU.
When I asked him why he doesn't just have one or two -big- power supplies in the unit, he said that he tried that, but the cost of the non-standard PSU was higher than all the ATX PSUs put together, and then some, and replacing the units when they eventually fail would be tricky, as opposed to just stocking more ATX PSUs.
I agree that it's a good idea, but until there's enough volume of large multi-output PSUs shipping, the cost of manufacture makes the product unworkable (unless you think big-picture and want to spend more up front for power savings over the whole unit's life).
Generally, the people who use the hardware aren't the ones building it, and buyers usually go for the lowest bid.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
You must be new here. /. We all run data centers with 3000 servers and program on apps with 10+ million LOC. We also all built something better than a 3d solar cell in the 5th grade.
There are no small guys... especially on
No comprende? Let me type that a little slower for you...
Well, it makes sense. Normal PCs run on essentially ambient air, and live for years even under heavy loads (games put a lot of load on systems) despite all the dust and cruft. Servers aren't that different in their hardware, so it makes sense they'd behave similarly. And there's a lot that can be done cheaply to reduce the problems that were seen. Dust, for instance. You can filter and scrub dust from the incoming air a lot cheaper than running a full-on AC system. In fact the DX system used on the one side of the test probably scrubbed the incoming air itself, which would explain the lower failure rate there. Reduce the dust, you reduce the build-up of the thermal-insulating layer on the equipment and keep cooling effectiveness from degrading. Humidity control can also be done cheaper than full-on AC, and wouldn't have to be complete. I don't think you'd need to hold humidity steady within tight parameters, just keep the maximum from going above say 50% and the minimum from going below 5%. Again I'll bet the DX system did just that automatically. I'd bet you could remove the sources of probably 80% of the extra failures on the free-cooling side while keeping 90% of the cost savings in the process.
The standard replacement cycle is about three years, so until they try that, this doesn't mean a lot. Also, what was the density of the data center? I still love the story of a datacenter with some DSLAMs that cooled left to right which were put next to each other in about 12 racks and the rightmost one caught fire once a week...
Also, I don't know the climate there, but in the regular climate here where it goes between -10 and +35 celsius (that's between 14 and 95 fahrenheit) and there's a good dose of humidity, the failure rate might be somewhat bigger...
The fluctuating humidity probably wouldn't be a problem in New Mexico either. The rest of us might have a problem.
I will rephrase your question. Would a .6% increase in the already tiny failure risk be noticeable to someone running a single server when their chances of failure were already so small to begin with that their server was far less likely to fail in the first place?
No, so yes, it is worth it from a cost perspective. They can take the money they save and replace the hardware twice as fast and their already small failure rate is less than half. This is a win all around and actually, the article never said what was the source of the increased failures, heat or particulate in the air. If the latter, this is a huge win for energy efficiency.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
Part of the problem is people are looking for very complicated solutions for very simple problems.
In retrofitting a standalone building, all you really need to do is reduce the amount of heat a building gains from the sun by improving it's R value and use sensible ducting to draw air through the building. I've seen some super energy efficient designs where each floor is vented, so that the building is itself a chimney, with cool air coming from vents from covered areas near the base, and enough size provided at the top to pull enough from the bottom, which is also easily aided by fans.
In building an entirely new datacenter, it would make sense to bury the server rooms, and cover the concrete structure with earth and solar panels. Combined with a flywheel load balancer, you could have an "off the grid" datacenter with the grid for backup. During the daylight hours, especially in the south, the panels can provide a good deal of the A/C and power necessary. At night the flywheel can continue powering the data center for a while, and turn fans without compressors to cool the equipment with night air.
This can all be done with existing technology. The trick is to convince people that green investment will lead to a return in the long run. I haven't personally looked at average rate increases in electricity, but the difference between efficient and additional construction expenses versus long term energy price fluctuations probably looks very good.
So this study is not actually useful for people who need to build data centers in more humid places then new mexico
Humidity only really matters for two reason - If too low, you get a lot of static buildup, and if too high, you get condensation.
Condensation only tends to happen on objects cooler than ambient, which doesn't really apply to running servers. Static matters a lot more, but you can raise humidity a lot cheaper than you can lower it, so, not as much of an issue there.
And as a bonus, more humid air can carry away more heat than the same volume of less humid air.
Antarctica would be kind of a neat place for a data center. You have all of the cold air you need and there is enough wind for power. Just have to find a way to keep it stable amidst moving ice.
One benefit to going DC is that you can wire your battery modules directly into the DC distribution grid for the CPUs (with appropriate charge and cutover circuits), and forgo the inefficiencies in converting AC to DC at the UPS, and then back out again, only to convert the AC back to DC at the CPU.
Having multiple of a commonly used voltage used in renewable energy also helps if, for example, you want to feed your datacenter directly from say wind or solar, in addition to a set of AC to DC converter.
Sun is also running a comparable experiment with Belgacom and allows you to log in to a live interface to view stats on in- and outlet temperatures and more at http://wikis.sun.com/display/freeaircooling/Free+Air+Cooling+Proof+of+Concept For more details and analysis see http://www.datacenterknowledge.com/archives/2008/09/18/intel-servers-do-fine-with-outside-air/ or http://securityandthe.net/2008/09/18/intel-sees-the-future-of-datacenters-and-it-does-not-include-airconditioning/
DC Knowledge also has a nice video of this experiment at http://www.datacenterknowledge.com/archives/2008/09/18/video-intels-air-side-economization-test/
You lose on density, though. Aisle space in front of the racks is fixed, you need a certain amount for humans to move in. Shallow, tall equipment means fewer units per rack. With a current-format rack you need say a 3'-deep area for the rack and a 3'-wide aisle for access. That's 50% equipment and 50% access. If the racks were only 1' deep instead, you'd be using 25% for equipment and 75% for access (since you still need that 3' wide aisle). And in that 25% of space for equipment you now get perhaps 25% of the amount of equipment since each one's using 4x more vertical space in the rack and rack height can't change (it's limited by basically how high off the floor a human can reach to get to the equipment).
To make up for that, you need more square footage of data center to hold the equipment. That increases operating costs, which is what we're trying to reduce.
Having very large PSUs is a pain in the ass. Failures tend to be catastrophic and dangerous. They're more expensive to build and maintain. (think basic economy of scale problems) They also may not be any more efficient than distributed conversion. You also tend to distribute much lower voltages with DC than you do with AC. (240vac vs 48vdc) This gives very high amperages which requires much thicker wiring. Copper is EXPENSIVE right now, this makes it a big factor in the cap-ex of building a new DC.
This is why a lot of work is going into improving the efficiency of commodity power supplies. Groups like 80plus.org are doing great things.
Also some other links:
http://www.treehugger.com/files/2007/07/secret_efficien.php
http://services.google.com/blog_resources/PSU_white_paper.pdf
I can say with much certainty that most of the big vendors are starting to warm up to this and know that needless cooling is not going to stand up to scrutiny much longer. In fact, Intel is not the only one looking at this. These standards that we apply for acceptable heat and humidity levels were a) never designed for IT equipment and b) were never actually tested. The come from old telecom standards and they were primarily assumptions based on very old technology. Anyone looking at datacenter eff is looking real hard at these and asking themselves, what are the real acceptable ranges for modern equipment, under modern conditions. When this is all said and done, the answers are going to be much more heat tolerance and far greater humidity tolerance in both directions.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
100% relative humidity is when the dewpoint is reached and water condenses out of the air (aka fog). The popular idea that 100% rel humidity = rain is not accurate.
Aluminum wiring is a FIRE hazard and was BANNED in all new
houses in the US due to it.
You might be able to get away with it outdoors, but it is
most likely a bad idea based on the indoor results.
http://www.physicsforums.com/showpost.php?s=7d306106c574b8acd101e052ab90be42&p=615606&postcount=6
http://books.google.com/books?id=2edigWaeGPUC&pg=PA175&lpg=PA175&dq=aluminum+wiring+ban&source=web&ots=l0eE26iMkt&sig=rVIgBVl0gXGlJicEHA_qW8s4zY0&hl=en&sa=X&oi=book_result&resnum=4&ct=result
Alot of areas you cannot even get insurance for the building
with aluminum wiring in it.
http://en.wikipedia.org/wiki/Aluminum_wiring#Hazard_insurance
google "32 trillion offshore needs IRS attention"
You're only half right. If you actually read any of the articles you linked to you'd know that.
Aluminum wire by itself is no hazard at all. It just doesn't do well when you connect it copper or other galvanically dissimilar materials that can cause corrosion. And there are some issues with dissimilar thermal expansion rates, but that's largely dependent on the terminal size and type.
You're right that the standard 14-10 AWG wiring used in homes is typically not aluminum, and that the wiring of that size that was aluminum and installed in the the 60s and 70s needs to be treated specially.
But aluminum was and still is commonly used in large-gauge wiring, starting around 8 AWG -- the ~2 AWG feed for many homes *is* aluminum. And it's entirely possible to safely wiring aluminum, even of smaller gauges, even of older alloy types, so long as you understand the limitations and use CO/ALR-rated devices.