Slashdot Mirror
Intel Shows Data Centers Can Get By (Mostly) With Little AC
Ted Samson IW writes "InfoWorld reports on an experiment in air economization, aka 'free cooling,' conducted by Intel. For 10 months, the chipmaker had 500 production servers, working at 90 percent utilization, cooled almost exclusively by outside air at a facility in New Mexico. Only when the temperature exceeded 90 degrees Fahrenheit did they crank on some artificial air conditioning. Intel did very little to address air-born contaminants and dust, and nothing at all to deal with fluctuating humidity. The result: a slightly higher failure rate — around 0.6 percent more — among the air-cooled servers compared to those in the company's main datacenter — and a potential savings of $2.87 million per year in a 10MW datacenter using free cooling over traditional cooling."
How about reducing the need for AC POWER as well by cutting down on the number of AC TO DC PSU's.
I do wonder how things could be improved with a decently sized stack... the higher an exit chimney, the more draw you'll get from the temperature differential. If your computer rooms are near the base of a decent sized office building, and you have a 20 story stack, I'd expect you could get away without any intake or exhaust fans.
Anyone here that can confirm or deny this?
Makes sense to me. The most efficent places to store data centers is in the northern US or Canada where you have sub-zero temperatures from November - March and ranging between 0-15 in April/May and Sept/Oct and the rest of the year 20-30+ (Celcius of course) With these lower temperatures they could run a data center entirely off outside air from September - May each year. Put a heppa filter in between to scrub out dirt and dust and vola, o'natural cooling solutions
I leave my systems on the deck.
I asked the president of an engineering firm that I work for about this. He ships racks of boxes, each holding DSP boards on backplanes, each backplane has it's own PSU.
When I asked him why he doesn't just have one or two -big- power supplies in the unit, he said that he tried that, but the cost of the non-standard PSU was higher than all the ATX PSUs put together, and then some, and replacing the units when they eventually fail would be tricky, as opposed to just stocking more ATX PSUs.
I agree that it's a good idea, but until there's enough volume of large multi-output PSUs shipping, the cost of manufacture makes the product unworkable (unless you think big-picture and want to spend more up front for power savings over the whole unit's life).
Generally, the people who use the hardware aren't the ones building it, and buyers usually go for the lowest bid.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
The savings should be more than enough to pay for replacement hardware, and even for upgrades. And stepping back and looking at the big picture tells me that there is at least one brilliant person at Intel--whoever though of doing this study is a genius!
--MarkusQ
You must be new here. /. We all run data centers with 3000 servers and program on apps with 10+ million LOC. We also all built something better than a 3d solar cell in the 5th grade.
There are no small guys... especially on
No comprende? Let me type that a little slower for you...
Well, it makes sense. Normal PCs run on essentially ambient air, and live for years even under heavy loads (games put a lot of load on systems) despite all the dust and cruft. Servers aren't that different in their hardware, so it makes sense they'd behave similarly. And there's a lot that can be done cheaply to reduce the problems that were seen. Dust, for instance. You can filter and scrub dust from the incoming air a lot cheaper than running a full-on AC system. In fact the DX system used on the one side of the test probably scrubbed the incoming air itself, which would explain the lower failure rate there. Reduce the dust, you reduce the build-up of the thermal-insulating layer on the equipment and keep cooling effectiveness from degrading. Humidity control can also be done cheaper than full-on AC, and wouldn't have to be complete. I don't think you'd need to hold humidity steady within tight parameters, just keep the maximum from going above say 50% and the minimum from going below 5%. Again I'll bet the DX system did just that automatically. I'd bet you could remove the sources of probably 80% of the extra failures on the free-cooling side while keeping 90% of the cost savings in the process.
The standard replacement cycle is about three years, so until they try that, this doesn't mean a lot. Also, what was the density of the data center? I still love the story of a datacenter with some DSLAMs that cooled left to right which were put next to each other in about 12 racks and the rightmost one caught fire once a week...
Also, I don't know the climate there, but in the regular climate here where it goes between -10 and +35 celsius (that's between 14 and 95 fahrenheit) and there's a good dose of humidity, the failure rate might be somewhat bigger...
I'd say that they will have to wait longer to get failures. Try to have a server running in that enviroment for 5 years and then we will see. I would not do it without having some good filters. But for a test it is a interesting experiment.
For datacenters in colder climates, you can already get cooling systems that cools the water using air only when the temperature is below a certain temperature(just forgot the number). When it gets above that level the water gets cooled like you normally do.
At work our old AC system was old and needed to be replaced and the new one does that. The outside temperature is so low that the water will be cooled with just air for half the year.
It was more expensive to install since it needed more and bigger cooling units(I belive they also talked about bigger slower fans that used less power) when just using air but it pays itself in a few years.
Another interesting experiment would be to use the heat again. I dont know if the water temperature is high enough so that you could use heat exchangers, perhaps as the first step on heating ingoing cold water.
The fluctuating humidity probably wouldn't be a problem in New Mexico either. The rest of us might have a problem.
A company I used to work for (SeaChange International) would ship systems that, in some cases, were large enough to be considered their own datacenter. Some customers would order -48 volt DC power supplies. They'd do their own wiring at the site, having one big AC-DC converter to handle the entire system. They were certainly more expensive than the ATX supplies.
SIG: HUP
I will rephrase your question. Would a .6% increase in the already tiny failure risk be noticeable to someone running a single server when their chances of failure were already so small to begin with that their server was far less likely to fail in the first place?
No, so yes, it is worth it from a cost perspective. They can take the money they save and replace the hardware twice as fast and their already small failure rate is less than half. This is a win all around and actually, the article never said what was the source of the increased failures, heat or particulate in the air. If the latter, this is a huge win for energy efficiency.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
What kind of air-conditioning is that? Here the rule of thumb looks like this - 10KW of electricity produce 7KW of heat, and it takes one-third of that (2.333KW) in electricity to move it out. Do you have any sources on this?:)
We all run data centers with 3000 servers and program on apps with 10+ million LOC. We also all built something better than a 3d solar cell in the 5th grade.
Pfft! I achieved a technological singularity 3 years ago. I am the datacenter.
Part of the problem is people are looking for very complicated solutions for very simple problems.
In retrofitting a standalone building, all you really need to do is reduce the amount of heat a building gains from the sun by improving it's R value and use sensible ducting to draw air through the building. I've seen some super energy efficient designs where each floor is vented, so that the building is itself a chimney, with cool air coming from vents from covered areas near the base, and enough size provided at the top to pull enough from the bottom, which is also easily aided by fans.
In building an entirely new datacenter, it would make sense to bury the server rooms, and cover the concrete structure with earth and solar panels. Combined with a flywheel load balancer, you could have an "off the grid" datacenter with the grid for backup. During the daylight hours, especially in the south, the panels can provide a good deal of the A/C and power necessary. At night the flywheel can continue powering the data center for a while, and turn fans without compressors to cool the equipment with night air.
This can all be done with existing technology. The trick is to convince people that green investment will lead to a return in the long run. I haven't personally looked at average rate increases in electricity, but the difference between efficient and additional construction expenses versus long term energy price fluctuations probably looks very good.
So this study is not actually useful for people who need to build data centers in more humid places then new mexico
Humidity only really matters for two reason - If too low, you get a lot of static buildup, and if too high, you get condensation.
Condensation only tends to happen on objects cooler than ambient, which doesn't really apply to running servers. Static matters a lot more, but you can raise humidity a lot cheaper than you can lower it, so, not as much of an issue there.
And as a bonus, more humid air can carry away more heat than the same volume of less humid air.
Antarctica would be kind of a neat place for a data center. You have all of the cold air you need and there is enough wind for power. Just have to find a way to keep it stable amidst moving ice.
Sun is also running a comparable experiment with Belgacom and allows you to log in to a live interface to view stats on in- and outlet temperatures and more at http://wikis.sun.com/display/freeaircooling/Free+Air+Cooling+Proof+of+Concept For more details and analysis see http://www.datacenterknowledge.com/archives/2008/09/18/intel-servers-do-fine-with-outside-air/ or http://securityandthe.net/2008/09/18/intel-sees-the-future-of-datacenters-and-it-does-not-include-airconditioning/
DC Knowledge also has a nice video of this experiment at http://www.datacenterknowledge.com/archives/2008/09/18/video-intels-air-side-economization-test/
Having very large PSUs is a pain in the ass. Failures tend to be catastrophic and dangerous. They're more expensive to build and maintain. (think basic economy of scale problems) They also may not be any more efficient than distributed conversion. You also tend to distribute much lower voltages with DC than you do with AC. (240vac vs 48vdc) This gives very high amperages which requires much thicker wiring. Copper is EXPENSIVE right now, this makes it a big factor in the cap-ex of building a new DC.
This is why a lot of work is going into improving the efficiency of commodity power supplies. Groups like 80plus.org are doing great things.
Also some other links:
http://www.treehugger.com/files/2007/07/secret_efficien.php
http://services.google.com/blog_resources/PSU_white_paper.pdf
There is another really smart thing you can do too. When it is hot inside and not hot outside yu can open a window. That seems obvious but how many office building have openable windows? For some reason Architects like to cool office space with AC even if there is "free" cool air out doors.
The reasons are things like: liability issues, chimney effects, people leaving the windows open even with the heat or a/c on, people leaving the windows open in the rain, bugs in the building, increased maintenance costs for more complex windows, etc. It turns out that architects aren't actually idiots and have thought about this.
This is even easier with computers. The servers would be happy to run at 95F and much of the time even in the American SW the outside air is cooler than 95F.
Your desktop can run at 95F ambient. If it has a variable speed fan it's probably screaming like a banshee. The key is how much heat can be dissipated, and at 95F ambient you can't move enough air through a dense computer system to cool the components down to a safe temperature. Even at 68F ambient people are have a lot of trouble moving enough air through modern super-dense racks to keep computers from seeing increased failure rates.
4 words "Single point of failure"
You mean like the power circuit that you are already connected to? That single point of failure has long ago been handled. Where the costs can be justified, run more than one power circuit, backup generators and UPS, etc. That's no different.
I'm personally more interested in the wasteful DC to AC and back conversion when considering small scale solar. Why in the world is the default option to run a wasteful inverter just to plug an AC to DC converter in to that? Almost everything I looked at for portable solar to power a laptop or netbook worked like that. A lot of netbooks could be run on a 10W solar panel with battery backup, or more reliably of course with more solar capacity.
A company I used to work for (SeaChange International) would ship systems that, in some cases, were large enough to be considered their own datacenter. Some customers would order -48 volt DC power supplies. They'd do their own wiring at the site, having one big AC-DC converter to handle the entire system. They were certainly more expensive than the ATX supplies.
-48V DC is nothing special in many telco applications. Sun equipment (which has been historically popular with telcos (they have lots of NEBS-certified hardware)) has DC power supplies as a standard option on a good portion of their servers.
Of course many other manufacturers also offer DC P/S options (and NEBS).
http://www.epanorama.net/wwwboard/messages/1142.html
The lower the DC voltage, the higher the current and line loss. And running 3-4 different voltages throughout the place leads to confusion and much higher costs (4 voltages == 4x the wire.) -48VDC systems have been common for decades... in the telco world. They just haven't been common for computer datacenters.
I can say with much certainty that most of the big vendors are starting to warm up to this and know that needless cooling is not going to stand up to scrutiny much longer. In fact, Intel is not the only one looking at this. These standards that we apply for acceptable heat and humidity levels were a) never designed for IT equipment and b) were never actually tested. The come from old telecom standards and they were primarily assumptions based on very old technology. Anyone looking at datacenter eff is looking real hard at these and asking themselves, what are the real acceptable ranges for modern equipment, under modern conditions. When this is all said and done, the answers are going to be much more heat tolerance and far greater humidity tolerance in both directions.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
If you want to go straight DC, you need to use the economies of scale, not replace AC power supplies with some alternate power scheme that still uses AC on the rack and DC into the server.
Instead, use large, very efficient AC-DC transformers and wire the rack DC.
If you convert AC to DC in bulk with more expensive but highly efficient equipment you will save significant money on the power conversion PLUS you can put that transformer outside in its own enclosure with a big metal heat exchanger for a case.
DC can be stepped down very easily and efficiently so various voltages are available from the transformer or from a seperate step-down box that doesnt create much heat because it is pretty efficient.
Now, you dont have to worry about the heat from the power supply and dont have to cool for it. You gain savings in efficiency and less AC use.
also, the transformer can very easily be cooled but an extremely simple ground loop and small pump can handle that for a few bucks per month.
EVERYTHING _M_U_S_T_ be air-conditioned at all times. From what we heard from France during their last heat wave a few years ago, air-conditioning isn't universal in the First World. Therefore, it must sound strange that air-conditioning is a inviolate moral imperative in all offices in the US. My wife has a sweater with her at work at all times even if it is July or August. Same for me. 100% wool. When it is 95 outside and 68 inside, I want nothing more than to hibernate -- like seriously drift off to sleep. I've worn gloves with the fingers cut out in July at my keyboard. I've sneaked in an incandescent lamp to warm my hands (please, sir, just a lump of coal?). I've gotten on my chair and stuffed paper towels in air ducts.
If management can't see that they are air-conditioning some of their people into productivity loss, not to mention pain, how much more likely are they to reduce air-conditioning on their precious equipment? No, doesn't matter whether one experiment shows it would save big money. The person who suggests reducing air-conditioning in the U.S. will be about as popular at his business as if he had suggested commissioning a portrait of Karl Marx on the lunch room wall. This just isn't a technical issue.
No. The reason AC was more convenient to move around is the ability to step it up and down with transformers. But in fact line losses are higher for a given voltage with AC than DC, for various reasons (e.g. peak voltage is higher, some of the power radiates). Nowadays, converting DC to DC is about as easy (it goes through a high frequency AC step on the way, however). A switching power supply actually converts AC (60Hz) to DC to AC (tens of kilohertz) to DC.
100% relative humidity is when the dewpoint is reached and water condenses out of the air (aka fog). The popular idea that 100% rel humidity = rain is not accurate.
Aluminum wiring is a FIRE hazard and was BANNED in all new
houses in the US due to it.
You might be able to get away with it outdoors, but it is
most likely a bad idea based on the indoor results.
http://www.physicsforums.com/showpost.php?s=7d306106c574b8acd101e052ab90be42&p=615606&postcount=6
http://books.google.com/books?id=2edigWaeGPUC&pg=PA175&lpg=PA175&dq=aluminum+wiring+ban&source=web&ots=l0eE26iMkt&sig=rVIgBVl0gXGlJicEHA_qW8s4zY0&hl=en&sa=X&oi=book_result&resnum=4&ct=result
Alot of areas you cannot even get insurance for the building
with aluminum wiring in it.
http://en.wikipedia.org/wiki/Aluminum_wiring#Hazard_insurance
google "32 trillion offshore needs IRS attention"
That's because aluminum significantly contracts and expands with temperature changes. When it does so in a residential setting, it will cause shorts and sparks and such in outlets and switches. The 1" wire (probably more like a crossbar) was probably specifically designed for electrical use, and had appropriate connectors and so on so that it was NOT a danger (as noted in the physicsforums post you linked to. Given the price of copper any more, the special work needed for aluminum is possibly worth it.
My blog. Good stuff (when I remember to update it). Read it.
You're only half right. If you actually read any of the articles you linked to you'd know that.
Aluminum wire by itself is no hazard at all. It just doesn't do well when you connect it copper or other galvanically dissimilar materials that can cause corrosion. And there are some issues with dissimilar thermal expansion rates, but that's largely dependent on the terminal size and type.
You're right that the standard 14-10 AWG wiring used in homes is typically not aluminum, and that the wiring of that size that was aluminum and installed in the the 60s and 70s needs to be treated specially.
But aluminum was and still is commonly used in large-gauge wiring, starting around 8 AWG -- the ~2 AWG feed for many homes *is* aluminum. And it's entirely possible to safely wiring aluminum, even of smaller gauges, even of older alloy types, so long as you understand the limitations and use CO/ALR-rated devices.
Computers run hot enough to get rid of moisture and one assumes that these data centers run around the clock.
But dust can be lethal to computers and in particular to power supplies and CPU fans. I clean my PCs guts at least twice a year and what comes out is amazing. Fans are great at collecting dust and they don't pump much air when coated with dust either.