Slashdot Mirror
Data Centers And DC Power
mstansberry writes "In the final article in a series on the price of power in the data center, IT pros weigh the pros and cons of direct current-powered servers. A limited number of companies make servers with the power supplies removed with DC power distributed to multiple machines from a single unit. It saves power by skipping an extra conversion from alternating current (AC). Telcos have been using this method for years, but some data center pros are leery of taking on the new systems. It's not something people are familiar with and if they break down, you have to hire a specialized engineer to come fix them. But if they're saving even half of what they're reported to save on the electric bill, companies could afford to hire the engineers." We've reported on previous articles in the series.
Tesla, you're fired. --Thomas Edison
try { do() || do_not(); } catch (JediException err) { yoda(err); }
I've always wondered (from a non-technical point of view) whether there was a benefit in having our homes wired up with two sockets (or maybe a 5 pin mains plug) giving standard AC voltage and a low-current DC voltage as well (12V?). So many devices only need low voltage, wouldn't we all benefit in having a power system in our houses in this way?
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
ac goes into data centres, systems run on dc. Either it gets distributed to each computer as ac and converted in a medium-sized box in the back of each system, or it gets converted in one big box and distributed to the systems as dc.
The question is of the efficiency saving of doing all the converting in a big box against the efficiency loss of piping it around the data centre as dc, and wether you get a large total net saving (which I suspect that you do, since even inside the data centre, it's not going far)
FGD 135
Nope, you save two conversions.
Without DC distribution, you have AC->DC->AC in the central UPS, and then AC->DC in each computer's power supply.
With DC distribution, you have AC->DC in the central UPS, and no conversion in the computers.
You get down from 3 conversions to 1.
I started working with IP in a small ISP. We were bought by a loal Telco and over the years have got used to having all our routers and switches running on DC current.
One thing telco companies do well is DC power, they have alot of skill in providing multiple DC feeds from DC power systems, with battery backup and generators all in line.
I would imagine that any big server farm would benefit from this kind of setup. Especially when you have people runnnig the lines that are as good as some of the guys in the telo world, they can really make the wiring look like a art in some places.
...this may or may not save a step.
:)
However, it does provide a few significant advantages.
Telcos use DC because it's easy to battery-back. Since all your gear is already running from the DC supply, there's no guesswork about whether your UPS will be able to handle the load. Each piece includes its own converters, so all you have to do is size the battery bank. Since most telcos aim for 8-hour runtimes on battery (long enough to discover and fix a generator problem), overkill is the order of the day.
There's also the point that you can run several small generators, instead of one large one. In an AC world, keeping multiple generators syncrhonized is nearly impossible on a small scale, so you just run one big one. If your setup grows, you rip out the old generator and replace it with a larger one. In DC, since all your generators feed the same battery bank, you can just tack on more capacity without trashing your original investment.
Using multiple generators provides cheaper redundancy too. In an AC setup if you wanted to be protected against a generator failure, you'd need two identical gensets, each large enough to run the whole load. With DC, say you had 5 generators but 4 could power the load. You still have no single point of failure, and you don't have to buy *double* the generating capacity.
Oh, and if a second generator fails, say you're down to 3, you're below the break-even point, but you're still limping along, with the operating generators assisting the batteries, extending your battery runtime long enough that you can probably fix one of the failed gensets. Oh, you found a spare generator at the rental place down the street? Switch a few rectifiers onto it and watch your charge status come back into the green. You just don't have that sort of versatility with AC.
DC is easier to noise-filter than AC. Keeping the high-frequency noise from switching converters off the AC input is something of a black art, and is hard to do effectively. You also have Power Factor (PF) issues when running large numbers of computers (or anything that uses switch-mode power supplies) from AC. Hence, your supplies have to be PF-corrected, which adds bulk and complexity, and reduces efficiency.
A DC-DC converter suffers none of those problems, going from your 48v battery bank down to the 12, 5, and 3.3 levels in your servers. It's easy to filter the switching noise because the input is DC, a big L-C filter works quite well. There's no such thing as power factor on DC, so the converters themselves are simpler and smaller, and run cooler.
One other huge benefit is that 48 volts is "low voltage" according to the NEC, so you can wire it yourself. You'll never have to let pole-climbers into your server room again.
Another advantage is that most DC-input equipment has a telco heritage, and supports dual inputs. Everything in telco has an "a-side" and a "b-side" power supply. It's only relatively recently that high-end datacomm gear has started to support multiple AC power inputs. History and experience are on your side with DC.
The slashdot story intro implies that the advantage of DC is that you
save a conversion step. Well, maybe you do, maybe you don't, but
counting the number of AC-to-DC and DC-to-AC conversions is very
misleading.
Converting 50 or 60 Hertz to DC is much more costly and less efficient
than converting in either direction at a higher frequency. Low
frequency rectification requires large filter capacitors, complex and
expensive inrush current limiting, and active power-factor correction.
By doing that front-end work in one place only, preferably from a
3-phase source, you save power and increase reliability. You probably
still want multiple 50/60Hz to DC rectifier stages, of course, but now
they can be in parallel (for redundancy), rather than each one
downstream of the other where a failure of either one will bring down
the system.
Just because you're distributing DC to the racks, doesn't mean you
don't have to convert it again. It typically gets converted to AC and
back to DC at least once, usually twice before it reaches CPU and
memory chips. That's equally true in data centers that distribute AC
or DC. The fact is, memory and CPU devices want very low DC voltages
and very high currents. To make matters worse, not all parts of the
system want exactly the same DC voltage, you almost always have to
have multiple supply rails. You can't distribute very low voltages,
because it would require wires as thick as your arm and they'd still
be too resistive and inductive, so instead you distribute the DC at,
typically, 48 volts. The subsequent conversion to low DC voltages has
to happen via an intermediate AC, but it's a high frequency AC, so it
can be done much more efficiently using ferrite magnetic components,
active rectification, and often resonant mode filters. This high
frequency AC is confined to the internals of a power supply unit, it
never travels over wires or between boxes, thus reducing typical
high-frequency problems such as RFI.
I haven't mentioned battery-backup (i.e. UPSs). They make the system
more complex, but don't change any of the fundamental concerns. Even
on a DC distribution system, the UPS system requires it's own
additional stages of DC->AC->DC conversion, both while charging
(standby) and while discharging (during AC power failure). This is
because battery charging has to have a precisely controlled current
envelope. And batteries don't discharge at the uniform and
well-regulatted voltage that your DC distribution wants. They need
regulators, and switchmode regulators (typically DC->AC->DC) are the
most efficient choice.
Does big iron still use 3-phase power?
Yes. Mainframes, large UNIX systems, and the storage boxes that connect to them still require three-phase. (I am a storage specialist.)
SirWired
Yes, real big iron still uses 3-phase power. I can only speak on behalf of large IBM system (zSeries, etc). These systems will accept 192VAC to 508VAC on the input, 3-phase Delta. This means no neutral required. Additionally, they will even run with one phase totally missing.
The first power conversion stage in any piece of their 'big iron' is a very large AC to DC converter, rated for a 350VDC output at over 42kW. Actually it's six 7.5kW converters paralled, and these are redundant/hot swappable. Totally modular, with no cable connections. This block is about 95% efficient.
This DC is then distributed to the rest of the system power supplies, with redundant cabling supplying all point of load converters. All point of load converters are also redundant and hot swap. These converters have a range of efficiencies, but are typically much better than industry standards. A DC/DC converter in the z9 can source 1000A alone on the CPU Vcore level (12 of these supplies are in the machine). Supplies are used for CPU nodes, I/O cages, blowers and refrigeration.
All blowers are 3-phase DC-brushless type, with the 3-phase synthesized off the 350VDC feeds. The blowers are usually 300W or larger, each.
The CPU refrigeration is also run by 3-phase compressors, this power also being synthesized off of 350VDC. This is done to allow a conventional off-the-shelf compressor to be run off any line voltage, and ride through phase losses (as this is seen by the bulk AC/DC converter instead).
The 'big iron' also supports built in UPS cabability, allowing you to connect battery packs directly to the bulk AC/DC converters. A machine will handle six 400V@2.5Ah battery packs connected to it. This feature is used to ensure a system such as a z9 has true 100% availability, and won't suffer a hard shutdown due to careless datacenter workers or electricians.
In short, the article is intend to address small white box systems that use $12 power supplies with very poor reliability and efficiencies.
And to another poster that brought up 3-phase being more efficient for power conversion...that's not really true these days, as everything requires power-factor correction. Nothing in the IT uses huge three-phase bridge rectifiers and phase-regulated primaries anymore.