Slashdot Mirror
Data Centers And DC Power
mstansberry writes "In the final article in a series on the price of power in the data center, IT pros weigh the pros and cons of direct current-powered servers. A limited number of companies make servers with the power supplies removed with DC power distributed to multiple machines from a single unit. It saves power by skipping an extra conversion from alternating current (AC). Telcos have been using this method for years, but some data center pros are leery of taking on the new systems. It's not something people are familiar with and if they break down, you have to hire a specialized engineer to come fix them. But if they're saving even half of what they're reported to save on the electric bill, companies could afford to hire the engineers." We've reported on previous articles in the series.
There is no standard Telco DC connector. This can cause headaches since you need special connectors and a AC-DC box for every type of DC appliance you have. Most every AC appliance uses one of a few types of industry standard AC connectors.
It isn't three phase like a HVAC blower motor, it is three separate legs using a common neutral, *** BUT ***, if each of the three legs draws a different amount of current, there is an imbalance on the neutral and computers hate that.
To be pedantic for a moment, Tesla quit after Edison screwed him over on a $50,000 bonus he was promised.
But you're sentiment is correct. Edison never really believed in AC power.
Javascript + Nintendo DSi = DSiCade
That said, there's no reason why the power couldn't come to your house as AC and then be turned into DC centrally by an efficient PSU in the basement (or wherever). The only minor problem is that DC is somewhat more dangerous than AC - if you touch a live AC wire you can pull away from it more easily than if you are in a DC circuit due to the effect on nerves.
I am TheRaven on Soylent News
...this may or may not save a step.
:)
However, it does provide a few significant advantages.
Telcos use DC because it's easy to battery-back. Since all your gear is already running from the DC supply, there's no guesswork about whether your UPS will be able to handle the load. Each piece includes its own converters, so all you have to do is size the battery bank. Since most telcos aim for 8-hour runtimes on battery (long enough to discover and fix a generator problem), overkill is the order of the day.
There's also the point that you can run several small generators, instead of one large one. In an AC world, keeping multiple generators syncrhonized is nearly impossible on a small scale, so you just run one big one. If your setup grows, you rip out the old generator and replace it with a larger one. In DC, since all your generators feed the same battery bank, you can just tack on more capacity without trashing your original investment.
Using multiple generators provides cheaper redundancy too. In an AC setup if you wanted to be protected against a generator failure, you'd need two identical gensets, each large enough to run the whole load. With DC, say you had 5 generators but 4 could power the load. You still have no single point of failure, and you don't have to buy *double* the generating capacity.
Oh, and if a second generator fails, say you're down to 3, you're below the break-even point, but you're still limping along, with the operating generators assisting the batteries, extending your battery runtime long enough that you can probably fix one of the failed gensets. Oh, you found a spare generator at the rental place down the street? Switch a few rectifiers onto it and watch your charge status come back into the green. You just don't have that sort of versatility with AC.
DC is easier to noise-filter than AC. Keeping the high-frequency noise from switching converters off the AC input is something of a black art, and is hard to do effectively. You also have Power Factor (PF) issues when running large numbers of computers (or anything that uses switch-mode power supplies) from AC. Hence, your supplies have to be PF-corrected, which adds bulk and complexity, and reduces efficiency.
A DC-DC converter suffers none of those problems, going from your 48v battery bank down to the 12, 5, and 3.3 levels in your servers. It's easy to filter the switching noise because the input is DC, a big L-C filter works quite well. There's no such thing as power factor on DC, so the converters themselves are simpler and smaller, and run cooler.
One other huge benefit is that 48 volts is "low voltage" according to the NEC, so you can wire it yourself. You'll never have to let pole-climbers into your server room again.
Another advantage is that most DC-input equipment has a telco heritage, and supports dual inputs. Everything in telco has an "a-side" and a "b-side" power supply. It's only relatively recently that high-end datacomm gear has started to support multiple AC power inputs. History and experience are on your side with DC.
Sure, but how often do the backup generators connect inside the UPS?
My understanding is that the UPS's will typically have a power source switch in front of them, not behind, and when the emergency generator kicks in, its power goes through the UPS just like the normal utility power.
There's a very good reason for that, too. Virtually every UPS will clean up the power feed, and backup generators are usually 'dirtier' power than mains power - the last thing you want is spikes and droops from the backup genny cooking your servers while you're under emergency conditions!
The argument for AC to DC conversion on each device is that individual power supplies provide isolation e.g. no direct current path from DC ground of one device to another device. This is why Ethernet is transformer coupled. Eliminates ground loops and propagation of damage during major hardware meltdowns, nearby lightning strike etc.
As long as you're feeding a site with AC (this is the only efficient way to transmit it from the Hoover dam to some farm in Iowa), then at some point, there is conversion from AC to DC before it gets to the circuits of any computer system.
Where there *could* be some benefit is where you have larger, more efficient converters very near the point of use. If you figure each power supply inside each box is 50% efficient, but a single big one is 75%, then you reap a net benefit (totally rhetorical - I have no idea how efficient they are).
Now, even if there isn't an efficiency gain, there are two other reasons to do this, which is entirely why telcos do it (not to save power). First, telco equipment has to run when utility power is lost. Every switch is powered off DC that comes straight from BATTERIES. There is no loss in service if AC power is lost, because the batteries are already in use. This explains why you can use your POTS line even during a brief power outage. We have such a box sitting in a machine room in attached building.
Secondly (and perhaps more germane to the concept of DC in non-telco datacenters), telco equipment is often housed in locations that have poor environmental controls. By positioning the not-so-efficient AC-to-DC conversion hardware away from the sensitive electronics, you reduce the heat load in the climate-controled space. This alone makes some sense for the direct DC feed concept.
Charles
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
1) Contacts tends to rust on the positive side.
2) Lower voltage means bigger current for the same power. This would require thicker, more expensive cables
3) DC-DC voltage conversion is, somewhat less efficient... Ok, I know switching mode power supplies are efficient, but this leads me to the last point:
4) No insulation between systems. That way, systems get more prone to ground loops...
"There is always an easy solution to every human problem -- neat, plausible, and wrong."
H. L. Mencken
I think there is at least one reason not to distribute DC inside the house: The same reason car battery contacts get yucky after some time.
AC prevents that galvanic(?) effect to occur on the house outlets.
There are definite advantages to DC power... but it can also be *hugely* annoying.
I've worked in DC powered labs.
There isn't really any concept of 'plug' in the DC powered world. Powering up a device usually entails reading it's current draw off the equipment, selecting the correct gauge of wire, cutting the correct length of wire, strip both ends, hook up to your DC distribution on one end and your equipment on the other, select about the right size fuse, plug it in... etc. It's a royal pain. Oh, and make sure you do it correctly, because it's not that hard to electrecute yourself...
Nearly every engineer I've ever worked with whose been exposed to DC powered labs has begged to return to the AC powered world... it's just MUCH easier to work with.
On the flip side though... telco racks rock! Nothing beats hex head rack screws... you can literally drive them in at a 45 degree angle with a power drill and it's OK. It makes going back to the world of crappy philips head data wrack screws that you occasionally have to drill out because the head has stripped very annoying.
But you're sentiment is correct. Edison never really believed in AC power.
Actually he was a pretty firm supporter of AC where the electric chair was concerned.
The original example of FUD!
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Granted, you may not need to carry a lot of amps at 2V. However, no matter what voltage/current you pick, it's much easier (in terms of wiring cost) to use higher voltage for electricity distribution.
I think what the main article was discussing is changing 120 AC into 120 DC centrally, but still having the 120 DC => 2v DC conversion done right where it's needed.
Does big iron still use 3-phase power?
Yes. Mainframes, large UNIX systems, and the storage boxes that connect to them still require three-phase. (I am a storage specialist.)
SirWired
True. But low voltage (under 50vDC nominal) doesn't require licensed electricians to run it. Clearly the extra buck for thicker copper outweighs the cost of paying an electrician for 8 hours to come extend a power feed. You've obviously never had to deal with licensed electricians.
How so? AC-DC switchmode power supplies start by rectifying the AC into a high DC voltage, and then perform internal DC-DC conversion to produce the output voltages. Even the ones that work straight from the AC are no more efficient once you include power factor correction.
Not true. Remember I said the telco power system is -48vDC with respect to ground? All the logic levels (12, 5, and 3.3v) in the cards are positive just like you're used to. The DC-DC converters are isolating; all they stipulate is that there be less than a 300v total differential between the inputs and the outputs. You're free to reference any part to ground, or leave it floating if your heart desires.
Telco grounding is insane anyway. Most places have #6AWG from each rack to a 1/0 aisle ground cable, and all the aisle grounds meet on a 750KCMil that runs the length of the building, over to the "office principal ground point". Track down a copy of TP76200MP and read up.
Easy.
First, DC actually is better for transmitting power over long distances. AC current tends to concentrate in the surface of the conductor, leading to higher current densities and larger ohmic losses.
So, why do we use AC almost everywhere? Transformers. It is relatively easy and efficient to use a transformer to change voltages of AC power. For large electrical lines, the voltage is cranked way up, which means the current is reduced. The less current, the smaller the losses due to resistance in the wires. So power is transmitted at high voltages, so the current and hence losses are low. Then, near the place where power is needed, transformers change the power to lower voltage, higher current. (This is because you can't have house wiring and appliances that won't arc or explode when hit with 13,800 V.)
Converting between high and low voltages with DC power is much more difficult, and requires more complex equipment. (An AC transformer is two pieces of wire wrapped around a chunk of iron.)
Let me be mroe specific on this problem.
If you have 120V and 20A coming into the house, and you convert it to 12V, you will de-facto allow 200A to go through the 12V circuit.
The problem is your circuit breaker would have to be on the 12V circuit, otherwise you could get a short, and instead of just blowing the breaker you will liquify your copper wire.
So for every circuit voltage you support you need a seperate circuit breaker, wiring, outlet, etc.
Plus a 1200W power supply will still need 100A at 12V, instead of 10A at 120V, so you would still need 6 gauge wire instead of your more normal 16 gauge wire, so all your house wiring will be four times as thick, and about as flexible as chicken wire.
But westinghouse did believe in it .. who then gave Tesla a job.
Westinghouse didn't give Tesla a job, he contracted with Tesla Electric Light & Manufacturing for R&D and licensed the AC patents. Eventually Tesla released Westinghouse from paying royalties to prevent the company from going under. (The AC/DC wars nearly bankrupt both Edison and Westinghouse.)
Though admittedly, Nikola was not much of a businessman
Indeed. He was always a little too paranoid. Instead of learning how to properly use the laws and courts to protect his work, he felt that the only option was to keep his work super-secret. The sad part about this is that we still don't fully understand some of his inventions. For example, take his electric car. How did he manage to power that thing at such high velocities given the technology of the day? The answer is still a mystery even today. (And a favorite of the free energy quacks, I might add.)
which is why while he was perhaps the most brilliant scientist to ever exist on this planet, he died virtually pennyless.
At least in part, that had to do with all the equipment he was purchasing to perform his grounded power experiments. He had this idea that he could run power through the Earth itself, allowing anything that touched the surface of the Earth to tap into the grid. Such a concept would have been a boon for electric vehicles. Sadly, his theories on the subject were later proven incorrect, meaning that he wasted his money and time on a dead end.
Javascript + Nintendo DSi = DSiCade
If you are reporting your teacher's comments properly, your teacher should be fired before someone is killed. Absolutely none of what you just said made the slightest bit of sense.
There are several ways in which electricity can kill you. It can heat you up and burn you. It can disrupt your brain and nervous system. But the most common thing is for it to stop your heart. (It turns out that the survival rate of folks with heart failure after CPR administration is extremely low, unless the heart failure was caused by electrical shock. Most causes of heart failure also prevent the heart from restarting.)
The other major safety concern with electricity is the risk of fire.
There's a tradeoff here. Your skin is basically a big resistor in the megohm range. Since the current that penetrates the skin is proportional to the voltage, high voltages cause a big danger of shock. On the other hand, a short is typically a small resistor. Since the heat dissipated by the short is proportional to the square of the current, high currents cause a big danger of fire. But the power delivered by a circuit is proportional to the product of the current and the voltage. So if you want to power something adequately, you need high current or high voltage or some compromise between the two.
The whole thing is further complicated by the galvanic reflex, which makes muscles contract when electricity reaches them through the skin. DC tends to penetrate deeper into the muscle tissue than AC, and thus causes more violent contraction. (Is this the "literal explosion" [sic] your teacher was referring to?) With low-enough frequency AC (and I'm not sure whether 50-70Hz is low enough) the galvanic reflex is down during the low-energy part of the cycle long enough that you can let go. Electric fences (at least properly designed ones) are very-low-frequency pulsed-current devices. This is so that you can release from the fence while the power is down.
Another major complication is defibrillation. 50-70Hz AC reaching the heart is much more effective at stopping it than DC, because it interferes with the heart's regulatory rhythm.
The executive summary? High voltages are dangerous (electrocution) and so are high currents (fire). House current is a reasonable compromise between these concerns for high-power devices. The AC/DC tradeoff is vastly overrated as a safety concern, and very complicated.
How to keep yourself safe when working theatre lighting? Here's some tips:
Be careful out there. Either listen more carefully to your teacher (if you got your stuff wrong), or publicly out the idiot (if you got it right). Electricity is really dangerous. I say this as someone who builds big rockets in his spare time. I was nearly killed several times as a kid while playing with electricity. Don't let it happen to you.
There are several approaches to power distribution. One is "telco type" -48VDC distribution. This is most appropriate when the configuration doesn't change much. Wiring usually involves big cables and screw lugs. Plugs aren't standardized. More importantly, there's no set of simple rules, like the UL/NEMA/NEC standards that govern plugs, outlets, wiring, and circuit breakers, that make 120V power distribution safe without having to measure everything.
In the 120VAC world, everything has been designed so that end users don't have to worry much about overloading the wiring. If they do, a circuit breaker will trip. An ordinary power plug, a "5-15P", can handle 15A, so if you have an outlet strip, there is a breaker to protect the plug and cord from overload, should the total load on the power strip exceed 15A. A 20A power strip must have a "L5-20P" plug, the big twist-lock type. As soon as you get away from 120VAC, you lose that designed-in idiot-proofing. (Europe is still struggling in this area, with too many different connectors, so you don't get the same level of idiot-proofing in the 220VAC part of the world.) So once you leave 120VAC, you're going to need power engineering skills. (Clamp-around ammeters are very useful, and yes, you can get them for DC.)
There's also 400Hz AC distribution, which allows for smaller transformers and filter caps in power supplies. 400Hz rackmount servers are available. Aircraft, military, and some mainframe systems use 400Hz. It's not a big win in this era of switching power supplies.
There's 3-phase power distribution. Here's a 3-phase outlet strip. More to the point, there's an efficiency gain in running a UPS from 3-phase power, and big UPSs are usually 3-phase, at least on the input side. Arguably, power should be 3-phase down to the point where it's rectified to DC, because 3-phase rectifiers need far less filtering, but nobody does this for small loads.
American Power Conversion has been pushing the idea of integrating power conversion, cable management, and cooling into standard racks. Classically, those are the big problems in big computer systems. Seymour Cray used to say that the big problems were "the thickness of the (wiring) mat" and "getting rid of the heat". By that standard, APC is now as much of a computer manufacturer as, say, Dell; neither makes motherboards or ICs, they just package gear from others. Which is a wierd thought.
All of this power is going to be converted again, at least once, and probably twice, before it hits the semiconductors. That's the job of point-of-load DC to DC converters, usually ICs on the board that do the final conversion. Typically, when you get to the computer, there's a conversion from the line voltage (120-240VAC, 48VDC, etc) to internal distribution voltages of 5-12VDC, then another conversion and regulation just before each device, usually downward to something like 3.3VDC. This keeps transient load changes from one device from affecting others. There may be on-chip regulation, too. The losses at those last stages of conversion are usually the biggest ones in the whole chain.
Yes, real big iron still uses 3-phase power. I can only speak on behalf of large IBM system (zSeries, etc). These systems will accept 192VAC to 508VAC on the input, 3-phase Delta. This means no neutral required. Additionally, they will even run with one phase totally missing.
The first power conversion stage in any piece of their 'big iron' is a very large AC to DC converter, rated for a 350VDC output at over 42kW. Actually it's six 7.5kW converters paralled, and these are redundant/hot swappable. Totally modular, with no cable connections. This block is about 95% efficient.
This DC is then distributed to the rest of the system power supplies, with redundant cabling supplying all point of load converters. All point of load converters are also redundant and hot swap. These converters have a range of efficiencies, but are typically much better than industry standards. A DC/DC converter in the z9 can source 1000A alone on the CPU Vcore level (12 of these supplies are in the machine). Supplies are used for CPU nodes, I/O cages, blowers and refrigeration.
All blowers are 3-phase DC-brushless type, with the 3-phase synthesized off the 350VDC feeds. The blowers are usually 300W or larger, each.
The CPU refrigeration is also run by 3-phase compressors, this power also being synthesized off of 350VDC. This is done to allow a conventional off-the-shelf compressor to be run off any line voltage, and ride through phase losses (as this is seen by the bulk AC/DC converter instead).
The 'big iron' also supports built in UPS cabability, allowing you to connect battery packs directly to the bulk AC/DC converters. A machine will handle six 400V@2.5Ah battery packs connected to it. This feature is used to ensure a system such as a z9 has true 100% availability, and won't suffer a hard shutdown due to careless datacenter workers or electricians.
In short, the article is intend to address small white box systems that use $12 power supplies with very poor reliability and efficiencies.
And to another poster that brought up 3-phase being more efficient for power conversion...that's not really true these days, as everything requires power-factor correction. Nothing in the IT uses huge three-phase bridge rectifiers and phase-regulated primaries anymore.
We seem to have moronic moderators today, as it's uncomprehensive why the above comment got modded as "Insightful".
For Christ's sake, this guy doesn't know what he's talking about! "DC power is naturally unstable", "unclean power" WTH?
Back to the original topic, the article is, as other mentioned already, 100% pure dribble. The major advantage of AC input power is that the power conversion (AC to DC system and from there down to 5V/3.3V/VCore/DDR/IO/etc), happens close to the loads.
AC voltage is 110V or more, therefore a power of 400W per system will give about 6.3A RMS (considering a conversion efficiency of 80%). So, one must design the wiring to the system to withstand let's say 7.5A. And then the last down conversion stages start from 12V only.
If you want to carry a DC voltage (e.g. -48V), then you must use an isolated DC/DC down to the system voltage because it's impractical to downconvert from -48V to every rail voltage needed in the system. The same 400W will be 9.3A out of the -48V input (for a 90% conversion efficiency).
Transmission power losses increase with the square of the passed current. The 9.3A versus 6.3A means one must use thicker wire, i.e. the wiring is costlier to carry the DC voltage and there is no obvious benefit.
In the end, if anyone wants to lower the electricity cost, he/she must invest in better off-line power supplies. For high power ones with PFC, cheapies have 65-70% efficiency, while a good design has more like 85-90%. This difference is significant.
Serban
The section that reads "These devices, a kind of switched-mode converter, generally perform the conversion by the following steps" is misleading. While it says "AC", if you click through the link, it actually says "The inverter stage converts DC ... to AC by switching it on and off ('chopping') at a frequency of tens or hundreds of kilohertz (kHz)". In other words, it's nothing like the orignal AC (60Hz), and the kHz chopping is necessary even if you start off with AC. So it's not redundant at all.
Eg. the typical Switchmode PSU has four stages:
- 60Hz AC
- half-wave rectified AC with a big capacitor (almost DC)
- kHz chopped AC
- final DC output
So, instead of hauling things around at stage 1, you haul them around at stage 2 instead.Hi everyone,
I am a datcenter manager that has had the opportunity to not only run but also build a datacenter from pretty much scratch. In my experience I have found that both DC and AC powered equipment both have their places in the environment. Neither system is perfect so by running hybrid you can get the most flexibility.
We recently moved our datacenter form a 10K sq ft facility down to a 1700 ft facility by doing a technology refresh and changing many of our key infrastructure methods. In the new facility I currently have 315 HP blade servers plus another 10-15 traditional rack type servers running. I have the capacity to add up to another 144 blades (assuming they are 1U) before I run out of floor and HVAC capacity. The power delivery method is hybrid. I run DC for the blades which are fed by Emerson Energy's Candeo XL rectifier stacks (originally designed for telco) and AC for everything else. To eliminate a lot of the under floor clutter I use a trough system instead of conduit for the various AC circuits. HVAC is provided by 4 Liebert 22TON units which keep my room at a comfy under floor temp of 66 degrees.
Adequate airflow is critical so we spent a lot of time planning tile placement. The key for proper cooling in this scenario was a high volume of airflow pushing the cooling to about 5.5ft up from the raised floor. This way my cooling isn't being sucked up by just the bottom half of the rack. Low voltage cabling is overhead.
We chose to power the blades DC for two reasons. First was the limited space I had for installing breaker boxes on the walls. The number of AC circuits I had was limited so I pulled fat feeds directly to the Candeo systems. A full rack of HP p-class blades would require 4 x 3phase 208 circuits per rack. My initial installation of blades would have consumed 144 of my 168 circuits leaving next to nothing to power my SAN/Network/Tape Library/etc equipment. The other reason was power supply efficiency. In the conversion of power from AC to DC the efficiency of the power supply must be taken into consideration. It's not just the number of conversions you do but the loss at each. Typical power supplies in servers run about 80% efficient while my Candeo as they are setup gets about 90%. For me this ultimately meant less heat and more available cooling, therefore I could bring in more servers under the existing HVAC.
I prefer a best of class mentality. IMHO there is no best universal solution. For those of you that use traditional rack mounts servers like Dell you can purchase these units with a DC option. I am not sure if HP offers a similar option but they might.
Len
p.s. Someone also made the comment about DC not generating noise in network cabling while AC does. This is not a totally true statement. Anytime you run a current through a conductor you will generate a magnetic field. Put this in parallel to another conductor and you will further induce a mag field (this is why any power runs that have to intersect low-voltage cabling should only intersect at 90 degree angles to avoid inductance). The big difference is the way DC cabling runs. In most DC circuits the feed and return lines run together so the proximity of the out of phase magnetic fields will cancel each other out. Don't believe me? I had this problem when we intially wired these Candeo systems up. The small feeds to the racks and the big mains that connected to the common buss bar were about a foot apart. Because the fields weren't cancelling, we were getting enough noise on the lines that it looked like there was AC leaking through the circuits (6volts p-p in some cases). By simply wire tieing the lines together, the proximity cancelled the fields out and everything was peachy.
The Tesla grounded power experiments have been used to develop and make feasible the very low frequency and very long range communication systems for submarines (emission only) using earth's crust upper part as resonator.
The reason that the terminals on your battery get "yucky" is the sulfuric acid leaking from behind the terminals onto the metal. A mixture of baking soda and water is good at cleaning corroded terminals because it neutralizes the acid, which then allows the ions to dissolve in water. If you were to dip the terminal of a corroded battery cable in mixture of baking soda and water, you'll notice that after a while the water turns a greenish-blue -- those are the copper ions that the acid has "liberated" from the metal of the terminal. This effect has nothing to do with AC vs DC and everything to do with leaky acid-cell batteries.
Copper and aluminum bus bars in AC power substations corrode just as much as they would if they were carrying DC; in fact, if you were to ever watch a substation being put together, every electrical connection is slathered with an anti-oxidation compound like "NoOx" (for copper) or "NoAlOx" (for aluminum) to prevent oxidatation that could then lead to hotspots and eventually fire.