Slashdot Mirror


Data Centers And DC Power

mstansberry writes "In the final article in a series on the price of power in the data center, IT pros weigh the pros and cons of direct current-powered servers. A limited number of companies make servers with the power supplies removed with DC power distributed to multiple machines from a single unit. It saves power by skipping an extra conversion from alternating current (AC). Telcos have been using this method for years, but some data center pros are leery of taking on the new systems. It's not something people are familiar with and if they break down, you have to hire a specialized engineer to come fix them. But if they're saving even half of what they're reported to save on the electric bill, companies could afford to hire the engineers." We've reported on previous articles in the series.

24 of 379 comments (clear)

  1. Forgot DC and AC power by mrm677 · · Score: 2, Interesting

    Does big iron still use 3-phase power?

    1. Re:Forgot DC and AC power by ThePowerGorilla · · Score: 3, Interesting

      Yes, real big iron still uses 3-phase power. I can only speak on behalf of large IBM system (zSeries, etc). These systems will accept 192VAC to 508VAC on the input, 3-phase Delta. This means no neutral required. Additionally, they will even run with one phase totally missing. The first power conversion stage in any piece of their 'big iron' is a very large AC to DC converter, rated for a 350VDC output at over 42kW. Actually it's six 7.5kW converters paralled, and these are redundant/hot swappable. Totally modular, with no cable connections. This block is about 95% efficient. This DC is then distributed to the rest of the system power supplies, with redundant cabling supplying all point of load converters. All point of load converters are also redundant and hot swap. These converters have a range of efficiencies, but are typically much better than industry standards. A DC/DC converter in the z9 can source 1000A alone on the CPU Vcore level (12 of these supplies are in the machine). Supplies are used for CPU nodes, I/O cages, blowers and refrigeration. All blowers are 3-phase DC-brushless type, with the 3-phase synthesized off the 350VDC feeds. The blowers are usually 300W or larger, each. The CPU refrigeration is also run by 3-phase compressors, this power also being synthesized off of 350VDC. This is done to allow a conventional off-the-shelf compressor to be run off any line voltage, and ride through phase losses (as this is seen by the bulk AC/DC converter instead). The 'big iron' also supports built in UPS cabability, allowing you to connect battery packs directly to the bulk AC/DC converters. A machine will handle six 400V@2.5Ah battery packs connected to it. This feature is used to ensure a system such as a z9 has true 100% availability, and won't suffer a hard shutdown due to careless datacenter workers or electricians. In short, the article is intend to address small white box systems that use $12 power supplies with very poor reliability and efficiencies. And to another poster that brought up 3-phase being more efficient for power conversion...that's not really true these days, as everything requires power-factor correction. Nothing in the IT uses huge three-phase bridge rectifiers and phase-regulated primaries anymore.

  2. Re:What about houses? by n0dalus · · Score: 4, Interesting

    I thinks it's a really good idea.
    If there were several pins, many different voltages would be possible, and a device could even use more than one voltage from one plug (eg, it could draw 2V for a relay, 4V for a power indicator, and the standard AC for the actual thing it's powering.)
    By not having to have transformers and big resistors inside all the household devices, there would be huge savings in power, things wouldn't get so hot, wouldn't need such big heatsinks, there would be far less electromagnetic radiation around the place (which is probably responsible for a lot of people getting sick etc), and it's safer too (devices that only need a small DC power source won't electrocute you when you drop them in some water.)

  3. Re:correction by ArsenneLupin · · Score: 5, Interesting
    Either way, there is still only one conversion: 110AC -> 12/-12/5vDC.

    Nope, you save two conversions.

    Without DC distribution, you have AC->DC->AC in the central UPS, and then AC->DC in each computer's power supply.

    With DC distribution, you have AC->DC in the central UPS, and no conversion in the computers.

    You get down from 3 conversions to 1.

  4. wasted servers by artg · · Score: 2, Interesting

    "His department conducted a study that said 80% of those servers were running at 5% to 15% utilization"

    Why such low utilization ?
    Any other industry would scrap 80% of that equipment to save costs and power.

  5. DC in Telco by Comen · · Score: 5, Interesting

    I started working with IP in a small ISP. We were bought by a loal Telco and over the years have got used to having all our routers and switches running on DC current.
    One thing telco companies do well is DC power, they have alot of skill in providing multiple DC feeds from DC power systems, with battery backup and generators all in line.
    I would imagine that any big server farm would benefit from this kind of setup. Especially when you have people runnnig the lines that are as good as some of the guys in the telo world, they can really make the wiring look like a art in some places.

  6. Re:What about houses? by Roger_Wilco · · Score: 2, Interesting

    I think the best way to do this is to use a smart bus, reminiscent of USB. You plug in the device, and it indicates "I'd like 5V DC", or whatever, and the other side provides the appropriate voltage. A powerbar would have to say "I'd like 120 VAC" or "I'd like raw AC power", then be smart enough to switch that to the desired voltage for each device.

    You could only have one plug on each wire from the smart hub.

    Costs would be higher due to all the electronics involved, but they'd come down with mass production. It should eventually be cheaper than running all those little black transformers for every device; they draw a certain amount of power even when the device is off.

  7. Re:What am I missing? by GigsVT · · Score: 4, Interesting

    The DC step in the middle is so you can chop it into a high frequency square wave. You save power and space because you can vary the duty cycle of the square wave for regulation, and the high frequency allows for smaller components.

    --
    I've had enough abrasive sigs. Kittens are cute and fuzzy.
  8. The first step by dsginter · · Score: 3, Interesting

    The first step will be in the home office. Have you taken a look at the rat's nest under most desks? Most of it is AC/DC conversion. If the industry could just arrive at a DC power standard, we could start with a single AC/DC "box" under the desk with a standard plug end for all DC peripheraps. Add daisy chaining and wireless USB or Bluetooth, and that nest is largely eliminated.

    At this point, we could start to build it into houses and other buildings.

    --
    More
  9. Don't you still need a PSU for 48v DC? by bigtrike · · Score: 3, Interesting

    Wouldn't you still need a power supply to convert to the various voltages required by a computer, which may change over time. Most DC power setups I've seen run at 48v, which still requires conversion to 12v, 5v, and 3.3v. You can buy a 48v power supply for most servers and other equipment today. With a switched power supply, you'd need larger capacitors or a higher switching frequency in order to smooth out the lower powered DC. It's very unlikely that you would eliminate any heat loss. I would assume that telecommunications equipment uses a 48v setup due to legacy issues and that it was a better idea before switching power supplies became cheap and efficient.

    The downside with DC is that lower voltages require much thicker wires, and you're at much greater risk for fire. Circuit breakers and other things are also more complicated and expensive since DC tends to weld things together.

    An EE I know just built a data center supplying 208v (2 branches of a 3 phase iirc) to all the racks. Almost all existing power supplies can take it, and it saves a bundle in wiring costs. I'm not sure about servers, but most desktop power supplies operate at a power factor of .8 or so, meaning 20% of the billable power is effectively wasted and could be recovered for a slight increase in cost.

  10. Re:specialized engineers? by gweihir · · Score: 2, Interesting

    ever seen someone drop a screwdriver between 5V and 0V buses on a 300A distribution system!?

    No problem at all. Turns immediately to metal vapour and will likely not even interrupt server operation.
    And with todays computers that would more likely be 12V/3000A ;-)

    Of course the human being standinge besides this events may suffer some serious damage....

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  11. Re:See, I told you so by nurb432 · · Score: 2, Interesting

    But westinghouse did believe in it .. who then gave Tesla a job.

    But later on he screwed Tesla anyway.

    Though admittedly, Nikola was not much of a businessman, which is why while he was perhaps the most brilliant scientist to ever exist on this planet, he died virtually pennyless.

    --
    ---- Booth was a patriot ----
  12. Re:What about houses? by Shakrai · · Score: 2, Interesting

    there would be far less electromagnetic radiation around the place (which is probably responsible for a lot of people getting sick etc)

    Since when was non-ionizing EM radiation dangerous? Besides, even if it was, I highly doubt that your computer PSU or little brick power supply for your cordless phone are wasting that much energy as EM. When you consider that most little brick power supplies are running most of the waste is lost to heat and not EM I highly doubt it amounts to much. Even your typical PC PSU runs less then 100-150 watts unless you are a dual processor 15 HD fiend.

    I'd be willing to bet that you get a larger dose of EM from your box fan, AC motor or refrigerator compressor.

    --
    I want peace on earth and goodwill toward man.
    We are the United States Government! We don't do that sort of thing.
  13. Re:DC power for data centers by paradxum · · Score: 2, Interesting

    I guess I find the amazing part is that with most modern switching power supplies, you can plug them into dc and have them just work... no special kit at all. I think it needs about 98VDC for a standard computer power supply.

    In a datacenter, it's much easier to provide DC. Most are set up against a large battery backup (to hold over till the generator kicks in.) So, You just simply don't run inverters. The AC->DC converters handle the rest (which they normally do to charge the batteries.) Plus, a AC-DC converter is "stupid simple" in Electrical engineering terms.

    I only see two drawbacks:
    1.) larger wires to handle the DC current without significant heating/loss (losses are in the form of heat in a wire)

    2.) Making sure your routers/switches are set up for DC. Many switches and routers DO NOT come with switching power supplies.... and need a different power supply to run on dc... but compnies like HP and Cisco have DC power supplies for their kit.

    I don't really think a data center needs to hire a "Power Engineer" I mean really, did you hire a new person when you switched from Parell-IDE to Serial-IDE??? (oh wait... I shouldn't ask questions I don't want answers to.)

  14. Re:What about churches? Very small rocks? by Da_Biz · · Score: 2, Interesting

    Transmitting DC over long distances doesn't work very well

    I've heard this before, but I haven't heard a terribly good explanation for why.

    HVDC Pacific Intertie between Oregon and California:
    http://en.wikipedia.org/wiki/Pacific_Intertie
    http://www.transmission.bpa.gov/cigresc14/Compendi um/PACIFIC.htm

    I worked on a transmission sales automation project at BPA, and I seem to recall some very good explanations for why they had a 2000+ MW, 400kV transmission hop.

  15. Re:What about houses? by Anonymous Coward · · Score: 1, Interesting

    You sir are incorrect. AC is dangerous because the alternating current causes you to grab tight on an object unable to let go. DC is far less dangerous, because it literally causes an explosion that knocks you free of contact with the source. We just went through this in lighting class at my school, and were discussing this as part of why major studios use all DC lighting, its far safer than AC lighting if there is a short.

  16. Re:What about houses? by evilviper · · Score: 2, Interesting
    The only minor problem is that DC is somewhat more dangerous than AC

    Yeah, Electric Chairs used AC power because it's LESS DANGEROUS, right?
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  17. Re:DC can be really annoying... by Myself · · Score: 2, Interesting

    You should look into the Anderson Powerpole plugs. They're made for DC, and can be assembled in a few ways to prevent accidental mixing of voltages. They've become a standard in amateur radio and emergency communications circles, for moving 12v around without the problems of large busbars or cigarette lighter sockets.

    The world could use a set of powerpole orientation and color standards. Hmm.

  18. Re:DC power outlets by Ignignot · · Score: 2, Interesting

    I wonder how much energy a household could save if we were all using DC outlets instead of AC.

    Well, none. Assuming you still have a washing machine, dryer, refrigerator, (possibly) electric heating, (possibly) electric oven, hair dryer, (possibly) electric water heater, and air conditioning, which in total consumes almost all of the power that is used in your house, and all of which is more efficient in AC. DC would benefit your electronics, which make up a much smaller portion of the total electric consumption. Some things can swing either way, but as a whole, your energy efficiency is much higher with AC than with DC. AC is much better at running motors than DC is. The real issue that AC to DC conversion has with respect to efficiency is that it produces harmonics on the power lines, which are undesirable and require some damping. The more AC to DC we use, the worse the harmonics are, and the more expensive it is to dampen them. But that's a really really small issue, in comparison to switching everything to DC.

    --
    I submitted this story last night, and it didn't get posted.
  19. Re:DC power for data centers by Anonymous Coward · · Score: 1, Interesting
    I don't really think a data center needs to hire a "Power Engineer"...

    A typical electrician knows almost nothing about DC power.
    This may be changing with the increase of large UPS installations but you are better off with someone that specializes in DC.

    Joe and Son's Electrical may have no problem installing a new breaker panel in your basement but don't let them try to run -48v to your server room.

    I speak from experience.

  20. Re:What about houses? by nido · · Score: 3, Interesting
    "Since when was non-ionizing EM radiation dangerous?"
    It isn't.

    don't be so sure... Living bodies are incredibly complex electrical devices. I think it arrogant to assume artificial electric fields cannot have an effect on their proper operation.

    Now if you look at the question of: Are there biological effects, the engineers and the physicists say absolutely not. Their view in general of what living systems consist of is that the cells are little plastic bags filled with minestrone soup. And you can then with that sort of a concept calculate the field strength and the frequencies you would need to produce an effect on the minestrone soup. And this is exactly the concept that was employed after it became apparent that radar systems could heat up the human body. The physicists that were involved in answering the question: Are there effects? And at what level do they occur? And what would be a safe level? Basically, they followed a basic precept which was to consider a spherical cow, a circular oval object filled with conducting solution and composed of a skin that is transparent to the radio frequency waves that microwave generators produce. And on that basis, they asked: How much does it take to heat this up? Where does the cow's temperature start to rise?

    And that number was calculated and confirmed in actual procedures in the lab using the spherical cow concept. They said, "OK, that's the number at which you are going to start heating people. Let's say that's not such a good idea and we'll set a level ten times lower as the safe levels."

    That level was applied for several decades to everything that concerned electromagnetic pollution. Of course, this is not correct. Any biologist can tell you that the body is much more complicated than that and the work I had done up to that point had involved the body's actual use of electric currents generated in the body that regulated certain things like healing. Wound healing is associated with a rather specific electrical current and voltage. So, the premise that was applied by the physicists and the engineers was erroneous from the start.

    That's number one. Number two, what would be the normal electromagnetic environment assuming that we're starting from scratch at Edison's time - and not Edison either because he went to DC current to light the light bulb. It was Nikolai Tesla who conceived of the system we presently use and who, incidentally, gets no credit for it: the 60 second electromagnetic field that is carried by power lines, the big lines that are strung across the country, and provides the current that comes into your home and appears in the wall socket and you use to run the coffee maker and the TV and all the rest of the things in the house 60 cycles. That didn't even exist one hundred years ago.
    (emphasis added) From an interview of Robert O. Becker, M. D., who was a pioneer in the study of natural electrical currents in the human body.

    --
    Learn the rules so you know how to break them properly.
    www.teslabox.com
  21. Re:specialized engineers? by jcaplan · · Score: 2, Interesting

    Well it looks like you've only started to answer the important question: What do I have to do to vaporize the screwdriver? Here's my stab at the problem:

    Boiling point of steel: 2500 C
    Specific heat of steel: 0.11 Kcal / Kg C
    Weight of metal in screwdriver: 0.1 Kg
    Needed change in temperature: boiling point - room temp = 2500 -25 = 2475 C

    Q = c*m *( delta T)

    where Q is heat added, c is specific heat and delta T is change in temperature.
    So, Q = 0.11Kcal/Kg/C * 0.1 Kg * 2475 C
    and a bit of arithmetic leads us to:
    Energy needed to bring to a rolling boil, Q= 27.225 Kcal

    A little bird (OK an online unit converter) told me that this is about 114,000 wattseconds

    Now we're getting somewhere...

    Lets assume a contact time of 0.1 s. (This is clearly a maximum, since we don't want the disappointment of the screwdriver melting and losing contact) Our power is now 1140 kilowatts- or approximately 1000 toasters worth, or, perhaps 10,000 computers which convinces me that the errors that you fellow Slashdotters have found in my math above are neatly canceling out.

    To conclude,

    Power = V * I (where I is current)
    so,
    I = Power /V

    Since we're using 5V
    I = Power / 5 V
    I = 1140 kilowatts / 5 V
    I = 228000 Amps

    So, the proposed 300 Amp power supply is off by a factor of 750 or so. We'll all have to head to one of Google's data centers to try this one out. Anyone out there able to arrange a guest tour?

    -Jon

    PS I know that there is some heat of vaporization to consider, as well, but that will be left as an exercise for the Slashdotter.

  22. Re:Saving a conversion step isn't the issue. by fred+fleenblat · · Score: 2, Interesting

    You almost seem to be arguing for power distribution to be higher frequency AC, so that the DC conversion componenents can be small, cheap and localized to the point of use. If the HF AC is well chosen for voltage and frequency then some of the intermediate conversion steps can be skipped as it will be "ready" for step down and rectification.

    50/60Hz originally was chosen because the low frequencies were easier to generate with the mechanical equipment available 100 years ago but also to avoid a lot of inductive loss into every bit of ferrous metal along the right of way; of course ridiculously high frequencies would actually radiate power away.

    I think this could actually work if the power cables were balanced and jacketed at all well, and the voltage was high enough (24v) to carry w/o huge cables. The frequency should probably be whatever switching PS's use these days, ~30Khz?

    Also I wonder if there is an increased electrocution risk from HF vs 60Hz.

  23. Re:Some clues on power distribution by IvyKing · · Score: 2, Interesting
    Arguably, power should be 3-phase down to the point where it's rectified to DC, because 3-phase rectifiers need far less filtering, but nobody does this for small loads.

    Especially if you can swing a 12-pulse rectifier - which gives much smoother DC and less harmonics on the AC side. This is becoming less of an issue with PFC SMPS's.

    Typically, when you get to the computer, there's a conversion from the line voltage (120-240VAC, 48VDC, etc) to internal distribution voltages of 5-12VDC, then another conversion and regulation just before each device, usually downward to something like 3.3VDC.

    The big reason for multiple conversions is that the latest and greatest chips use ~100W of power at a bit over 1V - so you 100A from the power supply - there is no way that you can transmit that much power at that low a voltage for more than a few millimeters with standard PC board traces. The solution is to use a buck converter right next to the CPU - and these converters can be very efficient (they have to be to stay cheap).

    What a lot of people are missing the baot on wrt DC/DC vs AC/DC conversion is that it is easier to get high efficiency with low voltage parts. The Rdson of MOSFET's rises rapidly when the Vds rating goes above 60V, silicon Schottky diodes are only good up to 200V (although SiC Schottkys are starting to show up). The fact that the input voltage is more or less constant means a lot less filtering and no need for PFC.