Facebook Putting Batteries On-Board Its Servers

1sockchuck writes "The data center of the future may have no central UPS units, and be filled with servers with on-board batteries. Facebook says it will adopt a new power distribution design that shifts the UPS and battery backup functions from the data center into the cabinet by adding a 12-volt battery to each server power supply, an approach pioneered by Google. Facebook says the move will slash its power bill and save millions in capital expenses on UPS systems and PDUs. Facebook acknowledged that these types of custom designs are limited to large companies, but called on server vendors and data center builders to adapt their offerings to make them available to smaller companies."

Comments

On board batteries fine, but 277 volt?

[TheLink]

From the article: "Facebook's new distribution scheme calls for 277 volt power to the servers. "We're working with power supply vendors to create a (server) power supply that will accept 277 volts on the input," said Michael."

Why 277 volts?

What about disposal?

[Turzyx]

As a rough estimate I would say mission critical servers get changed out every 3, maybe 4 years. I would imagine any cells would need to be at least laptop battery sized to run the server for an appreciable period of time, so what is going to happen when a server gets replaced? Keep the battery? I don't think so.

Re:What about disposal?

[temojen]

Those Pentium IIs are what gives me something to do... so many break/fix calls between them and the same company's over-complicated network.

My point was just that you can't assume everyone has the same refresh cycle, just like you shouldn't assume that all servers are in a datacentre, or that racks are only in a datacentre, or that all servers are on racks, etc. There's a lot of variety out there.

"mesh" thinking

[girlintraining]

The problem with mesh computing is that it doesn't save in energy costs. With a centralized UPS and power supply, improving efficiency requires that you upgrade one unit. This way, you have to upgrade a few hundred units. It's similar to why moving to electric cars is advocated despite their limited range and low performance: Because it's easier to upgrade a dozen power plants than a few hundred thousand cars, to take advantage of the latest technology.

The best solution?

[LWATCDR]

This is interesting to me in a couple of ways.
The idea is that it is cheaper to have just a battery instead of a UPS. A UPS will also have to have an inverter.
Okay I can see this but they why have it at the server level?
Remove the power supply from the server and put it at the rack level? Have a big redundant power supply for each rack and batteries for each rack?
Or why not use DC for the entire data center and put the battery at the Data Center level?
Seems to me that there may be more than one way to skin this cat and each have it's pluses. If you are using a large number of low load balanced servers where having any one go down isn't a disaster then putting the battery on the server would give you a good trade off. You are probably more likely to have a single server to fail than a more centralized system would but the odds of taking down the system would be tiny.
I would love to see a study of the benefits of each type of system with the trade offs.

Re:The best solution?

[hannson]

Google did it with good results

Re:The best solution?

[inKubus]

If you read the Google stuff, they have a number of reasons for it. Firstly, they wanted to minimize or eliminate conversion costs (Converting AC to DC and vice versa takes energy, sometimes 10-20% right off the top in heat. Secondly, they didn't want to have to do standard systems planning to deploy more capacity. With a central UPS, you have to worry about how many systems are plugged in, how many VA, etc. You have to do these calcuations and planning all the time with new hardware configurations. Google wants to be able to add capacity as fast as possible, so they mass produce a single computing "unit" that only needs power and network. All the costs are packaged into one unit. This minimizes everything from planning meetings to deployment patterns. If you have a given rack they run a certain amount of AC to it that will support x number of units. That's all the deployment engineer needs to know. The Google mainframe asks for more processors, the deployment person just loads up a rack and turns them on. Thirdly, the batteries and compute units are both on the same replacement cycle, so they will replace the entire unit at once, recycle the batteries, etc. Lastly, one of the largest costs in a generator set and UPS is the switching over to emergency power. Generators have to start up and come up to phase. This might not happen simultaneously for all the generators either. This could cause a huge brownout which would take everything out if say only one generator came on. Normally you would make sure everything is segmented but with the decentralized system you don't have to do that. The battery will make up for any shortage as well as a total loss. So your motherboard will have totally constant power no matter what.

Look for the google research on conversion losses, though. It's published out there somewhere.

I'm sure this looks great on Powerpoint

[SuperBanana]

Facebook says the move will slash its power bill and save millions in capital expenses on UPS systems and PDUs.

And it'll move the complexity and unreliability to the server. The whole idea behind centralized UPS's (and by the way, you still need PDUs) is that you have reliability, serviceability, and economies of scale and efficiency. Now you have to monitor and service the batteries in thousands of pieces of equipment. And guess what happens when one of those batteries fails by getting cooked? Sulfuric acid all over the place (yes, even the "sealed" lead acid batteries can fail and leak) instead of the batteries being in, say, a battery room. God help us if they use lithium-ion, which would introduce us to a world of server fires and water damage, since a lot of datacenters are now dry-pipe to save costs. Nevermind that batteries and their associated electronics take up space, and that space has to come from somewhere.

So, now you have each server getting more expensive, more complex with both hardware and software (server now needs its own battery power management) heavier, bigger, featuring toxic materials, and now non-standard, non-commodity design which vendors will charge more for as they specialize the equipment.

I'm sure this all looks great on a powerpoint slide simplified into "if we put batteries in our servers, we can throw out our expensive UPS and save money!" This is just another hot/stupid trend; just because Google is doing it, doesn't make it brilliant. I stopped believing everything google was doing was a Best Practice around the same time gmail started going down for hours (and for some users, more than a day) at a time on a regular basis.

I tuned out of the article around the point where the guy from Facebook complains about cosmetic features interfering with airflow. Uh, guess what, bud? Dell's pretty front panel has been optional (saving you a few bucks sometimes) for years.

Facebook is converting to solid state drives.

[Animats]

Facebook is also converting over to solid state drives. They will have relatively low power consumption per board. Putting both flash chips and a backup battery on each board makes sense.

Why is this limited to big institutions?

[jandrese]

I for one would be very interested in a standard for consumer UPSes that has them output 12v DC, and an ATX (or BTX) motherboard extension that allows it to take 12v DC in for its power needs. Failing that, a DC-DC power supply could be used.

The point being that it's dumb that a UPS has to invert the power coming out of it just so the power supply can rectify it back to DC. I'd much prefer saving the step and running DC straight from the UPS to the motherboard.

Come to think of it, the standard isn't necessary, a UPS manufacturer could do this today, although they would have to bundle the dummy power supply with the UPS. The cost could even be kept somewhat reasonable if you factor in the savings from not having to buy a power supply. The only major sticking point is that most UPS vendors put out a lot of distressingly bad products and the consumer trust issue will be a problem.