Slashdot Mirror
Facebook Putting Batteries On-Board Its Servers
1sockchuck writes "The data center of the future may have no central UPS units, and be filled with servers with on-board batteries. Facebook says it will adopt a new power distribution design that shifts the UPS and battery backup functions from the data center into the cabinet by adding a 12-volt battery to each server power supply, an approach pioneered by Google. Facebook says the move will slash its power bill and save millions in capital expenses on UPS systems and PDUs. Facebook acknowledged that these types of custom designs are limited to large companies, but called on server vendors and data center builders to adapt their offerings to make them available to smaller companies."
From the article: "Facebook's new distribution scheme calls for 277 volt power to the servers. "We're working with power supply vendors to create a (server) power supply that will accept 277 volts on the input," said Michael."
Why 277 volts?
How long is twelve volts going to keep a server running? A UPS would guarantee that you have enough time to finish transfers and close connections before shutting down into a safe mode, even give clients a warning before shutting down.
I'm imagining an A23 battery keeping a computer running for about 30 seconds, basically long enough for it to go "SHUTTING DOWN NOW KTHXBAI" and all your clients go WTF?
This right after they announced they were going public. This will definitely boost their stock price.
I think facebook has some good merit, however I am an avid anti facebook poster boy when it comes to destroying
relationships. If you thought it was easy before to hook up online and cheat on your spouse, well imagine now!!!
Girlfriend: "Hey, i got over 5000 friends online, isn't it wonderful???"
Boyfriend: "ummm...why are they all pictures of young good looking dudes wearing no shirts....?"
Girlfriend: "well that's because you can put anything on there..."
Boyfriend: "how long have you know these so called friends?"
Girlfriend: "Well not long, most are people that invited me to be their friend"
Boyfriend: "Why is this guy keep writing on your wall how hot you are....I am not sure i really like this..."
Girlfriend: "You always make such a big deal about nothing..."
What would the world be like if facebook went offline... I'm not sure I could continue living.
Only the State obtains its revenue by coercion. - Murray Rothbard
As a rough estimate I would say mission critical servers get changed out every 3, maybe 4 years. I would imagine any cells would need to be at least laptop battery sized to run the server for an appreciable period of time, so what is going to happen when a server gets replaced? Keep the battery? I don't think so.
Since batteries have to be replaced every few years that will not be any fun taking those servers out of the racks one by one to replace their batteries. One would hope they'd be changeable from the back or front, but I wouldn't bet on it.
Also, UPSs can be retained when you buy upgraded servers. But if it's built in, you get to buy it again.
And capacity? You can't just get a bigger UPS to run longer on battery. Although if you have true external (genny) power you just need something to hold for the cutover blip I suppose. I'd be in agreement that any server could benefit from an internal UPS that could hold it for say, 10 seconds.
Not a good idea on several fronts.
I work for the Department of Redundancy Department.
I wish George W. Bush would stop trolling Facebook.
Whats stupid about putting batteries inside servers?
The problem with mesh computing is that it doesn't save in energy costs. With a centralized UPS and power supply, improving efficiency requires that you upgrade one unit. This way, you have to upgrade a few hundred units. It's similar to why moving to electric cars is advocated despite their limited range and low performance: Because it's easier to upgrade a dozen power plants than a few hundred thousand cars, to take advantage of the latest technology.
#fuckbeta #iamslashdot #dicemustdie
The Voltage is not the problem, it is the Ampere Hours(AH). It can be achieved but there is a hidden problem of replacing the batteries after their Charge/Discharge cycle makes them ready for replacement. Say in about 3-4 Years, Facebook will die because of their Servers crashing out, one by one, if Battery condition is not monitored? Google may help, i definitely can.
This is interesting to me in a couple of ways.
The idea is that it is cheaper to have just a battery instead of a UPS. A UPS will also have to have an inverter.
Okay I can see this but they why have it at the server level?
Remove the power supply from the server and put it at the rack level? Have a big redundant power supply for each rack and batteries for each rack?
Or why not use DC for the entire data center and put the battery at the Data Center level?
Seems to me that there may be more than one way to skin this cat and each have it's pluses. If you are using a large number of low load balanced servers where having any one go down isn't a disaster then putting the battery on the server would give you a good trade off. You are probably more likely to have a single server to fail than a more centralized system would but the odds of taking down the system would be tiny.
I would love to see a study of the benefits of each type of system with the trade offs.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
I've found facebook chat relatively stable. Then again, I use it via Pidgin more often than not, rather then through FB itself, so maybe the problems you are seeing symptoms of lie in the client end. Try Pidgin's FB plugin )or other IM clients that have one) and see if you have any more luck.
Distributed!
Centralized!
Distributed!
Centralized!
Distr...
ad infinitum
That they anticipate frequent disruptions in the electrical grid?
That they regard their service as some kind of public utility?
That they think they are Google?
Facebook says the move will slash its power bill and save millions in capital expenses on UPS systems and PDUs.
And it'll move the complexity and unreliability to the server. The whole idea behind centralized UPS's (and by the way, you still need PDUs) is that you have reliability, serviceability, and economies of scale and efficiency. Now you have to monitor and service the batteries in thousands of pieces of equipment. And guess what happens when one of those batteries fails by getting cooked? Sulfuric acid all over the place (yes, even the "sealed" lead acid batteries can fail and leak) instead of the batteries being in, say, a battery room. God help us if they use lithium-ion, which would introduce us to a world of server fires and water damage, since a lot of datacenters are now dry-pipe to save costs. Nevermind that batteries and their associated electronics take up space, and that space has to come from somewhere.
So, now you have each server getting more expensive, more complex with both hardware and software (server now needs its own battery power management) heavier, bigger, featuring toxic materials, and now non-standard, non-commodity design which vendors will charge more for as they specialize the equipment.
I'm sure this all looks great on a powerpoint slide simplified into "if we put batteries in our servers, we can throw out our expensive UPS and save money!" This is just another hot/stupid trend; just because Google is doing it, doesn't make it brilliant. I stopped believing everything google was doing was a Best Practice around the same time gmail started going down for hours (and for some users, more than a day) at a time on a regular basis.
I tuned out of the article around the point where the guy from Facebook complains about cosmetic features interfering with airflow. Uh, guess what, bud? Dell's pretty front panel has been optional (saving you a few bucks sometimes) for years.
Please help metamoderate.
...this makes sense. If you distribute the UPS responsibility across all the servers, then a single failure will only take down one server, rather than a whole block.
On the other hand, they're probably going to need to hire a full-time person eventually just to go around and change all the batteries. They do go bad eventually and need to be changed.
(As for the people wondering why 12 volts - your computer at home may be plugged into a 110 or 220V household circuit, but the CPU, and everything else on the MoBo, doesn't use anywhere near as much.)
Modding "-1, Troll" is not a proper response if you disagree with me. Try reason.
I told the APC CEO this at a dinner about 7 or 8 years ago. This could have been a significant part of their wildly successful NetShelter product line, but no... too smart for their own good, those APC engineers.
Most servers sit there at 3% cpu until something strenuous occurs. You've still got the big fans blowing, drives spinning like mad, and lots of power getting sucked down. I'd like to be able to see these units able to reduce power in low-use times and seamlessly ramp up when demand hits. It bugs me that we leave servers running overnight at full clip simply because we don't want to come in early to turn them on for the early workers, don't trust they'll come back on after a powerdown due to IT voodoo, etc. It really drives me nuts when IT policy says desktops are to remain on overnight for patching at 3am. Waste of power.
Kwisatz Haderach
Sell the spice to CHOAM
This Mahdi took Shaddam's Throne
Facebook is also converting over to solid state drives. They will have relatively low power consumption per board. Putting both flash chips and a backup battery on each board makes sense.
How long is twelve volts going to keep a server running? A UPS would guarantee that you have enough time to finish transfers and close connections before shutting down into a safe mode, even give clients a warning before shutting down.
Volts are a measure of electrical potential, not capacity. You mean watt-hours, most likely.
How long is twelve volts going to keep a server running? A UPS would guarantee that you have enough time to finish transfers and close connections before shutting down into a safe mode, even give clients a warning before shutting down.
That depends on how big the UPS is. Many large-scale datacenter UPS's are only designed to ride out the time between when the power goes out and the generator is warmed up enough to take the load (tolerances are not 'right' when the engine is cold, so damage is caused, or the engine won't provide rated output.)
The argument is that in a "cloud", none of this matters- you only need to ride out a temporary power outage, or allow the machine to shut down properly so it doesn't have to be repaired software-wise. However, for the rest of us who don't live in poofy clouds and have non-cloud things like mail servers, yeah, you're right- the capacity in a server is pretty low.
Please help metamoderate.
This is idiotic. Why not just run the equipment on 48VDC (telco style) with extremely high efficiency DC SMPSes and heavy gauge wiring? Power that directly with a large bank of batteries.. no inverter.. no distributed battery mess (pressure discs do burst).. no capacity limits.. no server weight issues..
Not to mention, under this scheme you can no longer fully de-energize the datacenter (or parts of it). An EPO switch could cut the mains, but I certainly wouldn't want thousands of fully-charged batteries helping to fuel a fire.
And how do you measure the health of thousands of batteries? Automated transfer discharge tests? Sounds like a recipe for frequent node failures. A centralized power system can perform offline load tests, split-bank tests, etc. so you know how your batteries will perform when you need them.
And finally, when LIPO starts to become economical, a centralized battery system is a quick, easy, online replacement. Not necessarily so in this scheme (charge/discharge controllers are different).
I suppose there must be some advantage to this, but I can't see it. It feels like some non-technical VP went to a distributed computing conference and levied an ill researched directive on the company.
A government is a body of people notably ungoverned - AC
I used to work at a company that decided to install large, monolithic UPS systems after the power company hit them with a spike that took the entire system down for over a half hour. As they're a broadcasting company, they (rightly) felt that feeding their network affiliates nothing was not a good idea.
As a result, they have these UPS "rooms" that hum like the dickens when you're passing them in the hall, all with batteries that will need to be replaced regularly (just like the Google server battery systems, so it's the same problem no matter what). Reason for the hum?
The hum is caused by these giant transformers that step the power from DC to AC and create 110 volts of AC current at whatever amperage is required for normal devices. But there is a lot of wasted energy in doing that.
Computers and servers all run off of DC power. They plug into AC power and then run that AC through a "power supply" that converts that to DC that the computer can use. That takes power, but power is plentiful when it comes from the power company and you pay your bill on time. But when you take the power from the power company, then change it to DC to charge batteries and then take power from those batteries to change it to AC to power normal wall outlets only to take that through a server's power supply to change it to DC again for the computer to use it, you're looking at lots of wasted energy in just changing from AC to DC, back and then back again, as well as changing to the kind of voltage and amperage needed to run the microprocessor, power the memory and power the drive arrays.
So this is all about lowering consumption. And if you lower consumption, you lower your electricity costs.
The hobbyist magazines were all aflutter some years ago about using photovoltaic (solar) energy to power a house. But what everyone had to do (early on) was to change their appliances (or order special ones) to run on DC -- not because you couldn't make AC current from the DC output of the photovoltaic systems but because it took a lot of energy to do that and these hobbyists were all about trying to save so much energy that they could take themselves off the grid.
Here, on a large scale, you see the same idea. It's just more efficient to do this. And one of the big arguments in the early years of electrification was between DC power distribution (Thomas Edison) and AC power distribution (George Westinghouse and Nikola Tesla). We may wind up fighting these battles again in the near future.
Gods don't kill people, people with gods kill people.
Don't they require power too? It's all very well keeping your server up in the event of a power failure but unless you keep your routers (and the routers all the way to the backbone) up, what's the point?
IranAir Flight 655 never forget!
I for one would be very interested in a standard for consumer UPSes that has them output 12v DC, and an ATX (or BTX) motherboard extension that allows it to take 12v DC in for its power needs. Failing that, a DC-DC power supply could be used.
The point being that it's dumb that a UPS has to invert the power coming out of it just so the power supply can rectify it back to DC. I'd much prefer saving the step and running DC straight from the UPS to the motherboard.
Come to think of it, the standard isn't necessary, a UPS manufacturer could do this today, although they would have to bundle the dummy power supply with the UPS. The cost could even be kept somewhat reasonable if you factor in the savings from not having to buy a power supply. The only major sticking point is that most UPS vendors put out a lot of distressingly bad products and the consumer trust issue will be a problem.
I read the internet for the articles.
I remember seeing a power supply around 2004 that had one or more backup batteries that fit in trays in 5.25" drivebays so you could hot-swap them and they were internal to the server. A SOHO or retail server (for a handful of POS' ) with this and a couple of PCI multiport ethernet cards and a PCI docsis or DSL modem would do a lot to consolidate the IT equiplent and all it's power bricks and interconnections. Sadly I've not been able to find that type of power supply since.
Yes, computers operate on DC.
Yes, putting a DC battery in between the DC output of the PSU and the DC input of the mobo will have a UPS / Laptop battery effect on temporary mains voltage loss.
The problem is very few mobos only have 12VDC input. This won't take care of the vast majority of mobos that also require 5VDC
5VDC isn't practicably doable from lead-acid @ around 2 VDC per cell.
Yes, there are ways around this, but the only practical ones are external DC-DC conversion, *or* 12VDC only mobos, which use on board internal DC-DC concersion.
ALL mobos use some form of onboard DC-DC conversion, even if only for the CPU.
If you want to buy a 12VDC mobo, do a google for industrial computer mobos.
http://slashdot.org/~GuyFawkes/journal
...that the central switch that goes out of the net, and some other small devices, have no internal UPS... ^^
No use having all the servers still running, if the “glue” between them dies anyway.
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Another idea behind a UPS is _a_single_point_of_failure_. Moving the power backups to the individual servers eliminates that worry.
Large-scale UPS systems are generally built with excessive amounts of redundancy.
Plus, since the servers are already redundant, you don't need the redundancy on the UPS, inverters, etc., which should save money.
You're assuming the servers are redundant, and that there is zero potential for lost data. Even with that fancy software we saw a week or two ago, it was discussed how there was still the possibility for lost transactional data.
And since it's long-term, I'm willing to wager it won't be lead-acid, but NiMH. So no real maintenance issues.
Google is using lead-acid.
. And your "what happens if..." scenarios apply equally to a battery in a megawatt UPS or a battery in a server.
Actually, they don't apply even remotely equally: location, location, location. If you have a battery fire or leak in your UPS room, it doesn't affect your servers, and the UPS room is designed through fire code / standards to tolerate all this. Plus, a battery room is vented to deal with normal off-gassing from the batteries (or fumes if a battery fails.) If a battery in a server pisses sulfuric acid, you now have a big problem on your hands- corrosive liquid and vapor. Want to guess what sulfuric acid vapor does for MTBF's for fans and hard drives? Want to guess how much of a pain in the ass it is for staff if they have to respond to a toxic waste spill, not just a server failure?
As for battery management and 'specialized' power supplies, etc.: go check out a laptop. That wheel has already been invented, and better yet, has benefitted from mass-production.
Servers don't currently have batteries, chargers, etc in them. Adding them in affects reliability of the overall component, especially if you've got a new design on your hands. Power systems for laptops are not designed anything like servers- they have big differences in requirements.
Also, you're not understanding what 'specialized' means. A power supply for a server with a battery in it might have an extra cable or connector, as an example. That's enough to make that particular power supply more expensive, if only slightly. Now figure in that you now need the chassis modified to hold a battery (and they only come in so many sizes- if you want a custom size, $$$$), you need to place a charger/power controller circuit somewhere...so the chassis is completely different. That costs money. Why? Because of the cost of the extra components, the design is 'custom', the manufacturers make them in lower volume, and they're distributed differently. Anyone who does their own car repairs knows this well- if your car uses a component few other cars do, it's going to cost more, and probably take longer to arrive.
Shall we also talk about the number of laptops that catch fire every year? Servers generally don't have much in the way of highly reactive materials in them.
Please help metamoderate.
My Farmville will be much more efficient!
Not exactly sure what this will do to my vampire clan, though. Hmm...more energy?
That's nothing new. I've seen these kind of developments for years. There is even an extended Wikipedia article on it: http://en.wikipedia.org/wiki/Laptop
That saved me a few times, by using my laptop when power went down, I was able to
[network connection lost, router is down]
Run the servers on 240 (VAC) / 280 (VDC). Almost every computer power supply converts the incoming power 120/240 (VAC) into a 320 (V) or so internal bus via a voltage doubler for 120 (V), and a bridge rectifier circuit at 208-240 (V). This means with NO MODIFICATIONS you can power almost every single server on the market from 280 (VDC), with no problems. Additionally, you can use a few bridge rectifiers, which are very common and inexpensive parts, to automatically select from a 240 (VAC) line and a 280 (VDC) DC bus. This completely eliminates the DC to AC conversion in the UPS.
Why is this not done? This will not work if any load in the system expects AC, like every Air Conditioning Unit, every AC motor, and every "regular" transformer. Most consumers can't tell which loads are DC capable, and as such, the safety standards authorities severely discouraged UPS manufacturers from outputting DC waveforms or "almost" DC / square wave outputs. However, if one had the ability to specify which power supplies one was purchasing, one could run everything on 280 (VDC) for backup and use completely standard computer parts.
... already do this. There are power supplies available in various PC and rack form factors designed to run off of the 48Vdc CO battery.
One thing to consider (and I'm sure Google and others have worked the economics) is the maintenance costs of a centralized battery bank vs distributed batteries. Batteries don't last forever and some types require periodic attention (topping off the electrolyte in lead acid cells comes to mind). Although monitoring functions have become increasingly automated, someone still has to weight the costs of chasing hundreds or thousands of individual batteries around in a data center vs a couple of centralized battery rooms.
One great example of battery maintenance comes to mind: near my house, Comcast has a pole-mounted battery cabinet that I drive by a couple of times a week. Its on a main arterial, so the probability that service personnel are passing by it frequently should be pretty high. For about 6 months, I've noticed that this one box has a red warning light flashing. Distribute batteries all over the place and make their maintenance a small part of some technician's job description and some are sure to be missed.
Have gnu, will travel.
This has been common place on IBM equipment for years.
That last machine had batteries on each of i/o controllers (RAID) with dual memory cards so that card system can take a "hit" and even half the i/o controller could go, and you can still change the I/O controller and KEEP the data in transaction.
The cabinet itself had dual power supplies with batteries, each board used dual power from dual supplies. The cabinet took two 30A 480V 3-phase feeds, and supplied 400V DC to the boards, which then dropped the power down need rates.
My company is in the process of designing our new/next data center and we're highly considering a non-battery based UPS solution from Active Power. It's the old 'fly wheel' technology of the past, upgraded to the 21st century. The idea is that the mechanical flywheel can sustain the load for 10-14seconds while the generator kicks on. The theory is that if your generators can't start up and switch over within 10-14 seconds (because of a failure), then they're probably not going to start up/switchover within 45mins--the usually battery backup time in a data center. Some people are using this technology to supplement their traditional battery arrays and make them last longer. The jury is still out on the solution, but looks promising.
I hope facebook's lawyers know that Google has a patent application for this very idea. TFA didn't mention whether or not they are actually licensing these and other Google-patented techniques.
How many betteries will they need and what is the ecological impact or recycle strategy and life span of said batteries ?? It seems that this might save in the short term but be a real mess in the long term.
errr....umm...*whooosh* *whoosh* Is this thing on ?
In other news, Energizer announces a purchase of 20% of Facebook's outstanding stock.
Common voltage regulator gives you clean 5VDC from 12VDC-20VDC inputs.
New Economic Perspectives
http://www.guardian.co.uk/technology/2008/jan/14/facebook
Half the time I use an UPS to filter out power spikes. Not sure these onboard batteries would perform that trick as well as the big hunk located on the bottom of my racks ...
Obviously notebook CPUs are not as powerful as desktop systems but so what? If people really care about raw processing for specific jobs they wouldn't be using commodity processors on such a large scale. Your really doing it because your problems are such that you need/can easily distribute storage and memory.
The hum is caused by these giant transformers that step the power from DC to AC and create 110 volts of AC current at whatever amperage is required for normal devices. But there is a lot of wasted energy in doing that.
That is not the function of transformers. Perhaps you're thinking of an inverter.
If they're going to redesign the PSU and put it together again, maybe it's time to drop 12V. What components still require 12V other than optical drives, 3.5inch HDDs, and high end video cards? Those are all pretty much optional in servers like FB's.
Actually this is quite common in China as they have a shitty electrical grid which frequently have brown outs and government mandated power cuts. Each desktop PC has a lead acid battery inside. Just gives you enough juice to save your stuff and power down.
Where does the EPO button go?
That may be true in your case, however, there's no reason for AC/DC conversion to be inefficient. Shop around for AC inverters for solar installations and you'll find ~97% efficient units.
APC did a study on the efficiency of AC vs. DC as well, which shows a bare minimum of extra efficiency to be had:
http://www.apcmedia.com/salestools/SADE-5TNRLG_R5_EN.pdf
The reason this works with large companies has NOTHING to do with efficiency. It's the flexibility to add new servers, one-by-one, without having to upgrade a central UPS, or taking a number of systems offline when a UPS fails. It's the compartmentalization, combined with the unimportance of any one server in a distributed environment, that makes it work.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant