Slashdot Mirror
Google Reveals "Secret" Server Designs
Hugh Pickens writes "Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM or Sun Microsystems, but Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. For the first time, Google revealed the hardware at the core of its Internet might at a conference this week about data center efficiency. Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. 'This is much cheaper than huge centralized UPS,' says Google server designer Ben Jai. 'Therefore no wasted capacity.' Efficiency is a major financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: 'We were able to measure our actual usage to greater than 99.9 percent efficiency.' Google has patents on the built-in battery design, 'but I think we'd be willing to license them to vendors,' says Urs Hoelzle, Google's vice president of operations. Google has an obsessive focus on energy efficiency. 'Early on, there was an emphasis on the dollar per (search) query,' says Hoelzle. 'We were forced to focus. Revenue per query is very low.'"
From 2007, the modular data center patent (where the bottommost image of the article comes from). There's no lack of patents revealing piece by piece how their power management setup works.
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
My work here is dung.
The in-computer onboard UPS is not a new idea. I don't see how they could have gotten any patents on it since I used it have one of these (my day might still). The device I saw had a gel cell mounted on an 8-bit ISA card, full length. It had +5/12v pass through connectors for powering the drives and it powered the computer through the main bus. There was more logic to it, as it had some monitoring capabilities too.
What's next, patenting a hard drive on a plugin board? Been there, it was called the Hard Card and put a 20mb HDD in an 8 bit full length ISA slot, a truly neat idea for upgrading old XT computers back in the day. You could make them work with AT computers too by putting a regular disk controller, without a drive connected, on the bus too and the BIOS would see the XT controller and boot from it.
By some measurements they exceed the computing power of a mainframe, by others they don't.
look at the date the article was published.
"It would be wrong to refuse to face the fact that everything is fundamentally sick and sad."
Peter Huber in his book on energy policy introduces the concepts of the "energy pyramid" and "energy refining". The thesis that new forms of energy technology use more technology and are subsequently more useful. The pyramid levels include wood, coal, petroleum, electricity, computing and optical. When I read the book a few years ago I always found it curious that he included computing in the pyramid. But I hear about aggregate gigawatts of hundreds of mass server farms in the world, it may start making sense. The web has transformed human technology and the server farms are the battery of the web. When Huber wrote the book he used the example of the automobile as it started being mostly petroleum energy, then acquired more electricity sub systems, and now more computing.
A desert does not describe the temperature of a region but the (lack of) rainfall/moisture.
http://desertgardens.suite101.com/article.cfm/definition_of_a_desert (link found using Google).
And besides, put the containers underground and I'm pretty sure that "hot" you refer to becomes a non-issue as well.
They'd still have a computer there that is staggeringly efficient, especially since a computer's output energy is entirely heat - information is not energy, computers are all 0% efficient. Still, this isn't what they meant and the 99.9% figure probably comes from battery in/out figures.
Date Center Knowledge has videos of the secret server and a tour of one of the container data centers.
Actually, looking at the battery, ir looks like the same exact type of battery as you'd find in an APC small (450-800VA) UPS. We also used the same batteries for emergency power in our door access systems to power the controller when I was managing those at a small college. That type of battery is widely used to compensate for short term power outages.
I presume, given the amount of hardware shown (2 drives, 2 processors, motherboard, RAM) that the battery would probably last that given system about 7-10 minutes... plenty of time for the electric system to failover to the generator farm (you know they have more than 2 for redundancy.
As to the lifetime on those batteries... I was replacing them every 3-3.5 years, maybe 4 if I was lucky. It's a standard generic battery, and the failure rate on them is quite low.
I'd echo another user... If Google wanted to be smart, they wouldn't bother repairing a server when a component fails. Server obselescence at a company that can afford it is about 3-4 years... pretty close to the time for these batteries. They'd probably just pull the main power on it, and when a threshold of servers is "dead" in the container, they pull it offline for renovation... Either to repair the bad servers, or just retire everything.
Probably because they own these datacenters and can do what they want with them. The EM emissions are probably contained by the fact that the servers are all in a giant metal box. UL is optional, and if they don't want to go through it they don't have to. It's not like they're selling these servers to anyone.
I read the internet for the articles.
then you need to move your offices to the middle of a desert. Space problem solved :)
SMEs often get themselves a small server room, and don't plan for expansion. When the time comes to stick more servers in, they usually have to put them in an office instead, with non-redundant power, little cooling. You're not alone there, but it doesn't necessarily apply to datacentres.
Space at datacentres is often the least of their worries nowadays, (it used to be different), but power is the big problem. Even the DCs in the middle of the metropolis has enough space to fit a few servers, but they can't get the power to them if they did.
The article says that they use special motherboards that require 12V only, which is what the batteries put out. No conversion needed.
I don't think they are greener, but cheaper? Maybe. As for reliability, You have to keep in mind the whole map reduce framework is built around the idea that anything at anytime could fail. The compensate in numbers of servers and software for the lack of reliability of each one. No not every task or application is applicable to their set up. But, I believe them. I'm not into conspiracy theories.
Plus you also have to account for gradual scaling up & geographical distribution. Easy to do with additional low powered servers, difficult to do with giant expensive mainframes.
Well.. maybe. Or Maybe not. But Definitely not sort of.
You're Doing It Wrong(tm). A sealed cell will only vent hydrogen if overcharged (at the cost of increasingly reduced cell capacity - you're not filling it back up with water!). An intelligent charger will eliminate any routine hydrogen venting, leaving only the occasional bad battery or battery hooked to a broken charger venting. Google is probably OK with that.
A 12v battery. I never knew DC was more efficient than AC! WOW GOOGLE IS SO COOL!
I'm not sure if you are being sarcastic. A 12v battery in the power supply is more efficient than taking DC -> AC -> DC. That is what a UPS does, each conversion introduces loss. Having the battery in the power supply means there is no conversion so less power loss.
Dude... UPS. If you're using the battery, you don't *have* AC.
You do know that a UPS (for the kinds of servers Google is powering) puts out AC power, right?
Oh, you didn't?
A switch mode PSU takes AC, converts it to DC, switches it at a high frequency and then filters it back to DC at each rail voltage. They have obviously modded this PSU so that it can take DC directly in at a much lower voltage and still work so the PSU and UPS are combined. I find this neat.
The whole server is redundant.
See my journal, I write things there
You forget that fault tolerance is not of utmost importance to Google. I read an article somewhere that said, in essence, that since these are search results, and not financial transactions it is okay if some parts of the overall network don't know everything that every network knows. Having access to 95% (or 99%) of the data is still acceptable in the search world.
There are voltage regulators that can drop/boost the voltage to a predetermined voltage and do so with 90+% efficiency. Look for 'buckboost regulator' or 'switching regulator'.
Tom.
"If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth."
no it doesn't.
Plus they are cheaper to maintain, require less power per cycle, require less square feet to house.
Yeah, I actually know about these things.
The Kruger Dunning explains most post on
When I worked for a University, we bought a few of the largest IBM pSeries machines (power4 at the time). These were powerhouse machines 5 years ago. Each one had a dedicated 24" oversized rack cabinet, and then we had a couple racks just for disk. The 4 machines, and about 40T of Fibre channel disk (or was it DASD), I think it was a total of 128 core and 256GB of ram. I think we paid about a million for that setup.
As was mentioned elsewhere on the webs, the machine shown off by Google was based on Nocona CPUs.. those are atleast 4 years old now. Not likely what they're buying new now.
I bet you could get a base z10 for a few hundred thousand, but a fully loaded one? With a disk array of 750 drives? I bet 4 racks of disk from IBM would cost most of that 950k budget.
Found the patent:
Application number: 11/756,939
Publication number: US 2008/0030078 A1
Filing date: Jun 1, 2007
Inventors: William Whitted, Montgomery Sykora, Ken Krieger, Benchiao Jai, William Hamburgen, Jimmy Clidaras, Donald L. Beaty, Gerald Aigner
Assignee: Exaflop LLC [Note that Exaflop LLC's mailing address is the same as Google's.]
U.S. Classification
307066000
The most interesting parts, to me:
Figure 1, which shows an AC-DC converter, a battery, and a motherboard on a tray, in parallel.
And the following excerpts:
[0013] The system can further include a charger configured to charge the battery through a path connected across the DC bus. In some implementations, the single DC bus voltage is less than about 26 Volts. In some implementations, the single DC bus voltage is between about 10 Volts and about 15 Volts. In some implementations, the single DC bus voltage is about 13.65 Volts. In some implementations, the AC-to-DC conversion circuit regulates the DC output voltage signal to approximately 1 Volt above the maximum nominal charge voltage of the battery. The DC bus voltage can provide sufficient voltage for a linear regulator connected in series with the battery across the DC bus to trickle charge the battery to a fully charged state according to battery specifications.[...]
and this gem:
[0016] The system can further include at least one DC-DC converter configured to convert a voltage supplied on the DC bus to at least one additional DC voltage. In some implementations, the additional voltage is selected from the group consisting of: -5; 1; 3; 3.3; 5; 7.5; 10; about 18-20; and, about 20-26 Volts.
That last one, 16, is pretty specific: It basically comes out and says that there is no secondary regulation to 12V.
And so, I rest my case and declare that it is, indeed, a brilliant and simple design.
Kid-proof tablet..
Actually google does everything thrice (not unlike the Ramans). And returns the result that reaches it first. So in effect it is even more fault tolerant than the Mainframe. And it does them at different locations not on a single Facility (as opposed to a server or a 1AAA sized Container).
You are underestimating Google.