Slashdot Mirror
Google Reveals "Secret" Server Designs
Hugh Pickens writes "Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM or Sun Microsystems, but Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. For the first time, Google revealed the hardware at the core of its Internet might at a conference this week about data center efficiency. Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. 'This is much cheaper than huge centralized UPS,' says Google server designer Ben Jai. 'Therefore no wasted capacity.' Efficiency is a major financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: 'We were able to measure our actual usage to greater than 99.9 percent efficiency.' Google has patents on the built-in battery design, 'but I think we'd be willing to license them to vendors,' says Urs Hoelzle, Google's vice president of operations. Google has an obsessive focus on energy efficiency. 'Early on, there was an emphasis on the dollar per (search) query,' says Hoelzle. 'We were forced to focus. Revenue per query is very low.'"
From 2007, the modular data center patent (where the bottommost image of the article comes from). There's no lack of patents revealing piece by piece how their power management setup works.
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
My work here is dung.
The in-computer onboard UPS is not a new idea. I don't see how they could have gotten any patents on it since I used it have one of these (my day might still). The device I saw had a gel cell mounted on an 8-bit ISA card, full length. It had +5/12v pass through connectors for powering the drives and it powered the computer through the main bus. There was more logic to it, as it had some monitoring capabilities too.
What's next, patenting a hard drive on a plugin board? Been there, it was called the Hard Card and put a 20mb HDD in an 8 bit full length ISA slot, a truly neat idea for upgrading old XT computers back in the day. You could make them work with AT computers too by putting a regular disk controller, without a drive connected, on the bus too and the BIOS would see the XT controller and boot from it.
By some measurements they exceed the computing power of a mainframe, by others they don't.
look at the date the article was published.
"It would be wrong to refuse to face the fact that everything is fundamentally sick and sad."
A desert does not describe the temperature of a region but the (lack of) rainfall/moisture.
http://desertgardens.suite101.com/article.cfm/definition_of_a_desert (link found using Google).
And besides, put the containers underground and I'm pretty sure that "hot" you refer to becomes a non-issue as well.
They'd still have a computer there that is staggeringly efficient, especially since a computer's output energy is entirely heat - information is not energy, computers are all 0% efficient. Still, this isn't what they meant and the 99.9% figure probably comes from battery in/out figures.
Date Center Knowledge has videos of the secret server and a tour of one of the container data centers.
Actually, looking at the battery, ir looks like the same exact type of battery as you'd find in an APC small (450-800VA) UPS. We also used the same batteries for emergency power in our door access systems to power the controller when I was managing those at a small college. That type of battery is widely used to compensate for short term power outages.
I presume, given the amount of hardware shown (2 drives, 2 processors, motherboard, RAM) that the battery would probably last that given system about 7-10 minutes... plenty of time for the electric system to failover to the generator farm (you know they have more than 2 for redundancy.
As to the lifetime on those batteries... I was replacing them every 3-3.5 years, maybe 4 if I was lucky. It's a standard generic battery, and the failure rate on them is quite low.
I'd echo another user... If Google wanted to be smart, they wouldn't bother repairing a server when a component fails. Server obselescence at a company that can afford it is about 3-4 years... pretty close to the time for these batteries. They'd probably just pull the main power on it, and when a threshold of servers is "dead" in the container, they pull it offline for renovation... Either to repair the bad servers, or just retire everything.
Probably because they own these datacenters and can do what they want with them. The EM emissions are probably contained by the fact that the servers are all in a giant metal box. UL is optional, and if they don't want to go through it they don't have to. It's not like they're selling these servers to anyone.
I read the internet for the articles.
then you need to move your offices to the middle of a desert. Space problem solved :)
SMEs often get themselves a small server room, and don't plan for expansion. When the time comes to stick more servers in, they usually have to put them in an office instead, with non-redundant power, little cooling. You're not alone there, but it doesn't necessarily apply to datacentres.
Space at datacentres is often the least of their worries nowadays, (it used to be different), but power is the big problem. Even the DCs in the middle of the metropolis has enough space to fit a few servers, but they can't get the power to them if they did.
The article says that they use special motherboards that require 12V only, which is what the batteries put out. No conversion needed.
A 12v battery. I never knew DC was more efficient than AC! WOW GOOGLE IS SO COOL!
I'm not sure if you are being sarcastic. A 12v battery in the power supply is more efficient than taking DC -> AC -> DC. That is what a UPS does, each conversion introduces loss. Having the battery in the power supply means there is no conversion so less power loss.
You forget that fault tolerance is not of utmost importance to Google. I read an article somewhere that said, in essence, that since these are search results, and not financial transactions it is okay if some parts of the overall network don't know everything that every network knows. Having access to 95% (or 99%) of the data is still acceptable in the search world.
"If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth."
no it doesn't.
Plus they are cheaper to maintain, require less power per cycle, require less square feet to house.
Yeah, I actually know about these things.
The Kruger Dunning explains most post on
When I worked for a University, we bought a few of the largest IBM pSeries machines (power4 at the time). These were powerhouse machines 5 years ago. Each one had a dedicated 24" oversized rack cabinet, and then we had a couple racks just for disk. The 4 machines, and about 40T of Fibre channel disk (or was it DASD), I think it was a total of 128 core and 256GB of ram. I think we paid about a million for that setup.
As was mentioned elsewhere on the webs, the machine shown off by Google was based on Nocona CPUs.. those are atleast 4 years old now. Not likely what they're buying new now.
I bet you could get a base z10 for a few hundred thousand, but a fully loaded one? With a disk array of 750 drives? I bet 4 racks of disk from IBM would cost most of that 950k budget.