Slashdot Mirror
Google Reveals "Secret" Server Designs
Hugh Pickens writes "Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM or Sun Microsystems, but Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. For the first time, Google revealed the hardware at the core of its Internet might at a conference this week about data center efficiency. Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. 'This is much cheaper than huge centralized UPS,' says Google server designer Ben Jai. 'Therefore no wasted capacity.' Efficiency is a major financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: 'We were able to measure our actual usage to greater than 99.9 percent efficiency.' Google has patents on the built-in battery design, 'but I think we'd be willing to license them to vendors,' says Urs Hoelzle, Google's vice president of operations. Google has an obsessive focus on energy efficiency. 'Early on, there was an emphasis on the dollar per (search) query,' says Hoelzle. 'We were forced to focus. Revenue per query is very low.'"
I think Google may be selling themselves short. Once you start building standardized data centers in shipping containers with singular hookups between the container and the outside world, you've stopped building individual rack-mounted machines. Instead, you've begun building a much larger machine with thousands of networked components. In effect, Google is building the mainframes of the 21st century. No longer are we talking about dozens of mainboards hooked up via multi-gigabit backplanes. We're talking about complete computing elements wired up via a self-contained, high speed network with a combined computing power that far exceeds anything currently identified as a mainframe.
The industry needs to stop thinking of these systems as portable data centers, and start recognizing them for what they are: Incredibly advanced machines with massive, distributed computing power. And since high-end computing has been headed toward multiprocessing for some time now, the market is ripe for these sorts of solutions. It's not a "cloud". It's the new mainframe.
Javascript + Nintendo DSi = DSiCade
From 2007, the modular data center patent (where the bottommost image of the article comes from). There's no lack of patents revealing piece by piece how their power management setup works.
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
My work here is dung.
Get that man a beer.
We all know the searches are actually being done by a large amount of people in suspended animation, being fed the corpses of the previous people.
The thing about each server having its own battery is a cruel joke.
The in-computer onboard UPS is not a new idea. I don't see how they could have gotten any patents on it since I used it have one of these (my day might still). The device I saw had a gel cell mounted on an 8-bit ISA card, full length. It had +5/12v pass through connectors for powering the drives and it powered the computer through the main bus. There was more logic to it, as it had some monitoring capabilities too.
What's next, patenting a hard drive on a plugin board? Been there, it was called the Hard Card and put a 20mb HDD in an 8 bit full length ISA slot, a truly neat idea for upgrading old XT computers back in the day. You could make them work with AT computers too by putting a regular disk controller, without a drive connected, on the bus too and the BIOS would see the XT controller and boot from it.
Google claims they did the math and found it was cheaper with commodity hardware. I advise everyone else to do the same and run the calculations for themselves to determine the optimal hardware for their particular load. With out the specifics of their situation, its difficult to criticize in an intelligent fashion, other than a more generalized statement expressing surprise at their configuration.
Well.. maybe. Or Maybe not. But Definitely not sort of.
When the weather gets warmer, Google notices is that it's harder to keep servers cool.
Brilliant journalistic work there.
I've a few questions, if the data centre is built in the desert don't you have a number of issues?
* Latency, if you have all your data centre's located in essentially a single part of the USA (lets ignore the rest of the world for this.. regardless that there are no deserts in Europe for example) won't that increase latency quite a bit to the more further away places that want the search results?
* Bandwidth/redundancy, if you have all your eggs in one basket as it were aren't you going to have to pay extra to have lots of extra fibre laid down to be able to handle all that extra traffic? What about natural disasters, if you have all your data centres in a single location then surely you run the risk of things going pear shaped if it burns down, suffers earthquakes, aliens destroy the building etc.
* Cooling, because it's in the desert isn't a lot of the electricity that is generated going to be cooling not only the building because of the outside heat, but also the heat generated by the servers? Surely it makes more logical sense to build in a colder climate say further north and use hydroelectricity? (if you're talking of using exclusively non active polluting (and non radioactive) natural electricity solutions)
Greater than 99.9% efficiency? They likely made a mistake in their measurements.
Maybe they measured 99.92% efficiency.
That is greater than 99.9% efficiency and they aren't breaking any laws of thermodynamics.
look at the date the article was published.
"It would be wrong to refuse to face the fact that everything is fundamentally sick and sad."
This is a questionable number. The best DC-DC conversion is around 95% so they aren't including voltage conversions from the battery to what the system is actually using.
Many data centres expressly forbid UPSes or batteries bigger than a CMOS battery in installed systems - because when the fire department hits the Big Red Button, the power is meant to go OFF. IMMEDIATELY.
So while this is a nice idea, applying it outside Google may produce interesting negotiation problems ...
http://rocknerd.co.uk
I'm working on a solution. If only I can contact Oracle.
A desert does not describe the temperature of a region but the (lack of) rainfall/moisture.
http://desertgardens.suite101.com/article.cfm/definition_of_a_desert (link found using Google).
And besides, put the containers underground and I'm pretty sure that "hot" you refer to becomes a non-issue as well.
Or maybe they think bigger...
They're deploying containers of servers. Maybe when a container gets a to a certain age or a certain failure rate, they replace/refurbish the entire container.
I doubt they care if some of their nodes go down in a power outage as long as some percentage of them stay up.
Hundreds of thousands of servers == thousands of dead batteries each month, since those batteries don't last more than a few years.
I would imagine that the battery replacement schedule mimics the server obsolescence perfectly.
LOL, when the battery catches fire, time to replace the server.
Google is basically re-implementing the efficiency that already exists in a laptop.
You have a laptop with >1000 processors, consisting of several times that many cores, with its own built-in gigabit ethernet running on built-in gigabit switches?
I'd hate to sit next to you on an airplane!
Javascript + Nintendo DSi = DSiCade
A google mainframe would be stupid.
If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth.
There are only 2 reasons to use mainframes now.
1: Development cost. Building software that can scale on commodity hardware is expensive and difficult. It require top notch software developers and project managers. It make sense for Google to do it, because they use so much hardware(>100000 computers at last count).
2: Legacy support.
They'd still have a computer there that is staggeringly efficient, especially since a computer's output energy is entirely heat - information is not energy, computers are all 0% efficient. Still, this isn't what they meant and the 99.9% figure probably comes from battery in/out figures.
It's ok, appearently he stores it in his middle finger.
Date Center Knowledge has videos of the secret server and a tour of one of the container data centers.
I could design this PSU configuration, and I do electronics only as a hobby.
First, your main PSU delivers 12V in this scheme. Then this is stepped down to 5V and 3.3V for mainboard use, a design that is already employed by some Enermax PSUs, for example. For the 12V line, remember that +/-10% lower is acceptable. The lead-acid battery delivers up to 14V, so you need a step-down converter to 12V. In fact, you can design a switching regulator that steps the input voltage down to 13.2V (12V+10%), if it is larger, and just passes it through for 13.2V...10.8V with very, very low losses. A similar design can be done for 5% tolerances. Modern switching FETs go down to 4mR per transistor and you can do the transition from switching mode to pass-through mode very easily, e.g. with a small microcontroller that can then also do numerous monitoring and safety things. I had actually considerd such a design (purely analog though) for a lower-power, 12V external supply system myself some years ago, but a single UPS was so cheap that I did not went through with it.
I do not mean to belittle the what the Google folks do, though. The real ingeniuity is relaizing you can do it this way on a datacenter scale when nobody else does it. The engineering is then not too demanding, at least for folks that know what they are doing.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
The problem I have with running a motherboard directly from a 12V battery is that most batteries are 12V nominal; actual voltage varies quite a bit (10.5-13.8 for a Pb-Acid). So the question is how well do the 12V components cope with the lower/high voltage? Most of the logic should be OK; that's all 5/3.3/1.xV. I'm guessing the only stuff that really uses 12V anymore is actually disk drive circuitry(not technically on the MB).
I have a suspicion that you really don't want to be running a hard drive off a voltage supply that varies by up to 25%. They must have solved this somehow (step up + step down converter? But that is not efficient) but I really see no point in using 12V motherboards unless everything else can reliably run off the battery first. The home consumer may as well stick with getting 5V from the PSU and letting that dissipate the heat from the step down conversion until we're all using 5V disk drives. In which case, we can probably move to lower voltages (and lower voltage batteries); ~8V seems about right to get a stable 5V.
I have determined that my sig is indeterminate.