Slashdot Mirror
Google Reveals "Secret" Server Designs
Hugh Pickens writes "Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM or Sun Microsystems, but Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. For the first time, Google revealed the hardware at the core of its Internet might at a conference this week about data center efficiency. Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. 'This is much cheaper than huge centralized UPS,' says Google server designer Ben Jai. 'Therefore no wasted capacity.' Efficiency is a major financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: 'We were able to measure our actual usage to greater than 99.9 percent efficiency.' Google has patents on the built-in battery design, 'but I think we'd be willing to license them to vendors,' says Urs Hoelzle, Google's vice president of operations. Google has an obsessive focus on energy efficiency. 'Early on, there was an emphasis on the dollar per (search) query,' says Hoelzle. 'We were forced to focus. Revenue per query is very low.'"
I think Google may be selling themselves short. Once you start building standardized data centers in shipping containers with singular hookups between the container and the outside world, you've stopped building individual rack-mounted machines. Instead, you've begun building a much larger machine with thousands of networked components. In effect, Google is building the mainframes of the 21st century. No longer are we talking about dozens of mainboards hooked up via multi-gigabit backplanes. We're talking about complete computing elements wired up via a self-contained, high speed network with a combined computing power that far exceeds anything currently identified as a mainframe.
The industry needs to stop thinking of these systems as portable data centers, and start recognizing them for what they are: Incredibly advanced machines with massive, distributed computing power. And since high-end computing has been headed toward multiprocessing for some time now, the market is ripe for these sorts of solutions. It's not a "cloud". It's the new mainframe.
Javascript + Nintendo DSi = DSiCade
From 2007, the modular data center patent (where the bottommost image of the article comes from). There's no lack of patents revealing piece by piece how their power management setup works.
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
My work here is dung.
Get that man a beer.
I'm no guru of servers, but from my own limited experience in installing servers at the small to midsized company I work at, space is always a looming issue. And shrinking the size of the UPS you need can only save money and space in the long run; which any IT manager will tell you is a huge benefit and a great selling point.
Nothing to do but wait for a finished product at this point though.
Posting with out proof reading since 2001.
We all know the searches are actually being done by a large amount of people in suspended animation, being fed the corpses of the previous people.
The thing about each server having its own battery is a cruel joke.
The in-computer onboard UPS is not a new idea. I don't see how they could have gotten any patents on it since I used it have one of these (my day might still). The device I saw had a gel cell mounted on an 8-bit ISA card, full length. It had +5/12v pass through connectors for powering the drives and it powered the computer through the main bus. There was more logic to it, as it had some monitoring capabilities too.
What's next, patenting a hard drive on a plugin board? Been there, it was called the Hard Card and put a 20mb HDD in an 8 bit full length ISA slot, a truly neat idea for upgrading old XT computers back in the day. You could make them work with AT computers too by putting a regular disk controller, without a drive connected, on the bus too and the BIOS would see the XT controller and boot from it.
Google claims they did the math and found it was cheaper with commodity hardware. I advise everyone else to do the same and run the calculations for themselves to determine the optimal hardware for their particular load. With out the specifics of their situation, its difficult to criticize in an intelligent fashion, other than a more generalized statement expressing surprise at their configuration.
Well.. maybe. Or Maybe not. But Definitely not sort of.
When the weather gets warmer, Google notices is that it's harder to keep servers cool.
Brilliant journalistic work there.
I've a few questions, if the data centre is built in the desert don't you have a number of issues?
* Latency, if you have all your data centre's located in essentially a single part of the USA (lets ignore the rest of the world for this.. regardless that there are no deserts in Europe for example) won't that increase latency quite a bit to the more further away places that want the search results?
* Bandwidth/redundancy, if you have all your eggs in one basket as it were aren't you going to have to pay extra to have lots of extra fibre laid down to be able to handle all that extra traffic? What about natural disasters, if you have all your data centres in a single location then surely you run the risk of things going pear shaped if it burns down, suffers earthquakes, aliens destroy the building etc.
* Cooling, because it's in the desert isn't a lot of the electricity that is generated going to be cooling not only the building because of the outside heat, but also the heat generated by the servers? Surely it makes more logical sense to build in a colder climate say further north and use hydroelectricity? (if you're talking of using exclusively non active polluting (and non radioactive) natural electricity solutions)
Googles secret is that all there computers have battery.
I think, it is called a laptop.
Greater than 99.9% efficiency? They likely made a mistake in their measurements.
Maybe they measured 99.92% efficiency.
That is greater than 99.9% efficiency and they aren't breaking any laws of thermodynamics.
...why desktops didn't have a built in battery deal that lived in an expansion bay. If you could even keep RAM alive for extended periods even with the machine shut down that would be spiffy as an option, let alone as a little general UPS.
look at the date the article was published.
"It would be wrong to refuse to face the fact that everything is fundamentally sick and sad."
This is a questionable number. The best DC-DC conversion is around 95% so they aren't including voltage conversions from the battery to what the system is actually using.
wouldn't trust them any further than I can throw them
Given the reliability, it's likely that someone has already measured that particular parameter for you. Have you checked the data sheets?
I am one of many. My idea is not unique, nor do I expect my voice alone to sway you. I speak in a chorus of opinion.
Many data centres expressly forbid UPSes or batteries bigger than a CMOS battery in installed systems - because when the fire department hits the Big Red Button, the power is meant to go OFF. IMMEDIATELY.
So while this is a nice idea, applying it outside Google may produce interesting negotiation problems ...
http://rocknerd.co.uk
I'm working on a solution. If only I can contact Oracle.
Hundreds of thousands of servers == thousands of dead batteries each month, since those batteries don't last more than a few years.
Now I'd think their design could be gentle on the 12V batteries, since it's possible to design UPSes that don't murder batteries at the rate cheap store-bought UPSes do. But still, they must have an army of droids swapping out batteries on a continuous basis.
Or maybe they are more selective, and only swap out batteries on hosts that have suffered one or two outages. It only takes one or two instances of draining a gel cell to exhaustion before it is unusable.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
This is composed purely of commodity parts. The power supply is the same thing you'd buy for your desktop, those are SATA disks (not SAS), and that looks like a desktop motherboard (see the profile view where all the ports on the "back" are lined up in the same manner they would need for a standard desktop enclosure).
Only the battery is custom (or even non-consumer grade), and you can note that since the power goes through the PSU first, that's DC power. DC is significantly better than AC, since the PSU then has to convert AC-to-DC (which wastes power and generates needless heat). While you can get DC battery supplies for server-grade systems, these are not server-grade systems. Built-in DC battery backup therefore affords them the ability to keep the motherboards cheaper. Very smart.
Also, if you recall from a few months ago, Google has applied pressure on its suppliers (I'm not sure why Dell comes to mind...) to develop servers that can tolerate a significantly higher operating temperature (IIRC, they wanted at 20 degree (Fahrenheit?) boost). I wouldn't be surprised if the higher temperature cuts down on operating expenses more than smarter battery placement.
Use my userscript to add story images to Slashdot. There's no going back.
Dawned on me the other day how little innovation occurs in our industry EXCEPT by hungry companies. For example, Desktops and Laptops have not really changed, while both have a piss poor design. ABout 4 years ago, it dawned on me that a much better way to design these is to merge them. Basically, different cases where the laptop has keyboard and a monitor hookup while the desktop is sans the prior. The smart move is to move the battery OUT of the case and into the power supply. Right now, you do not get to buy variable amounts of batteries. But a company would do well to sell an external power supply with varying storage capacities, but with a simple 12V line. In this fashion, ppl can pick the parts for a laptop similar to a desktop, while the desktop gets to take advantage of the drop in prices of the laptop linage.
I prefer the "u" in honour as it seems to be missing these days.
Peter Huber in his book on energy policy introduces the concepts of the "energy pyramid" and "energy refining". The thesis that new forms of energy technology use more technology and are subsequently more useful. The pyramid levels include wood, coal, petroleum, electricity, computing and optical. When I read the book a few years ago I always found it curious that he included computing in the pyramid. But I hear about aggregate gigawatts of hundreds of mass server farms in the world, it may start making sense. The web has transformed human technology and the server farms are the battery of the web. When Huber wrote the book he used the example of the automobile as it started being mostly petroleum energy, then acquired more electricity sub systems, and now more computing.
A desert does not describe the temperature of a region but the (lack of) rainfall/moisture.
http://desertgardens.suite101.com/article.cfm/definition_of_a_desert (link found using Google).
And besides, put the containers underground and I'm pretty sure that "hot" you refer to becomes a non-issue as well.
"Google's designs supply only 12-volt power, with the necessary conversions taking place on the motherboard"
This seems to be a more interesting point than the battery part. 12V-only?
This means that there's some serious power conversion done on each of the motherboards, and with SMPS evolving at the rate that it is, this could be relevant to anything larger than a laptop.
How much exactly is gained by making such a big change, to a point where you'd need to redesign all of your motherboards, each time for each different chipset? (they mention they use both Intel and AMD)
Will this particular change make it into desktops? How much *more* efficient would it make the overall system?
Entomologically speaking, the spider is not a bug, it's a feature.
I'm a little surprised by the keyboard and mouse port and the two USB ports. If it uses USB, why not just use that for the keyboard and mouse? And why the second USB port? I suspect the second port doesn't consume extra energy directly, but it causes air resistance where they'd like to clear path to drag air across the RAM and CPUs.
And why the slots which will never get used? In quantities like Google buys, you'd think those would be left off.
Maybe they don't make any demands on Gigabyte (the manufacturer) and just buy a commodity board? When they're buying this many, you'd think Gigabyte would be happy to make a simpler board for them. On a trivial search, I don't see the ga-9ivdp for sale anywhere, but maybe it's just old.
So this sounds like one of those "so obvious, no one thought of it" questions - if Google is so concerned about precious mW that it standardizes on 12V hardware to reduce current losses of sending 5V & 3V power from the powersupply to the board, why do the CPU's have fans???? The side view of the chasis seems to suggest that with a few minor tweaks the units could rely on passive cooling and use the data centre / container fans for air flow.
1) Move hotter components like the CPUs to the front and replace fans with larger passive heat sinks.
2) RAM modules lined up to ensure proper airflow to the back of the chasis, chipset heat sinks lined up accordingly.
3) HD's laid over top of voltage regulators with appropriate heatsinks
4) power supply and battery at the rear.
Have the hot air return duct work arranged at the back of the rack with appropriate holes and seals so that the units make a good connection to maximize airflow.
I'm in my right mind and I have the answer to everything!
Google is basically re-implementing the efficiency that already exists in a laptop.
You have a laptop with >1000 processors, consisting of several times that many cores, with its own built-in gigabit ethernet running on built-in gigabit switches?
I'd hate to sit next to you on an airplane!
Javascript + Nintendo DSi = DSiCade
A google mainframe would be stupid.
If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth.
There are only 2 reasons to use mainframes now.
1: Development cost. Building software that can scale on commodity hardware is expensive and difficult. It require top notch software developers and project managers. It make sense for Google to do it, because they use so much hardware(>100000 computers at last count).
2: Legacy support.
They'd still have a computer there that is staggeringly efficient, especially since a computer's output energy is entirely heat - information is not energy, computers are all 0% efficient. Still, this isn't what they meant and the 99.9% figure probably comes from battery in/out figures.
I believe the joke was that the distance a DeskStar can be thrown may be published in the data sheets. Being such a common concern and all. :-)
Javascript + Nintendo DSi = DSiCade
It's ok, appearently he stores it in his middle finger.
Date Center Knowledge has videos of the secret server and a tour of one of the container data centers.
...imagining a Beowulf cluster of these?
Aww - nevermind.
Just because something is published on April 1st doesn't mean it's an April Fools joke. In the case of this article, it's clearly not.
You're seeing a connection where there is none. The two SATA cables run back behind the plate the drives are mounted on. Presumably, the mainboard connectors are back there as they're not visible on the rest of the mainboard.
Javascript + Nintendo DSi = DSiCade
Once upon a time, maybe 6 or 8 years ago now, I got to sit down with the CEO of APC and basically told him I wanted battery backed in-computer power-supplies, something small yet efficient. I wanted functionality like my laptop does, unplug PC, move it, plug it back in. Same for my servers (might have been when that whole Netshelter product line started up.
:-)
Ah, too bad I kept no notes, no logs, could have made a fortune suing Google.
Probably because they own these datacenters and can do what they want with them. The EM emissions are probably contained by the fact that the servers are all in a giant metal box. UL is optional, and if they don't want to go through it they don't have to. It's not like they're selling these servers to anyone.
I read the internet for the articles.
There's only so many places you can connect a battery to a PC and all of them have already been implemented by someone at some point. There's been motherboards with second power connectors, motherboards with battery connectors, power supplies with batteries, power supplies with battery connectors, DC power supplies connected to external batteries, integrated UPS systems which take in and put out AC and which are basically just hooked up in line with the power supply... Off the top of my head I immediately think of AS/400 systems which were offered with integrated UPS before they even renamed it to zSeries or whatever it is. (I always forget. AS/400 was a good, IBM-sounding name.) The solution which comes immediately to my mind for a google-style distributed data center would be to use something power-efficient hooked up to a PicoPSU hooked up to a SLA battery hooked up to a charger hooked up to your power source. Cheap, simple, and built with commodity parts. (They seem to sell a UPS charger unit where you can get the PicoPSU as well.)
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
How did this get marked informative?
I mean it's certainly true that Deserts are defined by lack of rainfall but since the GP said
"Build your data center in the desert and build 150 MW industrial solar thermal system to power it."
I think it's fair to assume they were talking about the stereotypical sunny and hot desert.
Secondly the reason it's cool underground is because soil is generally a very good insulator. I would suggest that it's a really bad idea to put things that are going to get hot inside a huge lump of insulating material.
I could design this PSU configuration, and I do electronics only as a hobby.
First, your main PSU delivers 12V in this scheme. Then this is stepped down to 5V and 3.3V for mainboard use, a design that is already employed by some Enermax PSUs, for example. For the 12V line, remember that +/-10% lower is acceptable. The lead-acid battery delivers up to 14V, so you need a step-down converter to 12V. In fact, you can design a switching regulator that steps the input voltage down to 13.2V (12V+10%), if it is larger, and just passes it through for 13.2V...10.8V with very, very low losses. A similar design can be done for 5% tolerances. Modern switching FETs go down to 4mR per transistor and you can do the transition from switching mode to pass-through mode very easily, e.g. with a small microcontroller that can then also do numerous monitoring and safety things. I had actually considerd such a design (purely analog though) for a lower-power, 12V external supply system myself some years ago, but a single UPS was so cheap that I did not went through with it.
I do not mean to belittle the what the Google folks do, though. The real ingeniuity is relaizing you can do it this way on a datacenter scale when nobody else does it. The engineering is then not too demanding, at least for folks that know what they are doing.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Arguably, APC has become a mainframe vendor. They sell rack systems with integrated power, cooling, and cable management. Add commodity motherboards, CPU parts, disk drives, and software, and you have a mainframe. It's not that different from what HP or SGI or IBM or Sun will sell you. Especially since the "mainframe" vendors have mostly moved to commodity CPU parts.
I've pointed out before that computing is becoming more like stationary engineering. Stationary engineers run and maintain all the equipment in building basements and penthouses. With containerized data centers, computing looks more and more like that.
I don't think they are greener, but cheaper? Maybe. As for reliability, You have to keep in mind the whole map reduce framework is built around the idea that anything at anytime could fail. The compensate in numbers of servers and software for the lack of reliability of each one. No not every task or application is applicable to their set up. But, I believe them. I'm not into conspiracy theories.
Plus you also have to account for gradual scaling up & geographical distribution. Easy to do with additional low powered servers, difficult to do with giant expensive mainframes.
Well.. maybe. Or Maybe not. But Definitely not sort of.
Have you ever even seen Mainframe pricing? No really have you?
It will cost you at least 10000$ to match the power of a single quad core intel/amd cpu.
And you do not want to run a mainframe(Or other computer that have a cpu bound task) for a decade. I think my current desktop computer have more power then avg mainframe
from a decade ago, and when I buy a new development workstation in then next decade, it will most likely have more cpu power then a 1 million $ mainframe you could buy today.
Just to set things in perspective: I am pretty sure, that google have more cpu power, more ram, more hd space and more aggregate io, then all mainframes in USA combined.
You're Doing It Wrong(tm). A sealed cell will only vent hydrogen if overcharged (at the cost of increasingly reduced cell capacity - you're not filling it back up with water!). An intelligent charger will eliminate any routine hydrogen venting, leaving only the occasional bad battery or battery hooked to a broken charger venting. Google is probably OK with that.
There is no possible way their solution is cheaper than a real mainframe (created for the task) when all costs are considered.
Nor is there any possible way their solution is more reliable, or more "green".
That depends on how you're measuring cost, reliability and "green"itude. Cost-wise, there's an enormous opportunity cost associated with going with a single mainframe vendor. Reliability... well, they've made the choice of having small, frequent failures that are cheap and easy to deal with rather than single large uncommon events that might put a division out of action all at once. Green credentials? Again, it's a trade-off. They've traded physical resource cost against energy cost.
Also, by doing it this way, they can take incremental improvements far more easily than they could with a mainframe installation. Once your mainframe is installed, that's it - you don't get to improve power efficiency or processing power ever again. With these, if you figure out how to get a percentage point improvement, you can roll it into the next build cycle, knowing that it'll probably be across half the company in a couple of years.
Oh, and you're slightly wrong about hard drives. They don't RAID them. They just chuck them.
Trash (Magnatek) power supply.
A couple of years ago, they announced that they had their own PSU design that was supposedly much more efficient than anything available on the market. If this is a cheap commodity PSU, it predates that.
A 12v battery. I never knew DC was more efficient than AC!
Dude... UPS. If you're using the battery, you don't *have* AC.
A good mainframe would last decades. Google's frankenframe (lets call it what it is) must be sloughing off parts like skin cells from a Texan with eczema.
And that, presumably, is just the way they like it, because if you upgrade something that hasn't failed yet, you lose whatever value was left in it.
Reality is the ultimate Rorschach.
The power of a thousand cores is no match for my asbestos underpants.
Ok. So, your load fits onto 5 mainframes. Now your requirement increases. What do you do? Do you buy number 6 now, and have it running at less than capacity for the next 18 months (or whatever)? That's a huge waste. Do you degrade your service for the next 9 months until number 6 would be at half capacity, then install? Again, you've wasted an opportunity, and number 6 is *still* not going to be at capacity.
Smaller computational units means better matching of demand to supply.
Reality is the ultimate Rorschach.
A 12v battery. I never knew DC was more efficient than AC! WOW GOOGLE IS SO COOL!
I'm not sure if you are being sarcastic. A 12v battery in the power supply is more efficient than taking DC -> AC -> DC. That is what a UPS does, each conversion introduces loss. Having the battery in the power supply means there is no conversion so less power loss.
Wait, my laptop has one of those too...
In other news, is anyone else surprised that a built-in UPS is so slow to catch on for the desktop when notebooks have had it by definition for years? Sure, powerful batteries are expensive, but you'll wish you had one when a power blackout destroys half a day's work. It's one reason why I hesitate to get a desktop PC.
Well, I'd hate to be ON that plane.... that system would be a cloud *in* a cloud... until the plane crashed from all the weight it.
Funny... captcha is "kerosene" (which some planes use, IIRC...)
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
Actually, they're not.
Laptops run slower than their PC counterparts.
Laptop drives run slower than their PC counterparts.
Laptops run hotter under load than their PC counterparts.
If you look carefully at the picture, they've found a 12v motherboard, tied a 12v battery directly into it, and used otherwise commodity parts. That's been the mantra for Google for as long as I can remember. Oddly enough, that was my mantra when I built up a big network. Lots and lots (and lots and lots) of cheap servers are better than a handful of really expensive ones. That saved our cumulative posteriors on more than one occasion.
I've spoken with some people who have personal knowledge of Google's equipment. They were setting up with RAID 01 or 10. I suspect with the two drive configuration, they're only setting up with RAID 0 now, and the redundancy is across multiple servers. I can confirm that they are using this open tray system for it's superior cooling.
I had considered open trays like this, except there's one huge downfall. You would have to be amazingly careful of what happens near the rack. If you are screwing something in, and the screw or screwdriver falls, that can become very bad very quickly. Did you see any fuses or breakers from the battery to the power supply?
Short of making the area around the rack a metal-free zone (no screws, screwdrivers, rings, keys, watches, etc), you'd seriously run the risk of shorting something out. I know I've been working up in the higher areas of a rack, and dropped screws. You listen to it rattle it's way down across several machines until it finally hits the floor. Since I use closed servers cases, it's never a problem. Maybe they don't have a big problem with it at Google, but I'd be terrified of it. Anyone who says they've spent any substantial time working in and around racks, and haven't ever dropped anything, are lying. I do love the idea for free airflow and better cooling, but ... well ... I like to keep magic smoke in it's place. :)
The one-battery-per-server is a nice idea though. I may look into that for future builds. Most PC's have 5v and 12v output. That power supply only indicated a 12v output, and didn't have any wires that indicated anything different.
Serious? Seriousness is well above my pay grade.
Only by converting it from DC power. Which is less efficient than using the DC power directly.
And is DC even any less efficient? I know it's more efficient to transmit AC power over long distances (i.e. power lines), but does that apply to short distances like these?
A switch mode PSU takes AC, converts it to DC, switches it at a high frequency and then filters it back to DC at each rail voltage. They have obviously modded this PSU so that it can take DC directly in at a much lower voltage and still work so the PSU and UPS are combined. I find this neat.
The whole server is redundant.
See my journal, I write things there
And you do not want to run a mainframe(Or other computer that have a cpu bound task) for a decade. I think my current desktop computer have more power then avg mainframe from a decade ago
You can run a mainframe for a decade or more because every part except the steel frame is hot-replacable. You upgrade the processors, memory, everything really every few years, without ever interrupting service. There's a reason they aren't cheap.
Even good minicomputers (or expensive servers, if you like that term berret) let you swap processors, I/O processors, memory, and sometimes motherboards while the machine is running. High-end mainframes just take that to the next level, by ensuring that every board is hot-replacable.
Of course, Google approach to the same problem (just hot-swap cheap commodity servers in and out of the cloud as units) may well be cheaper, in terms of hardware costs. I doubt it's cheaper if you include all of the related development costs, but sometimes that's a good trade-off to avoid vender lock-in.
Socialism: a lie told by totalitarians and believed by fools.
There are voltage regulators that can drop/boost the voltage to a predetermined voltage and do so with 90+% efficiency. Look for 'buckboost regulator' or 'switching regulator'.
Tom.
The problem I have with running a motherboard directly from a 12V battery is that most batteries are 12V nominal; actual voltage varies quite a bit (10.5-13.8 for a Pb-Acid). So the question is how well do the 12V components cope with the lower/high voltage? Most of the logic should be OK; that's all 5/3.3/1.xV. I'm guessing the only stuff that really uses 12V anymore is actually disk drive circuitry(not technically on the MB).
I have a suspicion that you really don't want to be running a hard drive off a voltage supply that varies by up to 25%. They must have solved this somehow (step up + step down converter? But that is not efficient) but I really see no point in using 12V motherboards unless everything else can reliably run off the battery first. The home consumer may as well stick with getting 5V from the PSU and letting that dissipate the heat from the step down conversion until we're all using 5V disk drives. In which case, we can probably move to lower voltages (and lower voltage batteries); ~8V seems about right to get a stable 5V.
I have determined that my sig is indeterminate.
"If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth."
no it doesn't.
Plus they are cheaper to maintain, require less power per cycle, require less square feet to house.
Yeah, I actually know about these things.
The Kruger Dunning explains most post on
Form a quick once over of from IBM ftp://ftp.software.ibm.com/common/ssi/pm/sp/n/zsd03005usen/ZSD03005USEN.PDF
There z10 can hold a max of 1.5 tb of Ram
Lets say for the load that is processed by each server, Google needs 8 gigs of memory. Which they can supply for $2,000 each.
No lets be generous to IBM and its reliability and say that we need twice as many google servers per the equivalent ibm reliability.
We can replace 375 (1.5 TB of ram /8gigs per google mache * 2googlemachines/ibm equvalence) google servers per z10.
Assuming each google server costs google $2000 to make, they would spend $750,000 on google servers. Now lets assume the IBM is better at power as well, to the tune of $10,000 per year and both expected lifetimes are 20 years. That comes out to a 20 year cost for google servers of $950,000.
If the ibm price for the z10, is greater than $950,000, then google should continue making their own servers. Otherwise, they should switch.
Obviously these are all ballpark figures, which I don't expect to be correct. There are quite a few variables and just because a mainframe may be more reliable and power efficient, it may not be the best choice even when dealing with hundreds ore even thousands of servers. Typically the price per performance unit ratio goes skyward as you move towards bigger and bigger servers.
Well.. maybe. Or Maybe not. But Definitely not sort of.
Fly the data center above the arctic circle in the northern hemisphere's summer, and fly it down below the antarctic circle in the southern hemisphere's summer, and you could do the solar thing 24 hours a day with cheaper cooling.
Modern high speed chips (which draw the bulk of the power in a typical PC) run thier core logic at much lower voltages. Typically somewhere between 1V and 2V though I think some may have gone below a volt now. Theese very low voltages have to be produced very close to the chip that uses them to avoid huge losses.
This means that modern PC motherboards take most of thier power at 12V anyway. The 5V and 3.3V lines really only serve to power the low speed chips and some of the interfaces between chips.
Given that I doubt there would be too much efficiancy loss from making a 12V only board. You could probablly even design it to hapilly deal with an input that was only approximately 12V without losing too much (since most of that 12V power is going to the input of switchers anyway).
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
When I worked for a University, we bought a few of the largest IBM pSeries machines (power4 at the time). These were powerhouse machines 5 years ago. Each one had a dedicated 24" oversized rack cabinet, and then we had a couple racks just for disk. The 4 machines, and about 40T of Fibre channel disk (or was it DASD), I think it was a total of 128 core and 256GB of ram. I think we paid about a million for that setup.
As was mentioned elsewhere on the webs, the machine shown off by Google was based on Nocona CPUs.. those are atleast 4 years old now. Not likely what they're buying new now.
I bet you could get a base z10 for a few hundred thousand, but a fully loaded one? With a disk array of 750 drives? I bet 4 racks of disk from IBM would cost most of that 950k budget.