Slashdot Mirror
How Many Google Machines, Really?
BoneThugND writes "I found this article on TNL.NET. It takes information from the S-1 Filing to reverse engineer how many machines Google has (hint: a lot more than 10,000).
'According to calculations by the IEE, in a paper about the Google cluster, a rack with 88 dual-CPU machines used to cost about $278,000. If you divide the $250 million figure from the S-1 filing by $278,000, you end up with a bit over 899 racks. Assuming that each rack holds 88 machines, you end up with 79,000 machines.'" An anonymous source claims
over 100,000.
No wonder I'm'a Googlin'
Here's what I do: Bitty Browser & Andromeda
That's $3159 per machine, and those are today's prices... They weren't so low a couple of years ago...
Can you imagine a beowul.... oh.. wait..
SysWear - Geek T-shirts (UK/Europe)
1) google is so pretty and smart
2) google is worth so much money
3) google has a huge rack!!
There was an article recently about how Google constantly understates various statistics about itself to mislead potential competitors. This article also said that the SEC would not allow them to do this once they became a publically traded company.
Seriously? What is the point of this article? What next? Linus found to prefer blue ink, over black ink?
========
CINC, 4th Penguin Legion
I don't think this is that strange: after all, that 10,000 machines figure is several years old. It's only logical that Google has expanded their facilities since then.
This space intentionally left blank.
SCO now knows how big an invoice to send Google! :-D
My rights don't need management.
According to calculations by the IEE, in a paper about the Google cluster, a rack with 88 dual-CPU machines used to cost about $278,000
Um, don't you think if you were buying 899 racks you might actually, you know, negotiate for a better price?
This isn't the only assumption in your analysis, and the problems with them will be compounded. What's the point of this, really?
Might just be me, but damn, don't you think this has raised the interested of our three letter entities? i mean, damn that is just some serious computing and indexing power on cheap, "disposable" hardware...with a filesystem that can keep track of that many machines? If i headed one of such entities, i'd sure want to know more about it!
Yes, but aside from dealing with hardware failures and other physical / logistical problems, there really isn't much of a difference between managing 45,000 computers and managing 80,000. They're both Really Big Numbers, and I'm sure whatever software they're using is scaleable enough to smoothly handle many more machines than that.
This space intentionally left blank.
Remember there's a little thing called "volume discount"...
It's gotta be more than that.
There are two kinds of people in the world: Those with good memory.
With all those TFlops, no wonder Google converts units so quickly.
You mean the PigeonRank(tm) technology is a hoax?
Well, let's count. I have two servers at home and 8 at work. They all run linux. Now, if everyone else in the world joins this thread, we can find out.
I wonder if google will start up a web-hosting business? I bet you can't beat their uptime guarantees. They could provide sql, cgi, etc, and build in multi-machine redundancy for your data just like they do for theirs. It'll be the google server platform, just one more step to replacing Microsoft as the evil monopoly.
The number of machines Google uses is considered a trade secret. By attempting to determine how many machines they have, you're in violation of the DMCA. I'm calling the FBI.
If you've ever read a white paper of Google's, you'd realize that they even tell people why they deal with massive clusters over mainframes: lower latency.
Sunny Dubey
...assuming 200W per server, which is probably low, but probably compensates for 79,000 being most likely an overestimate. However, that doesn't even begin to account for the energy used to keep the stuff cool.
Anyone know how many trees per second that would be? Conversion to clubbed-baby-seals-per-sec optional.
Please help metamoderate.
Racksaver was selling dual-machine 1U racks for several years and I owned a few of them. Think deep, not tall. Racksaver seems to have renamed itself Verari and only has dual-Opteron in a 1U now. Most dense configs seem to be blade-based these days. Verari advertises 132 processors in a single rack, but I suspect they are not king in this area.
If Google is innovating in this area, it could either be on price or in density.
A Pentium 4 dissipates around 85 W of heat. I don't know what the Xeon does, but let's be kind and say 50 W (wild guess). Using the article's "low end" estimate, that brings us to 4.7 MW!
I hope they have good ventilation...
42.
When you can just open "Computer Architecture: A Quantitavie Approach, 3rd Edition" by Hennessy and Patterson to page 855 and find out that in summary: ...
Google has 3 sites (two west coast, one east)
Each site connected with 1 OC48
Each OC48 hooks up to 2 Foundry BigIron 8000
80 Pc's per rack * 40 racks(at an example site)
= 3200 PC's.
A google site is not a homogenous set of PC's instead there are different types of PC's that are being upgraded on different cycles based on the price/performance ratio.
If you want more info get the patterson hennessy book that I mentioned. Not the other version they sell. This one rocks way harder. You get to learn fun things like Tomosulo's algorithm.
If I am violating any copy rights feel free to remove this post.
Mainframes are optimized for batch processing. Interactive queries do not take full advantage of their vaunted I/O capacity.
Moreover, while a mainframe may be a good way to host a single copy of a database that must remain internally consistent, that's not the problem Google is solving. It's trivial for them to run their search service off of thousands of replicated copies of the Internet index. Even the largest mainframe's storage I/O would be orders of magnitude smaller than the massively parallel I/O operations done by these thousands of PCs. Google has no reason to funnel all of the independent search queries into a single machine, so they shouln't buy a system architecture designed to do that.
Interesting People 2004/05:
I know for a FACT they passed 100,000 last November. One thing the Louis calculation may have missed is Google's obsession with low cost. For example read the company's technical white paper on the Google file system. It was designed so that Google could purchase the cheapest disks possible, expecting them to have a high failure rate. What happens when you factor cost obsession into his equation?
in how they recycle their gigantic heat output...perhaps move data center to the windy city, open up a homeless shelter next door, and put the hot air to good use for once. They might even get a tax break on this.
Better yet, open up a nursery (plant type) next door , build a green house, and piple 25% of the heat to it. Have you guys see the price of trees lately? Google could make a killing with the "recycling" plant.
All those machine, all that complexity and activity, all boiled down to one little box under a Google logo. The most useful input box on the internet.
Thanks Google!
Yeah it's kind of like:
Your wife has slept with 80 other men, or was it 200?
Either way, it's not good for you.
They better have at least 10^100 machines, or they will be getting a call from my lawyers.
word.
The cost of acquiring the machine is a fraction of the cost of owning it.
And lets not forget the overhead of 2 networks per machine and all the patch panels, wiring, switches. Toss in console management (which may not be on all machines at all time), monitoring and management of said machines. Oh, and one really tired guy running around.
Disks are going to fail at a rate of several hundred or thousand PER DAY, just statistically. (along with power supplies etc)
Toss in that in three years, ALL of those machines are obsolete.
That's huge.
I've got ~300 racks in a half full data center upstairs from me. All network cables run to a room below it to patch panels. Around 50% the size of the DC is cable management. Next to that is a room FILLED with chest high batteries - these are used during outages until the generators need to be kicked on. And a NOC takes up about 1/5th the space of the DC (monitoring systems worldwide, but it's got seating for maybe 40 people - tight and usually filled with 10 folks, but in a crunch we live up there).
So that $3159 is only a bit of it. And in 3 years, all those machines will likely be replaced for whatever $3k buys then. That's about to be a 2 CPU Athlon64 box. If Sun can pull a rabbit out of its ass, we'll have 8 and 16CPU Athlon64 boxes. At least with that, some of the CPUs can talk to each other really really really fast.
I was in Exodus - Toyama facility in Sunnyvale, CA back in 2001 and was talking to some of the data center techs, they were bitching because Google DOES stack 44 -half depth- servers in a rack, on EACH SIDE (aka 88 servers per rack indeed) and how the heat that produces is absolutely fucking insane and how he can't believe they don't meltdown. He was comlaining how frugal google was not giving the systems more room to breath.
--- www.f-theocean.com
Well, that depends on what sort of time portal they use. Now, a T1000 would probably be saturated by a time portal following the Terminator rules: one way only. But Google seems to favour Back to the Future rules, as shown by number of hits:
13,500,000 for back to the future
3,460,000 for terminator
This would make saturating a T1000 a lot easier, since you could saturate it while travelling back in time yourself, or maybe even while standing still in time. This would make Google's bandwidth infinite, as a measly T1000 would stand still. Unless it was using its own time portal to travel back in time to destroy Google, but that would create a paradox, since, as we all know, Google will become Skynet, which will create the T1000 in the first place.
What I'm trying to say is: I don't know, but I'm sure Google could do it.
Yes, it is true. We can't exist without polluting. However, I'm willing to bet, without doing the calaulations, that the pollution you personally generate by querying google is much less than what you generate browsing slashdot on your home computer.
Computers are useless. They can only give you answers.
-- Pablo Picasso
You don't; their Sales Director comes to you...
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
I'm not a grammer/spelling nazi.
Obviously.
"Your phone company can't just lose a few calls you made and not bill you for them."
Wait, what's wrong with that one?
G
His pricing in the summary may be a bit off.
:)
:) Well, not the Asus 1400r, those are built into a 1u case, but other machines we've built for servers are very easy to build into midtowers instead. Those machines don't get gobs of memory, but do get extras like nice sound cards and CD/DVD players. The price would be the same, as they'd probably still be attaching them to the same networking equipment. 132,000 servers, and 2,682 workstations and dev machines is probably fairly close to what they have.
Every article I've read about Google's servers says they use "commodity" parts, which means they buy pretty much the same stuff we buy. They also indicate that they use as much memory as possible, and don't use hard drives, or use the drives as little as possible. From my interview with Google, they asked quite a few questions about RAID0, RAID1 (and combinations of those), I'd believe they stick in two drives to ensure data doesn't get lost due to power outages.
We get good name brand parts wholesale, which I'd expect is what they do too. So, assuming 1u Asus, Tyan, or SuperMicro machines stuffed full of memory, with hard drives big enough to hold the OS plus an image of whatever they store in memory (ramdrives?), they'd require at most 3Gb (OS) + 4Gb (ramdrive backup). I don't recall seeing dual CPU's, but we'll go with that assumption.
The nice base machine we had settled on for quite a while was the Asus 1400r, which consisted of dual 1.4Ghz PIII's, 2Gb RAM, and 20Gb and 200Gb hard drives. Our cost was roughly $1500. They'd lower the drive cost, but incrase the memory cost, so they'd probably cost about $1700, but I'm sure Google got better pricing, buying the quantity they were getting.
The count of 88 machines per rack is a bit high. You get 80u's per standard rack, but you can't stuff it full of machines, unless you get very creative. I'd suspect they have 2 switches, and a few power management units per rack. The APC's we use take 8 machines per unit, and are 1u tall. There are other power management units, that don't take up rack space, which they may be using, but only the folks at Google really know.
Assuming the maximum density, and equipment that was available as "commodity" equipment at the time, they'd have 2 Cisco 2948's and 78 servers per rack.
$1700 * 78 (servers)
+
$3000 * 2 (switches)
+
$1000 (power management)
--------
$139,600 per rack (78 servers)
Lets not forget core networking equipment. That's worth a few bucks.
Each set of 39 servers would probably be connected to their routers via GigE fiber (I couldn't imageine them using 100baseT for this) Right now we're guestimating 1700 racks. They have locations in 3 cities, so we'll assume they have at least 9 routers. They'd probably use Cisco 12000's, or something along that line. Checking eBay, you can get a nice Cisco 12008 for just $27,000, but that's the smaller one. I've toured a few places who had them, and pointed at them citing them to be just over $1,000,000.
So....
$250,000,000 (ttl expenses)
- $ 9,000,000 (routers)
------
$241,000,000
/ $ 139,600
------
1726 racks
* 78 (machines per rack)
------
134,682 machines
Google has a couple thousand employees, but we've found that our servers make *VERY* nice workstations too.
I believe this to be a more fair estimate, than the story gave. They're quoting pricing for a nice fast *CURRENT* machine, but Google has said before that they buy commodity machines. They do like we do. We buy cheap (relatively) and lots of them, just like Google does. We didn't pattern ourselves after Google, we made this decision long before Google even existed.
When *WE* decided to go this router, we looked at many options. The "provider" we had, before we went on our own, leasing space and bandwidth directly from Tier 1 providers, opted for the monolythic sy
Serious? Seriousness is well above my pay grade.
You might also be interested to know that there are a lot of government buildings in Washington DC.
Nono, only those you have root access to. Post your pass as proof.
It would not be a very distributed DDOS and that would stop any attack quite quickly. Quite simply google's bandwidth providers (or the providers above them) would just unplug them. They may be global, but they probably have less than 40 datacenters. It would not be distributed enough to sufficiently attack. If you could take over the same number of machines with the same amount of bandwidth, but distributed globally on various subnets (say a massive virus), *then* you'd have a DDOS machine. As is, google's DDOS would be shut down quite quickly.
Photos.
You almost have it right... first, a couple of hookers come to you. Then, a few hours later, they are followed by the sales director.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
OK, it's 1-2-3-4-5 -- the same as my luggage!
"Save the whales, feed the hungry, free the mallocs" -- author unknown