How does Google do it?

Doc Tagle writes "With Google reportedly on the verge of going public, more and more people want to know what makes Google tick. The Observer, serves up the answers to our questions."

Comments

As a consultant

[elinenbe]

having been a consultant at their data center a year or so back I can attest that they had well over 50,000 machines. I am not sure about the 80GB drive per machine because from what I understood was they bought whatever drive at the time was the cheapest MB/$ and would replace any dead ones with the larger ones. Also, at any given time machines just die and many of them are not replaced or repaird for months. Their cluster accounts for all this...

Re:As a consultant

[_Sharp'r_]

But also realize that the data center you were at isn't their only one. I know of at least 7 physical locations and there are probably more out there.

But yeah, their racks of 4 servers/1U is pretty impressive when you see them lined up in row after row of racks. Their data centers have to bring in extra cooling because they are so densely packed.

Re:Soon to be everything

[richard_za]

Google already has spell check, and so does Gmail have a look at the screenshots on my blog. I believe they're looking at releasing it to the public in six months time, have a look at this article.

You may also find this interesting...

[lunar_legacy]

Another wonderful speculation about Google infrastructure which You can find it here.

Re:Google is faltering

[orthogonal]

Actually, they already have the fix implemented, and it's currently in the process of being rolled out. The upgraded system makes use of a split primary key which comprised of a "selector" subkey and a "segment" subkey. The selector key is shifted left by four bits and then arithmetically added to the segment key. This clever scheme expands the index by a factor of 16; Google will soon be able to host over 64 billion pages!

Ah, youthful mod!

You've been (humorously) trolled. I suggest posting in this thread to remove your "+1 Informative", or getting a friend to mod it "Funny".

What the parent is describing is not what Google will do, but what DOS did: the above scheme is how MS-DOS managed memory, except that the "selector" and "offset" were both 16-bit numbers under DOS. (Although "segment" was the more usual term for "selector".) The segment number was shifted left four places -- or put more simply but less graphically, multiplied by 16 -- and then added to the offset number, to give the whole or "flat" address:

segment (in hex): 0001
offset ( in hex): 0002

segment is multipled by 16 (shifted left 4 bits or one hex digit of multipled by 16)

segment: 0001x
offset: 0002
---------------
total: 00012

This allowed DOS to use 16-bit numbers to address 2^20 = 1 MB of memory, but since DOS reserved the upper 384 KB for the (remapped) BIOS and peripheral cards, programs were able to address at most 640 KB of memory; the parent's mention of "64 billion pages" is probably an allusion (increased several orders of magnitude) to this DOS limit.

Of course, this was a kludge, pure and simple, required because DOS machines were 16-bit. Among other things, it allowed the same memory locations (all but the very top and bottom memory addresses) to be addressable by several different addresses, and discovering pointer aliasing it required calculations that, by their very nature couldn't be done wholly in the machines (16-bit) registers.

Consider: segment 4, offset 0 is 4 * 16 + 0 = 64,
and segment 3, offset 16 is 3 * 16 + 16 = 64,
and segment 2, offset 32 is 2 * 16 + 32 = 64
and segment 1, offset 48 is 1 * 16 + 48 = 64
and segment 0, offset 64 is 0 * 16 + 64 = 64:

so all five segment:offset pairs are apparently different but actually point to the same memory location.