Slashdot Mirror
Google's Secret Plans For All That Dark Fiber?
beat.net writes "Robert X. Cringely details the plan for all the dark fiber Google has been buying up: "The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center. Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.""
"Maybe Google will end up becoming the first sentient AI, if storing and finding association patterns between data is the essence of conscious thinking. The amount of information that Google has at its disposal is staggering, and poised to continue its growth with the introduction of Google Mail. What makes Google more than an extra-big database is the software that sits under that database, and its ability to continue scaling up. Jason Kottke has a great post on the big-picture trajectory of Google's technical efforts, and hits an essential point by noting that Google's focus has always been about what people are doing - searching, talking, shopping, and soon, emailing. Google's focus is human activity and the relationships between trillions of interactions. When I think about that , and then think about how much the daily use of the web has come to rely on Google, my joke about the system becoming sentient, by intent or by accident, seems a little less funny. " source : http://www.holycola.net/archives/000423.html
a Beowulf cluster of these puppies...
...Oh, we don't really need to Google seem to be building one.
I used to have a better sig but it broke.
That's a nice idea but that thing must need some serious amount of power to run. Add the massive cooling system needed to keep the box runnning without melting. If they intend to just "drop" it anywhere... they have to think about security. You don't want some geek with a saw to steal your 3.5 PB array! Omni
Sounds like Google may be ready to go starbucks.
If most Google employees don't know about the storage container, how does THIS guy know about it???
I'm sure google has. It's not like you can't have another truck towing a generator following the truck towing the portable datacenter.
I used to work at a datacenter and we had a generator small enough that you could fit 12 of them in a shipping container, and the genny was enough to run a 500 machine datacenter for three days without refueling. The portable datacenter may well have a generator included.
Reinvent the wheel only at either a lower cost, greater effectiveness, or your own personal enrichment and satisfaction.
>>We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage
They're just getting ready to run Windows Vista when it comes out.
Once I was a four stone apology. Now I am two separate gorillas.
Network latency. You get a faster response from a server in your own locale than on the other side of the world. And if you're doing network applications that are intended to compete with traditional local applications, then you need low latency.
Cringley may be a fool, but he's almost right on this one. There's a saying in networking that you can't buy latency. The speed of light is just too low for Google's AJAX applications to take over the world - for many apps you can never get the latency low enough if you use only a few datacenters. So, the shipping container is irrelevant to the important part of this story. The key is that for Google to succeed in making online services as effective as desktop applications, they have to get the latency down. And there's only one way to do that, which is to move the servers close to the customers. To do that, they need a lot of data centers, and they need a lot of bandwidth between them, because when you connect they need to move your data to the nearest data center to you. So, they really do need to have a way to provide data centers quickly and easily to places all over the world. But Cringely doesn't seem to have realized why this is the only way Google can succeed in the long run. It appears you can buy latency after all if you spend enough. - Fzz
1W = the amount of power required to heat 1g of water 1 degree celsius in 1 second (1 J/sec).
1 cup of coffee: 0.2 litres (200g) heated from 10 to 100 degrees celsius (90 degrees) = 18 KJ.
250 KW: 14 cups of coffee per second.
The answer to "where do we put these puppies"?
Next to Starbucks.
This is the man who brought us the mathematically impossible 6.5 mile 802.11 link with a passive repeater. The repeater that he never showed to anybody. He also shows us an idealistic world of a community cable and telephone company that nobody's ever seemed to find evidence of.
Saying that, when it comes to technology at least, he is speculative is something of an understatement. Take what he says with an extremely large grain of salt.
...the puppy's on fire.
How are they going to cool these things?
With 300 data centers hosted at the important Internet peering points, and only 2-3 hops away from each user, Google will be easily able to offer a personal "Google Desktop" to each person, driven by FreeNX remote GUI technology (remember, NX can make X11, VNC and RDP run a multiple speeds with fractions of the bandwidth needed as compared to the protocols run natively).
Google will manage everything for its users: software upgrades, backups, search and organisation of personal data and files. Just like ISPs 20 years ago offered a monthly rate of 20 $US to connect to the internet (giving away a 2400 b/sec modem for a reduced price), Google could ask for a 20 $US fee (and give away a Google Thin Client embedded into a georgeous 17'' LCD screen that includes a EJ45 jack) to take care of people's computers.
I for one would sign in immediately.
So, Cringely is wrong. No need for AJAX office. It will all work with traditional GUI desktop programs, over an NX link that does not consume more than 40 kBits/sec for office productivity work.
So, Cringely is also right. The operating system doesn't matter to Google.
Even assuming the power and heat requirements of cramming that many opterons into that small a space could be dealt with, there's another, larger problem:
It's not fucking 1997 any more.
"Peering points" -- big, open-access traffic exchange handoffs like the old MAE-East and MAE-West used to be a big deal back in the late 90s, when OC-12 circuits were still rare and hideously expensive beasts, and Gigabit Ethernet was still a gleam in some 3Com engineer's eye.
In 2005, they simply don't matter. The big players (level3, MCI/Verizon, Qwest, SBC, etc) all exchange traffic over private fiber interconnects, and everyone else buys transit from the big guys directly or ponies up for a switch port at Equinox, PAIX/Switch&Data or some other 'carrier neutral' colocation center. Dropping a datacenter-in-a-box onto MAE-east or any of its surviving ilk would buy Google precisely nothing.
(And nevermind the fact that google is documented to own thousands upon thousands of unused square feet of datacenter space already: they went on a very well-thought-out buying spree in 2000-2001 when all the dot-com datacenter companies were going out of business, and are very well provisioned for the forseeable future as a result.)
Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...
News for Nerds. Stuff that Matters? Like hell.
> what's the point of putting network latency between all those shipping containers?
To remove the network latency between them and you.
They're not being used "for computing" in the sense you're envisioning. For one thing, 5000 Opterons is enough to tackle pretty much any problem you'd care to throw at it, so there's no need to talk to anyone else. For another thing, they wouldn't be doing big computations, they'd be doing massive numbers of small ones. Think Gmail. 3.5PB is enough to store an awful lot of email, and a few thousand Opterons can run rather a lot of simultaneous HTTP connections from people accessing the mail. Add in a fast network link (for talking to all those many people accessing the mail, and for replicating everything offsite), and you're set.
Cringeley's penchant for sensationalism aside, it's pretty clear that Google's got the expertise and the mindset to deal with problems that start with "if we had 10,000 fast CPUs, 10,000 hard disks, and 10,000 GB of RAM...". Google's rapidly expanding, and has been ever since they started. Back when Google fit in a closet, a new server constituted a big expansion. I'm not surprised that these days their unit of expansion is a tractor trailer with a few dozen racks in it. And if you've got something that packages up that nicely, it only makes sense to pepper the globe with capacity.
High-speed Road Trip (18.000KPH)
That's it! The opterons produce heat, which boils water which runs the turbines to generate electricity to power the opterons! My god man! It's the Google perpetual motion machine!
Sounds like Google is trying that out.
There's nothing that exotic about this. The military builds racks of electronics into shipping containers all the time. It's mostly a cable management and maintenance access problem. You have to be able to do everything from the front of the rack, which requires some design work but isn't rocket science.
Here's the link http://www.everything2.com/index.pl?node_id=176552 3&lastnode_id=1765513