Slashdot Mirror
Google's Secret Plans For All That Dark Fiber?
beat.net writes "Robert X. Cringely details the plan for all the dark fiber Google has been buying up: "The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center. Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.""
"Maybe Google will end up becoming the first sentient AI, if storing and finding association patterns between data is the essence of conscious thinking. The amount of information that Google has at its disposal is staggering, and poised to continue its growth with the introduction of Google Mail. What makes Google more than an extra-big database is the software that sits under that database, and its ability to continue scaling up. Jason Kottke has a great post on the big-picture trajectory of Google's technical efforts, and hits an essential point by noting that Google's focus has always been about what people are doing - searching, talking, shopping, and soon, emailing. Google's focus is human activity and the relationships between trillions of interactions. When I think about that , and then think about how much the daily use of the web has come to rely on Google, my joke about the system becoming sentient, by intent or by accident, seems a little less funny. " source : http://www.holycola.net/archives/000423.html
5000 Opterons? It makes sense to put those near power plants / ice bergs. That's at least 500 kW of heat dissipation.
Problem solved (portable nuclear reactor).
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
If an Opteron produces say, on average, 50W heat output (I know this isn't accurate, but just as an example), 5000 Opterons would produce 250kW of heat. That would require an air conditioning unit larger than the building used to house the container.
Cringley may be a fool, but he's almost right on this one. There's a saying in networking that you can't buy latency. The speed of light is just too low for Google's AJAX applications to take over the world - for many apps you can never get the latency low enough if you use only a few datacenters. So, the shipping container is irrelevant to the important part of this story. The key is that for Google to succeed in making online services as effective as desktop applications, they have to get the latency down. And there's only one way to do that, which is to move the servers close to the customers. To do that, they need a lot of data centers, and they need a lot of bandwidth between them, because when you connect they need to move your data to the nearest data center to you. So, they really do need to have a way to provide data centers quickly and easily to places all over the world. But Cringely doesn't seem to have realized why this is the only way Google can succeed in the long run. It appears you can buy latency after all if you spend enough. - Fzz
Actually, I'm curious how Cringeley thinks Google can get the hardware at the prices he quotes in the article. I'm sure he's given it some thought, but unless they're getting hardware at below-cost prices, I don't see how it can be done. The CPUs cost about $50 each to make, so that's $250k for chips. Then you need a few petabytes of disk. I don't know what the manufacturing cost is for disks, but I'd guess about $50 there too. Say $50 for a 500GB drive. That's a few thousand drives to reach the several petabytes, and there goes the rest of his half-million dollars. You still need motherboards, RAM, power supplies, chassis, racks, switches, etc.
I'm not saying he's wrong, but I'd be curious to hear where I've gone astray in my figuring.
Not to mention, of course, the enormous electrical requirements this thing would have, as you've commented. If we round the CPU's power consumption up to account for all the support machinery, and figure 100W per CPU, this neat little semi-load is going to want half a megawatt, plus cooling. Just the disk array will chew through 50kW or so. Even from a power plant's perspective, that's a pretty hefty chunk of juice.
High-speed Road Trip (18.000KPH)
Google cutting in on Akamai's territory here?
http://www.akamai.com/
Half the big boy websites I visit seem to run through these guys. They seem to provide fat throughput for mega sites, apparently hosted in a distributed geographical fashion. I could just be imagining these things, though, because I really don't have a clue.
Many of the things I used to go to Google for, I go to Wikipedia instead. Now there is a category for which I go to Wikipedia for and a category I go to Google for. Actually they were distinct before, but the category of things I go to Wikipedia for, I fancied Web Directories might be useful for except that they weren't very robust and got out of date.
At peak performance, one Opteron will draw (conservatively) 1 Amp, and use (more conservatively) 100 Watts. Double it to include the disks, etc, and we're probably still conservative at 200 W * 5000 CPUs = 1 Megawatt, which basically all gets converted to heat, all in a box that size. Surface area of the box?
40 * 40 * 40 feet -> 104 Watts/sqft out...
...the puppy's on fire.
How are they going to cool these things?
With 300 data centers hosted at the important Internet peering points, and only 2-3 hops away from each user, Google will be easily able to offer a personal "Google Desktop" to each person, driven by FreeNX remote GUI technology (remember, NX can make X11, VNC and RDP run a multiple speeds with fractions of the bandwidth needed as compared to the protocols run natively).
Google will manage everything for its users: software upgrades, backups, search and organisation of personal data and files. Just like ISPs 20 years ago offered a monthly rate of 20 $US to connect to the internet (giving away a 2400 b/sec modem for a reduced price), Google could ask for a 20 $US fee (and give away a Google Thin Client embedded into a georgeous 17'' LCD screen that includes a EJ45 jack) to take care of people's computers.
I for one would sign in immediately.
So, Cringely is wrong. No need for AJAX office. It will all work with traditional GUI desktop programs, over an NX link that does not consume more than 40 kBits/sec for office productivity work.
So, Cringely is also right. The operating system doesn't matter to Google.
Even assuming the power and heat requirements of cramming that many opterons into that small a space could be dealt with, there's another, larger problem:
It's not fucking 1997 any more.
"Peering points" -- big, open-access traffic exchange handoffs like the old MAE-East and MAE-West used to be a big deal back in the late 90s, when OC-12 circuits were still rare and hideously expensive beasts, and Gigabit Ethernet was still a gleam in some 3Com engineer's eye.
In 2005, they simply don't matter. The big players (level3, MCI/Verizon, Qwest, SBC, etc) all exchange traffic over private fiber interconnects, and everyone else buys transit from the big guys directly or ponies up for a switch port at Equinox, PAIX/Switch&Data or some other 'carrier neutral' colocation center. Dropping a datacenter-in-a-box onto MAE-east or any of its surviving ilk would buy Google precisely nothing.
(And nevermind the fact that google is documented to own thousands upon thousands of unused square feet of datacenter space already: they went on a very well-thought-out buying spree in 2000-2001 when all the dot-com datacenter companies were going out of business, and are very well provisioned for the forseeable future as a result.)
Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...
News for Nerds. Stuff that Matters? Like hell.
Actually this article came in on my rss reader yesterday, I did some quick back on the envelope counting, and I'm not sure you can get a container for $500.000. I read before that Google used to have racks with wheels under them, completely full with 1U servers and fully cabled. They only had to plug in power and network and they were all set. When the datacenter went bankrupt, they just wheeled the racks off to another location. So assuming that they are only using the containers for shipping:
;)
One 20Ft container is:
* Length (20Ft)
* Width (8Ft)
* Height (8.5Ft)
That means you can get about 12 * 19" racks in, using 4 rows, about 64U high. That means a total of 3072 servers, using dual socket, dual core opterons, that's 12288 cores. Each server with 8 memory sockets + 4 disks, that's 24,576GB of RAM (1G sticks) and 6,144,000GB of Storage (500G disks). With some guestimate figures on current prices, I'd say one of those container would be worth about $12,500,000.
But then again, from a quick Google, they have about $3 billion in cash, and that's a lot of containers....
PS: I'm european using metrics mostly, so they're might be a small conversion problem here and there
What if one day googlenet becomes a place of education? Or even a government that would help locate you into your new job?
Is doing what we're told bad altogether or are people going to open their eyes?
There are too many dead zones between major cities .
... At least they are cheaper than satellites and fly
Great distances of rural farmland .
http://www.locustworld.com/ is a awesome idea, and they have done some
great things, but there are many places off the coast where distances
between cities is greater than the range of WiFi unless u use the ballon trick .
But long shots in the midwest are going to have to route via
conventional telecom unless we setup telecommunications WiFi ballons .
www.21stcenturyairships.com
Someone has to pay for them
high enough to avoid all wind .
Ex-MislTech
google "32 trillion offshore needs IRS attention"
Back in 1991 I worked for a wireless company that tried using data containers for quickly deployed cellular switch and cell sites. The idea would be to prebuild these at a central location and then drop them at areas where they needed to go up.
The idea was good, except for a couple of problems.
These shipping containers are nothing but a giant metal box. Grounding can become an issue, so can accidnentally having the box be one of of the poles for a DC based power system. If you are near an active AM tower, the box becomes a giant antenna and it's virtually impossible to filter out the AM signal internally.
Last, and certainly not least, these shipping containers are vulnerable to rust and other problems due to exposure to the elements. That can take several years (5 or so) if the box is in perfect shape at the start, but if they are using used boxes then it can take less than 2 years for rust holes to be a problem.
Plus, physical security isn't all that good unless the walls are beefed up.
I'm hoping these are not "standard" shipping containers, just something that looks like them.
This grand experiment with shipping containers for cellular applications was an attempt to make it cheaper to deploy equipment to new locations. And, shipping containers (especially used) were a _LOT_ cheaper than fibrebond or other prefab buildings for that purpose. Of course, the fibrebond building had a lifespan a lot longer than 2 to 5 years. So, you get what you pay for.
. 62,400 repetitions make one truth -- Brave New World, Aldous Huxley
What's this really about?
It's about that Google want to be able to move Google computers around in any datacenter asap. We know that google uses a grid of single computers, that all compute the search results as fast as possible. All these computers create a space-problem (physical space) at any datacenter that Google owns. Also, shipping these computers around costs money. I bet google store this "secret package" just to be able to send it around anywhere where there suddenly is a problem with the network.. We all know how much money google loose if they are experiencing downtime...
The specifications [see footnote for a few other sites] state 146 mm) x (101.6 mm) x (25.4 mm) x 7 340 = 2.76551705 m^3,
and, running with the article's numbers, let's see how much of 20 feet cubed that is... (article: the most storage, memory and power support into a 20...foot box -- note that a BOX of course is less cubic area than a 20-foot cube)....
((146 mm) x (101.6 mm) x (25.4 mm) x 7 340) / (20 (feet^3)) = 4.88316565...
WHAT? it's not a fraction, but larger by a factor of 4+??? Just for the hard-drives? Even when we assumed a CUBE???
Man, I want some of the shit that guy's smoking. I was expecting to debunk with just the hard-drives taking an impossibly large percentage of the proposed 20-foot "box". But....man. Cringely must not have done even a basic sanity check. (And remember, I'm pretty sure he didn't have a 20 foot high, 20 foot wide box in mind, or he would have said cube. To a writer, a "20-foot box" sounds like an elongated storage container, e.g. 8x8x20 feet.... BTW that's the first hit for 20 foot storage container, I can only assume a writer would have such a thing in mind...)
English and math, people, English AND math.
Footnote:
Other sources for specifications: