Slashdot Mirror
Google's Secret Plans For All That Dark Fiber?
beat.net writes "Robert X. Cringely details the plan for all the dark fiber Google has been buying up: "The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center. Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.""
"Maybe Google will end up becoming the first sentient AI, if storing and finding association patterns between data is the essence of conscious thinking. The amount of information that Google has at its disposal is staggering, and poised to continue its growth with the introduction of Google Mail. What makes Google more than an extra-big database is the software that sits under that database, and its ability to continue scaling up. Jason Kottke has a great post on the big-picture trajectory of Google's technical efforts, and hits an essential point by noting that Google's focus has always been about what people are doing - searching, talking, shopping, and soon, emailing. Google's focus is human activity and the relationships between trillions of interactions. When I think about that , and then think about how much the daily use of the web has come to rely on Google, my joke about the system becoming sentient, by intent or by accident, seems a little less funny. " source : http://www.holycola.net/archives/000423.html
a Beowulf cluster of these puppies...
...Oh, we don't really need to Google seem to be building one.
I used to have a better sig but it broke.
Has anyone given any thought to how many of these peering points have excess power capacity for 5000 Opterons? Hmmmmm?
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
That's a nice idea but that thing must need some serious amount of power to run. Add the massive cooling system needed to keep the box runnning without melting. If they intend to just "drop" it anywhere... they have to think about security. You don't want some geek with a saw to steal your 3.5 PB array! Omni
Sounds like Google may be ready to go starbucks.
If most Google employees don't know about the storage container, how does THIS guy know about it???
5000 Opterons? It makes sense to put those near power plants / ice bergs. That's at least 500 kW of heat dissipation.
...I know what I want for Christmas this year.
>Google hired a pair of very bright industrial designers
:o)
I haven't yet met one that didn't think they were very bright. Industrial Designers invent stuff that takes 'ordinary' engineers years to throw away and build something else that will fly. No danger of anything happening here folks
Open Source Drum Kit, LPLC deve board - mjhdesigns.com
I don't understand how a few boxes full of Opterons automatically means taking over the Internet.
In my opinion, Google has penetrated the American market with its services as much as it can. It is probably looking to other places in the world to prop up its cash flow. You know, like a business, rather than a collection of world-domination-bent nerds?
Of course we want another Microsoft. We need something that can compete with MS.
If an Opteron produces say, on average, 50W heat output (I know this isn't accurate, but just as an example), 5000 Opterons would produce 250kW of heat. That would require an air conditioning unit larger than the building used to house the container.
If they were planing on building 300+ of these things, why not have built-in broadband wificasting ability... and just replace the internet without having to lay all that expensive cable?
The Admin and the Engineer
The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid.
Er, they plug a bunch of servers into the Internet and suddenly it's the Internet that's doing the processing and storage, not the servers? Sounds magical. Maybe I can plug my Playstation into the Internet and turn the entire Internet into a giant game.
Great Article. It just shows how quickly Google is becoming a global enterprise right under the nose of all the other huge companies such as Microsoft.
google.slashdot
>>We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage
They're just getting ready to run Windows Vista when it comes out.
Once I was a four stone apology. Now I am two separate gorillas.
Network latency. You get a faster response from a server in your own locale than on the other side of the world. And if you're doing network applications that are intended to compete with traditional local applications, then you need low latency.
Skype is in the same situation - they've been able to support so many users simply because their bandwidth is only used to setup the initial connection between the two parties, after that it's the telcos who are supporting and providing the infrastructure for the service that threatens them most. Now that Skype can make real money from its pay services, look for them to do something simliar to Google, to ensure the availability of their service.
Drag n' Drop DVD Recommendations
Cringley may be a fool, but he's almost right on this one. There's a saying in networking that you can't buy latency. The speed of light is just too low for Google's AJAX applications to take over the world - for many apps you can never get the latency low enough if you use only a few datacenters. So, the shipping container is irrelevant to the important part of this story. The key is that for Google to succeed in making online services as effective as desktop applications, they have to get the latency down. And there's only one way to do that, which is to move the servers close to the customers. To do that, they need a lot of data centers, and they need a lot of bandwidth between them, because when you connect they need to move your data to the nearest data center to you. So, they really do need to have a way to provide data centers quickly and easily to places all over the world. But Cringely doesn't seem to have realized why this is the only way Google can succeed in the long run. It appears you can buy latency after all if you spend enough. - Fzz
This is the man who brought us the mathematically impossible 6.5 mile 802.11 link with a passive repeater. The repeater that he never showed to anybody. He also shows us an idealistic world of a community cable and telephone company that nobody's ever seemed to find evidence of.
Saying that, when it comes to technology at least, he is speculative is something of an understatement. Take what he says with an extremely large grain of salt.
Google cutting in on Akamai's territory here?
http://www.akamai.com/
Half the big boy websites I visit seem to run through these guys. They seem to provide fat throughput for mega sites, apparently hosted in a distributed geographical fashion. I could just be imagining these things, though, because I really don't have a clue.
Many of the things I used to go to Google for, I go to Wikipedia instead. Now there is a category for which I go to Wikipedia for and a category I go to Google for. Actually they were distinct before, but the category of things I go to Wikipedia for, I fancied Web Directories might be useful for except that they weren't very robust and got out of date.
Now all we have to do is wait for some Google employee to play a Sony CD on this and these will become spam relays.
Perfect.
The alternative is everybody running their own stations in a massive wireless mesh network.
More than anything else, this seems to be a from of psycological warfare. Can you imagine the feelings of Bill G or Steve B when they read that article?
I mean if you are at Microsoft or Yahoo, where would you throw your money at? Office/Excel/Gamil like AJAX apps, this new 300 Uber Server threat or the next thing some bored reporter comes up with?
I don't know if this is report true or not, but I do know that Google is running a very effective psy-war campaign against their competitors.
Google's growth was in part made possible by heaps of commodity hardware. Hardware that was originally meant for standard lusers, cheap and unreliable. They built their systems for it and tolerate that. They change lots of haddrives in their datacenters and god knows what else.
What I'm trying to say is that for each of those googlecubes they need staff that regularly changes whatever hardware fails. With 3.5 Petabytes of storage and 5K processors it means that something will fail every single day that beast is powered. All that crammed inside 20/40 feet space (WTF does that mean?) means that heat will kill even more hardware.
So, yeah it should be possible, but not very likely.
fucktard is a tenderhearted description
...the puppy's on fire.
How are they going to cool these things?
With 300 data centers hosted at the important Internet peering points, and only 2-3 hops away from each user, Google will be easily able to offer a personal "Google Desktop" to each person, driven by FreeNX remote GUI technology (remember, NX can make X11, VNC and RDP run a multiple speeds with fractions of the bandwidth needed as compared to the protocols run natively).
Google will manage everything for its users: software upgrades, backups, search and organisation of personal data and files. Just like ISPs 20 years ago offered a monthly rate of 20 $US to connect to the internet (giving away a 2400 b/sec modem for a reduced price), Google could ask for a 20 $US fee (and give away a Google Thin Client embedded into a georgeous 17'' LCD screen that includes a EJ45 jack) to take care of people's computers.
I for one would sign in immediately.
So, Cringely is wrong. No need for AJAX office. It will all work with traditional GUI desktop programs, over an NX link that does not consume more than 40 kBits/sec for office productivity work.
So, Cringely is also right. The operating system doesn't matter to Google.
Even assuming the power and heat requirements of cramming that many opterons into that small a space could be dealt with, there's another, larger problem:
It's not fucking 1997 any more.
"Peering points" -- big, open-access traffic exchange handoffs like the old MAE-East and MAE-West used to be a big deal back in the late 90s, when OC-12 circuits were still rare and hideously expensive beasts, and Gigabit Ethernet was still a gleam in some 3Com engineer's eye.
In 2005, they simply don't matter. The big players (level3, MCI/Verizon, Qwest, SBC, etc) all exchange traffic over private fiber interconnects, and everyone else buys transit from the big guys directly or ponies up for a switch port at Equinox, PAIX/Switch&Data or some other 'carrier neutral' colocation center. Dropping a datacenter-in-a-box onto MAE-east or any of its surviving ilk would buy Google precisely nothing.
(And nevermind the fact that google is documented to own thousands upon thousands of unused square feet of datacenter space already: they went on a very well-thought-out buying spree in 2000-2001 when all the dot-com datacenter companies were going out of business, and are very well provisioned for the forseeable future as a result.)
Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...
News for Nerds. Stuff that Matters? Like hell.
> what's the point of putting network latency between all those shipping containers?
To remove the network latency between them and you.
They're not being used "for computing" in the sense you're envisioning. For one thing, 5000 Opterons is enough to tackle pretty much any problem you'd care to throw at it, so there's no need to talk to anyone else. For another thing, they wouldn't be doing big computations, they'd be doing massive numbers of small ones. Think Gmail. 3.5PB is enough to store an awful lot of email, and a few thousand Opterons can run rather a lot of simultaneous HTTP connections from people accessing the mail. Add in a fast network link (for talking to all those many people accessing the mail, and for replicating everything offsite), and you're set.
Cringeley's penchant for sensationalism aside, it's pretty clear that Google's got the expertise and the mindset to deal with problems that start with "if we had 10,000 fast CPUs, 10,000 hard disks, and 10,000 GB of RAM...". Google's rapidly expanding, and has been ever since they started. Back when Google fit in a closet, a new server constituted a big expansion. I'm not surprised that these days their unit of expansion is a tractor trailer with a few dozen racks in it. And if you've got something that packages up that nicely, it only makes sense to pepper the globe with capacity.
High-speed Road Trip (18.000KPH)
Actually this article came in on my rss reader yesterday, I did some quick back on the envelope counting, and I'm not sure you can get a container for $500.000. I read before that Google used to have racks with wheels under them, completely full with 1U servers and fully cabled. They only had to plug in power and network and they were all set. When the datacenter went bankrupt, they just wheeled the racks off to another location. So assuming that they are only using the containers for shipping:
;)
One 20Ft container is:
* Length (20Ft)
* Width (8Ft)
* Height (8.5Ft)
That means you can get about 12 * 19" racks in, using 4 rows, about 64U high. That means a total of 3072 servers, using dual socket, dual core opterons, that's 12288 cores. Each server with 8 memory sockets + 4 disks, that's 24,576GB of RAM (1G sticks) and 6,144,000GB of Storage (500G disks). With some guestimate figures on current prices, I'd say one of those container would be worth about $12,500,000.
But then again, from a quick Google, they have about $3 billion in cash, and that's a lot of containers....
PS: I'm european using metrics mostly, so they're might be a small conversion problem here and there
A 40 ft shipping container has a surface area of something more like:
(40 * 8) * 4 + (8 * 8) * 2 ==
1408 sq. ft.
which, for 1 megawatt, is more like 710 watts/ sq. ft.
"I assumed blithely that there were no elves out there in the darkness"
...will we still love them? I get a feeling Microsoft's monopoly will look like a minor bother compared to Google's omnipresence one day. Google, the Evil Empire?
Sounds like Google is trying that out.
There's nothing that exotic about this. The military builds racks of electronics into shipping containers all the time. It's mostly a cable management and maintenance access problem. You have to be able to do everything from the front of the rack, which requires some design work but isn't rocket science.
Even with liquid cooling it's a hard problem. For one thing you need 5000 little hoses, and beefy pumps to get the water through there at a reasonable speed. Opterons are specced to run up to about 85C (depends slightly on model / family). Suppose you've got incoming water at 10C, and heat it up all the way to 85C. That's 75C difference, or 313.5J/g of water you're taking away. That works out to 5.75 million grams of water per hour, or just under 6000 liters per hour. You can't just dump it into a lake or river or you'll completely nuke the resident ecosystem. It's a manageable number from the point of view of getting it through the machines, but it's still an awful lot of energy to get rid of.
The sort of temperature-differential energy recovery you speak of is technically possible but isn't efficient enough to substantially reduce the cluster's power requirements, and thus its need to vent waste heat.
High-speed Road Trip (18.000KPH)
By unplugging it.
My other car is first.
They don't have control. As long as there is one search company in business Google will not have control. You give them control but competition is so low barrier that google will not be "The One"
What if one day googlenet becomes a place of education? Or even a government that would help locate you into your new job?
Is doing what we're told bad altogether or are people going to open their eyes?
Google will win when google holds 1) the speed (see latency discussions above; taking their multiple redundant data centers and moving them closer to the customer solves that problem) and 2) the vast majority of the information itself.
In fact, I can see this working because the media companies themselves (the labels, the networks, the studios), the very ones who can't afford to have all those redundant servers and data and managing their own damn network, will be the ones to finance it all by buying local cache space. rather than serve up "Welcome Back, Kotter" from media1.abc_and_nick_at_nite_joint_server.com and killing that server, they'll serve it up from "abc_nick_at_nite_joint.googlemedia.com" and the DNS system will return the nearest google.com to the user -- boom, no latency and no 1 million hits all on the same server killing it in seconds from a public announcement of "first season of Friends is available now!".
A site served up by google in this way would survive a slashdotting without any second thoughts.
Google cache, google mail, google groups, google maps, google yellow pages, google-licensed 3rd party services serving up all the above google stuff, all financed by google advertising and all of the media paying for hosting on google's redundant servers rather than killing their own network servers...
with all of that information in google's hands, able to return the fastest searches around, the other search tools, *especially* microsoft's late entry into this market, simply won't matter.
google doesn't care about the search business as such anymore; they've already won it big enough to make the search market itself a commodity as much as microsoft made the OS a commodity, much as they kept insistent they weren't going to...
"But remember, most lynch mobs aren't this nice." (H.Simpson)
-- Joe
Back in 1991 I worked for a wireless company that tried using data containers for quickly deployed cellular switch and cell sites. The idea would be to prebuild these at a central location and then drop them at areas where they needed to go up.
The idea was good, except for a couple of problems.
These shipping containers are nothing but a giant metal box. Grounding can become an issue, so can accidnentally having the box be one of of the poles for a DC based power system. If you are near an active AM tower, the box becomes a giant antenna and it's virtually impossible to filter out the AM signal internally.
Last, and certainly not least, these shipping containers are vulnerable to rust and other problems due to exposure to the elements. That can take several years (5 or so) if the box is in perfect shape at the start, but if they are using used boxes then it can take less than 2 years for rust holes to be a problem.
Plus, physical security isn't all that good unless the walls are beefed up.
I'm hoping these are not "standard" shipping containers, just something that looks like them.
This grand experiment with shipping containers for cellular applications was an attempt to make it cheaper to deploy equipment to new locations. And, shipping containers (especially used) were a _LOT_ cheaper than fibrebond or other prefab buildings for that purpose. Of course, the fibrebond building had a lifespan a lot longer than 2 to 5 years. So, you get what you pay for.
. 62,400 repetitions make one truth -- Brave New World, Aldous Huxley
So you need to cool about 600KW worth of heat dissipation. A ton of ac/ is about 3.5 KW or in the neighborhood of 180 tons of a/c required that will take about 1.4 kw per ton to eject that energy. So you'll need another container-sized unit to hold the a/c and then some sort of radiator or cooling tower to eject the heat.
Clearly you're talking about serious energy density here with cooling which is on the order of what it took to cool a 637 class nuclear submarine underway in moderately cool water. Of course you'd really need TWO of those a/c units because you will want to protect that investment (still fits in the same footprint) and be ready to pay some substantial utility bills.
The crazy thing is, if it isn't in Google's cache today, it will be in the next couple of days once Google crawls this page.
Ironically, the Wikipedia server is so slow, I'll often Google whatever Wikipedia entry I'm looking for. It usually doesn't require any special syntax, e.g., "Tin Foil Hat" Wikipedia although sometimes it's handy to be more explicit, like site:en.wikipedia.org "tin foil hat" Both queries (plus I'm Feeling Lucky) return the desired search result in this instance. And not only does Google tend to return the desired page faster, it also makes for more precise Wikipedia searches.
What's this really about?
It's about that Google want to be able to move Google computers around in any datacenter asap. We know that google uses a grid of single computers, that all compute the search results as fast as possible. All these computers create a space-problem (physical space) at any datacenter that Google owns. Also, shipping these computers around costs money. I bet google store this "secret package" just to be able to send it around anywhere where there suddenly is a problem with the network.. We all know how much money google loose if they are experiencing downtime...
Maybe Google will end up becoming the first sentient AI,...
Not AI, try TV and broadcasting on demand. What else could you do with that much storage, CPU and memory?
My guess, and it is only a guess, GoogleTV is geting a lot bigger and going to carry some 500,000 to 1,000,000 full length movies and shows or something. And it will be so kewl for us to watch what we want and not what some clown wants us to watch. I suspect it will change ratings too as your not stuck picking from the least boring shows because there is nothing better on. No more pre-empting Enterprise.
http://news.com.com/GoogleTV+is+hiring/2100-1026 _3-5876654.html
Or maybe I am being wishful.
Here's the link http://www.everything2.com/index.pl?node_id=176552 3&lastnode_id=1765513
...I immediately thought of the crates from the 'Worms' games.
BytesTemplar.com
Gridnet did the same thing back in the 90s. Except they used small prefab buildings (the same type used for regen huts). Inside were racks of modems, routers, backup batteries, and SONET transmission equipment. They would buy or rent a small patch of land in a parking lot within the magical "zero miles" of the bell central office. Truck their pre-built "point of presence" to the location, have a crane move it into place, connect the power, and start ordering circuits. This allowed them to get modems into nearly every LATA within a matter of months.
Why then buy all the fiber ? There are areas of the country that aren't attractive to telcos/cable co.s.
I hope they succeed along with their WIFI access.
Good job boys.
One more point if anyone at Google is listening. How about a tax on chip companies to use this highspeed access. It would be nice if they could help support it since they are the ones who benefit.
It's a reference from Keith Laumer's novel "The Great Time Machine Hoax". Don't know if I spelled it quite right.
The higher the technology, the sharper that two-edged sword.
As for lots of hot water - if you are worried about heating up part of a small lake you just run the water through some open drains to cool off a bit first. If you are going to recirculate it you let it drop down from a height to get some evaporative cooling - the principle behind cooling towers.
Wikipedia is a big help in refining your searches enough to use Google. A few bad Google searches are all the proof I needed to believe Wiliam Gibson's assertion that for the first time in history, the new threat to civilzation is being overwhelmed by too much information to make meaningful choices.
In addition, I've noticed that any time I don't find much help on Google, invariably I can easily rephrase what I want as a series of very general questions, and then feeding a few of them to Ask Jeeves will get genuinely useful hits. The more I have only the simple, "obvious" lay-man's questions on the subject the more some of Google's alternatives are the way to go.
The last time for me was finding useful stuff on DIY satellite dishes. If you don't yet know any of the terms (which I didn't, then), like C-band, Ku-band, LNBF, Free To Air, and such, Google gives a huge number of useless to just plain evil links. It's not just lots of people who want to sell you a overpriced 'complete solution', but 6 year old, never updated web pages that want to sell you a dish 5 meters across at 30,000$, "Christian" Broadcasters who want you to help them buy more gear and have replaced all the standard terms with "It needs a new thingee, around here we just call it a Jesus-box, won't you please help?", and Utar Pradesh complaining about how Nepal either needs to translate their G2S's signal out of Hindi, or into it.
The only way around it starting from Google seems to be running across about 5 of the 'insider' terms, Wiki for every single one of them, look for what other terms are links, keep Wiki'ing, and thus fairly swiftly refine your search. Basically, without Wikipedia or some other shortcut, you have to get to about the depth where you know how (and WHY) global positioning works differently for civilian uses (like Onstar) and Military uses, and which idiots in Congress voted which way on it, just as a side effect of learning enough to build your own TV receiver (assuming you're already a fair solder jockey and don't need to learn what a MOSFET is). That's a terribly steep learning curve.
This is for a subject that's not really all that esoteric, but it has a half dozen facets, all of which Google can't sort out by simple searches - for just one example mistaking sites that are about beaming 'politally free' info into authoritarian countries for 'own instead of rent type free' consumer solutions. In the process, you are likely to still not know many things that might be of much more interest to most people wanting to build or just own a home satellite system, such as the existence of PC card recievers and motor controllers, or if you can combine satellite internet access with TV reception when you are trying to avoid just renting a dish.
Something like "How do I get satellite TV free?" on Ask Jeeves will get you that whole list of terms and some basic definitions and diagrams very quickly. Ask Jeeves seems to try and match the whole question if it can, before searching for phrases and keywords, and that leads to pages that are set up in question and answer format. In this case, the first link back when I tried it was to a "How things work" page that gave accurate and generally unbiased info, frequently updated.
(Warning, I haven't tried this lately, for all I know Ask Jeeves has gone out of business and I missed it).
Who is John Cabal?
The specifications [see footnote for a few other sites] state 146 mm) x (101.6 mm) x (25.4 mm) x 7 340 = 2.76551705 m^3,
and, running with the article's numbers, let's see how much of 20 feet cubed that is... (article: the most storage, memory and power support into a 20...foot box -- note that a BOX of course is less cubic area than a 20-foot cube)....
((146 mm) x (101.6 mm) x (25.4 mm) x 7 340) / (20 (feet^3)) = 4.88316565...
WHAT? it's not a fraction, but larger by a factor of 4+??? Just for the hard-drives? Even when we assumed a CUBE???
Man, I want some of the shit that guy's smoking. I was expecting to debunk with just the hard-drives taking an impossibly large percentage of the proposed 20-foot "box". But....man. Cringely must not have done even a basic sanity check. (And remember, I'm pretty sure he didn't have a 20 foot high, 20 foot wide box in mind, or he would have said cube. To a writer, a "20-foot box" sounds like an elongated storage container, e.g. 8x8x20 feet.... BTW that's the first hit for 20 foot storage container, I can only assume a writer would have such a thing in mind...)
English and math, people, English AND math.
Footnote:
Other sources for specifications:
Yeah, that's a common problem in our generation.
Grown up with TV and playstation our attention span has degraded to..Oh, something beeped, hold'on a sec.
Taken from http://www.incendiary.ws/node/194
Crmblznski's Limit, sometimes spelled Crizmblski's Limit, has its origins in Keith Laumer's novel "The Great Time Machine Hoax" [1].
The basic theorem is that there is a finite limit to the complexity of any given machine, which specifically precludes the operation of "a machine with sufficiently extensive memory banks, adequately cross-connected and supplied with a vast store of data, [that by its very essence] would be capable of performing prodigious intellectual feats simply by discovering and exploring relationships among apparently unrelated facts." The Limit is an irrational number, much like Pi, in that the total complexity of machine is wholy dependent upon both hardware and software designs.
Promote freedom; fight fascism.
I just dug out my copy of the book ... it was actually spelled "Crmblznski". My mistake.
Here's a link to the entire text of the novel, for anyone that's interested:
The Great Time Machine Hoax
The higher the technology, the sharper that two-edged sword.