Google Pulling Back the Veil On Its Custom-Built Data Centers

jfruh writes: In the mid-'00s, as Google scaled up its data centers to meet increasing demand, "we could not buy, for any price, a data-center network that would meet the requirements of our distributed systems," says Amin Vahdat, the company's networking technical lead. So they had to build their own software-defined networks inside what were essentially vast warehouse-sized computers. And now the company is starting to tell the world how they did it.

Comments

And you all still don't need it

[magamiako1]

Right now there's a massive push for even the smallest web projects to be "cloud scale" to this degree. No, the crazy custom shit that Google does in their datacenter is not something you need in YOUR datacenter.

Re:And you all still don't need it

[turkeydance]

Google is saying how Google did it. not objective.

Re:And you all still don't need it

[hawguy]

Right now there's a massive push for even the smallest web projects to be "cloud scale" to this degree. No, the crazy custom shit that Google does in their datacenter is not something you need in YOUR datacenter.

I thought the current push was for the smallest web projects to run on a public cloud like Google or Amazon, so if you need this kind of scale, you can have it, but you don't have to pay for it until you need it. Is there some other push to drive companies to create their own datacenters to give them the ability to scale?

I know my company is completely in the cloud, there are literally no servers in the "server room", just switches and a firewall. Oh, and two 1 Gig fiber connections from 2 providers.

Re:And you all still don't need it

[ShanghaiBill]

Right now there's a massive push for even the smallest web projects to be "cloud scale" to this degree.

No, the "push" is in the opposite direction. Unless you are in the business of running datacenters, you probably shouldn't be running your own datacenter. Google/Amazon/Azure can do it better than you can.

Re:And you all still don't need it

[haruchai]

Are those fiber links really from 2 independent providers? Where my company most needs redundancy, the fiber is owned & maintaned by a single provider and every one is a reseller.

Re:And you all still don't need it

[Bing Tsher E]

What could possibly go wrong with that?

Obviously the only requirement for a good data center is scalability.

Re:And you all still don't need it

[davester666]

Upload your data to both Google and Amazon! They'll fix you, but good.

Re:And you all still don't need it

[dave420]

You just made that up. Seriously. You start talking about web projects, then move seamlessly to datacenters while attempting to make the same point. Web projects are not datacenters.

Re:And you all still don't need it

[FridgeFreezer]

Some providers will offer redundant links, even geographically separated redundant links (fibres enter the building though 2 separate ducts, head off in 2 different directions back to 2 different exchanges via 2 different routes... of course, no-one wants to pay for that so they very rarely actually do it.

If you buy "dark fibre" then you needn't care about a one provider's hardware/network being subject to a particular failure as you're only using a length of glass they happened to provide. Again though, is spendy.

Re:And you all still don't need it

[haruchai]

One of the divisions of my organisation acquired quite a bit of dark fibre for almost next to nothing during the dotcom meltdown as providers were going out of business .
They even have 4 manholes in a 2 mile radius that's "theirs" - the covers have the old corporate name stamped on them.

Re:And you all still don't need it

[pnutjam]

Well...
Maybe not Azure...

Re:And you all still don't need it

[Anonymous Coward]

Quite the generalization there. You also left out Colos. And the fact that lots of IT shops run datacenters, and run them well.

Re:And you all still don't need it

[hawguy]

Are those fiber links really from 2 independent providers? Where my company most needs redundancy, the fiber is owned & maintaned by a single provider and every one is a reseller.

One fiber goes out to the street where it connects to AT&T's fiber ring, a single fiber cut could take out our building, but it would take 2 cuts to take out AT&T's ring.

The second fiber connection goes up to the roof where it's connected to a (microwave?) transmitter that hits an antenna about 5 miles away where the provider has their own backhaul to an IXP.

Re:And you all still don't need it

[phantomfive]

But hey, if someone wants to pay me to build it, I'll sure have fun doing it.

Screw this layout change

[Anonymous Coward]

Dice Holdings, Inc. is now DHI Group, Inc. As far as I can tell, the I still stands for inc, so now I suppose they're doubly incorporated. They certainly act like a corporation twice over.

Re:Screw this layout change

[MightyMartian]

Sooner or later Dice is going to render this site unusable. What a pack of fucking retards.

Re:Screw this layout change

[PIBM]

I think you give them too much credits; they've been trying for years and haven't yet succeeded :)

Re:Screw this layout change

[Bing Tsher E]

For some reason, it doesn't look any different to me than it did a year or a month or week ago. But I bitchslapped Slashdot's configuration so long ago that the last time I looked, I couldn't even find any configuration controls to do whatever I did. My goal at the time was 'simple' and they offered some 'simple' options if I recall. I haven't seen any share buttons.

But I am also hesitant to change ANYTHING in the settings for my account. This site is fragile in strange ways. That's been my experience in the past.

Pulled back veil

[belthize]

To expose a burka.

What a singularly uninformative article. I guess if you've been living in a cave the past 10 years you might not know that Google has a lot of computers and needs a lot of networking and had a project called Firehose (that didn't work) and one called Jupiter which apparently has.

I'm not intrigued by your ideas and I don't want to sign up for your fucking newsletter.

Re:Pulled back veil

[antiperimetaparalogo]

And the (also uninformative, just a bit less...) blog post this article was based on: http://googlecloudplatform.blo...

Re:Pulled back veil

[Junta]

Well, if you have a lot of awareness of the landscape, you can sort of fill in the pieces. Basically what they did was reveal enough to show that they may be clever, (the principles described are sound) but not enough to actually change the landscape. What they purport to have done is already an obvious high level strategy to anyone in the world who would have actually been able to pull it off and probably is already in play at some of their competitors. If you ask someone who actually recognizes the specific cited components, they can testify that it's a reasonable result though they are likely unable to say they could trivially reproduce the success. I would say the nearest competitor to Google with external evidence that they *could* possibly do this is IBM (their POWER servers used to support a proprietary interconnect that acted like this in the super computer space), though I don't know how much of that team is left or how capable IBM is at actually gathering those folks together for their cloud initiative.

The short of it is they used OpenFlow to enable them to construct an ethernet fabric that resembles a supercomputing fabric. I thought from the first time that I got a high level briefing on OpenFlow that the openflow controller role was quite reminiscent of the subnet manager concept that is ubiquitous to high speed fabrics (today basically Infiniband, but other fabrics historically did the same thing). By using OpenFlow, they get out of the trap of having to use equipment from the dwindling number of suppliers (if you want to buy a new fabric today, basically only three companies do it anymore, Mellanox, Intel, and Cray). They don't get *all* of the benefit of those fabrics, but they get the vast majority of the benefits that would be relevant to the Google workload, at siginificant reduced cost. Basically, in 'traditional' ethernet, there's a gread deal of gymnastics to have a loop-free network from the perspective of any particular participant while at the same time doing all sorts of things with aggregation and MSTP and other technologies to gain some benefit. In a supercomputing fabric, they go to town with mesh networking that doesn't give one crap that there are 'loops' in the network because the entire route of each packet is settled at the edge.

There are things in a modern Supercomputing fabric that I don't see mapping to OpenFlow, but I could see how a sufficiently invested company could negate the need of some of it in something designed particularly for their application.

my other datacenter's a footlocker

[ihtoit]

Seriously. I'm building my new data warehouse inside a wooden footlocker. All I need is an ATX extender and to finish building the drive frames (for 8x3.5" drives, 12x2.5" drives and two DVD burners), and the back arm for the VESA mount for the monitor, then it all gets bolted together and fired up. It looks fuckin' sweet.

Re:my other datacenter's a footlocker

[ihtoit]

...laugh all you want, it's finished.

F-ing Slashdot

[fox171171]

What the hell is with the stupid share thing? When I point at it, it looks like they are trying to tweet "F-ing"!

Re:F-ing Slashdot

[TheGratefulNet]

'share' is one thing I have filtered on ALL sites.

'social' is another. if your url has that string in it, it gets to the dev/null device.

I have zero patience for 'social networking' whores. it sucks I have to spend the first 5 minutes on any new site filtering and adding block entries to ABP and noscript but since its war, this is what we have to do to tame the (already messed up) internet.

and if a site can't show its conent in the first few minutes of my configuring filters, I just leave and never come back.

soon, slash will be that way for me. I've been here for a long time, but I can see it will end, probably this year.

fuck you, dice. just fuck you. sad how it ends like this.

Re:F-ing Slashdot

[Anonymous Coward]

Fucking stupid Share icon on every godddam article. Some MBA thought of this, didn't they?

Re:F-ing Slashdot

[Anonymous Coward]

Just make a /. clone. Problem solved.

Re:F-ing Slashdot

[dave420]

It sounds like you are making work for yourself, and then complaining about said work...

Re:F-ing Slashdot

[simonreid]

Oh the irony. Complaining about social media on a social media site.

WTF???

[djbckr]

Why the fuck are you fucking around with the /. UI? Do you have to change things for change's sake? If it works, stop changing it!

Re:WTF???

[Anonymous Coward]

Maybe they want to add unicode support?

HAHAHA The joke's on them.

Re:WTF???

[Anonymous Coward]

Are there new people that are coming to slashdot? I dont see much about slashdot anywhere else i frequent, and the folks I know that frequent it have been for over 10 years. Fuck, I've been checking almost daily for 16. Why the fuck do you need to change shit up.

Re:WTF???

[sinij]

If you must ask...

Because if the news article (and publication) isn't trending on social media, advertisers don't consider it relevant. Advertising-Media industry inbred to the point that likes/re-tweets is universally seen as a measure of relevance.

In /. case - you have a counter-culture to this, where big % of community signed out of social media for various valid reasons. This creates a problem for Dice, since 1000-comment 'This just in: Cowboy Neal Just Invented Warp Drive' article on /. would only be measured by social media activity. Which might be low.

Re:WTF???

[The Fifth Man]

>where big % of community signed out of social media for various valid reasons.

EXACTLY and it amazes most if not all of us that Dice could fundamentally misunderstand the user base so badly.

I can just imagine the discussions -- hey let's buy Slashdot, it's Facebook for tech nerds!

Please bring back "Read more"

[itsenrique]

I promise I won't accidentally share a story over social media instead of clicking "Read more", no dice Dice.

Re:Please bring back "Read more"

[Anonymous Coward]

"Read more" is retarded. You read a summary, then you click on "Read more", and instead of sending you to the full article it shows you the same summary. At least rename it to "View comments" or such.

They want more money

[itsenrique]

which means they want to draw in people who aren't hardcore adblockers and social media non participants.

Re:They want more money

[MightyMartian]

In other words, mindless rubes and hipsters. Soon they'll rename Slashdot iDot or iSlash, of iTwitfuckhamshitprick. It's going to get so bad even APK is going to walk, and then where will I get my hosts file from?

Re:They want more money

[sinij]

That is exaggeration, they simply want /. to be non-tech site where people don't know any better and don't block ads and over-share on social media. Too bad, they should have done their homework before buying into this community.

Custom built data centers

[TheCreeep]

Slashdot also pulling back the veil on its crappy share buttons.

Re:Custom built data centers

[MightyMartian]

Is "pulling back the veil" some sort of sodomy euphemism?

bullshit

[Gravis Zero]

"we could not buy, for any price, a data-center network that would meet the requirements of our distributed systems,"

bullshit. if you told a company that specialized in making toothpicks that you would give them a trillion dollars to build a datacenter with XYZ, they would make it happen. did you offer someone a trillion dollars?

Re:bullshit

[dave420]

Lighten up, Francis. To normal people "any price" translates to "any justifiable price", at least in this usage.

Re:bullshit

[Anonymous Coward]

Why do you constantly stalk-harass apk? Your post history's evidence you do. No denying it. You're obsessed with him doing better than you have in computing so you stalk him harassing him constantly like a psycho you're showing us you are by doing it. He's challenged you to do better. You prove you can't. You can't prove his lists of points favoring hosts files wrong. You've been shown agreeing with him that he's correct on them from recent replies of yours in exchanges with apk you've had. Dave420, keep smoking that dope. It's amusing seeing you fall into your dope brain pattern everytime of illogic logic calling others names when you are cornered failing due to your drug addled dim brain shorting out, lol! Smoking drugs is not being grown up stupid. It's being an imbecile destroying one's self with drugs. I love seeing you try to play smart only to fall into your usual illogic logic ad hominem attacks Dave420. It's proof the drugs have gotten to your already dim brain and when you get 'frustrated' your dull drug brain shorts out, hahahaha, forcing you to do the only thing you know in calling others names out of frustration at your own stupidity. Your post history again is your undoing on this count proving this point for me. You project a pattern of stupid, constantly.

Libraries of Congress

[darkain]

So, how many Libraries of Congress is that anyways? ... oh wait ... the Google blog post (ya'know, the actual artist, not the article talking about the article which was linked from the summary) actually states!

"Our current generation — Jupiter fabrics — can deliver more than 1 Petabit/sec of total bisection bandwidth. To put this in perspective, such capacity would be enough for 100,000 servers to exchange information at 10Gb/s each, enough to read the entire scanned contents of the Library of Congress in less than 1/10th of a second." = Source: http://googlecloudplatform.blo...

Re: Libraries of Congress

[Anonymous Coward]

But what if you loaded it onto as many Tb micro SD cards you could fit into a new 787 and fly them across the country.

Re:Libraries of Congress

[Whiteox]

Except that there is no agreement on how much data the Library of Congress actually has.
"Though some sources have suggested that 10 terabytes represents the total quantity of data stored at the Library of Congress, this is a significant underestimate, given that the Web Archiving program had by itself collected 525 terabytes of data as of July 2014. A slide from a September 2012 presentation by a Library of Congress storage engineer furthermore noted institutional storage capacity in excess of 27 petabytes, casting further doubt on the accuracy of the 10 terabyte number as applied to the entire holdings."
https://en.wikipedia.org/wiki/...