Slashdot Mirror

The USA doesn't have any locations (like Wyoming, Colorado or the Dakotas) with an abundance of cheap energy, high speed data connections and local cheap power.

How about selling political stability and business climate?

According to this page, a third of the 10 largest data centers in the world are in Chicago. So... I would guess that political stability and business climate are near the bottom of the list of priorities when it comes to locating a data center!

The EFF and Carpathia Hosting announced this morning that they're working together to assist users who stored non-infringing files on Megaupload. Users can go to MegaRetrieval.com to connect with the EFF, which will review the cases and try to help resolve issues through their free legal services.“EFF is troubled that so many lawful users of Megaupload.com had their property taken from them without warning and that the government has taken no steps to help them,” said Julie Samuels, Staff Attorney at EFF. “We think it’s important that these users have their voices heard as this process moves forward.”

28,000,000 / 800,000,000 = 0.035 Watts/user

If they only had one data centre.

They have a European data centre in Sweden, and has at least nine across the US
br.That's still only 0.35W per user, but an order of magnitude is huge no matter who you ask.

So does google. Apparently the best place for corporate HQ and the best place for a data center are somewhat opposite.

Data centers have always created very few jobs due to the high level of automation in these facilities. As a result, they don't appear to be a compelling candidate for economic development incentives, which have traditionally been all about job creation. But there's a political component to this. Data centers represent far more than jobs or bricks & mortar. They have become symbols of the new economy, a tangible sign that a community is making a successful transition to the digital economy. Governors and local legislators understand the value of a press conference to announce a new project from Google, Facebook or Apple. That's why North Carolina has hit the data center trifecta with projects for all three of those companies, and continues to offer aggressive incentives for new projects. We've been tracking this trend for years, and there are more states than ever before offering incentives for data centers. That competition will intensify as the Internet continues to transform our economy, and ensure that tax incentives for data center projects are here to stay.

You're referring to this report from way back in june. Neither Apple nor MS has confirmed it and iCloud was barely in beta at the time so it's hardly a solid fact. Maybe we'll get some more solid information on how it works now it has actually been released.

it's not impossible. Apple uses akamai for CDN.. so why not use AWS and Azure. Just because the compete in the consumer space doesn't mean they can't have partnerships for the infrastructure. More datacenters = better redundancy, and geo diversity also saves a lot of traffic on the backbones.. (seems inefficient, for example, for a California resident to download their music servers on the east coast.. even more so for customers overseas).

They're suing Sammy on one side, but buy a shit ton of parts on the other.

I don't see Xservs: http://www.datacenterknowledge.com/archives/2011/06/06/steve-jobs-provides-a-look-inside-the-idatacenter/

http://www.datacenterknowledge.com/archives/2010/07/21/the-technology-inside-apples-new-idatacenter/

Apple says that its “data center environment consists of MacOS X, IBM/AIX, Linux and SUN/Solaris systems.”
The Maiden facility will have a “heavy emphasis” on high availability technologies, including IBM’s HACMP and HAGEO solutions for high-availability clusters, Veritas Cluster Server, and Oracle’s DataGuard and Real Application Clusters.
Job candidates are also asked to be familiar with storage systems using IBM, NetApp and Data Domain, and data warehousing systems from Teradata.
Networking positions require a familiarity with Brocade and Qlogic switches.

it's not impossible. Apple uses akamai for CDN.. so why not use AWS and Azure. Just because the compete in the consumer space doesn't mean they can't have partnerships for the infrastructure. More datacenters = better redundancy, and geo diversity also saves a lot of traffic on the backbones.. (seems inefficient, for example, for a California resident to download their music servers on the east coast.. even more so for customers overseas).

They're suing Sammy on one side, but buy a shit ton of parts on the other.

I don't see Xservs: http://www.datacenterknowledge.com/archives/2011/06/06/steve-jobs-provides-a-look-inside-the-idatacenter/

http://www.datacenterknowledge.com/archives/2010/07/21/the-technology-inside-apples-new-idatacenter/

Apple says that its “data center environment consists of MacOS X, IBM/AIX, Linux and SUN/Solaris systems.”
The Maiden facility will have a “heavy emphasis” on high availability technologies, including IBM’s HACMP and HAGEO solutions for high-availability clusters, Veritas Cluster Server, and Oracle’s DataGuard and Real Application Clusters.
Job candidates are also asked to be familiar with storage systems using IBM, NetApp and Data Domain, and data warehousing systems from Teradata.
Networking positions require a familiarity with Brocade and Qlogic switches.

It's been noted before too, and there's even proof.

This story is apparently based off of observations made in june, when iCloud was first announced, and seems to concern just iMessage not the entirety of iCloud services. iCloud isn't even out yet, it's still in beta (real beta, not "Google beta".) As always without any kind of official confirmation or strong observable evidence this is just a rumor, but that won't stop everyone from reporting it as fact.

This is not the first time -- http://www.datacenterknowledge.com/archives/2009/06/11/lightning-strike-triggers-amazon-ec2-outage/ .

There are reports that Google has been testing servers using low-power many-core servers from Tilera and Quanta. Facebook is also test-driving Tilera chips and seeing promising results when using them on key-value pair apps like memcached. When you have 900,000 servers, you get plenty of attention from processor and server vendors.

As others have noted, there are many good examples of data center reusing waste heat. Here's a list of examples of server heat being recaptured to warm homes, offices, greenhouses and even swimming pools. This is common enough that The Green Grid recently released guidelines on the best way to integrate heat recapture in key efficiency metrics like PUE (Power Usage Effectiveness).

Here are some images of the system, which currently uses 672 cabinets and uses about 10 megawatts of power. The K system is more powerful than the next 5 systems combined. It's a big-ass system.

Flywheels are also common in datacenters, where they fill the gap between the limited run-time of battery powered UPSs and the long start time of diesel generators.

What are the odds that this iCloud service isn't run OSX server at it's core?

Well, some is OS X. But certainly not all.

Hint: It ain't HP or Dell, neither. Nor is it running a "free" (as in beer) Linux distro; so no snarky comments about OS X Server. Those to Linuces are know for their superior load-balancing software, and I would imagine that's what they are being used for.

Wait, is this that after-market for conversion like we saw for Y2K?

Just the reverse imho. Y2K had lots of buy-in and few real problems. Today we have lots of problems, from routing table explosion to address exhaustion, with very little buy-in to the only thing that remotely resembles a solution. Of course, the powers that be only give a thought to the easiest and most trivial of the problems, address exhaustion.

Also, forgive the poor phrasing, but can everyone in IPv6 see each other?

No. Unless he-cogent has been solved. I could login and check if I weren't so lazy. I'm sure this is not the only case of this problem.

The internet is basically a big collection of unidirectional traffic exchange (this is how BGP works). This *could* lead to a full graph (everyone can communicate with everyone else), but it doesn't. There are a few technical caveats, which sometimes interrupt connectivity (e.g. ghost routes, bgp loops, ...). But mostly there are political problems which prevent global routing from working (and I'm not talking about Iran and China, I'm talking disputes between companies.

Can we just ditch all that eHow and Experts Exchange junk all in one swoop?

No, in fact IPv6, by expanding everything, will probably expand this problem too. The alternative is censorship, let's not go there.

It's like a giant Reset Button for the Internet.

Unless we network engineers collectively and massively screw up, no it isn't. Nobody wants that. And we certainly don't want international censorship to be implemented as a result of the transition.

"Everything that matters will migrate because the people that care will do it. 15 years of legacy will fall away."

Again, no it won't. New and more troubling versions of all problems in IPv4 have already manifested and are affecting at least some backbones. Furthermore, running dual-stack has a lot of new problems as well.

Go Go Gadget Nevinyrral's Disk!

Go !

As the other reply says, it has a UPS+PSU builtin, it is actually a lot more efficient. Because a normal UPS converts the power coming into the UPS for the batteries, then it gets converted again when it is sent to the PSU, the PSU converts it again...

All of that is a waste.

If you have the UPS next to the PSU, you convert the power ones coming into the PSU, it sends power to the board/CPU and so on and also to the UPS if it needs to be charged. When the PSU does not get power, the UPS delivers power to the board/CPU, no conversion needed.

From an older article:
"This design provides Google with UPS efficiency of 99.9 percent, compared to a top end of 92 to 95 percent for the most efficient facilities using a central UPS."

http://www.datacenterknowledge.com/archives/2009/04/01/efficient-ups-aids-googles-extreme-pue/

Anybody know if the "Google web server" at the same website as in the article is actually real?

I mean, do they really have a 6-inch battery contraption hanging off the side of every one of their web servers?

Google is pretty innovative about stuff like this. They use their own in-house version of Linux on commodity hardware, thousands upon thousands of PCs in each data center. But they still use air cooling and air conditioning because, at the end of the day, it's the best bang for the buck.

Not any more. In the early days, as I understand it, Google bought the cheapest desktop PCs they could find, made them netboot, and filled warehouses with them. Their software would deal with hardware failures by routing around the broken server. It worked out cheaper to leave a broken server where it was, than to locate it and repair it or dispose of it. This made sense when Google was a certain size -- big enough to need a big cloud of servers, but too small to invest in custom hardware.

Nowadays, however, they have boards made in bulk to their own design, which slot into racks of their own design, and they take cooling very seriously -- because to Google, a 1% saving in energy costs represents millions of dollars. As an example, here's a story about Google patenting a novel approach to water cooling.

I would be *very* surprised if Google hasn't got someone investigating oil-submersion cooling, even if it's not in their production toolkit yet.

Google already has a battery on each motherboard, which is a cheap and efficient UPS solution.

This design provides Google with UPS efficiency of 99.9 percent, compared to a top end of 92 to 95 percent for the most efficient facilities using a central UPS.

Running the whole rack on a 12VDC power source would be the next logical step.

There's actually been a bit of a boom in these underground data bunkers in recent years, but there are also plenty of "nuke-proof" data center projects projects that have failed due to cost issues. It's more challenging than you think to prepare a subterranean environment to meet the standards for hosting servers for enterprises - who are usually the target customers due to the high cost of finishing the space. There are plenty of old military facilities available for retrofits, including one that's been listed on eEBay a few times. But the old military facilities are now competing against caves - the state of Missouri has been trying to pass economic incentives to convince developers to covnert the state's many limestone caves into disaster recovery centers.

There's actually been a bit of a boom in these underground data bunkers in recent years, but there are also plenty of "nuke-proof" data center projects projects that have failed due to cost issues. It's more challenging than you think to prepare a subterranean environment to meet the standards for hosting servers for enterprises - who are usually the target customers due to the high cost of finishing the space. There are plenty of old military facilities available for retrofits, including one that's been listed on eEBay a few times. But the old military facilities are now competing against caves - the state of Missouri has been trying to pass economic incentives to convince developers to covnert the state's many limestone caves into disaster recovery centers.

According to Savvis invested in a new ipv6-capable network in 2006, to be finished in 2008. Savvis hosts sourceforge / slashdot (from the whois record). Yet, according to the nanog grapevine in 2010, Savvis is not yet able to offer IPv6 to customers. Time to put 'working ipv6' on the checklist for your new hosting?

Facebook has five hundred million users. Is each user really worth a hundred dollars?

I'm not a businessman but I'm not so sure this is the correct way to think about this.

Everything depends on how much the market is penetrated for social in two ways: users and advertisers. Can they grow that revenue/profit? And if so, to what point? If Zuckerberg sneaks it into China then I think you're looking at a potential to increase that significantly. Facebook hosts its statistics so you can guess if it's got a half billion in revenue yearly at half a billion users and it scales perfectly, that's a dollar per year per user. Can it get up to a billion users? It's probably clear that in the long run as the younger generation matures, that penetration will slowly expand ... but there's no guarantee that Facebook remains the de facto standard that far out. You need to consider future growth.

The other factor, advertisers and game publishers, could also be troublesome. Is this a "Honeymoon Period" for advertisers where they're paying an unsustainable amount to Facebook for the time being just to gain exposure? Could the above assumptions about scaling with userbase actually be false if advertisers aren't willing to spend more than they are now once more users join?

Consider that these numbers put Facebook's Net Profit Margin at almost 30%. That's very high for the industry. They're in the same region as Google and Microsoft but as I stated above can it scale?

One last thing, you seem to think that Facebook's worth is only its users. They are also a large company with almost two thousand employees and are building infrastructure. Include that on your assets sheet.

Facebook is going public soon. What are the chances that this 'leaked' report is designed to pump up the stock, and therefore Goldman's profit?

I think the SEC would come down pretty hard on GS if they did that -- they have before for less. Misleading investors is very serious.

Data Center Knowledge has a photo feature with a bunch of images of the facility in Quincy and the container modules being assembled. You can see all the servers they pack into them.

And so did Yahoo!
Microsoft and Google aren't the only people playing this music.

http://www.datacenterknowledge.com/archives/2009/06/30/yahoos-fresh-air-computing-coop/

http://www.computerworld.com/s/article/9186618/Yahoo_opens_chicken_coop_green_data_center?source=rss_news

The $450 million number from Goldman is interesting because Facebook just announced plans to invest $450 million over the next 5 years in a huge new data center in North Carolina. Facebook's already spending about $50 million a year on leased data center space, and expects to spend about $200 million building its new Oregon server farm. It takes a lot of infrastructure and servers to support 500 million users.

The $450 million number from Goldman is interesting because Facebook just announced plans to invest $450 million over the next 5 years in a huge new data center in North Carolina. Facebook's already spending about $50 million a year on leased data center space, and expects to spend about $200 million building its new Oregon server farm. It takes a lot of infrastructure and servers to support 500 million users.

Here are some factoids gleaned from Apple’s job postings:

- Apple says that its “data center environment consists of MacOS X, IBM/AIX, Linux and SUN/Solaris systems.”
- The Maiden facility will have a “heavy emphasis” on high availability technologies, including IBM’s HACMP and HAGEO solutions for high-availability clusters, Veritas Cluster Server, and Oracle’s DataGuard and Real Application Clusters.
- Job candidates are also asked to be familiar with storage systems using IBM, NetApp and Data Domain, and data warehousing systems from Teradata.
- Networking positions require a familiarity with Brocade and Qlogic switches.

The linked article http://www.datacenterknowledge.com/archives/2010/09/14/111-8th-avenue-carrier-hotel-is-for-sale/ ends up as dark gray on black in Chrome though .

(Guess who actually RTA?)

Netcraft has been tracking the shifts in Wikileaks' infrastructure, and notes today that one of its post-Amazon hosts is Swedish ISP Bahnhof Internet, which operates the "James Bond Villain" data center housed in a nuke-proof bunker 100 feet beneath Stockholm.

I wouldn't use their cloud platform anyway.

They had Black Friday deals in the UK all last week and had extremely limited numbers of stock, but rather than do things first come first served, they had an unstable platform on which to run it which repeatedly threw back 503 errors due to the load, meaning those that got processed weren't necessarily in order (i.e. 200 get through, 200 rejected, 200 through or whatever load it was taking).

If they can't even scale their own services on their platform, what hope is there for anyone else being able to rely on them? It's not like they haven't had problems either:

http://www.datacenterknowledge.com/archives/2007/10/02/amazon-ec2-outage-wipes-out-data/

http://www.readwriteweb.com/archives/more_amazon_s3_downtime.php

Trusting Amazon as a hosting platform at least is foolish anyway.

Comcast says the issue with Level 3 is a peering dispute and says it "offered Level 3 the same terms it offers to Level 3s CDN competitors for the same traffic." The issue seems to be that the Level 3's addition of Netflix as a customer may have altered the balance of the traffic exchange between Level 3 and Comcast. In other words, Comcast says the volume of traffic is the issue, while Level 3 says the type of traffic is the issue.

500k sqft is a decent size datacenter for a single building, though there are a number of larger datacenter buildings, and many larger datacenter complexes (like Stone Mountain at 6 million sqft) or (DataPort at 3.5 million sqft).

500k sqft is a decent size datacenter for a single building, though there are a number of larger datacenter buildings, and many larger datacenter complexes (like Stone Mountain at 6 million sqft) or (DataPort at 3.5 million sqft).

Not much different than the first video".

one of the top ten is in phoenix, which gets unpleasantly hot, info and photos here: inside the phoenix one datacenter

Google purchased a Church near their DC in Iowa, as well.

Actually, from the article, you would have seen that the contract was signed back in 2005, when Virginia enjoyed the presence of Mark Warner, Democrat, and now US Senator for Virginia.

Amusingly, Aneesh Chopra, the current CTO of the Obama administration, was the Virginia Secretary of Technology starting shortly after this was signed, and he never added redundancy to the service contract. This was during Warner's tenure and during Tim Kaine's (D-Va) tenure.

Also, counter to your argument, it was actually Bob McDonnell, the current Republican Governor, that renegotiated the contract to include redundancy.

With all of that said, I do not think Northrop Grumman was the best fit for this job and after so many egregious failures, they deserve to have their contract reworked in VA's favor, but bureaucracy being what it is, regardless of party politics, I doubt this will change. I really feel like this kind of contract could have gone to a small-to-medium sized VA business that could have handled it extremely well, and locally, for much less. The real sad thing is that the guy who's largest job was to oversee this contract, and did nothing, is now the CTO for the entire country. I don't care what party you are, that's a scary thought.

When it comes to aggression versus servers, you can't beat the Gallery of Exploding Servers. Some serious mayhem inflicted upon hardware.

This seems to hold true for data centers as well. There were a flurry of data center power outages in the first week of July 2009. In June 2010 there were major outages at Intuit and Amazon Web Services.

This seems to hold true for data centers as well. There were a flurry of data center power outages in the first week of July 2009. In June 2010 there were major outages at Intuit and Amazon Web Services.

[bookmarked for future use under /. predictions come true catagory] -- done

So when you're pushing data as fast as you can through a socket, the old read(byte[]) or write(byte[]) are faster? Wow, no kidding.

You do NOT use java.nio (like Jetty's SelectChannelConnector) for maximum throughput. You use it to handle persistent connections, like all those long polling requests via AJAX which return on an event or timeout after a minute. This article is like recommending Apache with its hard limits on how many requests it can serve concurrently over newer, asynchronous servers like Nginx for static media servers with keep alive enabled.

The slides even mentions the C10K problem, but what it doesn't do is mention when to use either technology - async IO for concurrency and endless scaling, and synchronous IO for pushing a 10G Ethernet link to the limits. No wait, the nio setup can do that too, 700MB/s or 5.6Gbit/sec per core on 2008 hardware should be enough to max out anything you can buy now. It's great that synchronous IO can hit 1GB/s, a whopping 30% faster, but useful? I'd say no.

For most users, you don't use either API. Lets be honest here, writing highly concurrent software is hard, why reinvent the wheel when you can get off the shelve software that can do it better? You use Jetty and choose between the SelectChannelConnector or SocketConnector, or choose between Apache or Lighttpd/Nginx depending on the traffic pattern. What you do write is the bit that accepts a whole HTTP request and returns a HTTP response, everything before and after is magic.

Unless you're a file server, each 50k sized HTTP response will require enough work to make sure you run out of CPU or Disk IO long before you hit even the 100Mb/s ceiling in most rack switches. Even if your app is fast, 16 cores x 100ms per request x 50K is only 62 Mbits. Not 5600.

But if you need to scale in concurrent client count, there's no way around async IO. The latest name to watch is Netty. In Plurk Comet: Handling 100,000+ Concurrent Connections with Netty, it scales up to 100000 concurrent connections on a quad core server with 20% CPU load.

Just stop worrying about sockets already, and start worrying about your SQL server suffering a meltdown. Even if you get manage to grow into the Facebook, it's not like using synchronous IO will save you from deploying 30000 servers, it's the application code that's slow. Zero copy, one copy, "string concatenation style twenty copies response building" socket writes don't matter at all, memcpy is cheap compared to a few lines of interpreted code, servers are cheap compared to developers, and never mind the cost of the programming gods giving these presentations.

"Several courts have gone paperless in recent months, and the district clerk’s office has been converting files to electronic form and destroying paper versions, the paper said."

unlike the USA, MOST countries around the world have alternatives to requiring a printed copy. because.. that's just a better idea in the modern age.

How you managed to read that I was suggesting that there weren't any and I was somehow lamenting that fact is beyond me. How you turned my post questioning the choice of location into "telling others how to conduct their business" is mystifying. You must be great fun in person when someone asks "why?"

To summarize my post and possibly increase your reading comprehension: I questioned why anyone would build a data center in California. I provided an example of an alternative that I was familiar with.

Since you seem to think that I was suggesting that there aren't any big data center projects here, I'll do the name-drops: Google, Amazon, Facebook, Intel, and Mission West all have or are constructing data centers here. There are more, of course, but those are well-known.

Yes the engineers and customers are in Silicon Valley, that's fine. But we have this thing called the Internet which means it is no longer necessary to put data centers anywhere near the customers or engineers. And if it's not necessary, it makes absolutely no sense for servers to compete for space in one of the most expensive real estate markets in the world when there are billions of acres of wasteland that would work just as well for a server farm. I've used a thin client desktop with the servers hosted over 3000 miles away and the latency was better than it was when the server was hosted only 8 miles away. Only video games and the kind of hyperactive trading that led to this month's stock crashes have latency requirements which would be significantly impacted by having your server in central valley or the Midwest or Iceland or Offshore and you will save an enormous amount of real estate and energy costs. I'm convinced that high-level corporate decisions are still based on inertia and nineteenth century factory-whistle mentality.

No surprise this year. Jobs himself: check. New iPhone model: check. New firmware version: check. All is "déjà vu" ! The only possible surprise would be "OS4 for iPhone2G" (they said "no !"; now they should say "yes !").

Well, there may be the iPhone on Verizon and there still is something about that fucking large datacenter Apple has just finished to build. I have the suspicion that Apple will be announcing something about that. Like cloud storage and streaming and syncing for all of your music and data.

> The next year will be the "year of android" but Apple needs to
> tee up a home run next year (Unless there are unannounced
> features for the 4th generation iPhone).

Wrong. Lots of other important updates have come out between the major point-0 releases each summer. Lots of substantial features were introduced in 1.1, 1.1.3, 2.1, and 2.2 in particular. As just one example, the iTunes Music Store was added in 1.1. It's entirely possible--nay, likely--that Apple will introduce, say, major "cloud" features this year, even if there's nothing shown at WWDC. I'm sure a lot depends on when this bad boy comes online.