Slashdot Mirror

Three separate PCs, each running the same software on the same data, and if one gives a different answer, the entire machine gets taken offline and support paged.

The difference is, three PCs can be had for less than three thousand dollars, new, even with monitors and such. How much will one mainframe cost you?

Take the cost of the system and divide it by the cost of one minute of unscheduled downtime. There are environments where thirty unscheduled seconds a year is unacceptable. If one of your three PCs goes down, how long will it take the other two to get their act together, back out the mistake, and reply, correctly, to the request the third PC was handling? You don't know, do you?

By the way, a PC that costs a thousand dollars (monitor included) was designed to be thrown away. And machines in data centers don't have monitors. Statements like that make it sound like you've stood on a raised floor.

How about a design that lets you run applications 24 hours a day, 7 days a week, with no downtime required for system upgrades?

Does Google qualify? How about Amazon?

If Google loses an app server and a thousand queries hang, a thousand people shrug their shoulders and hit Ctrl-R to try again. If a Google DB server scribbles the database and crashes, Google will use a day-old copy while they resync. This won't cut it when a database that's a few seconds out of date means millions of dollars are unaccounted for.

There are areas where mainframes eat Unix systems for lunch.

Only if there's an irrational need for it to be exactly one machine.

There's hardly ever just one. Companies that need mainframes usually need several.

And here's something else you don't understand about mainframes: There are environments where your combined internal throughput should be in the terabyte-per-second range, maybe because once a year you have to buzz through tens of millions of extraordinarily complex tax returns, apply rule sets with over a century of changes in them, flag suspicious activity, reconcile them against bank and employer data you've been tabulating throughout the year, and generate a check or electronic deposit for most of them. And every day you're late is a day where you've kept two hundred and twenty five billion dollars in tax refunds out of circulation.

You gonna pull that plow with a team of oxes? Or are you gonna settle for ten thousand mice who spend half their day shouting "you still there?" at each other?

What happens if, say, that building explodes?

Funny you should ask. Long before you were born, many very serious adults dedicated a sizable portion of their careers to figuring out exactly what to do when this sort of once-per-country-per-decade disaster happens. Adults serious enough to consult geologists about seismic activity before they build their data centers. Serious enough that they receive regular reports on political stability in the countries that host their data centers. And bright enough to ask, "hey, what if we put some of our computers in a different building?"

I'll add a disclaimer, too: I work on a project which is currently deployed via Amazon EC2.

When they went down for two days last October, how much data did you lose? How much much damage was done to your reputation and revenue stream when they went down for two days in February?

Here's my disclaimer: I work for a company that processes

The cost analysis was really what did it versus our managed hosting plan (1/10th the cost per month). Auto scaling and healing of the application cluster was also a benefit. To scale with a traditional host meant getting locked into a contract for the added server(s).

One thing about ec2 is that it forces you to use best practices for disaster recovery. Instances don't commonly just "disappear" but you need to plan for it. Well tuned ec2 images can have your site up and restored from backup automatically within minutes.

ec2 / s3 is far from perfect and certainly won't meet everyone's needs. The downtime s3 has seen (like last weekend) would be devastating to some businesses. Of course even with a traditional host you may have downtime due to truck crash or other random act.

Ever pay attention to the render times, though?

Their infrastructure is scary-massive, from almost every report

This is a huge topic, since so many different strategies are being brought to bear. For data center operators, energy efficiency is a business imperative since the power bills are soaring. Here are some sources offering ongoing reading about Green Data Centers:

The Green Data Center Blog
Data Center Knowledge
Groves Green IT
The Big List of Green Technology Blogs

This is a huge topic, since so many different strategies are being brought to bear. For data center operators, energy efficiency is a business imperative since the power bills are soaring. Here are some sources offering ongoing reading about Green Data Centers:

The Green Data Center Blog
Data Center Knowledge
Groves Green IT
The Big List of Green Technology Blogs

This year the top 500 also tracks how much power is used by each system. Systems under development at Oak Ridge National Lab will reportedly have annual power bills of more than $30 million when they debut in 2012. See ComputerWorld and Data Center Knowledge for more.

There's more info on Switch Communications and its Vegas supercenter at The Register, Data Center Knowledge and the Silverback Migration blog. These stories talk about their cooling system, which handles 1,500 watts per SF with air cooling. Gotta love that video on the Switch web site.

If you're in Hawaii, you better be pimping the dickens out of your data center. Hawaii has the most expensive electricity in the United States, according to state-by-state energy prices, which show Hawaiian power prices at 16 to 17 cents per kW hour.

Too late. They're already putting data centers in caves. They'll soon be filled with nerds.

On the "how a data center gets built" front, last week I had a tour of a new $250 million data center facility in Virginia that is getting ready to open later this month. The facility manager provided a walk-through of the power and cooling infrastructure, explaining the company's approach to designing these systems for energy efficiency and scale. I shot video, which is now posted online. The data center operator, Terremark, separated most of the electrical infrastructure from the IT equipment, putting them on separate floors and housing the generators in a separate facility. They have 11 generators now, but will have 55 Caterpillar 2.25-megawatt units when the entire complex is finished.

It's not just data center designers that are in demand. There are a ton of listings for data center managers at the Data Center Jobs site.

Data center employment often comes up in discussions of economic development. Many communities are eager to attract data center projects, but struggle to define the economic benefits of these facilities. Jobs have always been the primary benchmark by which economic development projects are measured. Incentive packages offered by state and local governments are often based on the number of full-time jobs created by a new business. And do data centers really hire locally, or do trained data center engineers migrate from other existing data center hubs? In some cases, local officials try to stipulate local hires, which is a sticky wicket.

For those who couldn't sit through the 80-minute video (or don't have Silverlight), Gates said that in the future Microsoft's mega data centers will have many millions of servers". It currently has "hundreds of thousands" of servers, but expects to pack up to 300,000 into its new Chicago container farm. Gates also predicted that only a select number of companies (presumably including Microsoft and Google) will be able to compete on this scale.

Data Center Knowledge has a story on the downtime at The Planet, summarizing the information from the now Slashdotted forums. Only one of the company's six data centers was affected. The Planet has more than 50,000 servers in its network, meaning that one on five customers are offline.

http://www.datacenterknowledge.com/archives/2008/Jan/16/joyent_backup_services_down_for_three_days.html

There are also videos available showing tours of the Rackable ICE Cube 40-foot container and Sun's Project Blackbox (now renamed to the immensely more boring Sun MD) in a 20-foot container.

The Virtual Earth team is an early adopter of the Microsoft containers, and has posted pictures of what they look like on the inside. Note that they're using customized Forest containers from Verari Systems rather than Rackable or Sun (at least at this point).

The development of solar thermal power is of interest to the data center industry, where the push for "green" energy has thus far focused primarily on hydro and wind power, prompting Google and Microsoft to build huge facilities near Northwest dams andf MidWest windmills. Some companies would love to incorporate solar generation to supplement grid power, but photovoltaic doesn't produce enough capacity to make a significant dent in overall data center energy usage. Solar thermal can at least generate "utility-scale" power.

That's probably why Google has invested $10 million in eSolar, a solar thermal startup. The speculation is that Google would like to use solar thermal power in some locations to whittle down the percentage of its data center power bill that comes from coal, which would make it easier for the company to meet its promise of carbon neutrality.

Well, first of all, one of the ships' anchors was found caught on one of the damaged cables. But I think the best proof lies in the fact that one of the companies in question paid for the damages to the cables. Unless there is a better explanation presented or more evidence to the contrary, I think it's reasonable to assume that the mystery is solved, no?

The truth of the matter was that I had no idea what happened and, thus, no real opinion either way. I was merely skeptical that a conspiracy was responsible, which I think is an entirely reasonable position in this context. Plus it irks me when people mis-apply scientific/mathematical principals principals in everyday discussions to disingenuously infer a false appeal to authority.

-Grym

Iceland's not alone. Manitoba, Canada is shaping up as another region that is an getting attention from data center builders due to its climate and energy profile. Large power customers in Winnipeg paid an average of 3.6 cents per kilowatt hour in 2007, cheaper than the average rate in virtually every state in the U.S. except Idaho. That's all clean, green power from Manitoba Hydro, which operates 14 hydroelectric generating stations and also buys the output of a 99-megawatt wind farm.

Meanwhile, there have been a bunch of major project announcements in Dublin, including major data centers for Eircom and Digital Realty Trust. Dublin has an existing data center workforce, which may be the difference.

Meanwhile, there have been a bunch of major project announcements in Dublin, including major data centers for Eircom and Digital Realty Trust. Dublin has an existing data center workforce, which may be the difference.

This sure looks like a growth area for qualified developers. An audience poll at the Gartner Data Center conference in Las Vegas in November found that just 17 percent of attendees felt their developers are prepared for coding multi-core applications, compared to 64 percent who say they will need to train or hire developers for parallel processing. "We believe a minority of developers have the skills to write parallel code," said Gartner analyst Carl Claunch. I take the Gartner stuff with a grain of salt, but the audience poll was interesting.

McColl's blog is pretty interesting. He only recently started writing regularly again. High Scalability is another worthwhile resource in this area.

The new architecture from the Comcast/BitTorrent effort will be of great interest to content delivery networks (CDNs) who have been sorting out the best way that P2P can be used to assist in delivery of large files. Yesterday a CDN called Velocix announced a hybrid P2P streaming media service combining traditional caching with P2P delivery for live events. Velocix used to be CacheLogic, and worked with BitTorrent to develop the Cache Discovery Protocol, which lets ISPs cache the most popular torrent files, and then seed the files from servers within their network, reducing network traffic.

Interesting. EnterpriseDB was also in the news today for its partnership with Elastra, a startup that announced a "cloud server" that lets companies quickly create database applications on Amazon's utility computing platform. "In the future, enterprises will view massive capital investment in on-premise server infrastructure to support database applications as entirely optional," said Bob Zurek, chief technology officer of EnterpriseDB, which uses Elastra to run its EnterpriseDB Cloud Edition. Maybe all that IBM money has their head in the clouds.

According to Wired, Cogent felt Telia didn't provide "fat enough pipes." The capacity of peering connections is becoming a point of tension in a growing number of peering relationships. Video traffic is driving strong demand for 10 gigabit Ethernet connections for peering, but some major ISPs are apparently reluctant to upgrade, asserting that the financial benefits of big-pipe peering don't offset the short-term expense of network upgrades needed to support 10gigE. The economics of peering is a tricky business sometimes, and video traffic is complicating the equation.

At least they managed to block the site without knocking YouTube completely off the Web.

Google is currently building a big Data Center here in Belgium($340 million).

for more info
http://www.datacenterknowledge.com/archives/2007/Apr/27/google_data_center_project_in_belgium.html

And it looks like it is just the beginning of their European investment.

Microsoft's use of Akamai in 2003 gained attention when it made it appear that Microsoft's web site was running on Linux. In actuality it was just the Akamai caching servers using Linux. Like Google, they've since shifted to using more of their own network as well as Limelight and Savvis (now Level 3).

Microsoft is partnering with Limelight to build its own CDN network. They're probably the biggest of Limelight's 1,150 customers, but there are plenty of other big companies using Limelight. If the judge issues an injunction, they might have tough decisions, as Limelight has said an injunction might force them to shut down their CDN. Appeals would stretch things out, but customers don't like uncertainty.

One of Google's motivations for the Trans-Pacific capacity bandwidth is to support the new data centers it will be building in Asia, as discussed here on Slashdot last month and updated today at Data Center Knowledge. In recent months there have been reports that Google has been scouting multiple locations around the Pacific Rim for new facilities, and it could easily have one or more ready by the time the undersea cable is completed in 2010. Google likes strong connectivity between its data centers. The plan isn't to have a Google data center in California serve data more quickly to China and Japan, but to have the Google data centers in California and Asia sync the index, and have the Asian facility deliver lightning-fast results to China and Japan.

Better technical explanations of the event are available from the Renesys blog and Data Center Knowledge. The erroneous IP assignments spread across the net within 1 minute, 45 seconds of its announcement by Pakistan Telecom, according to a timeline by Renesys. It took about 80 minutes for YouTube to inform its providers that the route had been hijacked. YouTube says it is "investigating and working with others in the Internet community to prevent this from happening again."

It could be that those Wikipedia folks just like underground data centers. Wikia Search, the search engine from the founders of Wikipedia, houses its servers in an ultra-secure underground hosting facility in Iowa. There are a growing number of these "data bunkers" that are getting business from folks who are paranoid about security, some of whom do indeed want "nuke-proof" hosting. I'm not sure if that's the issue for Wikia ... it's more likely that it's the proximity for Wikia's Jeremie Miller (perhaps best known to Slashdot readers as the developer of the Jabber instant messaging program) who is based in Iowa.

Yahoo's move is the inevitable endgame in an ongoing arms race between major shared hosting firms, who have been super-sizing the disk space and data transfer on their accounts for two years. Here's the larger question: Is this just a marketing gimmick; a bright shiny "UNLIMITED" bauble to dangle in front of small business folk? Or is it an effective way to attract customers from HostGator who find that 1,000 gigs of disk space is simply not enough? Almost nobody needs this, but some might be influenced by it.

A consolidation of the Microsoft and Yahoo networks could shift a massive amount of infrastructure from open source technologies to Microsoft platforms.Microsoft said that "eliminating redundant infrastructure and duplicative operating costs will improve the financial performance of the combined entity." Yahoo has been a major player in several open soruce projects. Most of Yahoo's infrastructure runs on FreeBSD, and the lead developer of PHP, Rasmus Lerdorf, works as an engineer at Yahoo. Yahoo has also been a major contributor to Hadoop, an open source technology for distributed computing. Data Center Knowledge has more on the infrastructure implications.

From Data Center Knowledge:

Some cloud-based services could become so vital that they become candidates for government regulation, according to panelists at the event ... "Everyone who is trying to get into utility computing is getting big fast," said Jesse Robbins (of O"Reilly Radar). "They're all trying to get as big as they can as fast as they can to win the platform play, and this is going to create lock-in. The big companies are going to be viewed as either monopolies or utilities, both of which are regulated."

From Data Center Links:

Princeton's Ed Felten: "Possession of data implies control, and control implies power. Whomever owns the systems on which data resides has the ultimate control of how that data is retained and who has access."

But at least Wikia Search is hosted in a cool underground nuke-proof data bunker in the middle of Iowa.

It means that if the instance is shut down for whatever reason, any and all data stored on that instance is lost. While in theory it's possible to keep the instance running indefinitely, there have been cases where people's instances have been restarted resulting in data loss. http://www.datacenterknowledge.com/archives/2007/Oct/02/amazon_ec2_outage_wipes_out_data.html

The InfoBunker, the Iowa site mentioned in TFA, is one of a number of cold war missile and/or communications facilities being used as data centers. The PJM Interconnection, which runs the East Coast power grid, is setting up a data center in a Pennsylvania site once used for White House-to-Kremlin communications during the Cuban Missile Crisis. The Bunker in the UK is in a former Ministry of Defense command-and-control center. Ask.com is building a major data center in the Titan building in Moses Lake, Washington, a former missile control facility.

I'm not sure EPA is the right party to be advocating virtualization. The EnergyStar ratings and utility-level programs are more up their alley.

Actually, dams are serving as magnets for data center development, since hydro power is cheaper than other sources and provides the public relations advantage of being "greener" than coal or nuke power. That's why more than 2 million square feet of data center space is being planned in and around Quincy, Washington, a farm town of 5,000. Meanwhile, in northern NY state, HSBC is locating a $1 billion data center project in Cambria (another farm town of 5,000), where it will use hydro power from the Niagara river.

Actually, dams are serving as magnets for data center development, since hydro power is cheaper than other sources and provides the public relations advantage of being "greener" than coal or nuke power. That's why more than 2 million square feet of data center space is being planned in and around Quincy, Washington, a farm town of 5,000. Meanwhile, in northern NY state, HSBC is locating a $1 billion data center project in Cambria (another farm town of 5,000), where it will use hydro power from the Niagara river.