Inside Amazon's Data Centers

1sockchuck writes "Amazon Web Services usually doesn't say much about the data centers powering its cloud computing platform. But last week the company held a technology open house to discuss the company's infrastructure, sharing cost data and a glimpse of a modular data center design. The key point: AWS is growing like crazy. 'Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,' said AWS Engineer James Hamilton, whose presentation (PDF) is available online."

Comments

Re:PDF is heavily slashdotted

[jewelises]

Someone should host the PDF on S3.

Re:PDF is heavily slashdotted

[jewelises]

Coral Cache isn't working for me, but google has cached it.

What good is that

[Anonymous Coward]

"Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,"

What good is that when their network design is so flawed that a single human error can bring the whole service down for a week?
Amazon Web Services? No, thanks, I prefer reliability.

Re:What good is that

[nereid666]

http://tech.slashdot.org/story/11/04/29/1156220/Amazon-EC2-Failure-Post-Mortem

Re:Don't read this if you are a (non-tech) manager

[fuzzyfuzzyfungus]

I think that it was manager-safe enough. There were a few diagrams that you should be careful to skip over(a few had percentage numbers that weren't finance related!); but the rest had numbers with dollar signs in front of them safely ensconced next to a variety of recognizable corporate logos with a generic-but-high-tech photo of some server racks in the background.

A few good buzzphrases as well. I'm definitely going to ask my department about why they aren't being more proactive and action-oriented about getting our network "on the Moore's law path". We'll never be ready for exascale cloud-centric computing if they don't...

Re:PDF is heavily slashdotted

[l_bratch]

I have mirrored the PDF here too: http://www.tghost.co.uk/JamesHamilton_AmazonOpenHouse20110607.pdf

Re:any reason they don't buy larger servers?

[PickyH3D]

I'd bet that buying the cheaper, yet still powerful computers is probably more cost effective considering the rate that these machines burn out under their kinds of load. Why spend so much money on a machine that will be broken, when you can just buy multiple cheap ones instead? Then, when one burns out, you're still running at a higher capacity even with the negative of higher power draw.

Re:any reason they don't buy larger servers?

[tibit]

They didn't say if it's per a real or virtual server methinks, and I don't know if the assumption that they pay whatever anyone else would be paying holds water either. I'm sure they can get a lot more "server" for $1.5k than me or you would.

Re:any reason they don't buy larger servers?

[micron]

Looking at a 3-5 years TCO, and power costs where these data centers are located, power costs are noise in the equation.
Taking advantage of commodity pricing in the lower tiers is where the savings is at. Example, single socket systems are a lot cheaper on the procs and mainboards than dual sockets. Quad socket processors are significantly more expensive per proc..
At $0.10 per KwH, a 400W server is $350/year to power. Quad socket processors (Intel I7) can be as high as $4500 each!

Am I the only one?

[kannibul]

Am I the only one that thinks that cloud computing is really NOT going to take off? Amazon is spending a boatload of money, in the hopes that other people are going to be willing to offload a boatload of money...in the hopes that they can process data faster/better/etc off-site. Just the security issues alone concern me, but the additional bandwidth is what really gets things going... Just saying - from a business perspective (and my perspective as a network admin), it seems that Cloud Computing is a 90's .com bubble that will pop in the near future...can Amazon (or any other company) REALLY offer a reduction in costs, or an increase in performance that justifies the (obvious increase in) cost, especially when you look at the whole package? I think not. Same reason not every company has made the switch on the backend to free server-level operating systems (FreeBSD or Linux, namely) - it doesn't matter if the platform may perform better in every regard or has inheritantly better security and even can do the job just as well or better...it gets rejected simply because the costs will increase - in the case of linux/unix, because the skillset to operate and maintain it is more rare and rare = higher price...and unlike a product you purchase (optionally) once like software...support/maintenance is annual. Cloud computing...would be annual, vs a piece of hardware you own, with software that you own the rights to use

Re:Am I the only one?

[donnyspi]

Yeah, the thing is it already has taken off. They can hardly add additional capacity fast enough. AWS is a very profitable business for Amazon. Our company switched from a traditional data center setup to running fully on AWS and it did reduce our costs and increase our agility and scalability. That said, like anything it's not a fit for everybody.

Re:Am I the only one?

[silky1]

I would think staffing a team to maintain hardware in an IT Department is much more expensive than paying Amazon to run the backend. Thinking of many corporations, all have data centers, IT staff, hardware, software etc, and here is Amazon with a single data center capable of providing all this infrastructure at what has to be a competitive price for them to survive. This has to be a better option for many companies, especially small companies. My behemoth company is constantly looking for off shoring opportunities to further reduce costs so smaller companies may not have this as an option. This seems like a simple decision in using Amazon's services. Comparing to the 90's and dot.com bubble, at least this is a tangible business model and not a dream like many of those companies back then were.

Re:Am I the only one?

[noahm]

Well, they did say that "Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years". They wouldn't be doing that if it wasn't taking off.

Keep in mind that "cloud computing" is a really vague term that is used to describe all sorts of different technologies and platforms. It's still really immature, in terms of best practices and that sort of thing. There are still a lot of people throwing around the "cloud will solve all our problems" mantra.

At a previous job (in a computer science research lab), the director got all excited about Amazon's cloud stuff. He wanted my team to move everything to AWS, from our email service to our backups of user data. We researched pricing, thought about how it'd change our service model, and wrote up our findings. When we presented the results to him, he simply refused to believe that the AWS solution was actually significantly more expensive than what we had been doing. It was inconceivable to him that there were workloads that didn't see a major cost reduction when moved to the cloud. Eventually saner heads prevailed, and we were able to convince him that we shouldn't move all our stuff to AWS, but it took a lot of effort to overcome the buzzword enthusiasm.

That said, however, I really think there are situations where the AWS model makes a lot of sense. Your application must be able to scale to additional servers easily, transparently, and at least semi-automatically. It must be easy to deploy in multiple datacenters, so you can insulate yourself from outages like the big one that hit Amazon's East Coast region a few months ago. And it must be easy to scale your application down to fewer servers when the workload is lighter. If your usage patterns are spiky enough that you will occasionally need lots of capacity for a short time, but less capacity most of the time, then AWS is a really interesting option.

I haven't thought as much about Google AppEngine or Microsoft Azure, but they'd likely be similar. The big difference between them and AWS is that they essentially abstract away all the scaling stuff. It's a potentially good model, but you need to write your application specifically to target their environment. You need to do that for AWS as well, but the model is somewhat more similar to the traditional server-oriented model.

Re:Am I the only one?

[dohnut]

As people have said, it already has taken off. Personally, I use AWS for much humbler reasons than most. I use it for hosting my personal domain(s) -- web, DNS, SVN, etc. I used to just have a linux server in my basement do it all, but then you have a $500 machine to maintain, a static IP to provision, possible TOS violations with your ISP, poor upload speeds, etc. For a few dollars a month I can host everything I used to at home on a virtual linux server with redundant storage attached and excellent bandwidth. It was a no-brainer for me.

I also store important documents and other irreplaceable items (photos, video recordings, etc) with their S3 service. I still have things stored locally, but it gives me additional piece of mind to have it somewhere redundant and external. You don't need to use Amazon for that, but it is more convenient for me than backing stuff up to media and then physically locating it somewhere off-site.

As far as the recent newsworthy outage at AWS, I trust Amazon will learn from their mistakes and fix them more than I trust my local ISPs to do so. Also, that network outage did not affect any of my sites or data.

Re:Real world at scale.

[mlts]

I would disagree at the same rate figure.

What you pay for with the bigger servers is redundancy. The higher end servers that Oracle and IBM offer cost more, but they engineer for reliability, not absolute cheapness of price as in the commodity x86 market. Yes, you can improve uptime by adding redundancy on upper layers up to and including the backend app.

On one end, you have FB's solution where reliability isn't as much as issue as deploying fast. The top layer backend app handles the redundancy. On the other end, you have mainframes and IBM Parallel Sysplex. Most businesses end up somewhere in between.

Almost always, you get what you pay for when it comes to servers.

Somebody missed ...

[Anomalyst]

the opportunity to crow about how effective it might be in generating bitcoins.