Slashdot Mirror
Amazon's Cloud May Provision 50,000 VMs a Day
Dan Jones writes "It has been estimated that Amazon Web Services is provisioning some 50,000 EC2 server instances per day, or more than 18 million per year. But that may not be entirely accurate. A single Amazon Machine Image (the virtual machine) may be launched multiple times as an EC2 instance, thereby indicating that the true number of individual Amazon servers may be lower, perhaps much lower, than 50,000 per day. So, even if it's out by a factor of 10 that's still 1.8 million VMs per year. Is that sustainable? By way of comparison, In February of this year, Amazon announced S3 contained 40 billion objects. By August, the number was 64 billion objects. This indicates a growth of 4 billion S3 objects per month, giving a daily growth total of about 133 million new S3 objects per day. How big can the cloud get before it starts to rain?"
If history tells us anything, it is that there _will_ be a failure.
The massive cloud will block sunlight and cause temperatures to drop. We've found a remedy for global warming!
Are you saying rain as bad or rain as good?
I don't see the point of this entire article. Its too early to predict if the Amazon cloud will do anything meaningful or if its going to be a spectacular failure.
How big can the cloud get before it starts to rain?"
Clouds don't work like that, they let go their rain when they enter a pressure zone where they can no longer hold water.
If Amazon is centrally dispatching, then they deserve to fail. If not, then there's no reason why getting larger would necessarily cause any particular problem.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Cloud is bad enough. Starting up bullshit analogies with clouds and rain just muddy whatever you're talking about far, far more than is necessary.
who cares how many potential VMs the "cloud" can host. its methodone for most end users/devs real problems: inefficient code. the "just pitch machines at it until it runs fast!" mentality will catch up to us.
Good people go to bed earlier.
I've never really understood the fuss around VMs. Sure , they're useful if you want to test run an OS install or run a different OS on top of another. But otherwise whats the point? Instead of having app + OS you end up with app + VM + OS so how exactly is that benefiting anyone other than the power company for the extra electricity used?
Hallelujah it's raining VMs
1) They'll increase prices if it becomes more taxing
2) New instances will come slower, some vanishing because of increases prices
3) ????
4) Stable profit.
Lets give a 12 hour lifespan, and say 25K VMs at the same time.
At 5 VMs/physical host (I suspect it is MUCH denser actually), thats only 5K servers. At 50 servers/rack, its 100 racks.
Or, in translation, not THAT much.
Test your net with Netalyzr
My company tried to provision 10,000 amazon instances to perform scalability testing of our software that runs on many computers. The math was simple - 10,000 servers * $0.15 / hour = $1,500 / hour for testing. We liked the multiple OSes & versions (Linux - Redhat, SLES, Windows - 2000, 2003, 2008?) and software stacks (mysql, apache, websphere, sql server, iis, etc...) that we all available out of the box.
However, if you need more than 20 servers, you have to fill out a form. A sales rep and tech guy called to discuss our needs. It turns out that they could only handle around 1000 instance request across all data centers unless we "reserve" the machines at $300 / each, which blew the math - 10,000 servers * $300 = $3,000,000 to start.
Looking at the article, it is likely that people are re-requesting the same machine be started & stopped multiple times per day - 50,000 is probably off by an order of 10.
I went to an Amazon's AWS talk in NYC a couple months ago where they brought some start-ups in to talk about their projects, the cloud and how the cloud helped them build their applications faster and better. During the opening talk, the speaker showed some use-cases, one including the New York Stock Exchange and how, at the closing bell, they provision over 3000 EC2 instances to crunch numbers overnight to be ready for the next morning.
A guy from a startup that I was talking to before we were seated was talking about how his company keeps between 5 and 10 instances up all the time for their application (dynamically bringing them up and down to scale with demand) and how they frequently had 4 and 5 sets of these servers running on the side for testing (20-40 instances at a time). He was talking about the metrics they were using to keep track of their use and how it was flawed due to the fact that they had hundreds of instances a day going up and down all the time.
Just because 50,000 instances are started per day doesn't mean that those 50,000 instances are running for any period of time. I frequently bring up an instance, tweak some things, create an image, then bring it down... or bring up an instance to test something for 20 minutes, then bring it down. EC2 has really benefitted my QA/Testing/Experimentation in that I really have an unlimited pool of resources to play with. It's a much more robust system than I have at home with VMWare... vmware was a gamechanger for me since before that, I had 2 physical servers at home and stacks of 40GB and 60GB HDs with multliple versions of OSs on them.
Of course AWS isn't for everyone. EC2 can be expensive for what they offer and the biggest advantage to AWS's services are that they are on-demand and work really well with applications that need to scale up AND down in real-time. If you've got an application that doesn't require to-the-minute scaling responses, it's less expensive to get a physical dedicated server with Xen on it and create your own virtual infrastructure... although if you don't have the skills or time to learn the tools, then AWS offers a much better learning curve.
...spike
Ewwwwww, coconut...
"Objects" doesn't mean VMs, objects can be files, processes, etc.
I wonder if high performance computing like ray tracing ( http://www.raytracing.ch )
or the things you find at http://cnx.org/content/m19804/latest/ can be done with this service.
There are even more reasons to be very interested in VM and the Cloud model.
Remember VM's can run anywhere, so while they CAN run at Amazon, they don't have to. They can run local or at a competitor. You choose where they are at and can move them based on your needs when you want to.
Also Linux servers can live on the same box as Windows! And since all you care about is the applications, maybe you don't really care so much what OS it uses. Maybe the config needed to run the OS can be bundled with the application config and the users just sees that, and the OS is mostly pre configured to run that application. Maybe then you don't need a giant do everthing OS, just the OS to run your application and no overhead. Maybe you now don't have to care if MS plays nice with your app.
VM is about creating an isolated location for an OS and applications, gaining hardware indepedence at the same time. Mainframers have understoord and been doing this for years, Java virtual machines were another attempt at the same thing. I don't know all the nitty gritty of why some isolation schemes work better than others, all I know is VM and hypervisors seem to finally have gotten it right.
P.S. every time someone mentions VM a Mainframer loses his wings.
Amazon EC2 is extremely attractive to spammers. They set up disposable spam servers (often using stolen credit card numbers). Amazon's policy is to shut down the account when this type of activity is detected, but they don't take any steps to keep the spammer from returning and setting up a new account. Based on this practice, the "50,000 VM's per day" seems quite reasonable when you consider the volume of activity that most spammers tend to produce.
Tired of FB/Google censorship? Visit UNCENSORED!
But it does open avenues for introducing the much loved car analogies ... windscreen wipers, rain covers, flooded engines ... could go on all day.
The benefits are many, costs (cooling, power, less hardware) Redundancy, DR, HA, recovery time. In some cases OS licensing, Win Datacenter is priced by physical CPU socket, and then you can downgrade and back rev as needed. Also, MS-SQL enterprise is priced the same, physical socket. With the cost savings its hard NOT to virtualize these days.
And on the IT side, it's actually EASIER to manage a virtual infrastructure than a physical one. Want to change the hardware in a VM, add CPU/RAM/DISKS, it's a mouse click away.
So, i'm sorry for feeding the troll, but with it modded Insightful, i had to explain WHY VMs are a benefit.
Those who can, do.