Slashdot Mirror
Amazon Betas 'Elastic' Grid Computing Service
RebornData writes "I receieved an e-mail this morning inviting me to participate in a limited beta of Amazon EC2: the Amazon Elastic Compute Cloud. It's a grid computing service that allows you to create and upload your own Linux-based machine images and run them in Amazon's system, starting at $.10 per "instance hour" (each machine instance being equivalent to a 1.7GHz Xeon with 1.75GB of RAM, and 160GB disk). You can use their tools to create and start new instances dynamically to meet whatever your particular capacity needs are at any given moment. Fedora Core 3 and 4 are explicitly supported, but any distro based on the 2.6 kernel should work. The service documentation provides more technical details. Unfortunately, it appears that the beta is limited to existing Amazon S3 users, and is already full."
At $.10 per hour, that makes a single server instance about $72 per month. If you have minimal storage needs, it can compete with a low end leased server, plus it has other advantages not present in the physical leased box world.
Personally I don't have any need for a scalable system such as this, but it certainly opens the possibility for products or projects that may not otherwise be feasible.
Have a CPU intensive batch job that can broken up and distributed? Use these boxes during the run then eliminate them when it's done. Only pay for the time you use.
At a previous job I had a task that would have been perfect for a burst-able cloud like this. Example:
Every evening we had a large number of scanned tiff images that needed to be manipulated, and a short time window in which to do it. Tiff image manipulation takes a lot of CPU resources and time. We ended up purchasing a bunch of blade servers that sat idle for the 22 hours a day they we not running images. Something like what Amazon is offering may have been a very high performance and cost effective solution to that type of problem. The control via web services could automate the whole process.
Soccer Goal Plans
Compromised Amazon grid being used as a botnet to send spam. Letting people upload their own OS image is really asking for it.
ELOI, ELOI, LAMA SABACHTHANI!?
Just junk food for thought...
Ok, so what is just me, or do the first to links on the post point to the exact same spot?
Maybe they meant the Technical Documentation?
Could I buy some server time to get my initial compile to under a week?
what? So they are building computers for you and letting you use them for whatever you desire? Hmm. 0.10/hr thats $2.40/day. Thats $876/year. Not a bad deal. Heck its even cheaper than some website services that give you a dedicated server for $200/month.
This is the umpteenth grid service where anybody can buy huge gobs of computer time. The question is, is there really anybody out there who needs to do this and doesn't have their own hardware? Sun's grid effort has pretty much laid an egg. Perhaps I have the economics wrong, but isn't it more cost effective to build your own cluster out of discarded PCs?
You can also go fart around with Amazon's Web Services for fun and profit.
They rolled this out a few months back, when I was one of the brave few to sit through the presentation at a programmers conference in Santa Clara (for a free t-shirt and pen.) It was actually amazingly cool and I'm planning something of my own with it. (but I ain't telling you because I wouldn't want anyone tempted to swipe my neat idea, like thinkgeek did to me once already.)
A feeling of having made the same mistake before: Deja Foobar
I like their pricing a great deal. It's much, much, much cheaper than many of the alternatives (notable the Sun one) AND you do not have to build your apps to use some proprietary IPC that's no good outside of their cluster.
For example, lets say I had a MPICH (or even a custom) application that I wanted to run. I'm just some joe schoe, so I
can't use the cluster in my (academic) department. I can run my application for one hour using 1000 "computers" for about $100 USD.
That's pretty good. It would cost me $1000 to use the Sun N1 stuff AND I would have to use the N1 grid-engine to develop my app.
Can't wait to see what comes out of the Beta. People give Amazon a bad rap because they're not Google, but make no mistake: they are innovators too.
The concept of virtualization is so seductive.
In our server room we have recently begun virtualizing servers and as a result have begun to think not in terms of physical servers and hard disks anymore, but in terms of resource pools of storage and processing.
It's like we have been able to smelt our physical machines and from the molten resources forge anew.
The recoverability and fault-tolerance is amazing as well - if a physical box dies there is basically no interruption in service. If something goes awry with an image we can just pull it and restore from yesterday.
Seeing Amazon offering what seems to be more of an ocean of resource than a pool is very tantilizing.
I'm certainly not the first, but I wonder if indeed local operating systems and cpus will become something of an anacronism, and that most processing will someday occur via the internet: that it will become the world-wide-mainframe.
My Computer Music Tutorial Videos
Where's the obligatory link to where Barnes & Noble has it for cheaper?
Pricing
* Pay only for what you use.
* $0.10 per instance-hour consumed (or part of an hour consumed).
* $0.20 per GB of data transferred outside of Amazon (i.e., Internet traffic).
* $0.15 per GB-Month of Amazon S3 storage used for your images (charged by Amazon S3).
Data transferred within the Amazon EC2 environment, or between Amazon EC2 and Amazon S3, is free of charge (i.e., $0.00 per GB).
Amazon S3 usage is billed separately from Amazon EC2; charges for each service will be billed at the end of the month.
(Amazon EC2 is sold by Amazon Web Services LLC.)
Either that, or "Amazon Elastic Compute Cloud" is the name of a new nanobot swarm threat that came out of Brazil.
Where were you when the voynix came?
I'm surprised that Amazon isn't using any of the existing methods of creating virtual machine images, especially since it seems likely that they are using some sort of VM technology to do all of this. VMs are the easiest way to provide identical hardware to all of the instances; VMs also provide the tools to efficently load and unload images on demand. I wonder if we'll see different pricing depending on how CPU intensive an instance is. I can see using an instance as a server for less-popular online games; you'd only pay for those times when someone is playing. Depending on how fast multiple instances could be ramped up, you could also provide slashdot-effect insurance; just bring more instances on-line as more people arrived and shut them down as the peak passes.
Nothing for 6-digit uids?
Whoa!
"Amazon EC2: the Amazon Elastic Compute Cloud."
Say it with me: Amazon Elastic Compute Cloud.
For God's sake, it's like a Tom Wolfe book!
I used to work on grid computing at a Japanese technology incubator. The project eventually got shelved because there were just not enough applications which honestly benefit from it when you consider the pain involved in programming the application to distribute over the grid (I gather our American competitors are having the same problem), but the idea was taking your company's totally-idle-at-night fleet of desktops and turning them into a virtual Beowulf cluster. I think our biggest success, if you can call it that, was turning about 1,000 computers at 15 sites (hospitals, college labs, and the like) into what for two hours was the largest supercomputer in the prefecture by a factor of 10. (Gifu Prefecture... this is sort of like bragging that you have the biggest computer in Iowa, but hey, you take your wins where you can.)
The downside, aside from the 100% solvable issue of being able to wake up 1,000 computers in the middle of the night without needing a staff of 80 (which was what it took for our "successful" run), is that programming for a grid is hard and tedious, and none of the frameworks that I am familiar with really take it down to the level where it needs to be for "regular" programmers to be able to do it.
In short: a lovely technology, some practical uses, not exactly going to set the world on fire though.
Help poke pirates in the eyepatch, arr.
I remember when Sun's grid was out. I tried for days to get pricing, and eventually I was quoted something like $1000 per hour, except I couldnt get one hour.
I was trying out different parameter combinations in the uCLinux kernel to check compile sizes and functionality, and wanted horsepower to compile every iteration of the kernel. But The Sun deal was a joke, I could build a few Athlon beige boxes and do it cheaper.
This deal sounds good enough for me to take it out on a spin of a day or two, but I really need to check the bandwidth (both the speed and allowance). Despite this, for bigger tasks, I think its still more feasible to just build beige machines using Core Duo CPUs, some ram and disk and just use those.
When I was at college, I had an idea of putting up a bootp server in the library, getting all the machines to boot the linux image at night, and just using those machines for fun. All 100+ of them. I just didnt have the need for so much horsepower.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
It seems it is the right moment to revive the Multics project.
Maybe Computers will never be as intelligent as Humans.
For sure they won't ever become so stupid. [VR-1988]
Just junk food for thought...
Maybe that would explain why the last few times I been to the Amazon website that it was unavailable. The grid is sucking up all the available CPU cycles. So much for customer service.
This service is paired with the Amazon S3 storage service, which has a high-bandwidth connection to the servers. Data transfer between EC2 and S3 is free.
-R
At the point where you legitmately need 800 processors ... you're probably big enough to borrow the money to build it. Most small million dollar firms I know of run off of simple dual core AMD64s. At the point where you need a 50 million dollar cluster ... you're probably big enough to buy one.
It sounds all nice in practice to buy "cheap time" but it's cheap for a reason
1. You don't own the boxes
2. It's not leased, they could go down at any moment
3. All your data is remote
4. You're subject to your network and their network and all the networks in between
5. You don't own the boxes so you can't add hardware, change key software components, etc...
There are a lot of downsides to it and running a business from it is stupid.
Tom
Someday, I'll have a real sig.
Keep in mind that keeping around old images isn't really much of a burden - 10G on S3 is $1.50/mo and $2.00 to upload (unless you make it from a running EC2 instance!), and as it's compressed, the prices may be even lower. As for network security, what makes you think amazon hasn't properly firewalled these VMs to keep them from forging their IPs/MACs, and to prevent sniffing? They're already configured with a default-deny firewall that the user needs to lift with an API call.
A very good idea IMO. Say your game burns 30KB/s of bandwidth per player... it'll take (by my weak math) about 9 hours to use 1GB of bandwidth, for which you paid $0.20... on average, that's gotta be at most an average daily usage. So effectively, your cost per user is in the $0.20/day range, or about $6/month for bandwidth.
Wonder what sort of latency these clusters have, if they'd be suitable to host a BF2 server or something.
More data, damnit!
God there's more penis waving going on in this thread than in a gay porno.
By summer it was all gone...now shesmovedon. --
No grid needed its $0.00, all thise bribe^Wcampaign contributions to his pet congress critters are write-offs.
There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
"Account Authority Digital Signature"---one letter off, but close enough for me.
Premature optimization is the root of all evil
This is cool. The ultimate silent machine.
And think of the possibilities...run a virtual
machine inside your Amazon VM, totally encrypted
with ssh tunnels in and out...
Sorry, I appear to be drooling!
Maybe you're just too worried that you're gay and that you've picked up an immune deficiency. Take two Tylenols, go home, watch some Spike, and when you wake up, maybe you'll remember that you're a bigoted a**hole who needs to get some computing done for $.10 an hour.
This service could be extremely useful as a render farm for 3D graphics. It would be wonderful to be able to call up 1000 CPUs for a couple days or weeks at a time, without paying for them when not in use.
... now you only need 50 or 60 CPUs, so it starts to look more reasonable to just have them on-site.
:-).
g e/story/0,10801,105052,00.html
The problem is that movies output at something like 10M per frame (4000 pixels wide by 2000 pixels high by 48 bits per pixel div 8 bits per byte div 4 lossless compression factor), so you'd be talking about nearly a gigabyte for 4 seconds of footage (and my compression factor may be way off). So you can spin up 1000 CPUs for a half hour to generate five minutes of footage, and wait for weeks for your data to download over your T4 link. You could decide you just want to use this for proofs and cut the resolution in half on each dimension, cut to 24 bits per pixel, allow lossy compression, which cuts down the space by a factor of 16, but
Yeah, I've wondered about this exact issue in the past. I work at a place which specializes in building amazing giant clusters, and I have friends at Pixar and Dreamworks. But this one is going to be hard to carry. Movie houses keep their render farms hot for 6 months at a time, or if they're like Pixar, pretty much continuously. When ILM moved from Santa Rosa to Presidio, they basically installed a 10gbit fiber link (*) between to help with the transition period (when the render farm was in a different location from the workstations). Amazon isn't going to install a 10gbit fiber link for you
-scott
(*) http://www.computerworld.com/hardwaretopics/stora
Um, you do realize you have to pay for the CPU time you use, right? I estimate that every cent of RSA factoring prize money will cost you about $50 spent on Amazon Compute Cloud services. But with the capacity of EC2, I bet you could make up for it in volume!
Why? Just your data transfer charges and disk storage space (those are billed extra by Amazon) are going to cost more than the electricity to run your box overnight (and you already have the box, so its not like you're spending extra for it). Plus it takes time to upload all that video data - time which your machine is running anyway, so you might as well just use it to encode it yourself.
cpu cycles have become so commoditized that there's no point in going to the hassle of "leasing" a small virtual cluster. Most people have several boxes already - cluster them and be done with it.
Perhaps, but a one-off or two-off project that requires a million hours of CPU time in 6 months can be outsourced to someone who has the equipment easier and cheaper than setting up your own hardware. Setting up enough servers or even desktop-class workstations could take a big chunk of those 6 months. And it'll probably cost more than the Amazon service, in hardware and manpower.
I say this from experience - my employer specializes in an industry-specific data validation process, and has spent 20 years writing and perfecting a few million lines of code that are repeated over and over within this process. One client came to us and wanted to run this process against an unprecedented amount of data, and it came out to 2000 CPU-years of processing time, with a 1 year deadline. The physical space, network and power requirements for 500 quad-processor servers would overwhelm our facilities, not to mention our network staff, so I'd at least consider outsourcing the hardware at this scale. Alas, we're decidedly a Windows shop and this isn't an option for us yet... Mono may change that.
Most businesses would not "run from" this cluster on a daily basis, but use it for a specific task-oriented purpose like research, math, graphical rendering, etc. Something they do once or twice, or for that client that wants, "What you do best, just 10,000 times bigger."
I havent looked at the networking details; configuring virtual lans for isolation is possible, albeit tricky.
One thing about image size is it isnt mandatory to have guis on these things. I would go for something like DSL for a lightweight system image, they produce very small binaries. That only leaves the data. S3 stores bulk data nicely, but you also need a back end RDBMS, or you host mysql in a cluster all of your own.
I suspect that the next offering of amazon will be pay-to-use RDBMS with an availability guarantee; MySQL-7x24 or something, so you can use your own front end images against the back end.
I also think this service shows us how they are moving to host their own systems; fully virtualized, on-demand cloning. nice,
maybe you'll remember that you're a bigoted a**hole who needs to get some computing done for $.10 an hour.
I remember 3 of my friends in the 80's coming down with GRID before the first heterosexuals were diagnosed and the CDC decided that it was not a 'gay' disease. They all died. That is why the acronym "GRID" is burned into my conscience. Please don't call anyone an a**hole before you know them, that is acting stupid.
I'm sorry for being kurt, but your post hit me in a hard spot (I'm not a biggot [and I hate Spike]). I try to make light of past injuries. I'm assuming that you are young, so here is an FYI about was I was joking about.
Apologies. Didn't know the GRID reference. I thought you were pulling something out of your a**.