Slashdot Mirror
Amazon Betas 'Elastic' Grid Computing Service
RebornData writes "I receieved an e-mail this morning inviting me to participate in a limited beta of Amazon EC2: the Amazon Elastic Compute Cloud. It's a grid computing service that allows you to create and upload your own Linux-based machine images and run them in Amazon's system, starting at $.10 per "instance hour" (each machine instance being equivalent to a 1.7GHz Xeon with 1.75GB of RAM, and 160GB disk). You can use their tools to create and start new instances dynamically to meet whatever your particular capacity needs are at any given moment. Fedora Core 3 and 4 are explicitly supported, but any distro based on the 2.6 kernel should work. The service documentation provides more technical details. Unfortunately, it appears that the beta is limited to existing Amazon S3 users, and is already full."
At $.10 per hour, that makes a single server instance about $72 per month. If you have minimal storage needs, it can compete with a low end leased server, plus it has other advantages not present in the physical leased box world.
Personally I don't have any need for a scalable system such as this, but it certainly opens the possibility for products or projects that may not otherwise be feasible.
Have a CPU intensive batch job that can broken up and distributed? Use these boxes during the run then eliminate them when it's done. Only pay for the time you use.
At a previous job I had a task that would have been perfect for a burst-able cloud like this. Example:
Every evening we had a large number of scanned tiff images that needed to be manipulated, and a short time window in which to do it. Tiff image manipulation takes a lot of CPU resources and time. We ended up purchasing a bunch of blade servers that sat idle for the 22 hours a day they we not running images. Something like what Amazon is offering may have been a very high performance and cost effective solution to that type of problem. The control via web services could automate the whole process.
Soccer Goal Plans
Compromised Amazon grid being used as a botnet to send spam. Letting people upload their own OS image is really asking for it.
ELOI, ELOI, LAMA SABACHTHANI!?
Just junk food for thought...
Ok, so what is just me, or do the first to links on the post point to the exact same spot?
Maybe they meant the Technical Documentation?
Could I buy some server time to get my initial compile to under a week?
Sounds neet, but what practical applications could be made from this service? /. thinks this may be used for...
I'm not trolling, rather genuinely interested in what
This is not the greatest
what? So they are building computers for you and letting you use them for whatever you desire? Hmm. 0.10/hr thats $2.40/day. Thats $876/year. Not a bad deal. Heck its even cheaper than some website services that give you a dedicated server for $200/month.
This sounds better than Sun's grid computing solutions in that there is a lot more control of the machine (at long as its a 2.6 kernel) at a better price. I could use any distributed compiler package/library such as MPI?
This is the umpteenth grid service where anybody can buy huge gobs of computer time. The question is, is there really anybody out there who needs to do this and doesn't have their own hardware? Sun's grid effort has pretty much laid an egg. Perhaps I have the economics wrong, but isn't it more cost effective to build your own cluster out of discarded PCs?
You can also go fart around with Amazon's Web Services for fun and profit.
They rolled this out a few months back, when I was one of the brave few to sit through the presentation at a programmers conference in Santa Clara (for a free t-shirt and pen.) It was actually amazingly cool and I'm planning something of my own with it. (but I ain't telling you because I wouldn't want anyone tempted to swipe my neat idea, like thinkgeek did to me once already.)
A feeling of having made the same mistake before: Deja Foobar
I like their pricing a great deal. It's much, much, much cheaper than many of the alternatives (notable the Sun one) AND you do not have to build your apps to use some proprietary IPC that's no good outside of their cluster.
For example, lets say I had a MPICH (or even a custom) application that I wanted to run. I'm just some joe schoe, so I
can't use the cluster in my (academic) department. I can run my application for one hour using 1000 "computers" for about $100 USD.
That's pretty good. It would cost me $1000 to use the Sun N1 stuff AND I would have to use the N1 grid-engine to develop my app.
Can't wait to see what comes out of the Beta. People give Amazon a bad rap because they're not Google, but make no mistake: they are innovators too.
The concept of virtualization is so seductive.
In our server room we have recently begun virtualizing servers and as a result have begun to think not in terms of physical servers and hard disks anymore, but in terms of resource pools of storage and processing.
It's like we have been able to smelt our physical machines and from the molten resources forge anew.
The recoverability and fault-tolerance is amazing as well - if a physical box dies there is basically no interruption in service. If something goes awry with an image we can just pull it and restore from yesterday.
Seeing Amazon offering what seems to be more of an ocean of resource than a pool is very tantilizing.
I'm certainly not the first, but I wonder if indeed local operating systems and cpus will become something of an anacronism, and that most processing will someday occur via the internet: that it will become the world-wide-mainframe.
My Computer Music Tutorial Videos
Where's the obligatory link to where Barnes & Noble has it for cheaper?
Anyone else getting tired off all this Web 2.0 (tm) terminology?... "Compute cloud", "Grid computing service", etc. Every time I see this, I think Gay-Related Immune Deficiency computing service. Guess I'm showing my age. Anyone else remember the diet candy called Ayds?... wow, bet the marketing guys at that company had coronaries after the CDC press release. It's not good when a government agency names a fatal syndrome after your product.
Pricing
* Pay only for what you use.
* $0.10 per instance-hour consumed (or part of an hour consumed).
* $0.20 per GB of data transferred outside of Amazon (i.e., Internet traffic).
* $0.15 per GB-Month of Amazon S3 storage used for your images (charged by Amazon S3).
Data transferred within the Amazon EC2 environment, or between Amazon EC2 and Amazon S3, is free of charge (i.e., $0.00 per GB).
Amazon S3 usage is billed separately from Amazon EC2; charges for each service will be billed at the end of the month.
(Amazon EC2 is sold by Amazon Web Services LLC.)
Either that, or "Amazon Elastic Compute Cloud" is the name of a new nanobot swarm threat that came out of Brazil.
Where were you when the voynix came?
I'm surprised that Amazon isn't using any of the existing methods of creating virtual machine images, especially since it seems likely that they are using some sort of VM technology to do all of this. VMs are the easiest way to provide identical hardware to all of the instances; VMs also provide the tools to efficently load and unload images on demand. I wonder if we'll see different pricing depending on how CPU intensive an instance is. I can see using an instance as a server for less-popular online games; you'd only pay for those times when someone is playing. Depending on how fast multiple instances could be ramped up, you could also provide slashdot-effect insurance; just bring more instances on-line as more people arrived and shut them down as the peak passes.
Nothing for 6-digit uids?
Whoa!
"Amazon EC2: the Amazon Elastic Compute Cloud."
Say it with me: Amazon Elastic Compute Cloud.
For God's sake, it's like a Tom Wolfe book!
I used to work on grid computing at a Japanese technology incubator. The project eventually got shelved because there were just not enough applications which honestly benefit from it when you consider the pain involved in programming the application to distribute over the grid (I gather our American competitors are having the same problem), but the idea was taking your company's totally-idle-at-night fleet of desktops and turning them into a virtual Beowulf cluster. I think our biggest success, if you can call it that, was turning about 1,000 computers at 15 sites (hospitals, college labs, and the like) into what for two hours was the largest supercomputer in the prefecture by a factor of 10. (Gifu Prefecture... this is sort of like bragging that you have the biggest computer in Iowa, but hey, you take your wins where you can.)
The downside, aside from the 100% solvable issue of being able to wake up 1,000 computers in the middle of the night without needing a staff of 80 (which was what it took for our "successful" run), is that programming for a grid is hard and tedious, and none of the frameworks that I am familiar with really take it down to the level where it needs to be for "regular" programmers to be able to do it.
In short: a lovely technology, some practical uses, not exactly going to set the world on fire though.
Help poke pirates in the eyepatch, arr.
"The question is, is there really anybody out there who needs to do this and doesn't have their own hardware? "
http://www.irtc.org/
Next question?
1.7Ghz Xeon?
/proc/cpuinfo | grep "model name"
:-)
tom@bigbox ~ $ cat
model name : AMD Opteron(tm) Processor 885
model name : AMD Opteron(tm) Processor 885
model name : AMD Opteron(tm) Processor 885
model name : AMD Opteron(tm) Processor 885
1.75GB of ram?
tom@bigbox ~ $ free
total used free shared buffers cached
Mem: 6104840 700028 5404812 0 62092 399356
No thank you.
Tom
Someday, I'll have a real sig.
Network storage, a few wrapper scripts and you're away. Kick off pretty much any GUI app on "the grid" and have it plonk it's interface back on your display. With something like GridEngine you can designate some of the machines as OpenOffice servers, others as Thunderbird servers to take better advantage of shared libraries, filesystem buffers and CPU caches, or you can just have everything kick off on the least loaded machines. The result is that the user's GUI apps could be running on a dozen different machines completely transparently.
With Windows of course, your extra machines might as well be doorstops.
Deleted
I remember when Sun's grid was out. I tried for days to get pricing, and eventually I was quoted something like $1000 per hour, except I couldnt get one hour.
I was trying out different parameter combinations in the uCLinux kernel to check compile sizes and functionality, and wanted horsepower to compile every iteration of the kernel. But The Sun deal was a joke, I could build a few Athlon beige boxes and do it cheaper.
This deal sounds good enough for me to take it out on a spin of a day or two, but I really need to check the bandwidth (both the speed and allowance). Despite this, for bigger tasks, I think its still more feasible to just build beige machines using Core Duo CPUs, some ram and disk and just use those.
When I was at college, I had an idea of putting up a bootp server in the library, getting all the machines to boot the linux image at night, and just using those machines for fun. All 100+ of them. I just didnt have the need for so much horsepower.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
It seems it is the right moment to revive the Multics project.
Maybe Computers will never be as intelligent as Humans.
For sure they won't ever become so stupid. [VR-1988]
Maybe that would explain why the last few times I been to the Amazon website that it was unavailable. The grid is sucking up all the available CPU cycles. So much for customer service.
This service is paired with the Amazon S3 storage service, which has a high-bandwidth connection to the servers. Data transfer between EC2 and S3 is free.
-R
Keep in mind that keeping around old images isn't really much of a burden - 10G on S3 is $1.50/mo and $2.00 to upload (unless you make it from a running EC2 instance!), and as it's compressed, the prices may be even lower. As for network security, what makes you think amazon hasn't properly firewalled these VMs to keep them from forging their IPs/MACs, and to prevent sniffing? They're already configured with a default-deny firewall that the user needs to lift with an API call.
A very good idea IMO. Say your game burns 30KB/s of bandwidth per player... it'll take (by my weak math) about 9 hours to use 1GB of bandwidth, for which you paid $0.20... on average, that's gotta be at most an average daily usage. So effectively, your cost per user is in the $0.20/day range, or about $6/month for bandwidth.
Wonder what sort of latency these clusters have, if they'd be suitable to host a BF2 server or something.
More data, damnit!
Ah yes, I'v ebeen perdicting this for 5 years now, and I see that it's finally "coming of age".
The beginning of the end of Personal Storage.
All your data belongs to us.
Stock up on hard drives while you can.
Privacy is dead, long live privacy.
No grid needed its $0.00, all thise bribe^Wcampaign contributions to his pet congress critters are write-offs.
There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
This is cool. The ultimate silent machine.
And think of the possibilities...run a virtual
machine inside your Amazon VM, totally encrypted
with ssh tunnels in and out...
Sorry, I appear to be drooling!
The end of the installation section of the Gentoo Handbook says:
Sorry, but my definition of a working system includes the Apps, modules and libraries that I need to do the job: samba, X, NVIDIA-glx, KDE, KMail, Firefox, lcms, Cinepaint, Scribus, Imagemagick, PerlMagick, libusb, VueScan (not in portage), etc in the case of this machine. This Dual Athlon 2600 with 1GB RAM and 4 RAID5'd 7200RPM ATA-133 drives has been compiling over 12 hours, and has a lot more to go.
Incedentaly, that migration guide appears to work, just make sure
This service could be extremely useful as a render farm for 3D graphics. It would be wonderful to be able to call up 1000 CPUs for a couple days or weeks at a time, without paying for them when not in use.
... now you only need 50 or 60 CPUs, so it starts to look more reasonable to just have them on-site.
:-).
g e/story/0,10801,105052,00.html
The problem is that movies output at something like 10M per frame (4000 pixels wide by 2000 pixels high by 48 bits per pixel div 8 bits per byte div 4 lossless compression factor), so you'd be talking about nearly a gigabyte for 4 seconds of footage (and my compression factor may be way off). So you can spin up 1000 CPUs for a half hour to generate five minutes of footage, and wait for weeks for your data to download over your T4 link. You could decide you just want to use this for proofs and cut the resolution in half on each dimension, cut to 24 bits per pixel, allow lossy compression, which cuts down the space by a factor of 16, but
Yeah, I've wondered about this exact issue in the past. I work at a place which specializes in building amazing giant clusters, and I have friends at Pixar and Dreamworks. But this one is going to be hard to carry. Movie houses keep their render farms hot for 6 months at a time, or if they're like Pixar, pretty much continuously. When ILM moved from Santa Rosa to Presidio, they basically installed a 10gbit fiber link (*) between to help with the transition period (when the render farm was in a different location from the workstations). Amazon isn't going to install a 10gbit fiber link for you
-scott
(*) http://www.computerworld.com/hardwaretopics/stora
I believe they are piggy-backing Xen, with their tools being wrappers.
Um, you do realize you have to pay for the CPU time you use, right? I estimate that every cent of RSA factoring prize money will cost you about $50 spent on Amazon Compute Cloud services. But with the capacity of EC2, I bet you could make up for it in volume!
I havent looked at the networking details; configuring virtual lans for isolation is possible, albeit tricky.
One thing about image size is it isnt mandatory to have guis on these things. I would go for something like DSL for a lightweight system image, they produce very small binaries. That only leaves the data. S3 stores bulk data nicely, but you also need a back end RDBMS, or you host mysql in a cluster all of your own.
I suspect that the next offering of amazon will be pay-to-use RDBMS with an availability guarantee; MySQL-7x24 or something, so you can use your own front end images against the back end.
I also think this service shows us how they are moving to host their own systems; fully virtualized, on-demand cloning. nice,