Slashdot Mirror

That's odd, I was just re-reading their docs today since we were affected and they make it clear that AZ refers to the instances in the same Region (i.e. the us-east-1a,b,c,d you mentioned). See: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

Warning heeded, but I saw this on a blog post at commoncrawl.org.

This bucket is marked with Amazon Requester-Pays flag, which means all access to the bucket contents requires an an http request that is signed with your Amazon Customer Id. The bucket contents are accessible to everyone, but the Requester-Pays restriction ensures that if you access the contents of the bucket from outside the EC2 network, you are responsible for the resulting access charges. You don’t pay any access charges if you access the bucket from EC2, for example via a map-reduce job, but you still have to sign your access request. Details of the Requeser-Pays API can be found here: http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RequesterPaysBuckets.html

If I understood that right, at least getting started with the tutorial will not result in me coughing up $200. Correct me if I am mistaken.

If you want to setup a cloud deployment on, say, Amazon's EC2, it's quite easy, and then once you're up and running you can then decide if you want to buy support for that deployment.

If instead you visit RedHat's cloud page, you'll see no similar guide to getting started. As far as I can tell, this is because you need to have a RHEL license to even get access to their EC2 AMI files. As close as you can get for free is the RightScale AMI for CentOS.

A lot of technical decision are made through the path of least resistance for getting started. Right now, if EC2 is your cloud platform, and you want to deploy a simple setup that you can add support to later, Ubuntu is where you'll end up at. A someone who leans toward RedHat for servers, I've been frustrated lately that I can't do free proof of concept deployments of RHEL on EC2, and then add support to them only once the result has been signed off as working. And for always unpaid setups, people don't really trust the RightScale AMI images the way they do the official Ubuntu ones either.

Both yum and deb based distributions have the ability to bootstrap the whole system under a mount point other than root. This is for the benefit of their installer, as you can imagine. Simply apt-get/yum install the one package you need, say apache httpd, and the package management figures out all the dependencies. After installation, you chroot to the mount point (don't forget to mount /proc and /sys there too) and run the service you want.

Instructions on how to build Amazon EC2 AMI is very similar to this, so you might find that helpful.

• http://ec2ubuntu.googlecode.com/svn/trunk/bin/ec2ubuntu-build-ami
• http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html#install-operating-system

Of course, for the purpose of chroot, you don't need to install any new kernel. If you already know about cryptsetup and LUKS, you can then mount an encrypted disk image, install the packages, and chroot into it for the service to run.

After saying all this, I think you really should switch provider, given how unhappy you are with them. Even if you manage to get the whole Slashdot to side with you, your provider will not likely change the way they do things.

Amazon S3 provides an API for getting a BitTorrent tracker for any publicly-readable file you have hosted on S3. Look! Yet another legitimate use :)

Since Amazon S3 has the native ability to be a torrent for your content stored there, I would think serving the content with Bittorrent would be better then having each person pay to access the data.

Using BitTorrent with Amazon S3

http://docs.amazonwebservices.com/AmazonS3/latest/index.html?S3Torrent.html

Screenshots

As one of their two demos, they show how you can use the GUI to add an Amazon EC2 instance "in only five minutes" -- as opposed to typing one (1) command or using Amazon's own ElasticFox.

While most of what you say is true, that particular statement is not. EC2 provides a virtual machine, running the operating system of your choice - anything you could do on your single co-located server, you can do on an EC2 machine.

There are heaps of limitations compared to dedicated machines. Check out the Amazon forums, and you'll be surprised how far you'll be from getting the full sensation of the real thing.

You can for example only have 1 static public IP per instance. Bad for hosting multiple Web apps with SSL on each domain.

Load balancing multiple instances is also a bitch.

We considered it seriously for a high-volume high-availability Web site project, but kept running into dealbreaking limitations, and some we just happened to stumble upon during testing. Minor things that suddenly turn incredibly difficult to solve.

Add to that the cost.. In our cost comparisons we found EC2 costing the same or more than managed dedicated servers with tier 1 providers.

There's no question EC2, VMWare and other "cloud" platforms will be highly attractive, but not yet.

While most of what you say is true, that particular statement is not. EC2 provides a virtual machine, running the operating system of your choice - anything you could do on your single co-located server, you can do on an EC2 machine.

There are heaps of limitations compared to dedicated machines. Check out the Amazon forums, and you'll be surprised how far you'll be from getting the full sensation of the real thing.

You can for example only have 1 static public IP per instance. Bad for hosting multiple Web apps with SSL on each domain.

Load balancing multiple instances is also a bitch.

We considered it seriously for a high-volume high-availability Web site project, but kept running into dealbreaking limitations, and some we just happened to stumble upon during testing. Minor things that suddenly turn incredibly difficult to solve.

Add to that the cost.. In our cost comparisons we found EC2 costing the same or more than managed dedicated servers with tier 1 providers.

There's no question EC2, VMWare and other "cloud" platforms will be highly attractive, but not yet.

I thought of this problem myself for a while, when playing around with the idea to try out the "cloud". You could use pound, a lot of its use for cloud computing has been discussed in the interwebs already. Biggest point of concern will be if the load balancer keeps your ssl data encrypted.

So here's a little about what EC2 actually is, for those of you who don't know. You don't have to reply here, start your own comments ;)

The Elastic Compute Cloud was originally designed as a way to host applications that needed lots of CPUs, and the option to expand by adding more CPUs. It's a hosting service that lets you start up virtual machines to run any software you want: they have a wide variety of pre-packaged open-source operating systems you can pick to start up your VMs with.

Starting up a VM takes just a minute or two, and it's point-and-click thanks to the Firefox extension. Each VM comes in one of three sizes: small (webhead), large (database), and extra large (bigass database). They cost respectively $72, $288, and $576 a month (billed by the hour), plus bandwidth ($0.18/GB out, somewhat cheaper for data going in and there's a price break at 10 TB).

One of the concerns everyone raises with hosting on virtual machines is that if a VM instance goes down, you lose everything on it. It comes with hard drive storage (160 GB on the small size), but if something goes wrong, that data's gone.

I think the rejoinder here is that, on real hardware, if something goes wrong, your data's gone. You never set up an enterprise-level website on the assumption that any particular hardware has to survive. Single points of failure are always a mistake, and backups are always a necessity. When any machine explodes - real or virtual - the question is how fast your system recovers to "working well enough" (seconds, hopefully) and then how long it takes you to get it "back to normal" behind the scenes (hours, hopefully). Those answers shouldn't depend on whether there's a physical drive to yank out of a dead physical machine that may or may not retain valid data.

Which brings up what I think is one of the selling points of EC2: free fast bandwidth to S3, Amazon's near-infinite-size, redundantly-replicated data storage platform. That's a nice backup option to have available. That's part of why, if I were starting a new web service, I wouldn't host it on real hardware. I'd like not having to worry about backups, tapes, offsite copies... bleah, let someone else worry about it.

Slashdot hasn't run many stories on EC2 (none that I know of) because until now it's been a niche service. Without a way to guarantee that you can have a static IP, there had been a single point of failure: if your outward-facing VMs all went down, your only recourse was to start up more VMs on new, dynamically-assigned IPs, point your DNS to them, and wait hours for your users' DNS caches to expire. That meant that while it may have been a good service for sites that needed to do massive private computation, it was an unacceptable hosting service.

Now with static IPs, you basically set up your service to have several VMs which provide the outward-facing service (maybe running a webserver, or a reverse proxy for your internal webservers), and you point your public, static IPs at those. If one or more of them goes down, you start up new copies of those VMs and repoint the IPs to them. No DNS changes required.

I know there are other companies offering web hosting through virtual servers. Please share information about them, the more we all know the better.

So here's a little about what EC2 actually is, for those of you who don't know. You don't have to reply here, start your own comments ;)

The Elastic Compute Cloud was originally designed as a way to host applications that needed lots of CPUs, and the option to expand by adding more CPUs. It's a hosting service that lets you start up virtual machines to run any software you want: they have a wide variety of pre-packaged open-source operating systems you can pick to start up your VMs with.

Starting up a VM takes just a minute or two, and it's point-and-click thanks to the Firefox extension. Each VM comes in one of three sizes: small (webhead), large (database), and extra large (bigass database). They cost respectively $72, $288, and $576 a month (billed by the hour), plus bandwidth ($0.18/GB out, somewhat cheaper for data going in and there's a price break at 10 TB).

One of the concerns everyone raises with hosting on virtual machines is that if a VM instance goes down, you lose everything on it. It comes with hard drive storage (160 GB on the small size), but if something goes wrong, that data's gone.

I think the rejoinder here is that, on real hardware, if something goes wrong, your data's gone. You never set up an enterprise-level website on the assumption that any particular hardware has to survive. Single points of failure are always a mistake, and backups are always a necessity. When any machine explodes - real or virtual - the question is how fast your system recovers to "working well enough" (seconds, hopefully) and then how long it takes you to get it "back to normal" behind the scenes (hours, hopefully). Those answers shouldn't depend on whether there's a physical drive to yank out of a dead physical machine that may or may not retain valid data.

Which brings up what I think is one of the selling points of EC2: free fast bandwidth to S3, Amazon's near-infinite-size, redundantly-replicated data storage platform. That's a nice backup option to have available. That's part of why, if I were starting a new web service, I wouldn't host it on real hardware. I'd like not having to worry about backups, tapes, offsite copies... bleah, let someone else worry about it.

Slashdot hasn't run many stories on EC2 (none that I know of) because until now it's been a niche service. Without a way to guarantee that you can have a static IP, there had been a single point of failure: if your outward-facing VMs all went down, your only recourse was to start up more VMs on new, dynamically-assigned IPs, point your DNS to them, and wait hours for your users' DNS caches to expire. That meant that while it may have been a good service for sites that needed to do massive private computation, it was an unacceptable hosting service.

Now with static IPs, you basically set up your service to have several VMs which provide the outward-facing service (maybe running a webserver, or a reverse proxy for your internal webservers), and you point your public, static IPs at those. If one or more of them goes down, you start up new copies of those VMs and repoint the IPs to them. No DNS changes required.

I know there are other companies offering web hosting through virtual servers. Please share information about them, the more we all know the better.

I agree that some of the WS-* extensions are extremely useful, but only if you actually need the functionality/complexity.

The services I write are consumed by developers at educational institutions around the world, many using nothing more than a quick-and-dirty PHP script. They appreciate the simplicity and ease of use - our read-only resources can be explored directly inside a web browser, and I point folks to the RestTest Firefox plugin if they need to experiment with our writable services.

However, I would hardly define our web services as simple... We've just done a lot of design work to make them appear simple. Through careful definition of resources, well thought out URL schemes that are human readable/constructible, and liberal use of XLink to point out related resources, we've avoided much of need for the complexity of the WS-* world.

With regard to WS-Security and WS-Trust - I've read and appreciated their specs before. Both appear well intentioned and well designed. However, the rest of the WS-* stack is so distasteful that it taints the gems that can be found. I'd rather take the concepts behind the specs, and implement the minimal subset my application needs. For example, see Amazon's REST authentication mechanism - simple yet effective.

As for WS-* interop, the base specifications are so complex that a complete web service toolkit is required to even contemplate interoperability. Even worse are the slight incompatibilities between major vendors - Microsoft's generated WSDLs for .NET services don't always map well to IBM's stack, for example (at least as of 2005).

I may have worded my initial post too strongly. Here's when I would consider WS-*

• Internal web services consumed by other resources at a single organization
• Homogeneous development platform across servers and clients, or enough time to work around minor interop issues
• A demonstrated need for more than a few WS-* extensions

We've been using it for a few months now and its great.

With a single command we can export computing tasks from our main system to a customized instance at amazon and when complete, import the resulting data. All powered by a few simple bash scripts. We can fire up any number of tasks like this and massively increase our overall processing capacity whenever needed and then shut it all down when not.

So far, after several months of running multiple instances we've not had a single failure or data loss although even if an instance had died it would make little difference since we can easily just export the tasks again at any time.

There is also the handy EC2UI firefox plugin to manage your instances..
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609&categoryID=88

Once you get the hang of EC2 you will likely come up with all sorts of computing tasks you can 'out source' from your current systems. Overall I highly recommend it.

Been playing with Amazon Elastic Compute Cloud more than one year, like its simplicity and great deal of opportunities it provides for businesses and other type of clients. Forum provides good deal of advice and useful information (see http://developer.amazonwebservices.com/connect/forum.jspa?forumID=30 ) Resource center has all kinds of tools to get you running in very short period of time, including pre-configured images of operating systems (currently only Linux), called Public AMIs. There's also some good blogs ( http://ihatecubicle.blogspot.com/ ), that provide help on advanced things like persistence to external services (S3, Nirvanix etc). SQS provides messaging facility with simple API, so it's easy to work with.

Its your lucky day.
Let me introduce you to EC2

Ok, so what is just me, or do the first to links on the post point to the exact same spot?
Maybe they meant the Technical Documentation?