SWF Appears to be much more interested in helping me manage clusters of instances than about streamlining the lifecycle of a single customized spot instance from inception to termination.
Thanks for the links, aminator looks to be perfect for easily crafting job specific environments -- I'll probably include this in whatever solution I come up with. Asgard on the other hand, and correct me if I'm wrong, looks to be much more oriented to those who have a lot of things running for an indefinite time frame in the cloud.
Thanks!
I'm aware that EC2 is inherently scriptable, though the documentation is incredibly poor for some areas, and heavily favours those interested in long running instances. This post is about asking others what their workflow for short term spot instances is, and generating some collaboration and sharing of ideas on the subject. Looking through the other comments there is a PhD who wrote some of his own scripts using boto (complains about its docs -- trend here?), someone working on a product to do this (wonder why he sees a business case for this?) .
The comments in this thread are evidence enough that there is hardly any consensus on how to do this easily and elegantly.
To all those shouting RTFM, you've clearly never read the EC2 docs or tried to use them for this use case. They are hardly adequate, just take a look at their scientific computing page (http://aws.amazon.com/ec2/spot-and-science/) Not a single person here has said something along the lines of "RTFM -- I did and it allowed me to easily do something similar." Just saying RTFM because you can doesn't help, nor does it mean anything if the docs are inadequate for the use case in question.
This is more or less exactly the problem, their spot instances for science page is a friggin joke.[0] Their API seems reasonable for spinning up instances and I am now looking at writing some scripts to do this, however their docs avoid ever telling you that you can run scripts in the "user data" field when starting an instance... kind of a major hurdle that the command line tools don't make clear.
I've actually got something going now with the CLI tools + docker that makes getting an environment running pretty simple. I'm going to formalize it and post it online in the near future.
[0] http://aws.amazon.com/ec2/spot-and-science/
Virtualbox will not in any way help me. I don't own, and don't want to purchase or manage the hardware myself -- time tends to be short for researchers and an automated, easy, pay per use solution is very ideal.
Slashdot Mirror
SWF Appears to be much more interested in helping me manage clusters of instances than about streamlining the lifecycle of a single customized spot instance from inception to termination.
Right.
Thanks for the links, aminator looks to be perfect for easily crafting job specific environments -- I'll probably include this in whatever solution I come up with. Asgard on the other hand, and correct me if I'm wrong, looks to be much more oriented to those who have a lot of things running for an indefinite time frame in the cloud. Thanks!
I'm aware that EC2 is inherently scriptable, though the documentation is incredibly poor for some areas, and heavily favours those interested in long running instances. This post is about asking others what their workflow for short term spot instances is, and generating some collaboration and sharing of ideas on the subject. Looking through the other comments there is a PhD who wrote some of his own scripts using boto (complains about its docs -- trend here?), someone working on a product to do this (wonder why he sees a business case for this?) . The comments in this thread are evidence enough that there is hardly any consensus on how to do this easily and elegantly. To all those shouting RTFM, you've clearly never read the EC2 docs or tried to use them for this use case. They are hardly adequate, just take a look at their scientific computing page (http://aws.amazon.com/ec2/spot-and-science/) Not a single person here has said something along the lines of "RTFM -- I did and it allowed me to easily do something similar." Just saying RTFM because you can doesn't help, nor does it mean anything if the docs are inadequate for the use case in question.
This is more or less exactly the problem, their spot instances for science page is a friggin joke.[0] Their API seems reasonable for spinning up instances and I am now looking at writing some scripts to do this, however their docs avoid ever telling you that you can run scripts in the "user data" field when starting an instance... kind of a major hurdle that the command line tools don't make clear. I've actually got something going now with the CLI tools + docker that makes getting an environment running pretty simple. I'm going to formalize it and post it online in the near future. [0] http://aws.amazon.com/ec2/spot-and-science/
Virtualbox will not in any way help me. I don't own, and don't want to purchase or manage the hardware myself -- time tends to be short for researchers and an automated, easy, pay per use solution is very ideal.