Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: Scientific Computing Workflow For the Cloud?

Posted by Soulskill on from the science-as-a-service dept.

diab0lic writes "I have recently come into the situation where I need to run cloud computing on demand for my research. Amazon's EC2 Spot Instances are an ideal platform for this as I can requisition an appropriate instance for the given experiment {high cpu, high memory, GPU instance} depending on its needs. However I currently spin up the instance manually, set it up, run the experiment, and then terminate manually. This gets tedious monitoring experiments for completion, and I incur unnecessary costs if a job finishes while I'm sleeping, for example. The whole thing really should be automated. I'm looking for a workflow somewhat similar to this:
  1. Manually create Amazon machine image (AMI) for experiment.
  2. Issue command to start AMI on specified spot instance type.
  3. Automatically connect EBS to instance for result storage.
  4. Automatically run specified experiment, bonus if this can be parameterized.
  5. Have AMI automatically terminate itself upon experiment completion.

Something like docker that spun up on-demand spot instances of a specified type for each run and terminated said instance at run completion would be absolutely perfect. I also know HTCondor can back onto EC2 spot instances but I haven't really been able to find any concise information on how to set up a personal cloud — I also think this is slight overkill. Do any other Slashdot users have similar problems? How did you solve it? What is your workflow? Thanks!"

5 of 80 comments (clear)

  1. EC2 is scriptable by Anonymous Coward · · Score: 5, Informative

    EC2 is inherently scriptable. There's nothing stopping you from using the command-line tools to fire up an instance, and let it run, and store its results to S3, and then decommission the instance. You can even set the instances to terminate on shutdown, which deletes the instance EBS stores (if you're using EBS) and deletes the instance. Sounds like you just need to spend 30 minutes reading the docs.

  2. SC13 by jsimon12 · · Score: 1, Informative

    Bunch of papers at SC13 presented this year. Suggest sunny look them up.

    http://sc13.supercomputing.org/content/papers

  3. This is precisely what the AWS API is for. by gdek · · Score: 5, Informative

    Because your workflow is likely to be customized to your tasks, it should be straightforward to write these kinds of tools yourself, with any number of available toolkits, based on what language you're most comfortable using.

    There's the straight CLI: http://aws.amazon.com/cli/

    And lots of sample code for the various SDKs: http://aws.amazon.com/code

    Best to just dive in. If you have any development experience at all, even just scripting, you should be able to figure it out pretty quickly.

  4. One AMI image, ec2, and shell scripts by Lally+Singh · · Score: 3, Informative

    Here's how I ran my PhD simulations on EC2:
    - The AMI downloads a manifest file at startup.
        - The manifest has one record per line, two fields per record: the s3 URL of a .tar.gz to download, and the path to download it
    - The AMI then runs a shell script (/etc/run.sh) that's been put there by a manifest entry

    Shell scripts upload new files to s3 (e.g., /etc/run.sh) and have ec2 run new VMs. When the VMs are loaded, they're running everything I need, ready to go.
    Other shell scripts stopped/started experiments on these VMs.
    Other shell scripts shut down the VMs when I'm done.
    The scripts did little more than scan the appropriate machine list from the ec2 tools and ssh into them with a specific command.

    At the end, I had some of the experiment-specific scripts quickly have git clone/pull in files I was changing quickly per experiment.

    All of it worked really well for me. Nothing fancier than the ec2 command-line tools, bash, ssh, & git necessary.

    --
    Care about electronic freedom? Consider donating to the EFF!
  5. Starcluster does 80-90% of what you want by austingeekgirl · · Score: 3, Informative

    http://star.mit.edu/cluster/

    The rest of it is easily scriptable. I have some ebs based AMIs that on bootup, connects to a central server,
    registers itself (ticks up a text file, and adds itself to /etc/hosts).

    If you combine starcluster for generic cluster management with the existing Amazon provided tools
    http://blog.roozbehk.com/post/35277172460/installing-amazon-ec2-tools)
    this is really only a days worth of scripting and testing.

    There are also several public AMIs on Ec2 that are oriented towards scientific computing.
    http://www.google.com/search?q=ec2%20ami%20scientific

    This is my day job stuff.