Slashdot Mirror
Ask Slashdot: Do Any Development Shops Build-Test-Deploy On A Cloud Service?
bellwould (11363) writes "Our CTO has asked us to move our entire dev/test platform off of shared, off-site, hardware onto Amazon, Savvis or the like. Because we don't know enough about this, we're nervous about the costs like CPU: Jenkins tasks checks-out 1M lines of source, then builds, tests and test-deploys 23 product modules 24/7; as well, several Glassfish and Tomcat instances run integration and UI tests 24/7. Disk: large databases instances packed with test and simulation data. Of course, it's all backed up too. So before we start an in-depth review of what's available, what experiences are dev shops having doing stuff like this in the cloud?"
Atlassian is already trying to push their customers in this direction. Their Bamboo OnDemand offering spins up AWS instance as needed for builds. In this case, you could still host a local Bamboo instance and use elastic remote agents.
One thing I do like about this sort of setup is that it keeps you honest about deployment. Your build environment stands up a new instance every time your remote agent goes stale and is recovered to reduce costs.
I'm IT for a company that does this for 95% of dev/test/qa systems. It's worked out pretty well. Most servers are spun up and then chef'ed, used, then deleted after tests/whetever are complete. We do keep our code in house. SVN/GIT/ and Jenkins along with server build farms are all in house. The cloud services are expensive, but since IT has automated the deployment process for the cloud hosts, it works out better than keeping enough hardware in house to meed all test/qa needs. Plus less hardware in house equals less admin time which is a plus for us.
If the stuff (data, processes, etc.) you put in the cloud are in any way sensitive, I would be very hesitant to put that in the hands of another company because of privacy and security. Particularly depending on your terms of service agreements with your users. I would avoid putting your source control system in the cloud too because then it's more accessible by nefarious actors than if it's locked down internally. This is of course assuming you have good security standards and practices in place.
We're not at your scale, but we do everything with AWS and have found that it works well.
One thing you might want to do is reexamine your mentality around 24/7...you need to evaluate what really does need to run 24/7 and what needs to be available 24/7 (i.e. something that can tolerate the time it takes to spin up from an AMI).
For example, your Jenkins server could be configured with a master/slave arrangement that allows the main Jenkins server to be a small or medium instance that runs 24/7 and then when a build needs to happen, spin up a beefier slave to rip through it as fast as possible and then shutdown when done. Each build then has a fixed cost, regardless of whether it runs serially or in parallel.
Our main reasons for choosing to use the cloud were:
- We have remote workers, both permanent and a WFH policy...cloud makes it not matter where you're working from.
- Less maintenance...stuff mostly just works and most things are scripted rather than configured.
- We like the mentality of thinking of computing as a resource, not a collection of discreet machines. Running 5 builds in parallel is expensive when you think machines but costs the same as 5 serial builds when you follow the spin up, build, spin down philosophy.
Amazon has a detailed AWS cost estimator:
http://calculator.s3.amazonaws...
When we migrated to the cloud, our actual costs were within 15% of the estimated costs.
But really, the easiest thing to do is just build a test environment and try it -- you only pay for the time you use.
When we migrated to AWS we knocked 70% off our colocation bill (we had more space at the coloc than we needed, but it's hard to move production hardware to a smaller space without downtime, plus we had significant savings in equipment leases and maintenance contract costs).
Our dev/test hardware was aging and becoming unreliable (and no longer matched production since we moved to AWS), so we moved that up to AWS as well, but even after that migration our total AWS bill less than half what we paid at the colocation center. We only run the dev/test hardware during business hours, or on-demand as needed -- we set up a simple web interface that lets developers spin up test instances as needed. AWS keeps dropping prices, so we're even as we've grown, our costs have remained relatively constant.
Amazon charges for instances by the hours they're running and the type of instance. Think of an instance as a server, because that's what it is: an instance of a VM. You can find the prices for various services at http://aws.amazon.com/pricing/. What you want are EC2 pricing (for the VM instances) and EBS pricing (for the block storage for your disk volumes. For EC2 pricing figure out what size instances you need, then assume they'll be running 720 hours a month (30 days at 24 hours/day) and calculate the monthly cost. For EBS pricing take the number of gigabytes for each disk volume (each EC2 instance will need at least one volume for it's root filesystem) and multiply by the price (in dollars per gigabyte per month) to get your cost. You can manage instances the same way you would any other machine, other than usually needing to use SSH to get access and having to worry about firewalling (these are publicly-accessible machines, you can't shortcut on security by having them accessible only from within your own network).
The cost isn't actually too bad. For generic Linux, the largest general-purpose instance will, for a reserved instance on a 1-year commitment, cost you $987 up front and $59.04/month for runtime in the US West (Oregon) data center. An 8GB regular EBS volume will cost you $0.40/month for the space and $50/month for 1 billion IO requests. And not all instances need to be running all the time. You can, for instance, use on-demand instances for your testing systems and only start them when you're actually doing release testing, you'll need to pay for the EBS storage for their root volumes but you won't have any IO operations or run-time while the instance is stopped.
The downside, of course: if Amazon has an outage, you have an outage and you won't be able to do anything about it. This isn't as uncommon an occurrence as the sales guys would like you to believe. Your management has to accept this and agree that you guys aren't responsible for Amazon's outages or the first time an outage takes everything down it's going to be a horrible disaster for you. Note that some of the impact can be mitigated by having your servers hosted in different regions, but there's a cost impact from transferring data between regions. Availability zones... theoretically they let you mitigate problems, but it seems every time I hear of an AWS outage it's one where either the failure itself took out all the availability zones in the region or the outage was caused by a failure in the availability-zone failover process. This all isn't as major as it sounds, outages and failures happen running your own systems after all and you've dealt with that. It's more a matter of keeping your management in touch with the reality that, despite what the salescritters want everyone to believe, there is no magic AWS pixie dust that makes outages and failures just vanish into thin air.
Why would you care about reliability for continuous integration?
We use Amazon EC2 with spot nodes for our CI. After all, if a node dies - you can just restart the whole process on a new one. Sure, you'll lose some time, but given that 32CPU node with 64Gb of RAM can be bought for $0.3 per hour - we simply don't care.
I think you're just failing to on-board the new cloud paradigm going forward.
You probably haven't accounted for the synergized trending advantages.
AWS has some of the lowest cloud prices I've found anywhere. You can get AWS instances for under $3/month reserved according to what you need. 'Small' Linux instances cost about $15/month reserved last I checked. In fact they'll even give you a Micro instance free for a year without spending anything as part of their 'free tier'.
How did you come to the conclusion AWS was expensive?
When working for companies, everything was "in the cloud" already: on remote servers. It's not like I was running the stuff on my desktop.
SSH to Amazon or SSH to a box in the closet. Pretty much no difference to me.
I do not fail; I succeed at finding out what does not work.
I feel like I just read a week's worth of posts from LinkedIn connections.
Yes, we use Visual Studio 365 Azure Edition for our C++ projects. Our compile times are a little longer, but we're riding the latest wave of post-Enterprise active data web cloud assured technology.
This gives us all the advantages of future web technology developments as they happen with Microsoft's world-leading Software Engineering/Code ARTezan(R)(TM) Cratfperson paradigm.
As a bonus, all of our best-shored development consultants were able to migrate their legacy Visual Source Safe projects seamlessly using cloud-aware IE plugins.
Stick Men
EC2 likely too expensive.. [...] If it's all Java / JVM, then look at the Cloudbees offering
You do realize that Cloudbees runs in EC2, right?
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock