Slashdot Mirror
Ask Slashdot: Do Any Development Shops Build-Test-Deploy On A Cloud Service?
bellwould (11363) writes "Our CTO has asked us to move our entire dev/test platform off of shared, off-site, hardware onto Amazon, Savvis or the like. Because we don't know enough about this, we're nervous about the costs like CPU: Jenkins tasks checks-out 1M lines of source, then builds, tests and test-deploys 23 product modules 24/7; as well, several Glassfish and Tomcat instances run integration and UI tests 24/7. Disk: large databases instances packed with test and simulation data. Of course, it's all backed up too. So before we start an in-depth review of what's available, what experiences are dev shops having doing stuff like this in the cloud?"
Atlassian is already trying to push their customers in this direction. Their Bamboo OnDemand offering spins up AWS instance as needed for builds. In this case, you could still host a local Bamboo instance and use elastic remote agents.
One thing I do like about this sort of setup is that it keeps you honest about deployment. Your build environment stands up a new instance every time your remote agent goes stale and is recovered to reduce costs.
It is 50%+ cheaper if you use in-house hardware. This assumes that you are a trained system administrator and you purchase energy and cost efficient hardware. Also, your data will be yours and not Amazons.
If it's all Java / JVM, then look at the Cloudbees offering, or the Waratek JVM (high-density) on something cheaper than EC2. Unless you have a decent grasp of when your environment can be shut down, EC2 is almost certain to be a waste of money, especially for dev / test.
I'm IT for a company that does this for 95% of dev/test/qa systems. It's worked out pretty well. Most servers are spun up and then chef'ed, used, then deleted after tests/whetever are complete. We do keep our code in house. SVN/GIT/ and Jenkins along with server build farms are all in house. The cloud services are expensive, but since IT has automated the deployment process for the cloud hosts, it works out better than keeping enough hardware in house to meed all test/qa needs. Plus less hardware in house equals less admin time which is a plus for us.
If the stuff (data, processes, etc.) you put in the cloud are in any way sensitive, I would be very hesitant to put that in the hands of another company because of privacy and security. Particularly depending on your terms of service agreements with your users. I would avoid putting your source control system in the cloud too because then it's more accessible by nefarious actors than if it's locked down internally. This is of course assuming you have good security standards and practices in place.
We're not at your scale, but we do everything with AWS and have found that it works well.
One thing you might want to do is reexamine your mentality around 24/7...you need to evaluate what really does need to run 24/7 and what needs to be available 24/7 (i.e. something that can tolerate the time it takes to spin up from an AMI).
For example, your Jenkins server could be configured with a master/slave arrangement that allows the main Jenkins server to be a small or medium instance that runs 24/7 and then when a build needs to happen, spin up a beefier slave to rip through it as fast as possible and then shutdown when done. Each build then has a fixed cost, regardless of whether it runs serially or in parallel.
Our main reasons for choosing to use the cloud were:
- We have remote workers, both permanent and a WFH policy...cloud makes it not matter where you're working from.
- Less maintenance...stuff mostly just works and most things are scripted rather than configured.
- We like the mentality of thinking of computing as a resource, not a collection of discreet machines. Running 5 builds in parallel is expensive when you think machines but costs the same as 5 serial builds when you follow the spin up, build, spin down philosophy.
He doesn't want to manage stuff in house because it's hard. But wait, that's his job, and why he draws C-level pay. If you are not just occasionally using it, the whole advantage of "cloud" goes away, unless you replace it with the concept of "outsource". Which might be his goal all along, either way, I would look for a new job. Cloud would be great if you needed to load test from 1000 machines or something, but even for that there are simulators.
I want to delete my account but Slashdot doesn't allow it.
Amazon has a detailed AWS cost estimator:
http://calculator.s3.amazonaws...
When we migrated to the cloud, our actual costs were within 15% of the estimated costs.
But really, the easiest thing to do is just build a test environment and try it -- you only pay for the time you use.
When we migrated to AWS we knocked 70% off our colocation bill (we had more space at the coloc than we needed, but it's hard to move production hardware to a smaller space without downtime, plus we had significant savings in equipment leases and maintenance contract costs).
Our dev/test hardware was aging and becoming unreliable (and no longer matched production since we moved to AWS), so we moved that up to AWS as well, but even after that migration our total AWS bill less than half what we paid at the colocation center. We only run the dev/test hardware during business hours, or on-demand as needed -- we set up a simple web interface that lets developers spin up test instances as needed. AWS keeps dropping prices, so we're even as we've grown, our costs have remained relatively constant.
Amazon charges for instances by the hours they're running and the type of instance. Think of an instance as a server, because that's what it is: an instance of a VM. You can find the prices for various services at http://aws.amazon.com/pricing/. What you want are EC2 pricing (for the VM instances) and EBS pricing (for the block storage for your disk volumes. For EC2 pricing figure out what size instances you need, then assume they'll be running 720 hours a month (30 days at 24 hours/day) and calculate the monthly cost. For EBS pricing take the number of gigabytes for each disk volume (each EC2 instance will need at least one volume for it's root filesystem) and multiply by the price (in dollars per gigabyte per month) to get your cost. You can manage instances the same way you would any other machine, other than usually needing to use SSH to get access and having to worry about firewalling (these are publicly-accessible machines, you can't shortcut on security by having them accessible only from within your own network).
The cost isn't actually too bad. For generic Linux, the largest general-purpose instance will, for a reserved instance on a 1-year commitment, cost you $987 up front and $59.04/month for runtime in the US West (Oregon) data center. An 8GB regular EBS volume will cost you $0.40/month for the space and $50/month for 1 billion IO requests. And not all instances need to be running all the time. You can, for instance, use on-demand instances for your testing systems and only start them when you're actually doing release testing, you'll need to pay for the EBS storage for their root volumes but you won't have any IO operations or run-time while the instance is stopped.
The downside, of course: if Amazon has an outage, you have an outage and you won't be able to do anything about it. This isn't as uncommon an occurrence as the sales guys would like you to believe. Your management has to accept this and agree that you guys aren't responsible for Amazon's outages or the first time an outage takes everything down it's going to be a horrible disaster for you. Note that some of the impact can be mitigated by having your servers hosted in different regions, but there's a cost impact from transferring data between regions. Availability zones... theoretically they let you mitigate problems, but it seems every time I hear of an AWS outage it's one where either the failure itself took out all the availability zones in the region or the outage was caused by a failure in the availability-zone failover process. This all isn't as major as it sounds, outages and failures happen running your own systems after all and you've dealt with that. It's more a matter of keeping your management in touch with the reality that, despite what the salescritters want everyone to believe, there is no magic AWS pixie dust that makes outages and failures just vanish into thin air.
If you were asked to do something then fucking do it. Any sticker shock is the CTOs problem to explain.
In spirit, I agree with you.
In practice, I write two kinds of in-house app on a regular basis - Integrations, and reports (yeah, I know, reporting doesn't generally count as an app, but compiling the data that goes into them behind the scenes very much does). How exactly do you move an integration that hits two (or more!) local servers, which may well contain sensitive (even in the PCI/PII sense) data, to "the cloud"? And for reporting, the task sounds simple until you ask "reporting on what?" Unless you want response times measured in hours or days, that would includes a hidden requirement that I also build a complete mirror of my local data warehouse in the cloud. Even ignoring the cost of that much storage, bandwidth rapidly becomes an issue when talking about trying to keep hundreds of gigs of data as close to realtime-fresh as possible.
So no, you can't always just "fucking do it". The FP likely understands that, and simply hopes to use price as an argument rather than patiently explain to his CTO why playing "Buzzword Bingo" doesn't make a viable long-term IT strategy.
I've spent over ten years on dedicated servers, and have been very happy. Over the next year, I'll be moving into a private cloud scenario -- (not amazon or google, yuck. A local datacentre rolling their own.) I'll have some dedicated hardware (physical servers: CPU, RAM), and be sharing the rest of the cloud (storage, power, network, et cetera.).
It's interesting because there are no actual benefits to me in terms of performance, capacity, stability, or price by moving -- even backups aren't any more fluid. Of course, my platform and business model have been well-tuned over the years, and my sub-industry doesn't have the fluctuations that are typically heralded by cloud services.
So why am I moving? Abstracted hardware. I've reached that point where migrating from one dedicated server to another is a major undertaking. It's days of work, weeks of testing, and a huge risk to my business if I were to move any significant number of clients at one time; that means spreading it out over a year which means paying for the old and the new at the same time with zero additional revenue.
I've got no problem with resource management and capacity planning. I just have trouble actually growing through the transition points. Moving to a private cloud is likely to give me the convenience of being able to upgrade physical servers instantly without any worries -- it's the virtualization layers and load balancing mostly.
Wish me luck.
AWS has some of the lowest cloud prices I've found anywhere. You can get AWS instances for under $3/month reserved according to what you need. 'Small' Linux instances cost about $15/month reserved last I checked. In fact they'll even give you a Micro instance free for a year without spending anything as part of their 'free tier'.
How did you come to the conclusion AWS was expensive?
LOL.
Maybe if you don't have a complex, co-dependent environment.
Otherwise, you're just going to get fired.
There are two types of people in the world: Those who crave closure
We did this at my last job.
In short, it sucked.
More descriptive: It really sucked!
The boss didn't want to manage servers in house to save costs. So as a developer, we had to show up every day, boot our cloud instance up, sync the latest code to that instance, and begin development. Then before going home, you needed check in your code, shut down the instance, and go home.
Doesn't sound so bad, except for the time you had to waste EVERY DAY logging into AWS, booting the EC2 instance, restoring the RDS instance, syncing the code, doing basic readiness tests BEFORE you could even begin working.
Then there was always the fun part of the dev team leader changing out which RDS instance you needed to use, and forgetting to tell you about it.
Not to mention the time you (the developer) had to waste every day shutting down (syncing / checking in code, creating snapshots, closing RDS instances down, etc, etc, etc).
Then there is always the fun times when the cloud was down (yes, it DOES happen people!) or the internet connection was down (ISP issue, internal LAN issues, etc)... All of the time wasted managing the cloud instances (starting up, shutting down each day) could of been spent actually fixing things, and writing code, but I guess that wasn't cost effective enough for them.
When working for companies, everything was "in the cloud" already: on remote servers. It's not like I was running the stuff on my desktop.
SSH to Amazon or SSH to a box in the closet. Pretty much no difference to me.
I do not fail; I succeed at finding out what does not work.
Data center operations are expensive when you factor in power, gear and staff. But I don't think cloud solves those problems particularly well and it actually adds some more. Cloud data sits on someone else's secondary storage, and if you don't understand the implications of this you are not thinking hard enough. I think the decision to use cloud varies on a case-by-case basis and that you just have to measure it for yourself. It might make sense for development, but maybe you don't want your code on someone else's systems, for whatever reason. I like to write code on my cubieboard, an SoC platform that runs on 5 volt - runs Debian, can mount a laptop hard drive on it, has a dual core proc on it. Runs great for its purpose. If I try to do the same thing on a cloud system (and I have), the cost rises dramatically. But you can't run a high traffic web site off of a cubieboard. There's a line where cloud begins to make more sense. Depends on what you're looking for. But it won't replace the data center.
didn't I just read somewhere about Google doing something with this enterprise cloud thing?
the answer to the question is "it depends" - my gut says "no" but as others have pointed out, if you want to know if something will be a cost effective solution, you need to test
the game changing benefit of the "cloud" is the ability to scale up/down as needed ... SO from a financial viewpoint the question is similar to "Should we buy a building or rent office space?"
BTW my headline is from a Dilbert gobbledygook generator - which I'm 90% sure that 100% of CTO's use an undefined % of the time
It ain't what they call you. It's what you answer to. http://mylyceum.us/
Snapshots, installs, and so on are not so much random access limited as sustained sequential throughput; more streaming than random... SSD's tend to saturate SATA ports so you end up with tricks like raid to get more speed.
*shrug* So... there are different kinds of "high IO". Depends on what your app needs from a storage point of view.
The take home message is that if storage performance matters to your app(s) be sure you understand what kind of storage subsystem options your cloud dealer can give you.
For your research pleasure... (link has performance #'s for an interesting range of devices, worth a look if you're doing data intensive things).
Um, LOTS of stuff requires high IO.
Think of a qa VM. It has to do snapshots, installs, reverts. All of which are high IO. Especially if the build is a large install.
Yes, we use Visual Studio 365 Azure Edition for our C++ projects. Our compile times are a little longer, but we're riding the latest wave of post-Enterprise active data web cloud assured technology.
This gives us all the advantages of future web technology developments as they happen with Microsoft's world-leading Software Engineering/Code ARTezan(R)(TM) Cratfperson paradigm.
As a bonus, all of our best-shored development consultants were able to migrate their legacy Visual Source Safe projects seamlessly using cloud-aware IE plugins.
Stick Men