Slashdot Mirror
Ask Slashdot: Do Any Development Shops Build-Test-Deploy On A Cloud Service?
bellwould (11363) writes "Our CTO has asked us to move our entire dev/test platform off of shared, off-site, hardware onto Amazon, Savvis or the like. Because we don't know enough about this, we're nervous about the costs like CPU: Jenkins tasks checks-out 1M lines of source, then builds, tests and test-deploys 23 product modules 24/7; as well, several Glassfish and Tomcat instances run integration and UI tests 24/7. Disk: large databases instances packed with test and simulation data. Of course, it's all backed up too. So before we start an in-depth review of what's available, what experiences are dev shops having doing stuff like this in the cloud?"
Atlassian is already trying to push their customers in this direction. Their Bamboo OnDemand offering spins up AWS instance as needed for builds. In this case, you could still host a local Bamboo instance and use elastic remote agents.
One thing I do like about this sort of setup is that it keeps you honest about deployment. Your build environment stands up a new instance every time your remote agent goes stale and is recovered to reduce costs.
It is 50%+ cheaper if you use in-house hardware. This assumes that you are a trained system administrator and you purchase energy and cost efficient hardware. Also, your data will be yours and not Amazons.
You'd better get some serious analytics done on your platform before moving it to a service that has open-ended billing.
If it's all Java / JVM, then look at the Cloudbees offering, or the Waratek JVM (high-density) on something cheaper than EC2. Unless you have a decent grasp of when your environment can be shut down, EC2 is almost certain to be a waste of money, especially for dev / test.
Were you asked to do something or were you asked if doing something is a good idea?
If you were asked to do something then fucking do it. Any sticker shock is the CTOs problem to explain.
If your asked by your CTO if moving to AWS is a good idea for organizations where money is an issue the answer is typically NO. You can drink all of "the cloud" Kool-Aid you want... you just have to pay for it.
I'm IT for a company that does this for 95% of dev/test/qa systems. It's worked out pretty well. Most servers are spun up and then chef'ed, used, then deleted after tests/whetever are complete. We do keep our code in house. SVN/GIT/ and Jenkins along with server build farms are all in house. The cloud services are expensive, but since IT has automated the deployment process for the cloud hosts, it works out better than keeping enough hardware in house to meed all test/qa needs. Plus less hardware in house equals less admin time which is a plus for us.
If the stuff (data, processes, etc.) you put in the cloud are in any way sensitive, I would be very hesitant to put that in the hands of another company because of privacy and security. Particularly depending on your terms of service agreements with your users. I would avoid putting your source control system in the cloud too because then it's more accessible by nefarious actors than if it's locked down internally. This is of course assuming you have good security standards and practices in place.
Forget trying to figure the cost, it isn't worth your time. Just move it and look at the bill. With Amazon and Google, there is no commitment, so try it. Take the bill to the other guys and see if you can beat it, if so move, if not stay. If everyone is too expensive, switch back to hosting it yourself. If cost is the most important issue, then the cheapest is building your own cloud with commodity hardware and OpenStack. You get the redundancy of the cloud with the cheap of whitebox hardware. Think of OpenStack as the "RAID of everything".
We're not at your scale, but we do everything with AWS and have found that it works well.
One thing you might want to do is reexamine your mentality around 24/7...you need to evaluate what really does need to run 24/7 and what needs to be available 24/7 (i.e. something that can tolerate the time it takes to spin up from an AMI).
For example, your Jenkins server could be configured with a master/slave arrangement that allows the main Jenkins server to be a small or medium instance that runs 24/7 and then when a build needs to happen, spin up a beefier slave to rip through it as fast as possible and then shutdown when done. Each build then has a fixed cost, regardless of whether it runs serially or in parallel.
Our main reasons for choosing to use the cloud were:
- We have remote workers, both permanent and a WFH policy...cloud makes it not matter where you're working from.
- Less maintenance...stuff mostly just works and most things are scripted rather than configured.
- We like the mentality of thinking of computing as a resource, not a collection of discreet machines. Running 5 builds in parallel is expensive when you think machines but costs the same as 5 serial builds when you follow the spin up, build, spin down philosophy.
It's pretty easy to go with something like Atlassian Cloud to handle all your build stuff. It will fire off that various EC2 instances when you need them. It handles the basic deploys. Fairly reasonably priced.
What's not easy is setting up a secure EC2 environment that your production code will run in. I'm not saying EC2 isn't secure, I'm just saying you need to wear a lot of hats to really set it up well. You need to know network, firewall, unix, chef (or similar suite), messaging, storage NAS, and app server.
My experience with EC2 was it's cheaper if you had the skills to manage the environment and deployment (i.e. Chef). However, it becomes a lot less cheap if you have to buy a service to help you manage the entire EC2 lifespan (i.e. if something like Chef isn't clicking for the team.)
Mozilla spends a fair chunk of change on the clown: http://armenzg.blogspot.com/2014/04/mozillas-recent-ci-improvements-saves.html
He doesn't want to manage stuff in house because it's hard. But wait, that's his job, and why he draws C-level pay. If you are not just occasionally using it, the whole advantage of "cloud" goes away, unless you replace it with the concept of "outsource". Which might be his goal all along, either way, I would look for a new job. Cloud would be great if you needed to load test from 1000 machines or something, but even for that there are simulators.
I want to delete my account but Slashdot doesn't allow it.
Amazon has a detailed AWS cost estimator:
http://calculator.s3.amazonaws...
When we migrated to the cloud, our actual costs were within 15% of the estimated costs.
But really, the easiest thing to do is just build a test environment and try it -- you only pay for the time you use.
When we migrated to AWS we knocked 70% off our colocation bill (we had more space at the coloc than we needed, but it's hard to move production hardware to a smaller space without downtime, plus we had significant savings in equipment leases and maintenance contract costs).
Our dev/test hardware was aging and becoming unreliable (and no longer matched production since we moved to AWS), so we moved that up to AWS as well, but even after that migration our total AWS bill less than half what we paid at the colocation center. We only run the dev/test hardware during business hours, or on-demand as needed -- we set up a simple web interface that lets developers spin up test instances as needed. AWS keeps dropping prices, so we're even as we've grown, our costs have remained relatively constant.
I was just explaining this to someone the other day that thought AWS was going to save them money. It's not cheaper than running your own shop. The only advantage I see is that you don't have to house/cool/maintain hardware. You can just move your application to higher capacity, faster servers. You get additional power and network reliability.
If your dev/test platform is already off-site and working, then what is the compelling reason to interrupt everything and do the move? Where I am working today, the tried to move from AWS to Google's cloud and had tons of issues of reliability. We're back on AWS. Our usage model though, lends itself to AWS. We sell "application instances" which are deployed for customers to AWS for its up-time reliability. All our development happens in-house though.
It's not a magic bullet. If you're looking to save money and your place already has a cooled closet and redundant network and power, then it offers no incentive for you to move.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
Amazon charges for instances by the hours they're running and the type of instance. Think of an instance as a server, because that's what it is: an instance of a VM. You can find the prices for various services at http://aws.amazon.com/pricing/. What you want are EC2 pricing (for the VM instances) and EBS pricing (for the block storage for your disk volumes. For EC2 pricing figure out what size instances you need, then assume they'll be running 720 hours a month (30 days at 24 hours/day) and calculate the monthly cost. For EBS pricing take the number of gigabytes for each disk volume (each EC2 instance will need at least one volume for it's root filesystem) and multiply by the price (in dollars per gigabyte per month) to get your cost. You can manage instances the same way you would any other machine, other than usually needing to use SSH to get access and having to worry about firewalling (these are publicly-accessible machines, you can't shortcut on security by having them accessible only from within your own network).
The cost isn't actually too bad. For generic Linux, the largest general-purpose instance will, for a reserved instance on a 1-year commitment, cost you $987 up front and $59.04/month for runtime in the US West (Oregon) data center. An 8GB regular EBS volume will cost you $0.40/month for the space and $50/month for 1 billion IO requests. And not all instances need to be running all the time. You can, for instance, use on-demand instances for your testing systems and only start them when you're actually doing release testing, you'll need to pay for the EBS storage for their root volumes but you won't have any IO operations or run-time while the instance is stopped.
The downside, of course: if Amazon has an outage, you have an outage and you won't be able to do anything about it. This isn't as uncommon an occurrence as the sales guys would like you to believe. Your management has to accept this and agree that you guys aren't responsible for Amazon's outages or the first time an outage takes everything down it's going to be a horrible disaster for you. Note that some of the impact can be mitigated by having your servers hosted in different regions, but there's a cost impact from transferring data between regions. Availability zones... theoretically they let you mitigate problems, but it seems every time I hear of an AWS outage it's one where either the failure itself took out all the availability zones in the region or the outage was caused by a failure in the availability-zone failover process. This all isn't as major as it sounds, outages and failures happen running your own systems after all and you've dealt with that. It's more a matter of keeping your management in touch with the reality that, despite what the salescritters want everyone to believe, there is no magic AWS pixie dust that makes outages and failures just vanish into thin air.
I've spent over ten years on dedicated servers, and have been very happy. Over the next year, I'll be moving into a private cloud scenario -- (not amazon or google, yuck. A local datacentre rolling their own.) I'll have some dedicated hardware (physical servers: CPU, RAM), and be sharing the rest of the cloud (storage, power, network, et cetera.).
It's interesting because there are no actual benefits to me in terms of performance, capacity, stability, or price by moving -- even backups aren't any more fluid. Of course, my platform and business model have been well-tuned over the years, and my sub-industry doesn't have the fluctuations that are typically heralded by cloud services.
So why am I moving? Abstracted hardware. I've reached that point where migrating from one dedicated server to another is a major undertaking. It's days of work, weeks of testing, and a huge risk to my business if I were to move any significant number of clients at one time; that means spreading it out over a year which means paying for the old and the new at the same time with zero additional revenue.
I've got no problem with resource management and capacity planning. I just have trouble actually growing through the transition points. Moving to a private cloud is likely to give me the convenience of being able to upgrade physical servers instantly without any worries -- it's the virtualization layers and load balancing mostly.
Wish me luck.
Zoom out and think about the economics. If the load you are shifting onto a cloud instance fills a machine or set of machines to their capacity -- i.e. little downtime between check out, build, validate, then I don't see the value proposition. There is a fixed cost to run a box flat out for a given number of years (hardware + care and feeding). I haven't checked in a while but the cost to rent resources in the cloud typically amounted to paying off an equivalent dedicate resource in something like three years. If you buy a resource and run it 5 years, it's cheaper to host it yourself. If the load you are moving to the cloud doesn't keep a system busy then renting may be a better option -- empty machines are expensive.
The OpenStack infrastructure team is running largest cloud-based continuous deployment environment I've ever seen, and they're more than happy to give people introductions to it.
This signature can save you $400 on your car insurance!
AWS has some of the lowest cloud prices I've found anywhere. You can get AWS instances for under $3/month reserved according to what you need. 'Small' Linux instances cost about $15/month reserved last I checked. In fact they'll even give you a Micro instance free for a year without spending anything as part of their 'free tier'.
How did you come to the conclusion AWS was expensive?
We moved our development and production systems to AWS in 2008 and have been quite happy. It has allowed us to grow and scale with load on production and quickly test things in development. There a few things to keep in-mind. First, if you know your usage pattern and can drop some money upfront then utilize the reserved instances to save some money. Second, you will need a script management system of some kind to run on the virtual servers at boot/shutdown, I recommend using something like Rightscale or Scalr. Most importantly, don't do it without some experienced help for the setup, a day of a consultant's or system admin's time to walk through your migration plan with you will save you a lot of help down the road. Lastly, aws has a cost estimator that is quite helpful.
Here is a recent video from Google on how they are doing it: Google Cloud Platform Live: DevOps at Google Speed & Tools for You
Have your CTO talk to Skytap. (I don't work for them, I've proposed this at my company) . This is one company, there must be others, who's main product is virtual dev. environments. I presume they run at Amazon. The whole environment can apparently be saved and immediately cloned. So it seems that they take server instances one step further, where there can be several of them in a private network and the whole thing can be versioned and cloned. (sorry for the anonymous, I'll get my login now)
What purpose would a cloud service serve in this case? The best points about cloud services are high availability and reliability. Clients (customer or internal) can access the servers easily from anywhere, and you don't need to be concerned about practicalities like hard disks breaking (but do have a backup plan). The costs can be high, though.
About the Jenkins example, does it need the redundancy or availability provided by a cloud service? A simple setup would be a dedicated machine running the Jenkins jobs in the company's local network. If you need access from outside, just connect to the VPN that you probably already have. When the machine breaks (or a hard disk), just replace it, install the OS from an image, checkout the Jenkins configuration and put it running again. With a well-defined process this should be less than an hour of downtime, or a few days if you need to get the replacement parts. I assume all the data created by the Jenkins job is easily replaced by just running the tests again. If this is not the case, the process gets more complicated.
What I said about Jenkins may apply to everything else, too. Keeping the services running with only the occasional downtime is easy, the hard part is saving the data if it is critical. If your data is easily replaceable, using a cloud service may be overkill.
Cloud is good for reliability, scalability, and if your particular scenario meets certain criteria, sometimes cost. Overall the cloud would be usually be more expensive, but can be cheaper to use cloud and only pay for what you need if you have short periods of high load combined with long periods of little load. Thus cloud might be cheaper because rather than paying for, cooling, powering, and maintaining alot of high end servers waiting to handle a large load only occasionally, you pay for what you need with the cloud. I would speculate you wouldn't see such savings on a single server.
If you could find a development/CI hosting platform that meets your needs, that would probably be a better bet. Such a service might be using cloud behind the scenes, but they benefit from the efficiencies of scale.
Another option is a semi-dedicate VPS. There are some VPS services that give you root access and terms would allow you to use the box as you do now.
I think someone went into this with the assumption that costs would go down using cloud, but your management time is going to be the same or more. You don't have a physical box, but now you will spend more time figuring out the nuances of your cloud host.
The key is to seperate the goal from the premature jump to cloud:
-If the goal is to get rid of the physical box, then VPS is an option.
-If the goal is to get rid of the physical box and lower management time/costs, then a hosted development/build/integration service is a good option. That way you don't even have to manage the OS, updates to the software/services, or perform backups.
Hi, I work at Netflix, you may have heard of us.
dev, test, build, and prod run on Amazon (leaving aside the actual streaming, which comes from cache boxes closer to the customer). We've been pretty public about the process, and some of the issues.
Yes.
But as someone else suggested, it sounds like you need a CTO upgrade to go along with your migration to the cloud.
She blinded me with science, she tricked me with technology. ~ Thomas Dolby
We did this at my last job.
In short, it sucked.
More descriptive: It really sucked!
The boss didn't want to manage servers in house to save costs. So as a developer, we had to show up every day, boot our cloud instance up, sync the latest code to that instance, and begin development. Then before going home, you needed check in your code, shut down the instance, and go home.
Doesn't sound so bad, except for the time you had to waste EVERY DAY logging into AWS, booting the EC2 instance, restoring the RDS instance, syncing the code, doing basic readiness tests BEFORE you could even begin working.
Then there was always the fun part of the dev team leader changing out which RDS instance you needed to use, and forgetting to tell you about it.
Not to mention the time you (the developer) had to waste every day shutting down (syncing / checking in code, creating snapshots, closing RDS instances down, etc, etc, etc).
Then there is always the fun times when the cloud was down (yes, it DOES happen people!) or the internet connection was down (ISP issue, internal LAN issues, etc)... All of the time wasted managing the cloud instances (starting up, shutting down each day) could of been spent actually fixing things, and writing code, but I guess that wasn't cost effective enough for them.
If you are nervous about costs and overhead time required, you can try out some small apps on a PaaS site such as Heroku https://www.heroku.com/ Then graduate to doing it on Amazon or whatever, once you figure out what you need or don't need, or don't want to be bothered with.
When working for companies, everything was "in the cloud" already: on remote servers. It's not like I was running the stuff on my desktop.
SSH to Amazon or SSH to a box in the closet. Pretty much no difference to me.
I do not fail; I succeed at finding out what does not work.
So, cloud hosting is expensive versus standard hosting, you pay a premium for scalability. Simple as that.
Data center operations are expensive when you factor in power, gear and staff. But I don't think cloud solves those problems particularly well and it actually adds some more. Cloud data sits on someone else's secondary storage, and if you don't understand the implications of this you are not thinking hard enough. I think the decision to use cloud varies on a case-by-case basis and that you just have to measure it for yourself. It might make sense for development, but maybe you don't want your code on someone else's systems, for whatever reason. I like to write code on my cubieboard, an SoC platform that runs on 5 volt - runs Debian, can mount a laptop hard drive on it, has a dual core proc on it. Runs great for its purpose. If I try to do the same thing on a cloud system (and I have), the cost rises dramatically. But you can't run a high traffic web site off of a cubieboard. There's a line where cloud begins to make more sense. Depends on what you're looking for. But it won't replace the data center.
didn't I just read somewhere about Google doing something with this enterprise cloud thing?
the answer to the question is "it depends" - my gut says "no" but as others have pointed out, if you want to know if something will be a cost effective solution, you need to test
the game changing benefit of the "cloud" is the ability to scale up/down as needed ... SO from a financial viewpoint the question is similar to "Should we buy a building or rent office space?"
BTW my headline is from a Dilbert gobbledygook generator - which I'm 90% sure that 100% of CTO's use an undefined % of the time
It ain't what they call you. It's what you answer to. http://mylyceum.us/
I would recommend doing both. There are significant advantages of moving some of the development/test tools out to the cloud. However, it should only go as far as development and perhaps first stage testing which is probably what your CTO has in mind.
There's no reason why each development project has to pay for the physical space taken up by *shared* development tooling such as Jenkins, Common Git Repository, JIRA/Redmine/Trac and some database and application server that is used for functional testing.
However, a proof of concept system must be present locally even if it is a limited capacity otherwise you'd be wasting a lot of bandwidth going back and forth.
It would also help to design your application architecture so that you can theoretically run everything on a laptop (and provide a powerful laptop to do it). For the developers.
If your'e an IBM shop, you may want to look at JazzHub to manage things for you.
Archie - CIO-for-hire
Snapshots, installs, and so on are not so much random access limited as sustained sequential throughput; more streaming than random... SSD's tend to saturate SATA ports so you end up with tricks like raid to get more speed.
*shrug* So... there are different kinds of "high IO". Depends on what your app needs from a storage point of view.
The take home message is that if storage performance matters to your app(s) be sure you understand what kind of storage subsystem options your cloud dealer can give you.
For your research pleasure... (link has performance #'s for an interesting range of devices, worth a look if you're doing data intensive things).
Um, LOTS of stuff requires high IO.
Think of a qa VM. It has to do snapshots, installs, reverts. All of which are high IO. Especially if the build is a large install.
I'm in the process of standing up a new cloudy little provider and we don't count hours or minutes. Is that so wrong?
The assumption is that the Internet is open 24/7 so why should we be marking time when we know you want it 24/7? We would rather cultivate the developers and geeks as customers. We'll soon have one portal for instant gratification but we're also happy to hand-craft VMs in a private place for you too. And it's built around CloudStack4 so it should feel familiar to many.
Come talk to us while we're young and dumb, before we figure out that our prices are too cheap. 20Gb/s out to the net, BGP/4, IBM BladeCenters for hardware, and we also rent bare metal. Anything from 1vCPU/512MB up to 336 CPUs and 10.5TB of RAM in bare metal if you're willing to pay for it.
http://www.blinkinblox.com/
Disclaimer: Yup, its a commercial plug. I'm pretty much both the King and the garbage man.
Do not do it, period.
Network connections are not reliable enough for that, cloud services can (and will) go offline and the worst part, you will be putting your code on a external server on which you have no control. Your secret, revolutionary code free to NSA industrial espionage or worst. NEVER put important/sensitive data on a cloud.
Religion: The greatest weapon of mass destruction of all time
We're doing so and I would never go back after the experience. We automatically spin up CI tests for each push of a branch, can bring up sites on demand, etc. We feel free to actually branch as desired, and we can spawn as many sites as we need for any manual testing as desired. The larger you are the more benefit you'll see, but if you put some time into the infrastructure surrounding it, you will be happy. Our purpose is to develop and enhance features, not mess around with if we need 10 or 1000 test machines today. I can create a site from any given branch or start tests on any given branch in less than a minute with the extra infrastructure we've built. Once you realize how many internal headaches can be cut away, any cost for the cloud becomes trivial in comparison, making it an easy decision.
Because the CTO can spin this to the CEO as being technically better somehow while getting kickbacks from the Amazon sales rep ;-)
"When information is power, privacy is freedom" - Jah-Wren Ryel
Yes, we use Visual Studio 365 Azure Edition for our C++ projects. Our compile times are a little longer, but we're riding the latest wave of post-Enterprise active data web cloud assured technology.
This gives us all the advantages of future web technology developments as they happen with Microsoft's world-leading Software Engineering/Code ARTezan(R)(TM) Cratfperson paradigm.
As a bonus, all of our best-shored development consultants were able to migrate their legacy Visual Source Safe projects seamlessly using cloud-aware IE plugins.
Stick Men
Define "cloud" - If it means data that passes through the wavy outlined 'cloud' of uncertainty in the network diagram, no. If you mean a distributed computing solution, then yes.
In house "cloud" is the same as the remote "cloud", except that the uncertainty is removed and thus the iteration speed for the system's construction and refinement is faster. I do several contracts a year building such systems for CTOs who are somewhat clueless, but fortunately have an ace-in-the hole that always makes them look good: Me.
I am the cloud, an unknown, invisible, buzzword-compliant deployment AND documentation guru so who seeks neither fame, nor job security (hence the impeccable docs). I wander the skies among your ivory towers and for a moment obscure your vision, of the highest floors, leaving apparent heroes who have "saved the day" in my vaporous trial.
Once your continuous build, test, deployment systems are migrated to the core system, they can be quickly distributed in house, and the kinks worked out much faster than tackling the a more grand solution, THEN, I can easily knit the structures together across the globe, even as deployment systems to your clients themselves at an unbeatable price. I can graze wild blue yonder and because I'm not greedy, I don't crave acknowledgement. I don't mind holding the wires for your crack team of superheroes flying high in the sky beneath me as I drift silently along.
Only realize the truth: There is no cloud.
need a proof of concept first? though the workflow is far more simple then the old skool you are asking for.
https://www.openshift.com/
missing functionality? or want the oldskool? add it as you wish.
http://openshift.github.io/documentation/oo_cartridge_developers_guide.html
Step 1: Put everything on the cloud... Step 2: Fire American programmer... Step 3: Hire sweat-shop workers
Amazon is not cheap at all for your task and I am not sure you are looking into the right direction. In the last year I was doing feasibility studies for software in the cloud and I implemented already a system that relies on Amazon 100%. I have evaluated various technical solutions and various providers. My findings are the following: 1. Amazon is second to nothing when it comes to be elastic. If you want to scale from 10 to 100 (you name what you count: CPU/Memory/Storage/Band/DB...) you will have it in a minute. They can scale you inside the box and outside the box and this works perfect. Not the case with other cloud providers. 2. You get the goodies from 1 on a very hefty premium. Going on Amazon is more than twice more expensive than renting an enterprise class server in a datacenter. If you are not so fussy about the brand of the server this can be even 4 times more expensive. Of course you are on your own on the rented servers. All the maintenance is yours. For the kind of setup you describe this can cost you some money with the labor and it might cut the margins. 3. If you choose to use rented hardware the best is to use them as virtual machine farms. It might look awkward to have a machine that runs as visualization server only to one single virtual machine but this gives you that huge option to move the virtual machines as you see fit. It looks to me like the elasticity is not your most important goal so it might happen you will be better on rented hardware. And again you must know that, after all that study, in my case I went for Amazon. Still, I need that elasticity...
They are very new but I've worked with the founder, Jay Moorthi and have nothing but good experiences with him. Maybe I'll ping him to come and give an overview of how they help. https://www.solanolabs.com/
This is fucking awesome!!! Just kidding.
The point of cloud services is that you provision things as you need them. That means that everyone can test at the same time - with no conflicts - because everyone has whatever test instances they need.
But don't do development in the cloud: do it on your laptop. Every developer needs to have the whole application stack on their laptop/desktop for development and for running the integration (using mocks) tests locally. Then if your code changes work, check them in (to a cloud-based repository). When the code is checked in, Jenkins can build it and provision a cloud based testing environment on the fly and run all the integration tests with real (non-mocked) resources.
That way no one every has to worry that their tests interfere with someone else and you don't have to maintain servers to test against.
The promise of OpenStack is that it can abstract the infrastructure layer so companies get control over costs and the hardware used. (source: http://www.zdnet.com/red-hat-ceo-whitehurst-on-innovation-openstack-hadoop-7000015292/ - year ago)
I've seen advantages and disadvantages in both scenarios. It depends on the application and the profile of your production systems. As a rule of thumb, your test/dev systems should be as close as possible to the production machines; If you're deploying to cloud services, you should have your test and staging system running on the same platform/provider; If you're deploying to bare metal, you should have dedicated servers for testing and staging. The applications don't work by themselves, and eg. controller/driver problems in production systems are quite difficult to diagnose if you can't replicate the problem in staging; So it doesn't really make sense to test something on a cloud provider (with "emulated" hardware) if you may be needing to be on the lookout for specific problems on your own hardware. There is also the question of I/O - both local and network. Dedicated gear will always be faster than virtualized solutions, or at least cheaper for the same amount of IOPs. If you have an application that requires heavy I/O, usually cloud services are almost as expensive as they are useless. As an example, you can rent a couple of servers in Hetzner that will run *laps* around most EC2 instances, for a fraction of the price. Since usually test data is ephemeral, server reliability isn't really an issue - a competent Sysadmin team will provision a cluster of those by a fraction of the price of a cloud service. So, in short, if using cloud in production, its the wise move, if not its probably not what you're looking for.
Just how much of your product or service is yours once you've flung it into "the cloud" (modern marketing term for "somebody else's systems that YOU do not control, YOU cannot secure, and YOU must pay extra for").
Best advice: keep your resume up to date. The clown you serve probably has upper management convinced he's a genius and he's getting his raises and bonuses and "golden parachute"..... and then the underlings will get the blame when things eventually go very wrong and/or get very expensive. Even if they never form a circular firing squad, it's probably true that eventually (as a cost reduction measure) the in-house IT budget will go down (both to pay for the "cloud" services and also because after everything's "on the cloud" any in-house IT will look (to pea-brained executives and managers) like a redundant expense.
It's very generous of your CTO to give the IT staff of another company the ability to see exactly what you're doing in case they're curious. I wouldn't worry about it though. The chances that any of them are cousins or brothers of employees of your closest competitors are very slim.
There is also VMware's DaaS, which is the former Desktone to take a look at for an alternative.
We tried amazon and hated it. It was too expensive and too slow and individual servers
weren't as reliable and predictable as we expected.
I think it probably works ok for someone like netflix distributing it across hundreds of servers
but I wouldn't recommend it for someone with less than a couple dozen servers.
We switched to stormondemand (aka liquidweb) and have been much much happier.
They have solid state drives which help with i/o which is one of amazon's weaknesses.
They also have dedicated cloud servers where you can pick the specs and know what
you're getting. Oh, and you can actually call and talk to someone if you need to.
Given the way so much stuff - including internal to companies I've worked for, no way. Links work... if you're on the internal network, not outside. Software runs SO FAST... until you're not on the intranet, and then it's a dog.
And developers always get and test on the hottest machines or servers... never mind the 95% of the folks going to that site, or using that software, are 1-2 generations of hardware back, and again, it runs like a dog, or requires you to buy new hardware.
So if you did it on superdooperpowerfast cloud hardware, you'd still not see how *users* see it.
mark
Currently we are running CI builds through Travis CI and push to multiple AWS instances. We have fail over RDS setup as well. The costs have come down on AWS time and are not far out of line.
"Voices In My Head" The Unauthorized Biography