Slashdot Mirror
Ask Slashdot: How Often Do You Push To Production?
First time accepted submitter Stiletto writes "I work for a traditional 'old school' software company that is trying to move into web services, now competing with smaller, nimbler 'Web 2.0' companies. Unfortunately our release process is still stuck in the '90s. Paperwork and forms, sign-off meetings, and documentation approvals make it impossible to do even minor deployments to production faster than once a month. Major releases go out a couple of times a year. I've heard from colleagues in Bay Area companies who release weekly or daily (or even multiple times a day), allowing them to adapt quickly. Slashdotters, how often do you push software changes into production, and what best practices allow you to maintain that deployment rate without chaos?"
I push to production fairly often. Our company's Visual Studios have been configured with test cases that make testing easy, and therefore we can push to production much more often than those who do not use Visual Studio.
Why not talk to your colleagues and suggest ways of speeding things up?
It's good to talk.
Release when ready is quite a good frequency :-)
>> Slashdotters, how often do you push software changes into production,
3 times a month, generally.
>> and what best practices allow you to maintain that deployment rate without chaos?"
IMHO, you can't. Having a deadline every week, programmers cut corners to make dates. QA cuts corners to make the date. This much code changing without any "bake time" inevitably leads to an unstable code base, full of "corner cuts" from the last N releases.
Most companies I've worked for have a continuous deployment cycle. All changes, from small bug fixes to major releases go through a ticketing system. After the ticket has gone through all the steps (code review, QA, UAT) it goes into the deployment manager's queue, who then deploys the change to production depending on each ticket's priority. This means that in general, changes go out as soon as they are ready, sometimes up to two times a day for the same project.
I think customers don't really NEED the frequent updates that you're getting used to seeing with apps. I would think only the foolish really expect rapid releases. That's been a side-effect of "apps" - not what we expect of software in the business world. If you set and can stick to a release schedule, I think your clients will appreciate it.
End users find updating all the time a headache - only super-geeks like us like seeing every possible iteration of an application. The average user doesn't want to be bothered with YET ANOTHER UPDATE (ahem... Flash and Java). But they do appreciate when you have their security interests at heart (like an out-of-cycle Windows Update).
I would say every other month for 'service releases' - every six months for major releases. The biggest exception being "out of cycle" emergency releases when deemed necessary.
This really seems like it depends on WHAT your company does. Pushing to production a browser or a utility weekly or even daily is a lot easier and less disruptive than say a OS or a major software product thats used by hundreds of large companies. I would say the biggest obstacle is to get rid of the paper process, bring in git or something else for changes that everyone can view, and limit the signing offs to production, QA, or gold master (or whatever your company calls them) while keeping the dev stuff out of it so that the dev team can be more flexible. Also have a pretty good test lab/suite that will enable you to test stuff on the fly on multiple machines and configuration. Case in point Apple has a testing lab of EVERY machine they have made for the last 3 years in multiple configurations for their software. Granted they are a closed ecosystem so its easy for them to do that, but generally you could get a good roundup of machines to use that span from the insane configs to the "are you fing kidding me you are using that???" type stuff.
"Slashdot, where telling the truth is overrated but lying is insightful."
Or were you looking forward to having to explain why your database got cracked and leaked 2 million passwords or that tens of thousands of customer machines will have to be manually patched to repair the damage done by your last update?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Sounds like yours wants a process. You push as the business feels comfortable with it.
But think about this...
Lets say you are dealing with peoples financial statements. You want to make sure they are pretty good and accurate. If it is wrong people loose money and you get sued. Your audience dislikes sudden change. So you put process around that to insure that (test/qa/forms/etc).
Now lets say you are running a forum where it is the stats from some online game. It can be a little off. Your audience likes sudden new features. Stuff might not render correctly... But you can fix it tomorrow and the only gripe will be on a forum.
One business you have to be careful in creating 0 downtime and 0 mistakes. The other, a mistake here and there is tolerated and can be fixed at will.
Now I am not saying you can only create the first one where it takes a month or two for a push. I am saying those sorts of situations where caution is warranted. If you want a quicker process your process will need to be able to break things into small pieces and a bad ass testing system. Some situations they are contractually obligated to go at a particular rate. If you can not see the difference then I suggest you let your question simmer for a few years and get some exp doing it.
Slashdot posted this story earlier,
http://www.businessweek.com/articles/2012-10-04/facebook-the-making-of-1-billion-users
Interesting read on how Facebook, arguably the largest website challenge so far this century updates daily.
At $dayjob, we decide to release and our process takes about an hour. All the automated tests are run by Jenkins CI, and are run again during the release on every box being deployed to in order to ensure stability. We tend to deploy to User-Acceptance Testing boxes before full production boxes.
At the game company, we wrote a system that works like:
1) Tag release in git
2) Release is pushed to beta servers. Beta players get immediate updates.
3) Click button in Jenkins to run stable release
Completely automated, even down to restarting the servers in a staggered fashion to ensure that users always have a game to connect to (even if they have to disconnect in 20 minutes to receive the client update).
Automated testing, Continual Integration, and automated release processes (including cfengine3 and custom Perl scripts working with Git) come together to produce a painless release process. Since it's easy, we can do it whenever we want. As soon as it starts releasing bad code, we'll have to put process in place to ensure bad code does not reach our stable users.
The goal is to always be able to push to production. Have a continuous integration and test system that allows you to have confidence that you can always push the latest build to production. Automate as much of your testing as possible so your build and test process can produce something that a human test team actually has to work hard to break.
Also, you should pipeline your approval process. Always be in the process of approving a new build to go out while you're working on the next one. This will put a lot of pressure on this process to be faster and more efficient. Holding the people responsible for this process accountable for production breaks will put counter-pressure ensuring that they do not become more efficient at the expense of actually doing the job.
Need a Python, C++, Unix, Linux develop
Sometimes all of those meetings and paperwork serve a useful purpose when an application is critical. If a one-day build of instagram doesn't work, then the only consequence is that there are fewer grainy photographs of someone's cat. If a one-day build of a power distribution system doesn't work, then an entire city loses electricity.
-Glires
Push to production as soon as the (many) automated tests that you have pass. This means you should have comprehensive unit tests and tests that run in the browser, probably written in Selenium. You'll also want to script your release so that you can do it with the push of a button. Once the tests pass, and the mechanics of a release are trivial, there is little reason to hold up a release.
I worked for a top 500 website (East coast) for 7 years that did weekly releases. Since I left, they decided that wasn't fast enough and now release multiple times per week. I'm now self-employed on my own website and release within an hour of finishing development of a feature.
I started my development career writing firmware for laser printers. When you are shipping code on a physical product, the cost of bugs can be quite high. Especially when it leads to returns or recalls because customers are not satisfied. Our release cycles there were 6 months+. Quite appropriately, IMO.
On the web, the cost of bugs is much lower. In most cases it is the only cost of another release. Sometimes it could cost more because of downtime, but good automated test coverage mitigates that risk pretty well (especially if there is load testing involved). The worst case would be data-corruption, but I've never actually seen that in practice from a release, that has only been related to hardware failure or accidents in my experience.
I think the submitter IS talking to colleagues and looking for suggestions right here...
We do a major release about every 2 months, with maintenance stuff going every 2 weeks. There's pressure to speed this up. On the release side of things we're streamlining our documentation requirements and automating everything we can. We're also moving to a continuous delivery model for our regular test builds... if it's tagged, it goes to QA and an automagical report goes out with a list of resolved defects. This requires the developers to package things cleanly and consistently, but it also gives us an amazing amount of data to mine and I like to think that it helps to improve our overall quality.
This doesn't address requirements gathering or development methodologies, but I can't really speak to those.
As a customer I hate daily updates to my applications. Unless the application is in Alpha or Beta it is very disconcerting to see updates at that frequency. I only want to see an update that quick when something is really broken. If brokeness is a daily occurence then maybe you need to slow down.
Life is short. You seem like a talented person who knows what you want. If you current company does not modernize in the next 6 months then they are holding you back. The market is starved for really talented software engineers. Take the time to research your opportunities and then jump ship if the one you are on is still stuck in the 90's.
That's not true. If you have good automated regression, have reasonable coverage and do continuous integration you can cut the release as soon as you have a clean regression. It may take a few hours or a day to prep the release (sign-offs, paperwork, push, etc) but it can be done in parallel. There is always a balance. Prepping the release incurs costs in manpower and resources. Pushing daily may not be practical if cost/benefit is not there. For example if it takes 6 hours to prep the release then doing so for a wording change on the page that get 10 hits per day is not practical. on the other hand, if dev team cranks out dozen cool and exciting features daily it may be worth to staff release and QA automation teams to allow them to prep multiple releases per day.
I am a Release manager at Acquity Group and have worked for "Old School" software companies that have their eyes blazing at all the new web companies that release release release all day long.
Here is the abbreviated philosophic path to victory:
DevOps.
Your developers need to act like operations (knowing how the code is deployed and configuration settings, routes and the like) guys and your operations guys need to start coding (as in ruby for puppet and chef, automation and automation and automation)
This leads to...
Infrastructure as Code.
Hire a Release Engineer or convert a sysadmin to start automating builds. Now you start automating code deployments, you start automating infrastructure deployments so they are repeatable.
This leads to...
Test Automation.
Now you need to stop focusing on smoke testing and have test automation engineers write automated test code.
One more thing...
Automated Rollback
When things go nuts, with fast deploys you need fast rollback. Capistrano is a great tool for deploying this way, rollback is very easy.
Now you can ...
Continuous Delivery.
Great, you got this far, your builds, testing, deploys are automated!
For Developers:
Coding for Continuous Delivery is a different paradigm where unfinished code makes it to production. This means that in the production configuration, settings for the new code must be activated by switches to turn new feature sets on. You don't want that unfinished code mucking up your site, right?
People Processes:
Do you have CAB boards and ITIL processes? Great, make them faster and more as DEV/TEST/QA becomes automated and just focus on UAT/Prod environments. See this book: http://www.amazon.com/The-Visible-Ops-Handbook-Implementing/dp/0975568612/
I can also gloss over on waterfall/Agile and hybrid software models.
Finally, unless your culture wants to shift, it may be damned near impossible to change the culture unless management wants to. If it's doomed, it's doomed!
If he can't deploy to production in one step, he needs to fix that first.
I'm not talking about from dev box to production. I'm talking about the physical act of someone running a single command (or for the Winlazy, pressing a button) then walking away. All code checkouts from source control, database changes, app server code deployments, web server restarts—whatever—happen without user intervention.
He should also be able to roll back in one step.
For all the meetings, forms, etc., it sounds like there is A LOT of CYA in that company. In that case, it is cultural and can only be changed from the top. Until/unless the company becomes less risk adverse, there is no point in trying to become more "agile" (i.e., risk-accepting) except making your job easier. Build your tools/scripts/whatever to make it easier to do stuff.
tl;dr: If you want a more nimble company, switch jobs.
Yeah, right.
In my current company, in one year I have not seen any modification on the main product, if we don't count new launched product. (in fact no change on the real product, but change in the software used to manage the product are more frequent)
Oh, by the way, when I say launch of the production, it's really called a launch, with a launcher, and satellite on the top of it.
I invite you to read the GitHub blog post on how they deploy.
It depends on your business which metric is meaningful. E.g. for a global bank, quality is more important than time to market. Make sure that your business really gains something by playing release time against quality.
If parent bothered to post AC then they should have mentioned that they work at Facebook and have like 500,000 servers. See parent links for subtle indication.
Many a user of that site wishes they would STOP updating.
And don't want to do it myself.