Slashdot Mirror
Ask Slashdot: How Often Do You Push To Production?
First time accepted submitter Stiletto writes "I work for a traditional 'old school' software company that is trying to move into web services, now competing with smaller, nimbler 'Web 2.0' companies. Unfortunately our release process is still stuck in the '90s. Paperwork and forms, sign-off meetings, and documentation approvals make it impossible to do even minor deployments to production faster than once a month. Major releases go out a couple of times a year. I've heard from colleagues in Bay Area companies who release weekly or daily (or even multiple times a day), allowing them to adapt quickly. Slashdotters, how often do you push software changes into production, and what best practices allow you to maintain that deployment rate without chaos?"
Most companies I've worked for have a continuous deployment cycle. All changes, from small bug fixes to major releases go through a ticketing system. After the ticket has gone through all the steps (code review, QA, UAT) it goes into the deployment manager's queue, who then deploys the change to production depending on each ticket's priority. This means that in general, changes go out as soon as they are ready, sometimes up to two times a day for the same project.
Slashdot posted this story earlier,
http://www.businessweek.com/articles/2012-10-04/facebook-the-making-of-1-billion-users
Interesting read on how Facebook, arguably the largest website challenge so far this century updates daily.
At $dayjob, we decide to release and our process takes about an hour. All the automated tests are run by Jenkins CI, and are run again during the release on every box being deployed to in order to ensure stability. We tend to deploy to User-Acceptance Testing boxes before full production boxes.
At the game company, we wrote a system that works like:
1) Tag release in git
2) Release is pushed to beta servers. Beta players get immediate updates.
3) Click button in Jenkins to run stable release
Completely automated, even down to restarting the servers in a staggered fashion to ensure that users always have a game to connect to (even if they have to disconnect in 20 minutes to receive the client update).
Automated testing, Continual Integration, and automated release processes (including cfengine3 and custom Perl scripts working with Git) come together to produce a painless release process. Since it's easy, we can do it whenever we want. As soon as it starts releasing bad code, we'll have to put process in place to ensure bad code does not reach our stable users.
I invite you to read the GitHub blog post on how they deploy.
Absolutely
One-step prep/audit/deploy/post-deploy audit/rollback/post rollback audit are critical and not dependent on your tool stack, pretty much any stack can be made to perform this function with proper attitudes/process design/automation. (also needs some break-point analysis ~ things like DB schema changes complicate roll-backs if you don't want to lose production data - some low latency design shops leave error checking out of certain high traffic code areas on purpose and they'll break on a rollback. This is also true of internal product dependency with multiple-roll outs that overlap, I'm sure there's many examples).
Where/what you check is situation dependent ~ ie name/value config pairs, binary checksum's, target env state/settings, do you have enough drive space, etc ~ depends on your particular organizational weaknesses.
I'm a big fan of buttons, downstream users tend to regress as you make their environments easier until button clicking is your final stop. Besides, humans make typo's, just accept it, buttons rock.
Meetings/forms - I'd recommend pushing most of this to electronic data entry that is prefilled based on use case ~ saves a ton of time/work, gives you clearly defined queues and gives you CYA all at the same time. I'm a big fan of really limiting facetime meetings that don't involve some type of problem related brainstorming on a whiteboard. The rest are typically low bandwidth information exchanges that are best communicated through data systems.
Nimble company - You can have nimble areas in larger companies, the data-driven ins/outs are critical in that, you'll never escape CYA in a large corporate environment, just build it into your framework and make sure to publicly display your KPI's, people will go after easier targets. You need a champion though, its lonely on the bottom.
elephants/peanuts ~ we usually talk about monkeys/banana's but that works to. ;)