Slashdot Mirror
Ask Slashdot: How Often Do You Push To Production?
First time accepted submitter Stiletto writes "I work for a traditional 'old school' software company that is trying to move into web services, now competing with smaller, nimbler 'Web 2.0' companies. Unfortunately our release process is still stuck in the '90s. Paperwork and forms, sign-off meetings, and documentation approvals make it impossible to do even minor deployments to production faster than once a month. Major releases go out a couple of times a year. I've heard from colleagues in Bay Area companies who release weekly or daily (or even multiple times a day), allowing them to adapt quickly. Slashdotters, how often do you push software changes into production, and what best practices allow you to maintain that deployment rate without chaos?"
I push to production fairly often. Our company's Visual Studios have been configured with test cases that make testing easy, and therefore we can push to production much more often than those who do not use Visual Studio.
We have two stacks; one serves internal users and one serves external.
We've found it handy to push to the "stage" stack on Thursday, let it bake Friday + the weekend + Monday, and then push to production on Tuesday if there's no problems.
We keep track of anything that would affect a potential rollback. If there's a database migration, we back up the database before going live. Simple, easy, effective. And despite what James may say, rollback does exist. Not that we've had to do it more than once or twice.
We would push more frequently except we're too busy changing the world.
We did a lot of agile for years, so once a week for small things, assuming non-impacting and we tried for quarterly for significant impacting changes.
Really, it's up to the needs of the business and how long your development cycles are.
Several times a day, its the nightmare of undecided clients in web development
Why not talk to your colleagues and suggest ways of speeding things up?
It's good to talk.
Release when ready is quite a good frequency :-)
It s a governmental project, all code is tested in many environments before going to production.
I use rake a bit too but Finalbuilder has made so many tasks faster to do. Can push a button and make a new release now.
>> Slashdotters, how often do you push software changes into production,
3 times a month, generally.
>> and what best practices allow you to maintain that deployment rate without chaos?"
IMHO, you can't. Having a deadline every week, programmers cut corners to make dates. QA cuts corners to make the date. This much code changing without any "bake time" inevitably leads to an unstable code base, full of "corner cuts" from the last N releases.
1) Tag code in revision control.
2) Notify internal release documentation team, who puts some entries in some kind of finance/logistical database
3) Receive email within a few hours that the release requires "engineering approval"
4) Click "I approve". Sometimes multiple people have to do this.
Our production environment can export and compile from revision control.
That's pretty much it, and it all takes just a few hours.
It's worth noting that I work for a monstrously large company, but not exactly a software firm.. more hardware and engineering services.
Most companies I've worked for have a continuous deployment cycle. All changes, from small bug fixes to major releases go through a ticketing system. After the ticket has gone through all the steps (code review, QA, UAT) it goes into the deployment manager's queue, who then deploys the change to production depending on each ticket's priority. This means that in general, changes go out as soon as they are ready, sometimes up to two times a day for the same project.
Multiple times a day. Its the wild west. We do a bigger push weekly, but as needed a team of 8 developers push throughout day. On average once a day, but at the start of our produciton launch it was many times a day.
I think customers don't really NEED the frequent updates that you're getting used to seeing with apps. I would think only the foolish really expect rapid releases. That's been a side-effect of "apps" - not what we expect of software in the business world. If you set and can stick to a release schedule, I think your clients will appreciate it.
End users find updating all the time a headache - only super-geeks like us like seeing every possible iteration of an application. The average user doesn't want to be bothered with YET ANOTHER UPDATE (ahem... Flash and Java). But they do appreciate when you have their security interests at heart (like an out-of-cycle Windows Update).
I would say every other month for 'service releases' - every six months for major releases. The biggest exception being "out of cycle" emergency releases when deemed necessary.
make changes in production environment, ftw
This really seems like it depends on WHAT your company does. Pushing to production a browser or a utility weekly or even daily is a lot easier and less disruptive than say a OS or a major software product thats used by hundreds of large companies. I would say the biggest obstacle is to get rid of the paper process, bring in git or something else for changes that everyone can view, and limit the signing offs to production, QA, or gold master (or whatever your company calls them) while keeping the dev stuff out of it so that the dev team can be more flexible. Also have a pretty good test lab/suite that will enable you to test stuff on the fly on multiple machines and configuration. Case in point Apple has a testing lab of EVERY machine they have made for the last 3 years in multiple configurations for their software. Granted they are a closed ecosystem so its easy for them to do that, but generally you could get a good roundup of machines to use that span from the insane configs to the "are you fing kidding me you are using that???" type stuff.
"Slashdot, where telling the truth is overrated but lying is insightful."
Soon as our Hudson server does a build and all the unit tests pass, it automatically pushes it to production.
If you are in a startup and the cost of a bad push is $0, no problem.
If your SLA has penalties for downtime or misquotes, then I can understand why your process is more laborious.
If your pushing code the production once a day, you have no QA cycle whatsoever. For a larger infrastructure, there's nothing wrong with extended release cycles.
Daily releases should be for break fix only. Weekly or bi-weekly pushes to stage and test environments are pretty normal. For a full release cycle, monthly is about the fastest I'd expect, and it's not unusual to see quarterly or longer on stable infrastructure.
Or were you looking forward to having to explain why your database got cracked and leaked 2 million passwords or that tens of thousands of customer machines will have to be manually patched to repair the damage done by your last update?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
I'm in a small company, making a full blown new release takes us about two weeks of code, and two weeks of testing.
Bugfix releases can be pushed out to production within two hours.
And the best practice? Test, test, test. Have proper testing protocols and procedures.
Sounds like yours wants a process. You push as the business feels comfortable with it.
But think about this...
Lets say you are dealing with peoples financial statements. You want to make sure they are pretty good and accurate. If it is wrong people loose money and you get sued. Your audience dislikes sudden change. So you put process around that to insure that (test/qa/forms/etc).
Now lets say you are running a forum where it is the stats from some online game. It can be a little off. Your audience likes sudden new features. Stuff might not render correctly... But you can fix it tomorrow and the only gripe will be on a forum.
One business you have to be careful in creating 0 downtime and 0 mistakes. The other, a mistake here and there is tolerated and can be fixed at will.
Now I am not saying you can only create the first one where it takes a month or two for a push. I am saying those sorts of situations where caution is warranted. If you want a quicker process your process will need to be able to break things into small pieces and a bad ass testing system. Some situations they are contractually obligated to go at a particular rate. If you can not see the difference then I suggest you let your question simmer for a few years and get some exp doing it.
Slashdot posted this story earlier,
http://www.businessweek.com/articles/2012-10-04/facebook-the-making-of-1-billion-users
Interesting read on how Facebook, arguably the largest website challenge so far this century updates daily.
zipcar used to push Mon-Thurs between 7-8am. they've cut back to (supposedly) twice a week since i left. it was basically 'when the code is ready, or when management decides something has to go out'. if you're lucky you can hit their site during a release that involves db changes (a _lot_ of them do) and get a cached img - that means the site is actually down at the moment ;)
At $dayjob, we decide to release and our process takes about an hour. All the automated tests are run by Jenkins CI, and are run again during the release on every box being deployed to in order to ensure stability. We tend to deploy to User-Acceptance Testing boxes before full production boxes.
At the game company, we wrote a system that works like:
1) Tag release in git
2) Release is pushed to beta servers. Beta players get immediate updates.
3) Click button in Jenkins to run stable release
Completely automated, even down to restarting the servers in a staggered fashion to ensure that users always have a game to connect to (even if they have to disconnect in 20 minutes to receive the client update).
Automated testing, Continual Integration, and automated release processes (including cfengine3 and custom Perl scripts working with Git) come together to produce a painless release process. Since it's easy, we can do it whenever we want. As soon as it starts releasing bad code, we'll have to put process in place to ensure bad code does not reach our stable users.
The goal is to always be able to push to production. Have a continuous integration and test system that allows you to have confidence that you can always push the latest build to production. Automate as much of your testing as possible so your build and test process can produce something that a human test team actually has to work hard to break.
Also, you should pipeline your approval process. Always be in the process of approving a new build to go out while you're working on the next one. This will put a lot of pressure on this process to be faster and more efficient. Holding the people responsible for this process accountable for production breaks will put counter-pressure ensuring that they do not become more efficient at the expense of actually doing the job.
Need a Python, C++, Unix, Linux develop
Get rid of the clue less PHB's lot's of meetings is sign of being over loaded with PHB's.
Sometimes all of those meetings and paperwork serve a useful purpose when an application is critical. If a one-day build of instagram doesn't work, then the only consequence is that there are fewer grainy photographs of someone's cat. If a one-day build of a power distribution system doesn't work, then an entire city loses electricity.
-Glires
Push to production as soon as the (many) automated tests that you have pass. This means you should have comprehensive unit tests and tests that run in the browser, probably written in Selenium. You'll also want to script your release so that you can do it with the push of a button. Once the tests pass, and the mechanics of a release are trivial, there is little reason to hold up a release.
I worked for a top 500 website (East coast) for 7 years that did weekly releases. Since I left, they decided that wasn't fast enough and now release multiple times per week. I'm now self-employed on my own website and release within an hour of finishing development of a feature.
I started my development career writing firmware for laser printers. When you are shipping code on a physical product, the cost of bugs can be quite high. Especially when it leads to returns or recalls because customers are not satisfied. Our release cycles there were 6 months+. Quite appropriately, IMO.
On the web, the cost of bugs is much lower. In most cases it is the only cost of another release. Sometimes it could cost more because of downtime, but good automated test coverage mitigates that risk pretty well (especially if there is load testing involved). The worst case would be data-corruption, but I've never actually seen that in practice from a release, that has only been related to hardware failure or accidents in my experience.
What you need to be asking is, "How long is an acceptable test cycle?". Considering something "in production" usually implies some sort of testing, otherwise all the changes would just show up immediately right? So, ask yourself (or your organization) how much testing is enough? Equally important, how much is too much? These are the questions every software company struggles with. Spend too much time testing and your releases will always be behind. Spend too little time on it, and your "production" will look too much like test. Your own appetite for risk will determine where that line is.
I work at a company with many brands and MANY customer-facing websites. Some sites/services can be updated in minutes with little hassle or approval. Some need weeks with much paperwork. It depends on the risk. The apps are built in different ways (CMS or from scratch or with many proven components). The apps have different features (some can result in $million fines if there's a problem).
The tools are there to set up processes where every time I commit, it goes to production. You need a process that matches the risk you are willing to take.
I think the submitter IS talking to colleagues and looking for suggestions right here...
We do a major release about every 2 months, with maintenance stuff going every 2 weeks. There's pressure to speed this up. On the release side of things we're streamlining our documentation requirements and automating everything we can. We're also moving to a continuous delivery model for our regular test builds... if it's tagged, it goes to QA and an automagical report goes out with a list of resolved defects. This requires the developers to package things cleanly and consistently, but it also gives us an amazing amount of data to mine and I like to think that it helps to improve our overall quality.
This doesn't address requirements gathering or development methodologies, but I can't really speak to those.
I work for a marketing company - primarily in email, but the company handles everything. My area is databases, data warehousing and reporting. For our code, we have twice a week change management meetings if there are any changes to be discussed. Given that timeframe, officially we go to production as often as twice a week. In practice, we occasionally fudge a meeting and will get caught up after the code has been promoted to ensure that i's are dotted and t's crossed.
I do work for possibly the largest hotel reservation website. The whole company but especially the IT department prides itself on being agile and quick acting.
This means a developer can roll out whenever he thinks it's time. He'll roll out trunk which will contain his changes and everything else other devs checked in. There's a small amount of testing done to catch the obvious errors.
Real quality control is done through excellent monitoring and a quick roll-back option in case there is breakage...
Today there were about 40 code pushes. Not all to the website, some to backend systems etc. but still: 40 rollouts and no rollbacks.
For many this will sound like madness and others will immediately understand the business benefits this approach gives. While others are still testing stuff, this company is already giving the users new features which will make the product better and thus lead to more customers.
If you try to copy this approach, you'll most likely fail. The trick to make this amount of change work is in having people do incremental development of small features, excellent monitoring and absolutely quick and easy communication. Maybe you can find the people to pull it off, maybe you can't...
But knowing what other people in the web-space do will probably not apply to your company. As you said, the others are younger, nimbler and are used to it. For your company it's an uphill battle because you have an existing ingrained company-wide mentality which is completely different. I'm seeing a lot of companies trying to be agile yet in the end they only go through the motions without living it. They do the daily standups after lunch where everyone is talking for minutes on end... Yeah, you're doing it wrong, right there. The idea behind the quick-turnaround agile startups and established players is that they basically live the style. If something doesn't work for them, they chuck the process out the window and come up with something better.
Proven companies generally do roll out ITIL instead...
As a customer I hate daily updates to my applications. Unless the application is in Alpha or Beta it is very disconcerting to see updates at that frequency. I only want to see an update that quick when something is really broken. If brokeness is a daily occurence then maybe you need to slow down.
Migration requests to Production (signed off by the person who requested the change/bugfix/new dev/whatever) are supposed to be to QA (who does the migrations) by the prior Thursday but that can be modified as necessary :) This is for any of the dozens of systems we develop for - the several-hundred-page in-house ERP system written in .NET, the legacy Oracle Forms system being upgraded to .NET, or the myriad one-off special purpose web sites.
If it's an *emergency* it can be slipped into Production during the week as needed, but as the process has become more refined over the last year, they try and not do that unless absolutely necessary
Life is short. You seem like a talented person who knows what you want. If you current company does not modernize in the next 6 months then they are holding you back. The market is starved for really talented software engineers. Take the time to research your opportunities and then jump ship if the one you are on is still stuck in the 90's.
each way the release is tested and signed off on. little things are tested like making sure that the financial transactions are OK and customers are overcharged or undercharged. One time we had a case where we undercharged customers for years, then sent them a bill for a few years services and ended up in a lawsuit.
other times code works on developer laptops but somehow it doesn't work in production
i'm at the point where unless you're facebook or someone big i never report issues. i just use your competition. if you want to deploy buggy code, go ahead. you just better be the only guy doing whatever you're doing or watch your customers flee
I sometimes just work in the live environment while people are in and using it, depending on what I'm doing.. I'm very careful when I'm doing this of course, but sometimes the hotfix requires having the user's screwed up data to work with.
The rest of the time, I publish when I think I have something useful for the users.. Often a couple of times a day, but we're one of those very small nimble web 2.0 places, and the decision to publish is all mine.
We sort of break ours up. We have an implementation every M-Th, but M-W's are on the sub systems and Thursdays are fo the main system. I'm a team lead over one of the sub systems, we push to production on Monday. But the implementation team do a push four times per week. Sorry for any typos, sent this from my phone.
Pushing to production multiple times a day sounds like code-and-fix to me. Now, promotion to UAT/Test multiple times a day sounds much more reasonable. Pushes to production should be regular and frequent but not multiple times a day. In my company, we have a couple of change windows that production pushes happen in. Most times the windows come and go with no updates but when we do have code that's ready for production we schedule it for the next available window.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
We push updates to production almost weekly -- any time it is needed. How? Automate! Automate the build! Automate QA! Automate the forms! Automate the process! Automate production monitoring! The change management meetings last 5-10 minutes a day. This is for all of the infrastructure for a multi-billion dollar global organization.
And fire the people that want to sign off on documentation changes. I'm dead serious about this. If you cannot or will not do this, you are screwed. If you are not in a position to fire those people, but the people that are in that position won't, then go work for a company that will. The company you work for isn't long for this world anyway.
We have one weekend a month where critical processing happens. No changes on Saturday. Instead they go out Sunday. Nobody stops the train. The downside is that tolerance for production outages is very close to 0. If you are a screw up that wants job security, this ain't the place for you.
I am a Release manager at Acquity Group and have worked for "Old School" software companies that have their eyes blazing at all the new web companies that release release release all day long.
Here is the abbreviated philosophic path to victory:
DevOps.
Your developers need to act like operations (knowing how the code is deployed and configuration settings, routes and the like) guys and your operations guys need to start coding (as in ruby for puppet and chef, automation and automation and automation)
This leads to...
Infrastructure as Code.
Hire a Release Engineer or convert a sysadmin to start automating builds. Now you start automating code deployments, you start automating infrastructure deployments so they are repeatable.
This leads to...
Test Automation.
Now you need to stop focusing on smoke testing and have test automation engineers write automated test code.
One more thing...
Automated Rollback
When things go nuts, with fast deploys you need fast rollback. Capistrano is a great tool for deploying this way, rollback is very easy.
Now you can ...
Continuous Delivery.
Great, you got this far, your builds, testing, deploys are automated!
For Developers:
Coding for Continuous Delivery is a different paradigm where unfinished code makes it to production. This means that in the production configuration, settings for the new code must be activated by switches to turn new feature sets on. You don't want that unfinished code mucking up your site, right?
People Processes:
Do you have CAB boards and ITIL processes? Great, make them faster and more as DEV/TEST/QA becomes automated and just focus on UAT/Prod environments. See this book: http://www.amazon.com/The-Visible-Ops-Handbook-Implementing/dp/0975568612/
I can also gloss over on waterfall/Agile and hybrid software models.
Finally, unless your culture wants to shift, it may be damned near impossible to change the culture unless management wants to. If it's doomed, it's doomed!
If he can't deploy to production in one step, he needs to fix that first.
I'm not talking about from dev box to production. I'm talking about the physical act of someone running a single command (or for the Winlazy, pressing a button) then walking away. All code checkouts from source control, database changes, app server code deployments, web server restarts—whatever—happen without user intervention.
He should also be able to roll back in one step.
For all the meetings, forms, etc., it sounds like there is A LOT of CYA in that company. In that case, it is cultural and can only be changed from the top. Until/unless the company becomes less risk adverse, there is no point in trying to become more "agile" (i.e., risk-accepting) except making your job easier. Build your tools/scripts/whatever to make it easier to do stuff.
tl;dr: If you want a more nimble company, switch jobs.
Yeah, right.
At one point we were on a monthly push for a web-based application. I was stuck in documentation and got all my changes last minute. Once I was told, "The software is ready, are the Help docs?" I asked for a day to see what the software had morphed into to make sure the docs were still OK and to then write/fix what had to be fixed only for the most current changes/new features. My problem was with the monthly pushes, QA was locked in testing the code and stopped reviewing the docs as they had no time to check the code. After a number of release cycles, all the un-reported code changes had altered the software so much the docs were wrong. I could never explain that I had no talent for fiction. I was not sufficiently creative to rewrite the docs just so they could be wrong. Every single page was reviewed at least once. However, given that I was barely keeping up with new/fixes, a complete rewrite to match months of changes was not possible. The software is only as good as the Help.
Without any idea of what the product does, this isn't a meaningful question. Is this just a web site? A program used by people? A program used by other programs? How much persistent state is there? Do users need to know in advance before an update is made?
Look into Platform as a Service (PaaS) that has a release/lifecycle management capability.
We only have a production environment to develop code, and use an interpreted language, so it's as simple as editing a file and sending a few HUP commands.
Cardninal Rule: Don't fuck up.
It's worked mostly well for years now.
I want to delete my account but Slashdot doesn't allow it.
In my current company, in one year I have not seen any modification on the main product, if we don't count new launched product. (in fact no change on the real product, but change in the software used to manage the product are more frequent)
Oh, by the way, when I say launch of the production, it's really called a launch, with a launcher, and satellite on the top of it.
I invite you to read the GitHub blog post on how they deploy.
The company I work for is right smack in the middle of this transition. We had something akin to a SaaS model, before SaaS was a "thing". We have 40+ applications, some are desktop thick net .Net clients, others are web based, all of which interact with one another to varying degrees.
Myself and one other person were instrumental in getting the company to a point where it is possible to release in a semi-automated fashion.
Our challenges were similar to what you described - manual work with lots of process wrapped around it to ensure some modicum of governance, which often failed.
Our number one task was getting our software dependencies under control and automating building. We settled on Maven/Hudson/Nexus as the tools of choice. We have a corporate POM that defines many of our baselines that each of the software projects inherit from. We use Hudson both for automated builds as well as one touch deploys (some are even totally automated) to environments, including production (which is not automatic - it requires a human to initiate).
We then spent over a year "cleaning" our old ant build structures and refactoring into Maven. It cost a lot of money. A whole lot. Maven found numerous cyclical dependencies that Ant hid. We defined all the core libraries, versioned and released them, then built the apps outward from there.
Today, 99% of our software builds are totally automated (a few stragglers of low priority products have not yet been converted to Maven). We have total control of our dependency structure. We have a totally automated release process. We have a totally automated deployment process. It took a lot of work, and a lot of money.
The other side of the coin is our runtime environments. This has been a disaster. The teams that run these systems don't have the concepts of automating anything unless it comes shrinkwrapped form a vendor. Plus there was a management structure in place that encouraged manual work with large numbers of employees. Firefighting was highly rewarded (both in cash and prestige). Eventually, the balance has shifted. The management on this side has either been terminated or left when they saw the writing on the wall. Slowly this side is embracing virtualization and a move toward generic environments whose buildouts are automated. There is a long way left to go here. The goal eventually is that we can automatically provision a VM for a product and deploy to it as we need (think a mini-AMI model).
It is hard. Really hard. A lot of companies won't have the energy to go through with this, and it has a lot of ways to go wrong. We have gone through multiple executive level people during this as well as countless mid-level managers. This type of change is not just technical, itis a massive cultural change for a company.
But today, we have a totally automated process for code release all the way from the developers desktop through to production. Depending on the interactions of a piece of software (if it is an edge or core piece), it can moved daily or monthly. The technology itself can allow multiple production moves a day if people so wanted , but for customer reasons we usually limit this to once a day.
Every Wednesday is Preprod day, and every Sunday is Production Deploy. Production was done by a rotating group of about a dozen people, so you didn't have to work on Sunday too often.
It was a rare week that had nothing to deploy. It was also rare (thank Fudd) for us to have to do an emergency backout on Monday.
Speaking of which, it's good to have different teams testing and deploying. It's less likely that the deploy team will try to fix the bits after they're deployed in Prod, and maybe forget to fold the fix back into Development. Having teams with different privileges and different responsibilities helps preserve consistency, and tends to avoid the morass of twisty passages that often results from free-form development.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
It depends on your business which metric is meaningful. E.g. for a global bank, quality is more important than time to market. Make sure that your business really gains something by playing release time against quality.
Monthly release cycle isn't really bad, if your goal is stability. This gives your testing/QA team (you have one, right?) plenty of time to find problems, and developers time to fix any bugs the QA team found.
Shorter release cycles might be better if your goal is responsiveness, but in my experience faster than weekly is asking for trouble.
Also, make sure you have a process for handling those things that do legitimately need to move faster than the release cycle (e.g. "We just discovered a massive bug that's risking exposing customer credit card numbers!"). The process for doing this needs to have people (such as your QA and admin teams) empowered to say "no-go" for fixes like this, to prevent someone from just willy-nilly pushing stuff by declaring it to be in this "faster release cycle" process.
I am officially gone from
We'd like to push stuff to production faster, but our client usually only has one window a year when we can take production systems offline for upgrades.
As for keeping it organized...hah. Two weeks of pure chaos while the hardware folks are trying to get all their upgrades and replacements in place at the same time the software guys are trying to figure out when systems will be back together so they can do their upgrades, followed by a weekend of praying that everything comes back online the way it should.
I push to production fairly often, depending on the nature of the tickets I'm responding to. I'm generally maintaining three branches concurrently, the production branch, the point-release branch and the major release branch. Trivial fixes or small-but-critical fixes tend to go onto the production branch and get released as hot fixes. This can happen several times a day or not for several days, depending on how extensive the change is and thus how much testing is needed.
Fixes that are a bit more extensive and/or that will have visible impact to the users tend to get done on the point-release branch, which typically is merged back to production and released every few days to a week or two. (Naturally, fixes made to the production branch are merged to this branch as soon as they are released.)
Significant functionality changes, refactorings and other large-impact changes are performed on the major release branch -- with the intermediate changes merged in, naturally. This gets merged to production per our roadmap schedule, hopefully after being fully tested by the beta users. (Yeah, right.)
Amen to that, brothers and sisters! Amen to that!
-kgj
There is no one frequency that is good for every application- it depends far more on what you're doing and you should be focused on delivering that, rather than a pretty arbitrary metric. Look at the value of what you're doing. What is the cost of things breaking? Don't forget, MTBF *includes* the time to fix the failure.
Think about live broadcasting. If you're selling informing people of what is happening right now, you better be able to recover from a link down quickly, even if it means not keeping records- but if you're selling an archival service, data loss is a problem, and some delay is acceptable.
This shirt covers the methodology I've sadly seen far too often:
http://www.thinkgeek.com/product/f141/ [Most Interesting Coder]
Invalid Checksum. Retrying.
You should have a script that will pull the release candidate out of version control into a new directory, build it, and run tests. Or as close to it as possible. I use Perforce (recommended), I use the mouse to pull my candidate into a directory, cd into it, then type "make all test". I work in a cygwin environment under Windows.
You should use a label when you pull out your release candidate. Even if it's the tips, label those tips before pulling the release candidate out.
All checkins to your version control should refer to the trouble ticket it applies to. All bugfixes, enhancements, and new features should have a ticket.
IMHO, without this your walking down the tunnel, and the headlamp of the oncoming train is burned out.
Doing this, I could do a bugfix and a release in a few hours on my last project. The actual "get the code and make all test" took about 30 minutes. Other projects had a 3+ hour compile, plus a 20 minute link, plus about 4 hours of testing. But during that 8 hour process I did about 30 seconds of work, then moved on to something else.
The answer you are looking for is: Lower company standards. Having worked for companies large and small, including start-ups at various stages, I can say from experience that frequent releases work because management has reasonably low standards, and is perfectly okay with the last deployment breaking things, and needing to be hot-fixed by a follow-up deployment just days later. That's just the model.
I know the model you're talking about, too. A more solid company, building its web products like it would a critical internal service... Going through design phases and in-depth code review, and extensive compatibility testing, to make sure anything that is deployed will work 100% (or almost), and is good enough that it'll never need to be held together with duck tape and band-aides because the feature was tacked on ad-hock by one dev in a few hours, which is really what you're talking about at these "nimble" startups.
There are some ways to get the best of both. The Amazon model is to develop a public API, then develop unit tests which FULLY exercise that API. Then all the quick development behind the scenes doesn't change the API, and regression-free code is assured in seconds, as you run all those unit tests against the new code. That also instills more discipline, as the documentation is law, and it'll be clear which code is to blame when there's a breakage, and less perpetuation of bugs and flaws to keep other bad code from breaking when they're fixed.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
We try to complete a sprint every 2-3 weeks and then push to production. Occasionally (more often than I would like) there are hotfixes that need to be done in between. Scrum with agile software development.
As often as we need. We push based on features. Although we sometimes slip into batching up production moves. I find it is much easier to push often when you work with people that are unafraid of branching based on feature requests.
I'm just glad we don't do the "quarterly push" like some do. It sure saves us a lot of troubleshooting in the chance that the push introduces a production error.
As other have pointed out, the speed is less important than making sure the process matches the product. How important are security and unit testing? How many people are working on this website? What does downtime cost? How easily can you roll back changes? These are the sorts of questions you should be asking and getting answered (internally) rather than "How can I push out updates twice a day?"
Really, if your organization is large and slow to adapt to the new way of going things, perhaps the powers that be should look at giving the web development team some independence. A skunk works type of set up where a small team can work on the technology and get things moving. So long as they check in every so often and don't go off the rails, the web team could be left alone.
This is what scientific study can tell us:
- Reviews can find 90% of the bugs.
- Review should be done by a single person and it should not take more than an hour (not much can be found after that)
- Any other testing phase can find about (or at most) 25% of the bugs.
Key to fast development is in unit tests and code reviews. With those in place, mostly misunderstanding of the business logic can get past you. And testing takes less than an hour. If you want to get rid of the business logic part also, look into user experience, it will tell you what customer really needs, when customer itself doesn't know it. Also add some CI, automate any other validation tests you might need (e.g. server startup time) and you are good to go. Also investigate Lean.
Imagine this: A developer makes a commit, computers run thousands of tests, all tests pass and release is published automatically or a nice package is provided for distribution. (or developer makes a commit, there is an error and CI informs the developer directly and the problem is quickly fixed). A developer complains that he has to wait for days before he gets to see the requirements, because architects what to review those first. So you have a meeting and notice that you can actually save days by arranging the process a little and still get the same value you used to get before.
Next, don't believe any of that. Life is not that easy. People are stupid, they might have a bad day because their favorite juice was sold out, or you might suffer from a faulty software etc. Life is complex. Something that works for someone, doesn't work for everyone. And environment around you constantly changes. If you don't change, you die. So try things out. Reject solutions that don't work, continue with good solutions until they become bad. Read books (peopleware, management 3.0, Clean code, Working Effectively with Legacy Code, The Design of Everyday Things, etc.), get ideas from other companies and organizations, and try them. It is impossible to tell what works and what doesn't because of the complexity.
Why do you have to accelerate releases just because you implement web services? The services are all about interactions with your clients and collaborators and are always attached to value generating processes which are intimately related to your business models. The business models of these web 2.0 strapons are completely different from yours.
I work for a smallish team (~100) in a megacorp that ships a service that has a significant number of customers.
We have 2 major code branches - the developer branch and the production branch.
- The production branch sees management approved point fixes that get collected and rolled out every month after a huge number of tests (automated and manual).
- The developer branch gets continuous feature additions. We collect all features from the developer branch and move it to production once every 6 months.
The details below pertain to our developer branch:
Like a lot of other people have suggested, we have spent time on robust test automation. We split that up into a smallish set that runs before any code check in. All checked in code runs through a significant battery of tests nightly.
We fork the developer branch and deploy it on a self hosted environment every week.
After self hosting features for ~5.5 months, we initiate rolling out the new features to production - takes about a fortnight.
Hope that helps.
Well, whatever you do, please don't follow Google's lead, at least with respect to Google Groups.
They seem to push a new version at least weekly, and sometimes daily.
It's a coin-toss whether it will actually work from one day to the next. Features will just stop working for a few days, until somebody notices and fixes. Or fixes the fix that was put in to fix the fix.
Latest is that Firefox 9.0 is "obsolete". If you are running anything older than a fresh-out-of-the-box (or Torrent) Linux release, you are probably using an "obsolete" browser.
At one prior job, it was official written policy to NEVER upgrade to the latest release of ANYTHING, on the grounds that it had no track record. So it doesn't matter how often you release.
Releasing on a fixed time schedule has never made sense to me. Target, yes, but if it's ready a week early, or needs a week longer, better to do it right than to do it at a fixed time.
You're not eating the cost of burning releases to media.
Push early, push often, but have a rock solid, bulletproof rollback mechanism.
Seriously, that's what staging is for. push it, roll it back, push it, roll it back.
Since you don't have to stick to a golden master, media reproduction schedule,
push a feature as soon as it is ready.
If your organization can't get past big quarterly releases, then they will not succeed with more nimble competition.
The organization has to adapt. So wither the management adapts, or gets replaced.
My Heart Is A Flower
Hourly. As soon as a feature is ready.
At Verkkokauppa.com we deploy several times a day. Our team is fortunate enough to work on our own system, so for small features or bug fixes the turnaround can be as little as minutes.
For us the most important tool is Git. We have a logging mechanism in our deployments scripts whish logs who deployed which version of the code base (SHA1 hash) and when. If we do manage to break something we simply immediately return to the last known good version by checking it out and deploying it. Then we have a lockout policy that prevents futher deployment until the bug is fixed.
On rare occations we have had to do reverts, but only a couple of times over the course of several years.
Other valuable assets that keep us agile deployment-wise is having the whole team physically in one place, and most of our stakeholders in the same building. We also use Jabber internally to make it even easier to communicate on those occations when face-to-face is not possible.
We also use Scrum, but this is a rather recent development, and we did mange the same deployment policy before we switched to Scrum. A continuous integration environment is in use, and does help keep things in order. But for minor features and tweaks with few stakeholders it's not entierly uncommon to simply deploy and communicate with the users directly to check if everything works as expected.
The fact that there is one installation of the system is a major asset. We really never have to deal with versioning. The current version is simply the HEAD commit on the master branch.
.: Max Romantschuk
Many a user of that site wishes they would STOP updating.
At netflix, each app owner (either person or group) figures out what release process and frequency make sense for them. Some apps get pushed extremely rarely -- probably for actively developed systems, the least frequent I see around here is every other week. Other apps get pushed more quickly. Lots of people who push weekly; lots of people who push ad-hoc, whenever there's a change they want to get into production. Our Engineering Tools people are also finally starting to support continuous deployment, so it'll only get faster.
I work at an online email marketing company of about 50 people.
We have weekly hotfixes for low/medium impact bugs. We can update all servers in 10 minutes for high impact bugs. We have a feature upgrade release every 6 to 9 months. We could release earlier but we don't want too many communication moments with our customers. The second reason is that we want to deliver a finished product which takes some time, documentation and marketing must be changed, etc.
We don't communicate our maintenance schedule for bugfixes. When we run an update the web browser ui is locked, afterwards a user can pick up where they left. An update is about 20 seconds and ~0% impact. In general we apply the update to a few servers earlier so we catch obvious bugs with less impact. Needless to say we make builds from a VCS and deploy automatically.
For SaaS, you want to have a solid 'operations' department which is in sync with development.
nosig today
Just call me King PRODFIX....everything else is throwaway work. BTW -- that's a joke :-)
We have a particular software from a particular vendor who only pushes an update once a year. We've already pointed out several bugs and quirks in the software that have become "feature requests" for the next release (due in January 2013). On the other hand, we also have software from a vendor who pushes an update every six weeks, and then we have to get that update installed for every user that has the software because every update changes the file format and it isn't ever backward compatible! . So, if you're a major software vendor, my suggestion would be to favor structure and stability over frequent and minor patches or releases, but not so infrequent that it doesn't seem like you're paying attention to your market.
I used to push out a release whenever it was needed - I'd typically go from bug report to release in about an hour in the past.
Of course now that we've gone "agile", that's once a week *at most* now.
We've started following a lot more rules and procedures. The irony is that there is a very good reason to: It's to keep defects to a minimum. Now, I was already doing a pretty good job at that - our bug tracking system showed my bug density being lower than average. We've all gotten more disciplined with testing and reviews, so the overall amount of new defects has gone down. Now, that's all good and well, except the *impact* of each defect is several orders of magnitude bigger now. Where previously a defect was fixed in an hour, it will now sustain itself for at least a week. In the name of preventing possible, hypothetical defects, we're now allowing known, observed defects to exist for over a hundred times as long. Even if we catch three times as many bugs prior to release as before, that's still a massive 30-fold DECREASE in overall perceived quality.
In the name of agile... we're doing the exact opposite of what the Agile Manifesto tries to convey.
Oh, and of course we have release cycles now. Which means that in the rare occasion that there's a bug in a library that's used by 10 applications, it's going to linger in there for ages, because you can't just push out the updated library to all the apps. Where previously there was ONE AND ONLY ONE version of this library, it's getting pretty close to being a different one for each app out there now. Fix a library bug, and months later other apps still have the same issue.
Something, somewhere, has gone terribly wrong. But I consider myself lucky. At least those ridiculous 30-minute long daily stand-up meetings have stopped.
I interposed an executable which selected which version of software to run based on program and UID. This allowed my to control to a fine degree who ran what.
I would do beta tests by putting the noisy people in a list so they got the new version. If no one complained I rolled it out to everyone a few days later. Changing versions for an individual user takes seconds and no one else is affected. So I could back out very easily.
I'd actually have multiple versions w/ different users for each. It was a million line code base of other people's code and pretty buggy. So having two different bug fixes in the same day to a single program was not unusual. I did this to avoid unwanted interactions.
Generally weekly, although we sometimes delay a week if there's something big that needs extra testing. Sometimes we deploy a build not on the normal day if something is really important. Those two balance each other out fairly well such that it still averages to once a week. Testing and stability is fairly important for us, as it's a insurance policy creation & editing system, and getting things right is usually more important than getting things fast. There's many more legal issues we can get into if we do something wrong than if we add in some new helpful code. We're required by auditing companies to do the following: have 2 developers review all code and documentation. It runs through a first, smaller set of scripts to confirm nothing is horribly wrong, then all tasks are tested by the unit test team, then the QA panel reviews both and determines if it approves of the risks involved. It determines if new training or documentation needs to be developed and sent out to the agent force. The work then goes to the regression region where a second, much more rigorous set of scripts is run, then it's deployed to production. Time from development to deployment assuming no bugs/referbacks is ~2 weeks.
It depends on the business, some can loose 50% of their customers and not care, and/or know their customers don't have any choice; while others would be devasted by lossing a single one. I have worked for both types of business and can tell you that THIS is what affects how often code goes to production (unless politics or extreme egos are involved, in which case all you can do is wait for those people to die/leave/quit/be fired).
NEVER (unless life or limb is invovled) have a Hard-fixed schedule, software can NOT be made to fit within a schedule unless it is so padded it becomes unrealistic (I have seen this, where two or three versions are done before the first goes to production since the padding was so large before the code could go to production). Finally, keep in mind that there are times when having a huge padding and slow production path is quite acceptable, think in terms of software that can affect life or limb.
And don't want to do it myself.
Unless you take into account the business you work for and the environment you work in then you've asked an incomplete question.
In a small-ish company you can push out updates quicker because in many ways there's less risk: if there's a problem it either (a) doesn't impact as many people or (b) is easy to roll back. That's a generalization for sure and everyone can pull out exceptions I'd bet, but I think it's a fair generalization.
In a large corporation though, especially one where there may be government regulations that have to be met and SLA's that have to be maintained or there can be real financial repercussions in short order (fines, give-backs to clients, etc.) the story tends to be different and considerably more conservative. It's not a question of nimbleness or doing things the "new" way or anything like that, it's simple risk assessment.
Naturally, the other big factor, and actually I should have said this first because in many ways it's the bigger point, is what the application actually is. I mean, if you're talking about a timesheet application for your workers than you probably can get away with a lot of mistakes. Try it with your key line-of-business POS application though... you know, the one that if it's down for more than two hours your business goes under?... in those cases, you likely don't want to be risking it any more frequently than you really have to.
Me, I work for a very large corporation, and I'm lead on a couple of very large, complex and extremely important applications. We have no set deployment schedule generally, but on average we do a release once every two months or so as driven by business requirements. That FEELS about right to me, given the environment, the applications and my 20+ years of professional experience in IT.
YMMV of course, and in fact it probably SHOULD since I'd contend there's no single right answer... but what I've mentioned here are probably the key considerations in any case.
If a pion (n-) collides with a proton in the woods & noone is there to hear it, does lamdba decay into the source pa
For my company, it's every 7 weeks with an 11-week lead time ... but it's going to be different for everyone depending on their business requirements. For my company, we must QA/Beta every development release before it makes it to production for general availability. To this end, I have produced a schedule:
* 3 week scrum
* 2 week first-pass QA
* 2 week second-pass QA w/customer Beta
*** dev team starts the next scrum cycle here, overlapping with the 4-week marketing period that follows
* 4 week marketing (documentation, training, customer announcements, press releases, graphic/video collateral production)
At the end of the marketing period, we release on a specific, scheduled date/time for the service window. The advantages are:
* gives our customers structure without surprises
* clear communication both externally and internally
* affords the development team 4 weeks (during QA) to work on internal, non-customer-facing projects for refactoring, tools, monitoring, etc.
* gives plenty of time for the sales/marketing folks to ramp up in preparation for the release
* allows us to get crucial features to customers in as little as 5 weeks by way of the beta program (great for sales prospects on our long sales cycle)
* allows us to maintain a development calendar for the entire year with exact dates that we know we will meet which gives our sales/marketing team visibility into what is coming and when, and where they might want to shift the schedule to re-prioritize as needed
Ahead of each scrum I do a little planning for what major application features we will be adding, I break down each major feature into a set of smaller tasks, must-haves and nice-to-haves, and I ensure that the must-haves can all be achieved within 3 weeks given our resources. We have always been able to complete all the must haves and many nice-to-haves for each scrum cycle. We have never completed all the nice-to-haves, and may never go back to do them unless an internal/customer request has enough pressure behind it to add it to a future scrum cycle.
In terms of ensuring that this is all possible, it comes down to understanding the business requirements early in the project proposal and then delegating the balance of the project management to a single person who understands our application, understands the requirements, and can break the project down into pieces that can be absorbed into the agile methodology. Documentation is kept to a bare minimum until the scrum cycle is completed. We get on-time satisfaction of business requirements every time.
But it wasn't always this way. Two years ago we were still stuck in the classic waterfall project management model and as a result progress with the application stagnated by comparison to our competitors. Management was stuck in their 1980's mentality of planning everything before writing a single line of source code. I personally had to champion an agile approach to software development and demonstrate the advantages that came with it to get the buy-in necessary. At some point you need to "just do it" and get yourself out of the rut.
I hate Agile with a passion it is the exact opposite of everything I believe and know to be true.
The only thing worse are people who when pushed are only able to justify their position by citing who all else is doing it too.
At the end of the day the premise of your question is wrong. The ONLY objective function is PROFIT.
If chaos makes your firm more money then your a moron if you don't immediatly ship code once it successfully compiles.
I actually wasn't talking about Agile the process but agile the adjective (hence the quotes). Perhaps I should've said 'flexible' or 'responsive'.
Yeah, right.
Our erp system has 2 scheduled releases from trunk per year with our biggest customer (top 20 telco).
Unscheduled releases i.e. fixes and small change requests are done once per week in avg. Deployment is done by simple scripts, could be done by a monkey. Downtime for an upgrade is 1-2 minutes per node. It's not a cluster technically, but a bunch of identical nodes.
DB schema changes are kept and versioned in a specialized DB (includes multilevel projects and branching). The necessary upgrade scripts are generated on deployment. Biggest pitfall is a typo in an sql script that will block the upgrade, so cant run unattended.
We push to prod every two weeks. Others in different parts of my company do daily pushes.
I work at a game company and we do about once a week, once every 24 hours for hotfixes. We can't really do sooner for several reasons but the foremost is that players want to play our games and they can't do that if we're bringing the servers down. We use some heavy automation (and always getting heavier. Currently transitioning from in-house + bash scripts to bash scripts + git + Puppet) which acts as a force multiplier for your teams.
However, PLEASE be careful when doing automation and sending builds quicker. Not everything has to be automated and many times the issue is not getting the software out fast enough it's instead getting the STABLE software out fast enough. Nobody's gonna die if you take an extra hour to explain to another team what you've done to make sure it all makes sense to everybody (and if you're in a company where that DOES matter I'd recommend leaving it...that level of stress isn't good for your heart). Automating a complex system is a long, arduous, and minefield-laden task that's going to have some screw-ups and rethinks along the way.
My personal opinion is that you should use Puppet/cfengine/chef/whatever-you-like to not only help out with continuous configuration management but *also* as a means of codifying knowledge at various steps in the process. Don't just have one team do your automation, have everyone get in on it and get familiar. From my experience this has been easiest with Puppet due to its syntax and modules (aka I can hand off a module to the programmer team, they can punch it up, then I run it and if I want to know how it works Puppet will tell me everything I want to know). You get everyone speaking the same language, working on the process together this way which works as a sanity check and as a knowledge repository (you know, just in case god forbid your admin gets hit by a bus or your low level grunts win the lottery).
"Just a fox, a whisper."
There are a few different questions implicit in the question:
1) are you making changes often enough to satisfy your customers
2) are you able to compose your efforts sufficiently to make good releases often
I think the ideal situation is to have customers who will accept that they are part of an organic process of improvement, that will accept occasional glitches, and a company that tries to minimise glitches and tries to improve its software. I is arguably the case that for software that is going through rapid iteration at least every third update will be one that your customers don't like even if you managed to deploy it hassle-free.
So if you are the MD/CTO of a small web software outfit, who is better than you to decide when to decide to merge testing into stable, run loader.sql across your 50+ Postresql instances and apachectl restart? Jenkins and co. can make it run nice, but if you haven't got the chops for that decision -- from every point of view from knowing your code base to understanding your customers -- I'd suggest you can't answer the question.
The only correct answer to which is "when the customer and the code is sort of ready".
Developers are responsible for supporting production, which means they can release whenever they want, as long as they're willing to deal with the consequences. This has lead to a release frequency which ranges from 1-20 times per day, depending on how critical the application is, what is being added, and how complex the change is.
The most important thing to us is that people who are treated like adults act like adults. Trust your developers to release responsibly, and (with a little teaching/learning) they will.
I've found that for web development today, weekly or twice a week is around about the sweet spot for non-trivial web-development.
Of course you need a modern pipeline. Good devtools, a framework, test first and a lead architect and scrum master who knows what he's doing. But the days of paper driven management in software development are totally over, except maybe for spaceships, military hardware, nuclear power plants and perhaps medical gear.
If you're doing web-dev management with a manually maintained papertrail, you're doing it wrong. It's that simple. And as an experienced scrum-master I'd say push-to-production once per week is minimum.
My 2 cents.
We suffer more in our imagination than in reality. - Seneca
Just leaving that here....
You just code the changes in your own library, then copy to the production library.... Or just edit in production, because the program object doesn't update until you compile.
Mind you I am the sole IT employee of a small company, so the entire process is under my control. FWIW It's a data-driven B2B web app kind of thing.
I prefer to split up change requests - including configuration changes - into individual numbered releases, which are tagged in SVN and documented on the login page visible to internal users, so the evolution of the app - and my contribution - is visible, especially for the people who wouldn't otherwise see what I'm doing.
I'm at a large company (you're probably one of our customers, and you're actually happy about it) where some teams push every change to production "instantly", where instant is probably 1-2 hours. We have build servers and automatic approval pipeline queues. When a change goes into the repository, it gets picked up by the build system, which compiles it and runs it against automated unit tests and integration tests, and verifies its health as it deploys to servers a few at a time. Assuming everything passes at that first stage, the change moves to the next stage, where it may have a different set of data, differently configured servers, or more rigorous integration tests gating its automatic approval to the next stage, and so on until it reaches production. My team uses a human-operated approval button for some of the stage promotions, because we prefer to release certain sets of features all together, but we still aim to release those sets every few weeks.
I work at a cowboy shop where we push whenever we want (note: it is a publication company that only has 2 developers). When I started (so there only being the one developer) the guy basically edited code on production. Over the last 3 years I have been making changes to the process (at first the other developer was relunctant, but now that we are close to almost becoming "proper", he is completely onboard). My first step was actually setting up a proper dev box.
/ahem most interesting man in the world voice "I don't test often.. but when I do, it's in production"
I work on a telephony infrastructure (voip pbx + websites), and production releases are as often as we need it.
Even if every minute of bugs or downtime costs money in terms of disconnected customers, gain is by being able to react and fix quickly. Sometimes I do write a quick code (usually fix) on live system, of course knowing what you do and what's on the stake makes you pay attention to details, taking a look on semicolons at the end of line before hitting save.
Also having control of replicating changes to all servers is a good tool, you can do a live test on small set of users, and if there are no errors just replicate changes to all servers.
However normal "feature" releases are usually from couple of times per week to once per month, depending on the size of features.
Rollbacks are always in mind, but you can't rollback everything, new call logs and recordings are created live - so either implement some backup of raw data and be prepared to manually fix broken parts later or you may just loose something important.
I find "Adapt Quickly" is a metaphor for "insufficiently tested" or "it worked on my machine"
If I had a dollar for every "tiny fix" that required hours of hacking in prod to resolve with the release team sitting around waiting for the wunderkind to figure it out, for it then to be reverted and done again the next weekend. So much better when properly tested with signoffs and accountability.
Yay me!
I'll take it whoever came up with this question obviously has little or no QA department. The company I work for sometimes has thousands of automatic and manual scripts to get through before a "release" can be approved. It can and does take months to complete this. The idea we could ever promote once a week is a nightmare to consider.
If you don't allow for proper testing you may break something and not realize it until months later. This could cause serious problems such as capturing invalid data, to something simple as an odd gui quirk. In one case you may open yourself up to lawsuits and in the other you may find it nearly impossible to track down when and why something broke.
Don't be so eager to get your fixes out the door...they don't always just fix something.
Slashdotters, how often do you push software changes into production, and what best practices allow you to maintain that deployment rate without chaos?
As often as we need to. This could be immediate (for small tweaks), middle of the day, nighttime if the deployment is going to have direct system performance impact.
.. ie. structured :), or else .. you can do it our old-skool way .. have experienced people who know about the business testing the changes.
.. keep them informed. You can accomplish this by letting the clients set their own priorities. Once the clients have the lists of work they want, they can decide what they want done and in what order. When the ducks are in a row, and things are flowing smoothly .. not only is the client in control, but they'll develop their own understanding of the development process and understand why work is being delayed. Getting clients into the project management system is also helpful.
.. because mistakes happen. Have a process for getting things done quickly when emergencies happen.
Single person responsible for deployment, use some sort of locking mechanics to ensure that what's deployed to live is precisely what's deployed to test. Give the clients the opportunity to UAT everything, and have environments setup for them with live data that will allow this. Automated testing is useful (if your software is structured in such a way to allow this
But by far the most important thing is to not set deadlines. And people might say that it's unrealistic, but the truth is setting deadlines is unrealistic. When does it go out to live? When it's ready, and not a minute before. Estimates are fine, and communicate with the clients
But the most fundamental thing is to take small bites of the apple. Don't do things in one hit. Break projects up into bite-sized manageable tasks. Then if something breaks, it breaks small. And be prepared to be nimble. No matter what processes are in place, there's always got to be work-arounds to get things done quickly and effectively
It's actually quite easy. But people need to take personal responsibility for their work and communication is king.
We usually make several production releases every day. We have a complete configuration management system (conform) that totally automates building a server and releasing software. We have a complete dev environment and two test environments. We test code in pre-production first. But the key is our automation. It automates releases and rollbacks. Without complete automation (and competence), we would be submerged in paper work and bleary-eyed midnight releases like other teams in the company.
Almost every day.
But we are a team of two, working on a private company intranet.
I suggest you read Getting Real.
We update our stuff on the fly, daily. But then, our systems are used internally. So as long as we don't cause a total production stoppage, having one form on one of the pages return an error instead of spitting back the expected report isn't going to cause the apocolypse. We do our best to be nimble, with the expecation that sometimes this causes rushed changes to break things.
Implement a Continuous Integration (CI) tech stack. The pieces of the stack are interchangeable depending on your technology. Look up Continuous Integration by Martin Fowler or the Wikipedia link or whatever to get your feet wet.
To properly implement a CI stack, you need these key pieces:
1.) A good SCM that works for your needs. Implement the best practices for your particular SCM. For instance, SVN has its own documentation that addresses recommended practices. I've noticed browsing forums that people tend to make proper, logical SCM something more complicated than it's meant to be. If you don't read and digest the docs and attain an understanding of the fundamentals, you end up in a lot of the snowballed situations you'll find on forums like stackoverflow.
2.) A good build tool. I primarily work with Java EE and Java SE, so Maven is my personal choice. If properly implemented along with a good project hierarchy, a build tool can make building 10-100 projects as simple as typing in a single command or clicking a button. Other tools that could be used are msbuild, make, bat scripts, rake, ant, etc. A great build tool like Maven can also output documentation such as Javadoc (API docs).
3.) A good Continuous Integration server. Opinions will vary on which CI server is better than the other. That all depends on your needs and development platform of course. I like Apache Continuum and Jenkins. But, there are other good ones too: bamboo, Hudson, Cruise Control, Anthill, BuildBot, TeamCity, Rational Team Concert, Team Foundation Server, Luntbuild, Vulcan, etc. Implementing a CI server will allow you to:
a.) Check in code to SCM, which, via something like a post-commit hook script in SVN, can
b.) Trigger your CI server to check out the project via some kind of script (Maven has the simple tag, so no script required)
c.) Build the project (or multi-module/nested projects via a root project)
d.) Output the results. Some CI servers like Jenkins allow you to see the build's progress in real-time in very nice presentation that management will understand and like.
e.) Notify a team if the build passed. If the build failed, the team is notified of who broke the build. Some businesses/teams take this one step further by hooking X10 devices up to the CI server, which allows you to use stoplights, glowing orbs, dart guns, etc. This technique is known as Extreme Notification.
f.) Send reports to interested/required parties. This can involve emails, IM, text, etc.
4.) An extra piece that I implement Maven repository manager, since I use Java. I use Apache Archiva as a gateway to download all my project's dependencies. I also house customized 3rd-party framework artifacts and in-house artifacts here too. This cuts down on bandwidth, download speed, use of unapproved artifacts. This also allows you to have your CI server publish your automatically-built artifacts and docs to the artifact repository. Once your artifacts have been published the CI server or the artifact repository or both can alert a team that new artifacts have been published. Archiva implements RSS to do this. From here, Q/A teams or testing programmers, programmers, etc. can grab these artifact snapshots or releases for whatever tasks they need to complete.
5.) The proper hardware to run your CI server. You need a box that can handle whatever load you're going to put on it. The build queue on a CI server can get quite large. Also, some CI servers can do distributed builds.
With all these pieces in place AND each implemented properly, you can ensure that your code meant for release is good. The fact that you can get management, coders, and Q/A rapid feedback during your entire build life-cycle is very powerful.
Major releases Quarterly. Anything delivered by vendors (outsourced) agencies, is weekly. The system I manage, we can push constantly. There might be days where I am making changes and pushing to production all day long, then days where I do nothing to production.
We are Ruby shop and push to production every 2 weeks. We are looking to make that shorter and do full blown continuous delivery. We have 95% coverage of the user interface automated with a lot of complicated scenarios. IMHO it is a must to automated tests to push to a feature rich application to production so often along with environments and automated continuous integration.
I would invite you to learn how Etsy releases multiple times per day:
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
There was a recent Wired mag. article about the guy at Facebook who does this. This way if you release dog food, only 1% of your customers have to go to the vet.
I come here for the love