Slashdot Mirror
How Do You Manage Dev/Test/Production Environments?
An anonymous reader writes "I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs through the different environments. I'm finding as our company grows (we currently host 50+ sites) it is cumbersome to manage all sites by hacking away at the command prompt. I would like to find a solution with a relatively easy-to-use user interface that provisions dev/test/live environments. The Aegir project is a close fit, but is only for Drupal sites and still under heavy development. Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
I do the same as Slashdot.org does - Make the changes on live code, except a little downtime and weird effects and then try to fix
it - while actually never fixing it. After all the results are not that significant:
- if someone posts about it on a thread, mods will -1 offtopic it and no one will hear your complain
- many people will "lol fail" at the weird effects, like when kdawson decides to merge two different stories together
You may be a new system administrator, but you are not a n00b.
A n00b wouldn't realize he was a n00b.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Create separate SVN deploys as separate environments. Deploy them as subdomains. If they require database access, create a test database they can share or separate test databases for each environment. Make sure the database class in the source is written as DB.bkp so when you deploy it, your deployed DB class won't be overwritten by changes to the source DB class.
This is my sig. There are many like it but this one is mine.
There's really only so much you can do generically. I'm really happy with phing. I use the dbdeploy task to keep my databases in a similar state. I build on a local machine, deploy via ssh and then migrate the database.
I'd suggest that rather than checkout at each level you create a continous integration machine using something like cruise control or bamboo, then push out build tarballs and migrate the database.
Never heard of a loghost eh?
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Still not figured out an efficient way to version MSSQL and MySQL databases using OSS, though. Open to suggestions!
sig:- (wit >= sarcasm)
If you're able to script deployments from a configuration management host you can script against your CVS (SVN, SourceSafe, whatever-you're-using).
There are a lot of ways to automate the management of what file version is in each environment but a smart choice is to tie things to an issue tracking system. My company uses MKS (http://mks.com) but BugTracker or BugZilla will do just as well.
Your scripted interface can check-out/export the specified version from controlled source and FTP/SFTP/XCOPY/whatever to the specified destination environment. For issue-tracker backed systems you can even have this processes driven by issue-id to automatically select the correct version based on issues to be elevated. Additionally, the closing task for the elevation process can then update the issue tracking system as needed.
Many issue tracking systems will allow you to integrate your source management and deployment management tools. It's a beautiful thing when you get it set up.
My office has been taken over by iPod people.
Most important thing is to treat your code and data separately.
Code:
Dev -> Test -> Production
Data:
Production -> Test -> Dev
Many developers forget to test and develop with real and current data, allowing problems to slip further downstream than they should.
And make sure you backup you Dev code and you Production Data.
If you are in the unix/linux world take a look at puppet. You provision out a set of nodes (allows node inheritance) and manage all your scripts, config files, etc from one central location (called the puppet master). Changes propagate to all servers that the change applied to automatically. It is built around keeping the configuration files in a versioned repository and is ready to use today.
We utilize a number of tools depending on the site, but generally:
Version Control (Subversion) for management of the code base (PHP, CSS, HTML, Ruby, PERL,...) - http://subversion.tigris.org/
BCFG2 for management of the system(s) patches and configurations (Uses svn for managing the files) - http://trac.mcs.anl.gov/projects/bcfg2
Capistrano/Webistrano for deployment (Webistrano is a nice GUI to capistrano - http://www.capify.org/ / http://labs.peritor.com/webistrano
However, all of the tools above mean nothing without defining very good standards and practices for your organization. Only you and your organization can figure those out...
It's hosted Subversion, with a slick web interface that walks you through darn near everything. You can configure development / test / production servers that can be accessed via FTP or SFTP and deploy new builds to any of them with just a couple of clicks. It integrates with Basecamp for project management, and it is really cheap - it sounds like either their Garden or Field plans would meet your needs, and they're both under $50/month.
Check them out here.
Not affiliated with them in any way, other than as a satisfied customer.
Blogging Weight Loss, Distance Education, and more at verlin.com
Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.
Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.
You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.
I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.
Its amazing, how this seemingly obvious question, always gets weird and overly complex answers.
Think about how every unix os handles this. Packaging!
Without getting into a flame war about the merits of any packaging systems:
- Use your native distributions packaging system.
- Create a naming convention for pkgs ( ie, web-fronted-php-1.2.4, web-prod-configs-1.27 )
- Use meta-packages ( packages, whose only purpose is to list out what makes out a complete systems )
- Make the developers package their software, or write scripts for them to do so easily ( this is a lot easier than it seems )
- Put your packages in different repositories ( dev for dev servers, stg for staging systems,qa for qa systems , prod for production, etc et c
- Use other system management tools to deploy said packages ( either your native package manager, or puppet, cfgengine, func, sshcmd scripts, etc )
And the pluses? you always know absolutely whats running on your system. You can always reproduce and clone a systems.
It takes discipline, but this is how its done in large environments.
-
I'm not aware of any systems that do all this for you while still being flexible enough to accommodate lots of unique requirements. The script-based systems I've employed over the years all followed some basic rules:
I've successfully managed hundreds of sites and web apps in this manner, with minimal fuss. Virtualization adds in extra complexity in some respects, but makes other things easier as each VPS can have its own customized environment. As long as everything in dev, test, and prod uses the same base VPS environment, problems should be minimal.
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
Develop 4 Environment Structures
Development (DEV)
Integration Testing (INTEG)
Acceptance (ACPT)
Production (PROD)
For each system create a migration script that generically does the following:
(We will use SOURCE and DEST for environments. You migrate from DEV->INTEG->ACPT->PROD)
The migration script as it's core does the following:
1) STOP Existing Services and Databases (SOURCE and DEST)
2) BUILD your deployment package from SOURCE (This means finalizing commits to an SVN, Creating a dump of SOURCE databases etc.) If this is a long process then you can leave the DEST running and STOP DEST at the end of the build phase. I do this as builds for my world can take 2-3 days.
3) CONDITION your deployment package to be configured for DEST environment (simple find and replace scripts to correct database names, IP address, etc. These should be config files that are read and processes.) This is common if there are different security SAPs, Certificates, Etc that need to be configured. For instance you may not have SSL enabled in DEV but you might in INTEG or ACPT.
4) BACKUP DEST information as an install package(this is identical to the BUILD done on the source. This BACKUP can be deployed to restore the previous version.) This should be the same function you ran on SOURCE with a different destination (say Backups verus Deploys)
5) MIGRATE the install package from SOURCE to DEST
START DEST
6) TEST TEST and RETEST
7) If all tests pass then APPROVE. This is the green light to re-start the SOURCE services so development can move on.
That is a brief of my suggestion.
DEV is obvious ... yeah...
INTEG is where you look for defects and resolve defects. Primary testing.
ACPT is where user and BL acceptance testing occurs and should mirror PROD in services available.
PROD
I handle about 790+ applications across 2000+ pieces of hardware so this may appear to be overkill for some but it can be as simple as 4 running instances on a single box with a /DEV/ /IT/ /ACPT/ /PROD/
Directory structure with MYSQL running 4 different databases. The "Script" could be as simple as dropping the DEST database and copying the SOURCE database with a new name. Other options are creating modification SQLS for instance that are applied onto the exist database.
e.g. STOP, UPDATE, START
to preserve existing data. In the case of Drupal your DEV might pull a nightly build and kick out a weekly IT, a biweekly ACPT, and a monthly PROD update.
JUST REMEMBER THAT YOU MUST MAKE SURE THE PROCESS IS ALWAYS REVERSABLE!!
The script to deploy needs to handle failure. There has to be a good backout.
You should have a method to backup and restore the current state. Integrate that into the script. Always backup Before you do changes and AGAIN after you change. DEV may need to look at the failed deploy data (perhaps a substitution or patch failed, they need to find out why.)
Before Backup and After Backup in the migration script.
And always 'shake out' a deployment in each environment level to make sure problems to propogate. You find problems in IT, you test to make sure what you found in IT is resolved in ACPT. Your testers should NOT normally be finding and filing new defects in ACPT environments with the exception of inter-application communication that might not be available in earlier environments. (Great example might be ACPT has the ability to connect to say a marketing companies databases where you use dummy databases in IT and DEV.) 80/20 is the norm for IT/ACPT that I see.
Good luck. Use scripts that are consistent and invest in a good migration method. It works great for mainframes and works great in the distributed world too.
A special condition is needed for final production as you may need temporary redirects to be applied for online services (commonly called Gone Fishing pages or Under Construction Redirects)
-=[ Who Is John Galt? ]=-
Just roll them into one. It's even got a catchy name.
Where eln failed is in how his post turned into nothing more than a personal attack against the parent story poster. In the open source world there are users, documenters, developers and visionaries. And guess what - a majority of those are users and most users will never contribute to your project.
Simply attacking the guy with crass, harsh statements is not in the vein of "The Gift Culture."
So, yes, eln's comment is a troll comment.
As for a "moral obligation?" That's laughable. If you give someone something for free don't expect them to do anything for you. Maybe that person doesn't have the time to invest in giving back at the moment. Making inflammatory comments will certainly push them away from your base of constituents. And that means less users. So attacking people who don't know is counter-productive and does not serve the OSS causes or beliefs.
If the someone feels to compelled to "give" or "give back" to the open source community - in whatever manner - count the community fortunate. Expecting anything is counter to the ideals of "The Gift Culture."
Please reread ESR's book.
I have adapted my system from the 5 years I professionally did it.
First of all, it's a 3-stage system.
You have a couple of live servers, a identical staging server, and the user machines.
Every system has a clone of the files. the servers have rsync copies of the stage server files.
And the users all sync to the stage with GIT.
Everyone has a local clone of the stage server software too, so he can test server-side code right on his machine.
That's important in every company where people could do conflicting (and even big, global) patches.
The stage server then has validity tests running. Compilations and unit test cases wherever possible. Including the database, the server side code, and rendering test pages in all relevant browsers, to diff the rendered versions (images) of the pages. (There's a app for that in Firefox, but otherwise it's desktop automation.)
There's an red alert box in the test case overview when something fails. Which gets checked every evening, before pushing anything to the live servers at night.
The only thing that turns out to be a bit hard, is to test the client-side logic (e.g. of web-apps) in a transparent manner (= keeping the software configurations and serveride code the same to be able to rely on it).
Then there is a emergency push and and emergency direct live update mechanism, for cases when you quickly have to fix something that got overlooked. (Which usually should result in a new test case to be written, to catch all such problems.)
A well-integrated project management system is very important. At the end of my first company, it was a self-written one with good integration. But in the beginning, something like Trac might suffice.
Then very important is, to have a knowledge base for all things that need to be remembered. Like a meta-documentation. Workflows and procedures. Why the mysql server will not restart on a reboot of stage server clones. Little hooks and mantraps like that. I recommend a Wiki.
And last but not least, never ever forget to have a Bugzilla. If you're good, you can integrate Bugzilla, the test validations and the task/project management into one system. Making the validity tests create bugs in Bugzilla, and bugs being the same as tasks (which makes test-driven development easier).
Yet this all is completely worthless, if your colleagues don't use it! ;) ...". It's the mother of all PHB stories. ^^
Unfortunately, I learned, that when someone can/em> do something wrong, he will.
So if you can't lock down possibilities to only those required, you have to be very very careful with who you hire. Especially with "web development", where you get sinology students who learned HTML while working as a taxi driver, stating that they are "professional web developers with 5 years of experience", while honestly believing that. And team leaders believing it too, because they are just as "competent". Because they themselves either started as something an simple as link collectors, or the boss of the company does not know shit about his business, and hired those types. They then usually get promoted to "Head of
The key is: Make them like to work the proper way. If nothing helps, money can always push them in the right direction. It's called "bonus". :)
And making it their project too, by also embracing their decisions!
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Basically, the writer said he wanted to "give back" by using the software.
That is an interesting take on what what "I would much rather use something OSS so I can give back to the community" means. Sure, it could mean what you said, but I suspect it was meant more in the spirit of having the opportunity to report any issues with the program back to the author(s). Perhaps I am reading into his statement as well...I don't know. What I do know is that he clearly hit one of your hot buttons.