How Do You Manage Dev/Test/Production Environments?

An anonymous reader writes "I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs through the different environments. I'm finding as our company grows (we currently host 50+ sites) it is cumbersome to manage all sites by hacking away at the command prompt. I would like to find a solution with a relatively easy-to-use user interface that provisions dev/test/live environments. The Aegir project is a close fit, but is only for Drupal sites and still under heavy development. Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"

Comments

How slashdot does it

[sopssa]

How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"

I do the same as Slashdot.org does - Make the changes on live code, except a little downtime and weird effects and then try to fix

it - while actually never fixing it. After all the results are not that significant:

- if someone posts about it on a thread, mods will -1 offtopic it and no one will hear your complain
- many people will "lol fail" at the weird effects, like when kdawson decides to merge two different stories together

Re:How slashdot does it

[cayenne8]

"I do the same as Slashdot.org does - Make the changes on live code, except a little downtime and weird effects and then try to fix"

That's not that far from the truth in MANY places and projects I've seen.

I've actually come to the conclusion, that on many govt/DoD projects, that the dev. environment in fact becomes the test and production environment!!

I learned that it really pays, when spec'ing out the hardware and software that you need, to get as much as they will pay for for the 'dev' machines....because, it will inevitably become the production server as soon as stuff is working on it, the deadline hits, and there is suddenly no more funding for a proper test/prod environment.

You are not a n00b

[davidwr]

You may be a new system administrator, but you are not a n00b.

A n00b wouldn't realize he was a n00b.

Re:You are not a n00b

[hydroponx]

OR they're Link.....

Re:You are not a n00b

[Eil]

If he is indeed allowing FTP logins over the public Internet (as the submission suggests), he is a n00b whether or not he realizes it.

Re:You are not a n00b

[nametaken]

Or maybe they're not over public internet, or are tunneled, or they're sftp and he just calls it ftp? We dunno.

Separate SVN deploys

[Foofoobar]

Create separate SVN deploys as separate environments. Deploy them as subdomains. If they require database access, create a test database they can share or separate test databases for each environment. Make sure the database class in the source is written as DB.bkp so when you deploy it, your deployed DB class won't be overwritten by changes to the source DB class.

Re:Separate SVN deploys

[Foofoobar]

Git does not have integration with Apache and other tools that developers still find useful. TRAC integrates with Subversion as do several other tool. You also cannot coordinate Git with your IDE. Don't get me wrong, it is definitely where version control will be in the future but the tools to support it have to get there first before widespread adoption should be advised for day to day use.

Re:Separate SVN deploys

[AlXtreme]

Subversion's security models in UNIX and Linux are exceptionally poor, and typically wind up storing passwords in clear-text without properly notifying you.

Auth token caching can be easily disabled and svn export, not svn checkout, should be used for deploying test/prod environments (like I've seen way too many people do).

Git (or any other distributed version control system) is great if you are into distributed development, but don't blame the tool when you don't know how to use it properly or expect it to be something that it's not.

Re:Separate SVN deploys

[garaged]

http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt
trac does have git support too, etcetera

Re:Separate SVN deploys

[nahdude812]

svn+ssh doesn't store anything in clear text. If that's a security concern for you, there's already a solution in place. Git is not the be-all and end-all solution to source control; it does many things very well, but there are a few things it does very poorly (repository control; with git, developers have a local copy of the repository which means that a stolen laptop comes with complete revision history). When the systems you're working on have certain sensitivities (legal, patent, security, etc), this can be a major weakness.

We do something very similar to the original submitter at work. We have 10-15 project branches open at a time. For us, we make sure that our code is subdirectory-agnostic (meaning it can run on the root of the website, or it can run out of a subdirectory). We use directory paths for branches, and we use internal DNS records for environments. http://de-appname/branchname would be a development branch while http://va-appname/branchname would be a validation branch.

For our lifecycle, we have development, validation, staging, pre production, and production. Development and validation are the only branching locations; staging, pre, and production are each single-path locations (though staging is a branch reserved for this purpose, pre production and production are /trunk)

On http://de-appname/ and http://va-appname/ there is essentially a directory listing along with the fully qualified branch name, revision, most recent contributor's name (even spelled out by looking up their record in LDAP), commit time, and most recent log message. Developers get a drop-down menu next to project branches which they can use to update the working copy there on that shared server (does a little ajax call and shows you the result in real time as though you were at a terminal). You can also create clean checkouts and even create new branches (either off of trunk or off of another branch). Finally you can even close a branch through this interface; it deletes the branch with a meaningful log message, and cleans up the files on that server. All through a web interface, no need to remotely log into a machine for this purpose. There's no reason for someone to administer this, each developer creates a working copy when and where he thinks it makes sense for himself.

Because our back end is SAP, we don't have to deal with multiple database environments. There's no "create a new copy of SAP" - indeed when this is something that's organizationally important (testing a major upgrade), it's a multiple day long process. We have a fixed set of data environments, and these are tied to hostname (going back to de-appname, va-appname, stage-appname, etc). If there were multiple database environments to worry about (eg, you wanted to be able to effectively branch database environments too), it wouldn't be a huge deal to set up a template database and have the same scripts we use to manage branches through a web API clone that database and update config in the appropriate app.

The key that I'm trying to get at is that you should create a web based tool to allow developers to manage this themselves. The developers will thank you because they'll be able to get what they want faster than you could have provided it, and they'll have control over when where and why working copies of their work in progress appear.

Re:Have the hosts email problems to an email accou

[BitZtream]

Never heard of a loghost eh?

Leverage your issue tracking and cvs

[dkh2]

If you're able to script deployments from a configuration management host you can script against your CVS (SVN, SourceSafe, whatever-you're-using).

There are a lot of ways to automate the management of what file version is in each environment but a smart choice is to tie things to an issue tracking system. My company uses MKS (http://mks.com) but BugTracker or BugZilla will do just as well.

Your scripted interface can check-out/export the specified version from controlled source and FTP/SFTP/XCOPY/whatever to the specified destination environment. For issue-tracker backed systems you can even have this processes driven by issue-id to automatically select the correct version based on issues to be elevated. Additionally, the closing task for the elevation process can then update the issue tracking system as needed.

Many issue tracking systems will allow you to integrate your source management and deployment management tools. It's a beautiful thing when you get it set up.

Most important thing in my book

[BlueBoxSW.com]

Most important thing is to treat your code and data separately.

Code:

Dev -> Test -> Production

Data:

Production -> Test -> Dev

Many developers forget to test and develop with real and current data, allowing problems to slip further downstream than they should.

And make sure you backup you Dev code and you Production Data.

Re:Most important thing in my book

[zztong]

Testing with real data is not necessarily a good practice. Consider sensitive data, such as social security numbers. Auditors may ding your development practices for providing developers access to information they do not need. You need realistic data, not necessarily the real data. If you're bringing real data from prod back to test and dev, consider having something scrub the data.

Tools, Practices and Standards

[HogGeek]

We utilize a number of tools depending on the site, but generally:

Version Control (Subversion) for management of the code base (PHP, CSS, HTML, Ruby, PERL,...) - http://subversion.tigris.org/
BCFG2 for management of the system(s) patches and configurations (Uses svn for managing the files) - http://trac.mcs.anl.gov/projects/bcfg2
Capistrano/Webistrano for deployment (Webistrano is a nice GUI to capistrano - http://www.capify.org/ / http://labs.peritor.com/webistrano

However, all of the tools above mean nothing without defining very good standards and practices for your organization. Only you and your organization can figure those out...

Check out Springloops

[Fortunato_NC]

It's hosted Subversion, with a slick web interface that walks you through darn near everything. You can configure development / test / production servers that can be accessed via FTP or SFTP and deploy new builds to any of them with just a couple of clicks. It integrates with Basecamp for project management, and it is really cheap - it sounds like either their Garden or Field plans would meet your needs, and they're both under $50/month.

Check them out here.

Not affiliated with them in any way, other than as a satisfied customer.

Look at Capistrano, steal ideas from Rails

[bokmann]

Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.

Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.

You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.

I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.

Packaging Packaging Packaging...

[keepper]

Its amazing, how this seemingly obvious question, always gets weird and overly complex answers.

Think about how every unix os handles this. Packaging!

Without getting into a flame war about the merits of any packaging systems:

- Use your native distributions packaging system.
- Create a naming convention for pkgs ( ie, web-fronted-php-1.2.4, web-prod-configs-1.27 )
- Use meta-packages ( packages, whose only purpose is to list out what makes out a complete systems )
- Make the developers package their software, or write scripts for them to do so easily ( this is a lot easier than it seems )
- Put your packages in different repositories ( dev for dev servers, stg for staging systems,qa for qa systems , prod for production, etc et c
- Use other system management tools to deploy said packages ( either your native package manager, or puppet, cfgengine, func, sshcmd scripts, etc )

And the pluses? you always know absolutely whats running on your system. You can always reproduce and clone a systems.

It takes discipline, but this is how its done in large environments.

-

Quick Brief

[kenp2002]

Develop 4 Environment Structures

Development (DEV)
Integration Testing (INTEG)
Acceptance (ACPT)
Production (PROD)

For each system create a migration script that generically does the following:
(We will use SOURCE and DEST for environments. You migrate from DEV->INTEG->ACPT->PROD)

The migration script as it's core does the following:

1) STOP Existing Services and Databases (SOURCE and DEST)

2) BUILD your deployment package from SOURCE (This means finalizing commits to an SVN, Creating a dump of SOURCE databases etc.) If this is a long process then you can leave the DEST running and STOP DEST at the end of the build phase. I do this as builds for my world can take 2-3 days.

3) CONDITION your deployment package to be configured for DEST environment (simple find and replace scripts to correct database names, IP address, etc. These should be config files that are read and processes.) This is common if there are different security SAPs, Certificates, Etc that need to be configured. For instance you may not have SSL enabled in DEV but you might in INTEG or ACPT.

4) BACKUP DEST information as an install package(this is identical to the BUILD done on the source. This BACKUP can be deployed to restore the previous version.) This should be the same function you ran on SOURCE with a different destination (say Backups verus Deploys)

5) MIGRATE the install package from SOURCE to DEST
START DEST

6) TEST TEST and RETEST

7) If all tests pass then APPROVE. This is the green light to re-start the SOURCE services so development can move on.

That is a brief of my suggestion.

DEV is obvious
INTEG is where you look for defects and resolve defects. Primary testing.
ACPT is where user and BL acceptance testing occurs and should mirror PROD in services available.
PROD ... yeah...

I handle about 790+ applications across 2000+ pieces of hardware so this may appear to be overkill for some but it can be as simple as 4 running instances on a single box with a /DEV/ /IT/ /ACPT/ /PROD/

Directory structure with MYSQL running 4 different databases. The "Script" could be as simple as dropping the DEST database and copying the SOURCE database with a new name. Other options are creating modification SQLS for instance that are applied onto the exist database.

e.g. STOP, UPDATE, START

to preserve existing data. In the case of Drupal your DEV might pull a nightly build and kick out a weekly IT, a biweekly ACPT, and a monthly PROD update.

JUST REMEMBER THAT YOU MUST MAKE SURE THE PROCESS IS ALWAYS REVERSABLE!!

The script to deploy needs to handle failure. There has to be a good backout.

You should have a method to backup and restore the current state. Integrate that into the script. Always backup Before you do changes and AGAIN after you change. DEV may need to look at the failed deploy data (perhaps a substitution or patch failed, they need to find out why.)

Before Backup and After Backup in the migration script.

And always 'shake out' a deployment in each environment level to make sure problems to propogate. You find problems in IT, you test to make sure what you found in IT is resolved in ACPT. Your testers should NOT normally be finding and filing new defects in ACPT environments with the exception of inter-application communication that might not be available in earlier environments. (Great example might be ACPT has the ability to connect to say a marketing companies databases where you use dummy databases in IT and DEV.) 80/20 is the norm for IT/ACPT that I see.

Good luck. Use scripts that are consistent and invest in a good migration method. It works great for mainframes and works great in the distributed world too.

A special condition is needed for final production as you may need temporary redirects to be applied for online services (commonly called Gone Fishing pages or Under Construction Redirects)

Re:Mod Parent Up!

[Like2Byte]

Where eln failed is in how his post turned into nothing more than a personal attack against the parent story poster. In the open source world there are users, documenters, developers and visionaries. And guess what - a majority of those are users and most users will never contribute to your project.

Simply attacking the guy with crass, harsh statements is not in the vein of "The Gift Culture."

So, yes, eln's comment is a troll comment.

As for a "moral obligation?" That's laughable. If you give someone something for free don't expect them to do anything for you. Maybe that person doesn't have the time to invest in giving back at the moment. Making inflammatory comments will certainly push them away from your base of constituents. And that means less users. So attacking people who don't know is counter-productive and does not serve the OSS causes or beliefs.

If the someone feels to compelled to "give" or "give back" to the open source community - in whatever manner - count the community fortunate. Expecting anything is counter to the ideals of "The Gift Culture."

Please reread ESR's book.