Slashdot Mirror
How Do You Manage Dev/Test/Production Environments?
An anonymous reader writes "I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs through the different environments. I'm finding as our company grows (we currently host 50+ sites) it is cumbersome to manage all sites by hacking away at the command prompt. I would like to find a solution with a relatively easy-to-use user interface that provisions dev/test/live environments. The Aegir project is a close fit, but is only for Drupal sites and still under heavy development. Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
I do the same as Slashdot.org does - Make the changes on live code, except a little downtime and weird effects and then try to fix
it - while actually never fixing it. After all the results are not that significant:
- if someone posts about it on a thread, mods will -1 offtopic it and no one will hear your complain
- many people will "lol fail" at the weird effects, like when kdawson decides to merge two different stories together
You may be a new system administrator, but you are not a n00b.
A n00b wouldn't realize he was a n00b.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Create separate SVN deploys as separate environments. Deploy them as subdomains. If they require database access, create a test database they can share or separate test databases for each environment. Make sure the database class in the source is written as DB.bkp so when you deploy it, your deployed DB class won't be overwritten by changes to the source DB class.
This is my sig. There are many like it but this one is mine.
There's really only so much you can do generically. I'm really happy with phing. I use the dbdeploy task to keep my databases in a similar state. I build on a local machine, deploy via ssh and then migrate the database.
I'd suggest that rather than checkout at each level you create a continous integration machine using something like cruise control or bamboo, then push out build tarballs and migrate the database.
I use a config file that sets a global variable that all scripts reference to determine what site (dev/test/prod) the code is running on. I also use CVS and RPM to handle code management and pushes.
When I did this years ago, each server would run scripts to read logs, etc and if they found something bad they would email me with what they found.
Simple and scalable
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
Not sure if this is what you're looking for, but a common solution is a configuration management tool.
Try puppet http://reductivelabs.com/products/puppet
It's simple, fast, and written in ruby
We do it the unix way: duct tape svn, sqsh, rsync, and sendmail together with shell scripts. Reconciliation of what went where can be a little hairy, documentation is sparse, and some safeguards I'd like to see are not there, but it's a good base. I'm actually taking a break from documenting the deployment system at this very moment...
Have a perfect virtual machine image ready. You can bring up a new server in about 5 minutes.
Put all of your application workspaces in an easily escapable situation involving an overly elaborate and exotic death.
But, if in doubt, add laser beams.
Never heard of a loghost eh?
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
/sarcasm (in case you couldn't tell)
If there's not a project to fit your bill, develop it internally and release it as an OSS project. It'll add some nice OSS experience to your resume and also add visibility to your employer. If it succeeds, it'll be a big deal for your company. If it doesn't succeed, at least you got the project done. Sounds like everyone wins.
I've never actually done this (my employer balks at the suggestion), but I'd love to have that sort of opportunity.
I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs [...]
If you have a WYSIWYG front end done DIY style then you need to CYA and RTFM, simply because the newer style AJAX IDEs dont support IDEA. Make sure that you mind your P's and Q's, or the FBI will make you MIA thanks to the P.A.T.R.I.O.T. act. It's pretty much a PEBKAC issue. Oh, did I mention that you should leverage as many TLA's as possible?
Still not figured out an efficient way to version MSSQL and MySQL databases using OSS, though. Open to suggestions!
sig:- (wit >= sarcasm)
If you're able to script deployments from a configuration management host you can script against your CVS (SVN, SourceSafe, whatever-you're-using).
There are a lot of ways to automate the management of what file version is in each environment but a smart choice is to tie things to an issue tracking system. My company uses MKS (http://mks.com) but BugTracker or BugZilla will do just as well.
Your scripted interface can check-out/export the specified version from controlled source and FTP/SFTP/XCOPY/whatever to the specified destination environment. For issue-tracker backed systems you can even have this processes driven by issue-id to automatically select the correct version based on issues to be elevated. Additionally, the closing task for the elevation process can then update the issue tracking system as needed.
Many issue tracking systems will allow you to integrate your source management and deployment management tools. It's a beautiful thing when you get it set up.
My office has been taken over by iPod people.
Ant.
Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
I'm sure you have a legitimate problem, and there are lots of ways to solve it, but this line just cracks me up. You COULD write it yourself or pay someone but if you use someone else's Open Source work (note: nothing is said about contributing to an OSS project, just using it) you'd be "giving back to the community.
Translation: I have a problem, and I don't want to spend any of my own time or money to solve it, so I'm going to try and butter up the people on Slashdot in hope of taking advantage of the free labor force that is the OSS community.
Simply using Open Source software is not giving back to the community...using open source software is what gives you the moral imperative to give back to the community, which you can do through contributing code, documentation, beta testing, providing support on the mailing lists, or whatever.
Or I would if I were in management. For some reason they won't promote me here.
Best Slashdot Co
Most important thing is to treat your code and data separately.
Code:
Dev -> Test -> Production
Data:
Production -> Test -> Dev
Many developers forget to test and develop with real and current data, allowing problems to slip further downstream than they should.
And make sure you backup you Dev code and you Production Data.
If you are in the unix/linux world take a look at puppet. You provision out a set of nodes (allows node inheritance) and manage all your scripts, config files, etc from one central location (called the puppet master). Changes propagate to all servers that the change applied to automatically. It is built around keeping the configuration files in a versioned repository and is ready to use today.
We utilize a number of tools depending on the site, but generally:
Version Control (Subversion) for management of the code base (PHP, CSS, HTML, Ruby, PERL,...) - http://subversion.tigris.org/
BCFG2 for management of the system(s) patches and configurations (Uses svn for managing the files) - http://trac.mcs.anl.gov/projects/bcfg2
Capistrano/Webistrano for deployment (Webistrano is a nice GUI to capistrano - http://www.capify.org/ / http://labs.peritor.com/webistrano
However, all of the tools above mean nothing without defining very good standards and practices for your organization. Only you and your organization can figure those out...
If you have some $$ off the shelf tools like vmware lab manager will kick some serious butt in this type of environment.
It's hosted Subversion, with a slick web interface that walks you through darn near everything. You can configure development / test / production servers that can be accessed via FTP or SFTP and deploy new builds to any of them with just a couple of clicks. It integrates with Basecamp for project management, and it is really cheap - it sounds like either their Garden or Field plans would meet your needs, and they're both under $50/month.
Check them out here.
Not affiliated with them in any way, other than as a satisfied customer.
Blogging Weight Loss, Distance Education, and more at verlin.com
Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.
Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.
You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.
I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.
. . . everything else comes after that. A small illustration:
When I was system admin for a small brokerage, one of my first tasks was to determine the hardware configuration of every server. There was one particular server that I needed to shutdown in the process. I asked every employee (it was that small) if there were any critical services on that machine. All agreed it was ok to take it off line. For the next 15 minutes, while the machine rebooted, no trading happened because the main program was linking to some libraries that were served off of that server.
I immediately put a new task at the top of my to-do list: reconfiguring the network. Thereafter, production was done on one network and development on another. The router between them would not allow nfs mounts. Production users were not given accounts on development machines. Developers were no longer given the root password, but it was kept in a safe for emergencies.
I know that wasn't what you were asking, but that is the first thing I would take care of.
I do mine with ssh, bash and git, for the moment. I'm looking at moving to something like puppet for system configuration, though. I've also heard good things about cobbler for initial provisioning, but it's mainly aimed at an RHEL environment and that's not what we're using.
The 2009 Utah Open Source Conference had several good presentations on infrastructure automation. See, in particular, Phil Windley's slides on puppet and cobbler (hopefully audio and maybe video will be available soon).
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
The trick then is to move on to another outfit just before it hits the fan. Don't worry about your customers - if they are running web based businesses, chances are most of them will have gone down the tubes in a year or so. Long before they get anywhere near release 1.0.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
People are supposed to TEST this stuff first!?
Did he forget the Sarcasm Mark ~, or does he not know about it?
TPS reports with lots of cover letters.
that /. allowed religious discussions
Its amazing, how this seemingly obvious question, always gets weird and overly complex answers.
Think about how every unix os handles this. Packaging!
Without getting into a flame war about the merits of any packaging systems:
- Use your native distributions packaging system.
- Create a naming convention for pkgs ( ie, web-fronted-php-1.2.4, web-prod-configs-1.27 )
- Use meta-packages ( packages, whose only purpose is to list out what makes out a complete systems )
- Make the developers package their software, or write scripts for them to do so easily ( this is a lot easier than it seems )
- Put your packages in different repositories ( dev for dev servers, stg for staging systems,qa for qa systems , prod for production, etc et c
- Use other system management tools to deploy said packages ( either your native package manager, or puppet, cfgengine, func, sshcmd scripts, etc )
And the pluses? you always know absolutely whats running on your system. You can always reproduce and clone a systems.
It takes discipline, but this is how its done in large environments.
-
Fabric.
http://www.gjcp.net/articles/fabric/
Saves so much hassle and buggering about.
Deploy a continuous build server. You've already done most of the work by writing all of these scripts. It will transfer nicely over to Hudson or maybe CruiseControl. As a bonus, if you guys use Selenium tests, you can automate those too.
Also, get a ticketing system (Jira/Trac/whatever) with a configurable workflow that you're comfortable with so you can track deployments and approvals/disapprovals. The workflow should be configurable because you never know if you'll start adding environments or release steps (your company's packaged releases won't really get deployed to production as the final step or you might add a customer environment).
Finally print out instructions for the developers and post them on a wall or wiki where they are plainly visible. It's all useless if you're the only one who knows how the system works.
OSS CMS !? Call me old fashioned (and it won't be the first time), but I will use plain HTML, thank you.
Carefully. And you?
"But this one goes to 11!"
Ever heard of Unixware!
remember this WAS 10 years ago!
(Now, get off my lawn!)
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
CVS (of whatever flavor) can help you do this. It's a pain in the ass, and everybody will hate it, but it works.
I've done this with virtual machines as well. It's kinda whizzy to do, but probably overkill.
The simplest way for me was to simply use rsync. Rigid delineation between live and test/dev environments is important. Use a completely separate database (not just a different schema), and if possible a completely separate database server. Changes to the database schema should be encapsulated in update scripts and tightly controlled and thoroughly tested in the test environment. Use a database that supports transactions and use them. Updating the live site should be performed by updating a clone of the live site in another directory. That way if everything goes tits up for some unexpected reason you can revert back to the old site while you lick your wounds. Virtual machines definitely make this all techy and bitchin', but editing httpd.conf and restarting Apache also works.
The best solution is going to be customized to the needs of the project. Most projects don't need a dev/test/live arrangement. Dev and live are sufficient. The most important thing is to establish a framework of how changes are to be made to the code base or database, and stick to it. CVS will help enforce this, but at the cost of having to use CVS.
Potato chips are a by-yourself food.
How can this be a troll? This is so true. I would however grant the extending the user base for an OSS project isn't exactly a bad thing, nor is having your company exposed to more OSS - but I sure would call it "giving back to the community".
Never let a mediocre career stand in the way of a good time
The simple answer? Virtual Machines. If you have to stay with linux, go with vmware or for a free solution, KVM. See http://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
If you want to run LAMP on open solaris/solaris, ZFS has very robust and easy to manage virtual machines called zones. Sun also provides enterprise ops center software that can be used to manage the zones via a gui. Copy/create/rollback, etc..
After that, smart system administration is required to keep things easy to manage.
How you choose to separate data, apps, and the OS will depend on your requirements, but in general, keeping them separate is a good idea.
Another good idea, is a /mnt/safe or area mounted inside the prod/test/dev boxes that is nfs shared between them. Often times, I'll make a request of my sys admin "Please refresh test and dev with prod". So I copy changes or other work to /mnt/safe and then he overwrites dev and test with a recent zfs snapshot (virtual machine snapshot) of production.
I see you use the word check-in/out. I'm assuming you have subversion or something similar running that you use to check in/out to a new location. Do your developers need access to a CVS? If so, I'd just build it into the virtual machine, so each developer has their own subversion installation.
The only thing you need to do when using zones/virtual machines (at least in zfs) is change the hostname and IP, but that is easily scripted.
That's interesting. At the moment we have a loghost, and all logs of all applications go to that syslog server. Now we face the problem of allowing access to those logs to developers. Say you have 50 production apps logging to that logserver, do you know some software (best would be a webapp) that can be configured to let developers login and see the logs for the application they are responsible for? We could simply share the log files with a samba share, but a webapp that has some kind of integrated tail, deep linking to specific lines, color highlighting and stuff like that would be über cool.
I just don't trust anything that bleeds for five days and doesn't die.
Develop 4 Environment Structures
Development (DEV)
Integration Testing (INTEG)
Acceptance (ACPT)
Production (PROD)
For each system create a migration script that generically does the following:
(We will use SOURCE and DEST for environments. You migrate from DEV->INTEG->ACPT->PROD)
The migration script as it's core does the following:
1) STOP Existing Services and Databases (SOURCE and DEST)
2) BUILD your deployment package from SOURCE (This means finalizing commits to an SVN, Creating a dump of SOURCE databases etc.) If this is a long process then you can leave the DEST running and STOP DEST at the end of the build phase. I do this as builds for my world can take 2-3 days.
3) CONDITION your deployment package to be configured for DEST environment (simple find and replace scripts to correct database names, IP address, etc. These should be config files that are read and processes.) This is common if there are different security SAPs, Certificates, Etc that need to be configured. For instance you may not have SSL enabled in DEV but you might in INTEG or ACPT.
4) BACKUP DEST information as an install package(this is identical to the BUILD done on the source. This BACKUP can be deployed to restore the previous version.) This should be the same function you ran on SOURCE with a different destination (say Backups verus Deploys)
5) MIGRATE the install package from SOURCE to DEST
START DEST
6) TEST TEST and RETEST
7) If all tests pass then APPROVE. This is the green light to re-start the SOURCE services so development can move on.
That is a brief of my suggestion.
DEV is obvious ... yeah...
INTEG is where you look for defects and resolve defects. Primary testing.
ACPT is where user and BL acceptance testing occurs and should mirror PROD in services available.
PROD
I handle about 790+ applications across 2000+ pieces of hardware so this may appear to be overkill for some but it can be as simple as 4 running instances on a single box with a /DEV/ /IT/ /ACPT/ /PROD/
Directory structure with MYSQL running 4 different databases. The "Script" could be as simple as dropping the DEST database and copying the SOURCE database with a new name. Other options are creating modification SQLS for instance that are applied onto the exist database.
e.g. STOP, UPDATE, START
to preserve existing data. In the case of Drupal your DEV might pull a nightly build and kick out a weekly IT, a biweekly ACPT, and a monthly PROD update.
JUST REMEMBER THAT YOU MUST MAKE SURE THE PROCESS IS ALWAYS REVERSABLE!!
The script to deploy needs to handle failure. There has to be a good backout.
You should have a method to backup and restore the current state. Integrate that into the script. Always backup Before you do changes and AGAIN after you change. DEV may need to look at the failed deploy data (perhaps a substitution or patch failed, they need to find out why.)
Before Backup and After Backup in the migration script.
And always 'shake out' a deployment in each environment level to make sure problems to propogate. You find problems in IT, you test to make sure what you found in IT is resolved in ACPT. Your testers should NOT normally be finding and filing new defects in ACPT environments with the exception of inter-application communication that might not be available in earlier environments. (Great example might be ACPT has the ability to connect to say a marketing companies databases where you use dummy databases in IT and DEV.) 80/20 is the norm for IT/ACPT that I see.
Good luck. Use scripts that are consistent and invest in a good migration method. It works great for mainframes and works great in the distributed world too.
A special condition is needed for final production as you may need temporary redirects to be applied for online services (commonly called Gone Fishing pages or Under Construction Redirects)
-=[ Who Is John Galt? ]=-
I would suggest using some virtualization technologies for that. Something that would make it easy to deploy multiple copies of the same template, easily manage different large scale architectures, and such. I have personally used 3tera's AppLogic, and have had a lot of great experiences there. With a few physical servers you can manage multiple separate VM's, create templates, automate functionality... blah blah. Good luck finding the best solution for you though.
Just roll them into one. It's even got a catchy name.
Web devs need to have security enforced or they won't think about it for their sites. Shut off FTP and enforce SFTP only. If bandwidth is a factor is choosing FTP over SFTP, at the very least, use kerberized FTP. Make certain that phpMyAdmin is behind https and that authentication is required. Yes, this means they have to use two passwords. Tough.
I'm just now reading "Pro PHP Security" (Snyder & Southwell, Apress), and it's got a lot of good information - hands-on examples, best practices and technical background that is useful whether you support PHP or not. It covers both local and web-based attacks such as XSS, SQL injection, vuln exploits, etc.
Among other things, it suggests you set up virtual servers for each domain user. You could use FreeBSD 'jails', linux virtualization tools, etc. - the book is agnostic on which ones you use, and doesn't cover a lot of detail in this area, at least so far. Virtualization of this type almost completely eliminates the ability of any client user accessing anything outside their virtual server space.
It also suggests that you automate the creation of new domains and hosts, with a form-entry input of some kind, perhaps a well-secured web-based front end, that assures everything gets done properly. Such a parametric front end (of whichever type) helps in preventing the sysadmin from forgetting or purposely ignoring certain setup tasks. I have not kept up to date in this area, because it's not something I do any more, but I'm sure there are some packages that do most or all this work.
You might even use webmin for some of this.
I also have on my shelf the Cisco book "Data Center Fundamentals" available directly from Cisco. It's $60 but has a slew of information.
I'm sure there are other books with more information, I just don't have them off the top of my head.
It's easier to be a result of the past, but more fun to be a cause of the future! http://www.spacefinancegroup.com/
Segue in to a new paradigm and experience increased synergy - consolidate already!
Kidding, naturally.
www.isoHunt.com
Try splunk. This should do exactly what you need - a way to "grep" through all the logs intelligently.
The Dev environment has better hardware than the Production environment. God forbid Dev and Test actually have the same specs so, you know, you can TEST ON IT!
Naming of servers end in "dev" for Development, "tst" for Test, and "prd" for Prod, except for where they end in "d", "t", "p", "pg", "02"... Unless it's SQL Server, in which case it's ending in "Dev", "Tst", "Prod" ... mostly.
All three of them have different names for Roles granting the same access to the same tables. And I have to fill out authorization forms before the databases are built, so good luck on schema names. And we're in a project where the technical spec was just rewritten yesterday, and process testing (that thing after dev) was supposed to finish 2 weeks from now. We had to start writing test cases two weeks ago, before I had access to the database, and I might as well throw a good chunk of the work away.
I modded, so posting as AC.
A minor technique we use is a versioned install structure, with symlinks to the current version. Eg., /opt/application_root /application_1023 (1023 is our build number) /application_1034 /application_1045 /current ==> /application_1045 (symlink to one of the install trees)
This allows easy roll back if an upgrade fails without having to reload from staging. Our update scripts build the new directory, stop the server processes, re-aim the symlink, and restart the server processes. All the execution scripts use the path /opt/application_root/current/...
One solution that I have implemented at several commpanies is to use Hudson and the Hudson Promoted Builds plugin. Read this brief introduction to the concept.
I've found it's useful to put any env-specific properties in external properties files, and then make a copy for each env. On each environment, there's a one-time exercise of creating symbolic links to point to the appropriate files. ...
e.g.
ln -s db.properties.dev db.properties
ln -s server.properties.dev server.properties
Then just use the links in the app code.
I work for a virtualization company that might be able to solve some of your problems. But I don't dare mention it because I don't want to be labeled a spammer. Do some searches, form your own opinion.
I have adapted my system from the 5 years I professionally did it.
First of all, it's a 3-stage system.
You have a couple of live servers, a identical staging server, and the user machines.
Every system has a clone of the files. the servers have rsync copies of the stage server files.
And the users all sync to the stage with GIT.
Everyone has a local clone of the stage server software too, so he can test server-side code right on his machine.
That's important in every company where people could do conflicting (and even big, global) patches.
The stage server then has validity tests running. Compilations and unit test cases wherever possible. Including the database, the server side code, and rendering test pages in all relevant browsers, to diff the rendered versions (images) of the pages. (There's a app for that in Firefox, but otherwise it's desktop automation.)
There's an red alert box in the test case overview when something fails. Which gets checked every evening, before pushing anything to the live servers at night.
The only thing that turns out to be a bit hard, is to test the client-side logic (e.g. of web-apps) in a transparent manner (= keeping the software configurations and serveride code the same to be able to rely on it).
Then there is a emergency push and and emergency direct live update mechanism, for cases when you quickly have to fix something that got overlooked. (Which usually should result in a new test case to be written, to catch all such problems.)
A well-integrated project management system is very important. At the end of my first company, it was a self-written one with good integration. But in the beginning, something like Trac might suffice.
Then very important is, to have a knowledge base for all things that need to be remembered. Like a meta-documentation. Workflows and procedures. Why the mysql server will not restart on a reboot of stage server clones. Little hooks and mantraps like that. I recommend a Wiki.
And last but not least, never ever forget to have a Bugzilla. If you're good, you can integrate Bugzilla, the test validations and the task/project management into one system. Making the validity tests create bugs in Bugzilla, and bugs being the same as tasks (which makes test-driven development easier).
Yet this all is completely worthless, if your colleagues don't use it! ;) ...". It's the mother of all PHB stories. ^^
Unfortunately, I learned, that when someone can/em> do something wrong, he will.
So if you can't lock down possibilities to only those required, you have to be very very careful with who you hire. Especially with "web development", where you get sinology students who learned HTML while working as a taxi driver, stating that they are "professional web developers with 5 years of experience", while honestly believing that. And team leaders believing it too, because they are just as "competent". Because they themselves either started as something an simple as link collectors, or the boss of the company does not know shit about his business, and hired those types. They then usually get promoted to "Head of
The key is: Make them like to work the proper way. If nothing helps, money can always push them in the right direction. It's called "bonus". :)
And making it their project too, by also embracing their decisions!
Any sufficiently advanced intelligence is indistinguishable from stupidity.
I am in a similar situation but perhaps a few more steps down the path. We have environments for internal and external clients to develop, test, approve and release code. These environments all run the same version of the base platform, and are all considered production systems. We have other environments to test platform level code. These are specifically to assist in the development and release process of individual projects.
We wrote a framework application that is a single access point to publish and commit code, and move it from environment to environment. Each movement requires approval, and this can be customized for each customer. To place something on the first environment, very like a sandbox, we just copy it out there. In order to get further a commit must occur. Each commit creates a new version of the project. That version must then be marked before it can be loaded on a different environment. For the second environment, test, the project manager marks the version, indicating it is ready for testing. The tester loads the version, maintaining control over his testing environment. Each loading requires the version be previously marked. For release into production QA marks it as acceptable and the PM coordinates the release (load). When a new version is created it can move along the path and then eventually replace the original version.
Our projects consist of jsps, java and configuration files, and sometimes other files. They are stored in CVS, each project on it's own branch. The project specific java is compiled on the fly in order to ensure compatibility with the existing codebase. A compilation failure results in a loading failure. We have a custom class loader written to handle the hotswapping of the resulting class files. Approximately 100 versions get loaded to various environments every single day, and we only restart the JVM maybe once every month.
The only other significant aspect is that we CVS tag all currently loaded files so that we can reproduce any environment's contents at any time.
Check out Atlassian Bamboo. It's a build server that can plug in to your repository and automate builds and deployments.
The deployment framework is a series of modules which are designed to allow developers to easily stage Drupal data from one site to another. This includes content (nodes, taxonomy, users, etc) as well as configuration (views, content types, system settings, etc.) Not only can it push new content, it can also push updates to existing content. Deploy automatically manages dependencies between nodes (IE nodereferences) as well as between other objects. It is designed to have a rich API which can be easily extended to be used in a variety of situations. Check out the screencast for a demo!
http://drupal.org/project/deploy
Don't ever enable FTP logins. FTP should only be used for anonymous access. EVER.
(See subject.)
There are no trails. There are no trees out here.
syslogd on every modern unix is capable of routing to a specific log file for a specific app. If the basic syslogd isn't enough, your loghost can run syslog-ng or any of the other more powerful syslog daemons. You only have to replace the one on the server, the other clients should just be forwarding EVERYTHING to it.
Of course at this sort of level, you'd probably save yourself a metric assload of trouble if you implemented a proper network monitoring/management server.
Myself, having only 15 or so hosts to deal with, most of which aren't chattery just use ssh + a colorizer script I wrote for my purposes. I typically leave it running on a spare monitor all the time.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
I think you're looking for a build tool. At work we use Maven to do some of what you're doing but I think Ant would be a better choice for you. Don't worry about the fact that they are written for Java.
They allow you to move environment specific settings into a configuration file. You tell it the different addresses, passwords and usernames to use in your development, test and production environments. You then ask for a build for the test environment and all those settings are inserted where they need to go. It comes with a lot of pre-defined tasks for checking out sources, initializing databases and publishing the files on an FTP. You can pick and choose which actions are run for which environment. Finally, if you run into a specific problem that won't be solved by one of the many, many, pre-defined tasks, you can write you're own task or launch a script.
#1 is version control. Without that you are lost. SVN, git, CVS, whatever as long as it's working for you.
#2 is simplicity. Without it, your systems will simplify themselves in a sub-optimal manner (i.e. dev server becomes production server).
Here's what I do:
- Developers work and test on their local machines and commit using version control to a central repository
- Dev site is a sub-folder of the production site - easy way to make sure every single damn variable is the same
- When I want to upload to dev environment:
rm -r dev (to remove all files in current dev site)
svn export file://(path) (to get a complete copy of latest code)
- When I want to upload to production environment:
source upload.sh, where upload.sh copies the existing site into a backup directory for quick access in case of disaster, then copies the dev site up into production, then re-copies a couple special files that differ between dev and production back (.htaccess, analytics.php)
Good luck!
>... it is cumbersome to manage all sites by hacking away at the command prompt
you should probably get another job. Maybe Sysadmin in a Windows-only shop.
With no personal offense to the OP, (and noting that this is Drupal), I think the OP is trying a little to hard or suffers from inexperience. My first Drupal server hosts 100+ sites and Dev/Test/Production was rarely an issue-- which is to say, what is the OP doing that requires that level of segmentation? It's simply not that difficult on the scale mentioned.
For large sites, of course, Drupal dev/test/production is another matter-- and there is a Drupal group that handles such questions and considerations. Reading it would be useful to the OP; for most people here, it is likely so Drupal specific as to give no lessons not already familiar.
For small-to-medium sites, keeping separate dev/test/production copies on separate subdomains, flipping between them as necessary, and maintaining a backup schedule of everything is practical. Module management is a different story, but my choice is to multisite (and rsync)-- custom modules go in each sites's custom modules directory. Beyond this, again, we descend into Drupal-specific discussion that would probably be best on drupal.org-- where it has already been occurring for years.
The general question-- well, away from Drupal-- is sort of platform dependent, isn't it? On Drupal (up to D6), there is an annoying possibility of data structure collision if you have different versions running, for instance-- no easy way to merge databases unless you plan for it and, say, index the db entries (odd/even). Other systems are better at merging different copies. One should probably write a system that plans in advance.
But for 50 sites, I ... just don't have a "good question here," as most of the issues should be manageable...
after you spend two days reading UNIX man pages. So there's my answer. Read-The-Fine-Man pages.
svn or whatever works fine for that. It is the motherfucking data that is the problem, at least with drupal.
Splunk http://www.splunk.com might do it...
Virtualization, Virtualization, Virtualization.
Which spacific technology you use is irrelevent, but any resonable one should have a way to clone VMs.
Clone production, Make modifications, test, push (the entire vm) back out. You should also be running your webserver and any DB on seperate VMs. This allows you to use clone-management to deploy site-changes without running the risk of affecting the DB. Changes to the DB are more complex, but so far I have found that to be inevitable and should always be cloned, and backed up beforehand (ask MS/Danger/T-Mobile).
Put everything in test, not configured properly for production, until such time as enough people start using test that it becomes production on its own. This usually happens slowly and organically, and usually in the middle of the night. Once you have at least 2-3 different groups screaming at you over the lack of availability of your test system you can be reasonably confident that it is now production.
My only advise is:
Dont let the app developers allow the marketoids to run production deliverables from development systems...
I'm amazed -- no one yet seems to have figured out one very obvious possibility:
You don't need development machines. Let them develop on their workstations. They're developers, they'll figure it out.
Use Git. Or, if you must use SVN, use developer-specific branches -- and after using this for awhile, it should be very obvious why you should use Git. The point is that each developer should be checking into version control often, without having to worry about causing problems for other users. Want someone to take a look at your code? Let them merge your code (or just check out your branch) and run it on their local machine.
Then, one staging and one production. Or more, if you need to be testing multiple versions company-wide. VM images to spawn each (so staging can be cloned from production, or vice versa), and Capistrano to deploy.
Granted, this wasn't used with a particularly large team, but we didn't have anything which even hinted at scalability issues down the road.
Now, if your team is so large that you need many test and many production servers, I apologize, but it does look like something of a WTF when you have web developers who can't figure out how to set up a webserver on their machine. I mean, every Mac comes with Ruby On Rails out of the box!
Don't thank God, thank a doctor!
We do this for many many Drupal sites on many horizontal web nodes via bzr + ant. By 'sites' I mean no multi-site; each 'site' gets its own Drupal instance. By 'Drupal instance', I mean the 'Drupal instance' is an ant-powered deploy from a branch in bzr comprised of vendor branches (core + modules) merged in plus customizations by our shop. Each environment gets a branch, and we merge code upstream (dev -> tst -> prd).
The only thing 'shared' across the infrastructure is the web services and frameworks on the webapp nodes. Ant is great at auto-magic MySQL db provisioning, Drush calls to pound the schema, APC cache flushes, Memcached bops, etc. Also I would throw myself off a bridge if I had to manage all the complex merges across our branches and dealing with updating the vendor branches.
Others here also made the comment wrt code up, content down. Live it, love it; SERIOUSLY! Refresh often, and give your devs anonymized slices of the db for them to keep on a laptop they will undoubtedly leave in a cab. Were currently bending ant to perform the downstream refreshes + sanitizes. Looks very promising.
Also if youre not able to bastardize ant to do what you want it to do, look at ant-contrib to further extend the tool.
http://bazaar-vcs.org/en/
http://ant.apache.org/
http://ant-contrib.sourceforge.net/
Slightly OT: The J2EE guys at $employer prefer a maven+ant+svn approach. YMMV.
Have fun. These are very interesting toys to play with, tbh.
Here is our entire processes from test to prod deployment
1) Unit tests
2) regression tests (hacky, but use junit to invoke and interact with a bootstrapped version of our service)
3) (provided everything goes well)
4) Deploy to Alpha stage
5) run remote regression tests. This is integration testing. Does it work with the db? All config is good? VIP settings ok?
4) deploy to devo. This is essentially a mirror of the production website
5) Run remote regression tests (as per #2, but this is against the real mccoy). This allows us to detect configuration problems as well as how the code interacts with other incoming requests.
6) promote to pre-prod. This is a box (or set of) operating with prod data but not serving prod traffic. The only difference between this and live traffic is merely the fact that our VIP doesn't know about it
7) run preprod regression tests
8) Deploy to prod.
9) Run prod regression tests
10) Monitor for fatal errors
All of our boxes have insane monitors on them. Disk usage, CPU usage, RAM usage, TCP socket usage, process count usage. Pretty much everything that defines the box as normal. Anything goes out of whack: Page the on-call.
FYI I work for a VERY large ecommerce site. But sadly, very few teams are this rigorous
How Do You Manage Dev/Test/Production Environments?
Very carefully.
this is such a nebulous question. you want your dev/qa/pre-prod to emulate your production environment as much as possible. this subject in itself could fill a book on best practices, techniques, and the like. Easiest said by saying: keep all developed code separate from 3rd party application code. packages/versioning/repositories are a good start. make things relocatable, have one installer, and have it take multiple environmental variables. ie - make environment variables 'run time', don't make the same mistake everyone makes - and make them 'build-time'.
Best of Luck.
We're like rats, in some experiment! -- George Costanza
There is no magic solution - you are talking about managing multiple environments with different requirements and technologies in some meaningful, automated way.
You're looking at home-brew here.
What you want to aim for is
0) Stop using multiple technologies if you can. If that's not an option, it just makes more work.
1) Clearly define policies regarding development, testing, and release. These have nothing to do with tools. You build and select your tools based on these policies.
2) Automated pushbutton deployment. You want your code releases of each new version of a site to be automated. You also want rolling back to the previous version to be automated. This applies for CI, QA, and whatever other stages you want, all the way to Production.
3) Automated deployment should involve at a minimum tagging a given revision and pushing it to the correct environment.
4) You can use commit hooks or some other method against TRUNK to run a CI server that continually does regression testing and other funky stuff... as well as just shows you a live version of what's in trunk "right now".
5) When working towards a target release,developers need to include any necessary scripts to update (and rollback, if necessary) their respective databases.
6) Config data... can be handled by having a separate /config folder for each environment, version controlled separately - and where access and change control are again strictly defined and limited, and well documented. this would automatically be inserted by your pushbutton deployment process.
First, I can't see the need for a GUI or management tool. God gave us bash to make us happy and productive and we should use it. Managing a mix of sites, is an ideal use of scripting. What we (small web dev company 100+ sites) do: bazaar (bzr) for repos. I much prefer it to svn. Cleaner and smarter. Uses sftp by default. Create a repo: 'bzr init ; bzr add', and anyone with access to that system can co a copy. Each developer checks out the devel code to their local systems. We have our nameservers set to handle custom naming, so like example.dev maps to 127.0.0.1. All the db stuff goes to a db server and everyone (in house and contractors), have access to that. Get to use all your desktop tools, etc. The windows guys have local versions of Apache and php, so everybody's happy on that end. Next level is staging/testing. That server is a checkout of the repo. Clients have limited access. When the time comes, a copy of the site is made, site is grepped for no-no's and FIXME's, the new code is "cleaned", and then copied to production. This is all scripted. Permissions are reset, development stuff (docs, fla's, psd's, etc) are removed. We have a naming scheme, so certain directories are blown away on the live site. The scripts issues some reminders of stuff to make sure get done at the last minute and post launch. Loop, gotop(), and tweak script.
One of the things we do that solves a lot of issues is that every night we export our production data to a file, zip it, stash it a few places,
and then we copy it to our Test site, drop the db and reload all the data.
Tells us fer sure the export was good and it makes sure we work on live data.
Another thing it guarantees is that you will write the correct scripts to make any db changes on production because you have to make those changes every day on test until they go live.
One of my co-workers suggested it a few years a go and I wouldn't go any other way.Making a backup is only half the solution. Guaranteeing the restore is the entire solution.
There are many ways to do the things you describe. I personally make extensive use of Puppet.
This is a great solution for your configuration files, but note (directly) your code. This is where your distribution's packaging system comes in.
Build packages of your code for your OS package manager (be it RPM, portage, apt, whatever... it's usually not that difficult). Give the packages version numbers based on svn revision, if you need that granularity. Create an automated mechanism to build your package and insert it into a local repository.
Tell puppet to ensure that your 'dev' environment is always using the latest package. Tell puppet to ensure that your production and test environments are running whichever specific version they're supposed to be running.
A downside of puppet is that it's a 'pull' based system, by default every 30 minutes. For most situations, this is adequate - but not all. You might also investigate Func as, at the very least, a convenient way to tell a group of notes to phone back home to puppet on demand.
we are a prefession online store, you can see more photos and price in our website which is show in the photos
if you are interested in our produ
We offer kinds of Newest Style Handbag,Brand Handbag,Fashion Handbags,
Ladies' Leather Handbag,Replica Handbag--AmmonOnline
We ship to worldwide by EMS,TNT,DHL,UPS.
We supply you with smooth and fast services, and do dorp shipping.
Welcome to visit our factory.
Please visit our Website:www.tntshoes.com or products Album,
Contact us now, We can send you more details.
OUR WEBSITE:
YAHOO:shoppertrade@yahoo.com.cn
MSN:shoppertrade@hotmail.com
HTTP://www.tntshoes.comCOOGI D&G diesel ED HARDY lrg etc $33-50 free shipping. Jersey NBA Jersey MLB NLBM nike puma adidas $12-30 free shiping.
OUR WEBSITE:
YAHOO:shoppertrade@yahoo.com.cn
MSN:shoppertrade@hotmail.com
HTTP://www.tntshoes.com
Don't take shortcuts. It just bites you in the end. Of course, this puts some burden on the dev teams. No environmental config in the project. There are many ways to accomplish this. In Java, we put properties files into the server's classpath. Figure out a similar mechanism for your platform.
I cant believe no-one has mentioned puppet to this guy. Works like a dream...
Allthough Aegir is in active development each release is stable enough for production use and migrations between releases are supported. Allthough right now it only supports Drupal the basic infrastructure is in place to support other LAMP applications in the future.
My company runs a virtual lab setup that is build on top of vmlogix labmanager. It handles automated rollouts of test machines but you're still on your own to handle the applications logic.
I've felt the pain of attempting to stage Drupal set-ups, and it's horrid.
Whilst people say "Just migrate the database from dev->staging->live" it's not that easy when it comes to the live server, because that will have changes mixed in with what you want to make changes to (e.g., user-generated content on the live site creates nodes, which share a node-numbering namespace with your dev-created new content). This is additionally complicated by Drupal putting all the config in the database, so you'll need to make changes to that when you push it live each time (since the dev server will have dev modules enabled that you won't want on the live site, say).
Separating the database changes you've made from the ones made on the live site (to allow you to merge them) leads to a whole set of options...
Drupal has lots and lots of different methods of working around this problem (just search for "staging" and "deployment" on drupal.org), but none of them really seems to work that painlessly -- they either require you to write all your changes as SQL scripts or to not use particular features (e.g., CCK).
Do all CMSes have this problem, or is it just because of the lack of separation which Drupal provides between live data, your own created data (from dev) and config?
You should taking a look at Nolio at noliosoft.com. It's a pretty full featured automation platform that should be able to handle everything you might need.
I will remind you that the forums are staffed by unpaid volunteers who take *their* time to answer *your* questions, "as they can."
If you ask a question that has already been asked and answered elsewhere, and you do not take *YOUR* time to find the answer because you'd rather "crowdsource="FREELOAD on *THEIR* time, you get what you pay for.
That said, Drupal is FOSS and the Forums, as constituted, are not perfect. There is much discussion of this and if you really care, you can learn something and volunteer for the appropriate team and help make it better. Or you can buy a subscription to Acquia.
And yes, you should upgrade to D6, not because it is "shiny," but because it contains numerous improvements and simplifications that are a result of the community process.
Examine syslog-ng, which can make a more sophisticated parsing of specific types of logs to specific targets, and set their permissions, with more resolution than syslog typically permits. It's feasible to have the same logs sent to multiple targets, and to make those logs accessible for various web clients.
The principles are pretty similar, so somebody working in a small company can look at the above process, understand it, and then apply the most important points to his own environment.
IANAL but write like a drunk one.
Owen aion gold
metin2 yang will be
wow gold cheap
aion gold possible
world of warcraft gold to
aion4gold return to
cheap aion gold the national
team Aion Kina
I'll forward this all to my boss. I'm sure my review will go better now that he knows the truth.