Slashdot Mirror
Who Owns Deployments - Dev or IT?
txpenguin asks: "I am IT manager for a small software company. We host several generations of our applications in a fairly complex environment. Our systems are very much inter-dependent (clustering, replication, heavily loaded, and so forth), and bad changes tend to have a domino effect. Additionally, it seems that there are always those who need to be 'in the loop', but aren't aware of changes which affect them. There is a constant battle between IT and Development regarding who should handle the deployment of new code releases and database schema changes to production systems. Dev doesn't understand the systems, and IT does not know the code well. How do you handle this at your company? What protocols seem to work best? Can there be a middle ground?"
These kinds of things where there are two opposing sides always have the same answer. Unless one side is teh debil or something.
You have to compromise. That's it. Middle ground. There are no other solutions to or ways around this problem. As you describe it, each side has access to and knowledge of half the problem. Half plus half is whole!
So, meet with the guys in Dev. If you want to be beaureaucratic and official about it, create a "deployment team" consisting of an equal number of members from each side that will sit down, discuss, and supervise all necessary changes to production systems. Hell, send someone to a project management class if you need to.
Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it. Too bad. To place how serious this issue is in overcoming the political terms: everyone in both departments needs to be cooperating or unemployed.
So there you go. Just like any other relationship, business or otherwise: sit down and talk it over. Problems solved!
ACs are modded -6. I don't read you, I don't mod you, I don't see you. Don't like it? Don't be a coward.
IT should only be involved in the maintenance of your company's network. The developers need to own the entire product, and if that means bringing in some database experts who can handle deployments, then that's a must. If you are a consumer of your own product (and a lot of companies do that as a way to be one of their own wins) then IT should handle the network right up to the point where your network starts hitting the product network.
Of course you will need to have separated the networks in order for this to be any use at all. Putting your production servers on the same network as your business servers will be catastrophic if anything were to ever happen. In the worst case you'd lose both and be totally dead in the water.
The problem is not "You've got peanutbutter in my jelly", but rather that you either need to hire devs who can do network stuff themselves or you need to bring a network admin onboard who can work solely on the deployment system. If you can't afford that, then you will need to use the current IT guys you've got, but they need to know that they are to keep the two networks separate.
IT should own the deployments.
Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
Middle Ground = "Production Operations" group. Familiar with the code and your application as well as the IT infrastructure.
...or at least, should, but we're close enough, is to just provide releases at certain points and when we make a release provide one script to update from (at least) the previous version.
That script (which probably calls other scripts in turn) should run backups, install new files, update the database schema, import any new data, the lot. Sure it's not trivial to produce such a script and they can end up taking on a life of their own but it saves the devs from having to talk to the IT monkeys.
When your team rolls out an application update that was created by an external company, Microsoft for example, do their coders pop in and lend a hand? No. The dev team should write an application effectively as a third party external to the rest of the company. If they're writing things that the IT dept can't roll out without their help then they're not doing a good enough job.
http://twitter.com/onion2k
This is why many companies start a Production Services team. Generally this means the hiring of a Build and Release Engineer or Manager who has an IT background and an understanding of Software Development.
The alignment of the Production team varies. At some companies they report to the development organization (e.g. to the Manager or Director of Software Engineering) and at other companies they report to Quality Assurance.
I would suggest that you check out the site: http://cmcrossroads.com/
This is why you should have at least 3 enviroments, dev, uat, prod (or similar) the dev's can carryout the promotion between dev and uat and in doing so iron out any bugs with their code / the promotion process. The entire promotion to production should be fully handed over to the production / IT team. At the end of the day you dont want developers anywhere near the production system.
It depends on the the makeup of your organisation:
Do you do ITIL?
Do you have SOX obligations or something similar if outside the USA?
Are you agile?
Can you automate your releases?
What technologies do you use?
My organisation is a large user of IT but we are not an IT company (we're a University). We have a network of about 8000 Computers (Solaris, Linux and Windows) and are heavy users of Oracle products.
We have a focus on workflow automation at all levels in the business and have around 200 IT staff. Our developers prepare releases but a dedicated team does the migrations. We use a workflow system to automate migration requests etc.
We use the releases to the user acceptance test areas as a test of the release package - Is everything there? Does it compile? etc. Some of our releases are fully automated ie PHP applications but our ERP releases are almost completely manual. We are currently running a project to increase the level of release automation.
Our process works because we treat the release package as a deliverable; we test the release mechanism along with the code in the acceptance test environment and we keep an audit trail of what is released and when.
In summary, we programmers and other IT people only want to only release tested and approved code; and to achieve this with good processes. I think a good process is a process that achieves this goal with the least amount of effort and overhead.
The man in black fled across the desert, and the gunslinger followed (SK)
You work for a small company. Any advice presented by /. will fail to solicit the correct response---except this one. This fight is between department heads, and should be resolved between them as businessmen. Failing that, whoever controls both managers should make the decision. This is not a software issue, but a business decision outside your control.
What those who want activist courts fear is rule by the people.
This scenario has always avoided the "pissing match" that inevitably occurs between Dev and IT, in my opinion. Clearly defined roles.
In a previous shop, we treated our in-house developer code releases like any other vendor release. Just because some code was written by the guy down the hall, shouldn't mean that I can't ask for the same level of documentation/support I get from another Tier 1 vendor.
In art classes they teach students to draw the space around the objects they are trying to depict. It's a useful skill in many areas.
Rather than imagining that there is this atomic transition point that one side or the other must own, look more closely at what happens when changes are put into production, zooming in until you have enough detail that every piece naturally belongs to one team or the other.
Then look at how this would play out in the real world, to find the "frothy" or "tangled" parts (well, IT should do this, then Dev should do that, then IT should do two more things, then it's Dev's turn again). These parts should be sorted out by requiring documentation (or scripting) to flow one way or the other, so that the process can be performed by one group without the direct involvement of the other.
In short, the problem here is the granularity of your question.
--MarkusQ
I could not agree more. We have a fairly large projects where the devs do the deployments and I can tell you its all a big mess. Since the devs have the right to do deployments, naturally they can make small changes to the production environment invisible to the operation team. Since there are quite some incidents occuring on the production environment, the dev team tends to fix the problems on the production environment on the spot because "oh, its just a matter of fixing this and that", therefore the acceptance environment is not used for testing as it should be.
Although I know it would put considerabe workload on operation team to take over deployments, I'm sure there would be less incidents and more stability on the production system. sure the deployment process would slow down updates, but would give a chance for a more transparent and controllable process with clear ownership of responsibility, etc.
Now that the project is for more than half a year in production, I see no real chance to change the deployment ownership, but still I wish it could be done.
Most projects of a size to really worry about this stuff would have an Integration/Deployment team. Essentially they are the "middle ware" of the Application Dev cycle. Dev don't know about the realities of real implementation and have all sorts of bad habits which don't scale to production and IT/production don't know about the deeper technicalities otherwise they would leave and get jobs in Development!!
Your Integration Team should be able to keep your dev/project "real", get them doing your Dev environments before the project goes into deployment otherwise its a real PITA. Good deployment teams should also produce a finished result with documentation which should be to a standard that the IT/Production can truly use. Best of luck with your project!
John A
...you should make a decision and stick to it. It doesn't really matter which team is in charge, so long as one of them is and that the other team knows it.
Bob
Listen to my latest album here
We're talking about actual development here. That is, we're not talking about taking example code from Sun's website, changing a few class names and calling it a new and finished product. We're not talking about taking snippets of code, unattributed, from various incompatibly-licensed open source projects and combining them into a single crashing pile of shit.
y -to-trick-our-clients-with-Sun's-example-code".
We've seen shit like that the few times we've dealt with Indian software firms, and frankly their development practices are just plain unacceptable. And I'd hardly call it "development". It's more a case of "let's-fuck-up-because-we're-clueless-and-then-tr
I work in my city (population approx. 200,000) government's IT department as a developer. As someone else mentioned, the best way to handle code deployment is over a three-tiered system. At DIT, we have Dev, Stage, and Prod for the Internet, Intranet, and Applications sites. Developers should be given full read/write access to dev, in which they do all of their work. In no way should the world, nor the client, see dev. Upon completion, we promote our work to staging (via sending a promotion request, or doing the promotion yourself, if you happen to have access to staging like myself). At staging, the client (for us it's 99% internal work, so someone else within City Hall) can review the work, request additional changes, etc. After the staging process is done, a request is sent to the guys in ops (IT) to promote to production.
Basically, to answer your question: IT should handle code deployment, but developers should be involved in the process.
in our company. Once the developer(s) have a stable release, passes developer unit testing, it goes to QA (Quality Assurance) for systems testing and User Acceptance Testing. Once the user signs off QA asks dev for deployment procedures/scripts. Once Deployed the end user is asked for final sign off on the deployment. Software development is a TEAM effort requiring developers, DBAs, System Admins, QA, and end users. The only one of these that may not be IT is end users.
Meddle thou not in the affairs of Dragons, for thou art crunchy and with most anything.
Each week we hold a meeting (call CCB - Change Control Board) where the people who do the development talk about what their projects that need to go in production in the upcoming week.
All software & hardware changes will be discussed; what the original problem was, what the solution is to do, what changes to make, what effects to expect, what the rollback plan is, emgency support, and who's all signed off on the changes & tested them. It's then approved or denied (for lack of planning, support, user sign off, etc.)
Works really well, even if it's bureaucratic.
III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIII
in bigger companies, it's a CM team that handles these things. neither dev nor it can do the deployment. CM will testify that the release has been tested (which can be done by a testing group in dev, or a test department), and they will have a handle on everything that is installed right now and has ever been. They have the final word on whether something should be deployed. They build the packages going to the clients, if need be. And you can always go to them and ask "where's the code that's running right now?".
they'll also keep track of what documentation corresponds to what. you usually can't trust the developpers for things like that.
you need a CM team, even if they're not full-time CM.
In our organization, we have a small group of admins who focus support on just internal business systems, including a number of in-house developed web apps. Our philosophy is we support the whole application, including deployments. When something breaks at 2 am, we're on call and anyone on the team is expected to be able to respond, even if the problem is on the application side of things. The developers are just an escalation point.
This means we are involved in a lot of details of the development of these applications. We maintain the configuration of all systems used to develop the applications short of the developers' individual workstations and actual code on development. So developers work with us to make the changes they need on the development systems, then document and request these changes and any new code get applied to QA. Finally they submit a request to production using the same set of documented changes--this request is reviewed by their project manager, then approved. In each case, we have the ability to review the changes and sometimes get involved if the requested changes may cause issues for the servers or open a security hole, etc.
The real key to a system such as this is documentation. When something breaks, we document how we fixed it. When we build a new server, we document the steps we used to load it. These documents we freely share with the developers and encourage their feedback. At the same time, the developers are expected to document their application, any scheduled tasks that must happen, any routine maintenance tasks that must be done, etc. This is on top of the aforementioned documentation of configuration changes and code deployments.
In practice, this system doesn't always work...especially as the number of systems and applications has grown over time and thus the complexity of supporting these, both for the developers and for us admins. But in general it does, and even when things don't quite mesh between the two teams, it's usually much easier to resolve such conflicts because of the overall framework we have in place.
I work for a small company, and I am in the convenient position of being in charge of both ops and dev. I came up in very much of a slash role (admin/developer), and I know both sides very well. In my experience, the real answer is neither operations nor development "owns" the deployment. The company "owns" the deployment. Usually there are several groups that have a part to play in making a deployment successful (dev, QA, training, ops, marketing, etc). The main thing is cooperation. We have a process that lays out the things that need to be done in a deployment along with the group responsible. This lets everyone do what they're best at. It takes a bit of work to get this, but it's much more important to have a successful deployment than to get an ego boost for "winning" the argument.
...that IT should own deployments. That being said, this assumes that DEV has made strict guidelines clear instructions, and foolproofed their release. There also needs to be a fully tested system in place for rolling out changes and rolling them back.
I work for a small software company that provides software for doing exactly this, and run into these questions and scenarios at every client I go to. Even when the instructions for deploying a release are 20 pages long and extremely complicated and labor intensive, you still don't see development applying it. My job is to make it a simple hand off process where DEV packages the release (in our software) and hands it off to the deployment team, who in turn executes the job against the production environment. If there are problems deployment can roll the changes back and let DEV debug any issues.
I gotta have more cowbell.
Sooner said than done, of course... but if applications are so interdependent that the combined system is fragile unless everything is just exactly the right version and installed just so and configured just so and located in just exactly the right place in the directory and has all the other configurable settings of everything else in the system just so... then it wasn't well designed.
In the Good Old Days an application was a single file, you copied it to your system, and ran it. Over the years--I tend to blame the loosey-goosey early PC culture for this--applications wouldn't run unless a slew of other things were installed and configured properly, and all of a sudden everything needed to be Just So. And we fell into the black darkness of "installer programs." Which generally had the property that they would work fine on virgin systems, but often failed if any other application had been "installed."
I used to work for a Fortune 500 company where things were seriously out of control. One of the ways you knew it is that people were always sending out memos saying to remember that version 10.8.5 of application X would only work if application Y was 6.2.3b and the firmware was rev 8.1.9c and the OS was rev 7.1b-2.... I worked in a department where the department head thought everything was OK because everything on _her_ PC was hand-installed file by file by her developers... and everything worked. But an ordinary layperson trying to install the company's products in the documented way frequently failed.
What I'm saying is that you may not have an organizational problem, but a technical problem. If things are so complicated that neither IT nor development knows how to make them work, maybe things need to be made less complicated.
Like I said, easy to say, hard to do...
"How to Do Nothing," kids activities, back in print!
...The IT person(s) who will be hadling deployment should be involved at all stages of Dev. They should attend all development meetings, and be part of all discussions that will affect deployment.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
Why would anyone want control of a balky, high profile and critical process? Glory? Future generations of IT managers speaking your name in tones of hushed awe?
Kidding aside, I've worked both ends of that stick. If anybody is goign to "own" a deployment it's going to be IT. But owning doesn't mean you don't share. IT and development both have roles in the deployment that, if you must insist, belong to one or the other. IT should make development aware of any deployment constraints, the development people should make sure it is possible to do the deployment under the conditions set by IT and provide an appropriate procedure, scripts etc. IT should take that procedure and test it, then put it into the context of a roll out plan, then do the plan.
The truth is in a sufficiently complex, sufficiently custom system, there is no clear line between IT and development. If you really care about doing it right, you form a team with members from each department, and you promise to horsewhip anybody who gets snooty about which department they come from. It doesn't wait for hand-off from development, development doesn't wash its hands once a procedure has been written. So make sure there are people from both functions in the game for the long haul.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
we have someone in our company that is a middle ground between the dba's/IT and the dev's. He is a coder by training but he supports the code the developers write, deploys it, checks for problems after QA signs off on it, etc.
the same people who write the code where I work also take care that it deploys properly. Sure someone else pushes the buttons and writes some user information but until its in the field at all locations the developer is actively involved.
code goes from development, q&a, field test, deployment.
once the code reaches q&a it does not change unless q&a rejects, but afterward the developer(s) are responsible if problems arise during install and work with support to fix it. they never lose ownership.
this might not work in well disciplined shops as it would be easy for the support staff to just bury developers in all sorts of stupid issues. This also may not work in pc centric development groups which seem for some reason to have a different idea on development from mini and main framers. (really, we make jokes about the pc centric groups - if something takes forever to get done, even simple stuff, we say "it moves as the speed of X" (x being the name of the pc development group)
just wondering when did IT and development become separate entities? wouldn't that make the job all that more difficult?
* Winners compare their achievements to their goals, losers compare theirs to that of others.
I work for a very large software company and we're going through this battle right now for an internal application.
The developers are fine at making improvements to the application interface and providing new features, but completely and utterly clueless when it comes to system architecture, security, using hardcoded hostnames, etc. etc. The more we close the developers out of the production deployment, the better this runs. We also have to increasingly review their work to make sure they aren't doing things like hardcoding hostnames, or writing data to local disk instead of the rdbms (writing data to the disk kills horizontal scalability, high availability, etc)
The relationship has turned quite sour and there is a big wall between both sides with a liaison sitting on top relaying messages back and forth. It's an entirely unhealthy working relationship but at least the developers aren't sneaking things into the production environment anymore that have no business being there.
If you are a SaaS company, you should be striving for a system that makes deployments so streamlined that they are able to be done by the system itself.
We use subversion to control our source, and the moment we tag a release for testing MSBuild scripts take the tag out, compile it, unit test it, deploy it to the testers environment, and resolve the bugzilla bug created for the release. When testers are finished testing, this bug gets marked verified and the staging published release gets copied into a stable folder, where at 3am the following day it will be updated on the server and any upgrade scripts that are required get run.
If new systems are to be added, IT builds the system, gives it an address, installs the required programs, installs the 3am update script and performs the initial checkout. The system will then come online at 3am the next morning (when it runs all the scripts required).
If it is an emergency the IT staff (or the developers) can manually run the update script, causing the system to be fully up to date and online immeadiatly. The only issue is that this could bring production down for a few seconds (all production systems running the same network must be running the same version [ok, that is a generalization, this isn't always required, it depends on the changes], so if the update script is fired off, it must be run on every production system at the same time: production will be down for the duration of the update from the first system to the last).
Plenty of other posters have pointed out that you sound a like an operation that is a bit small for the full Software Development Process. However if you're asking I suspect you're a growing company, in which case you need to get a Process in place, and soon or you will experience the full agony of a chaotic IT environment. (NB That's where I work now - I've worked sane places too) Fairly typical Process: 1. Dev receive Requirements and Defects from the Business, and code to them, unit testing their code (). 2. Code is delivered to Operations with a 'Release Note' or equivalent covering how to deploy the code to Environments 3. Operations deliver (deploy) the code to test environment(s). Link and Acceptance testing is performed - does it meet Requirements? are key defects resolved? Plus regression testing - does it break the existing system? Test sign it off if it clears these tests. 4. Operations deploy the code to the Production system on sign off. You inevitably end up with tensions in the Business vs IT, plus the divisions between the priorities of Dev, Test and Operations. Sounds like you are at the stage of not having any well-defined roles/teams for these responsibilities. I'll detail the Operations breakdown too. Operations: As others pointed out this breaks down into various teams. DBAs, Sysadmins, Change Management, Release Management, Operators, depending on your site. Operations are responsible for the stability and smooth running of the Production system - they must accept change, but control it. Since I work there, and you specifically address the subject, I'll detail Release Management too Release/Change Management Usually end up the Gatekeepers on changes. They'll need to be familiar with the whole system, and resistant to the pressure they'll receive from all sides. They need to know what versions of code are where, and be able to reject bad code when it turns up, but be flexible enough to make sure Test have something to test. They need to be experts on everything your IT does. No jobsworths here, and good generalists are rare. Since you'll inevitably go through a period of chaos if you are growing I'll mention that staff turnover here is very high - unless you get in contractors, and pay highly for them. The Change Management role, sometimes covering the Release role too, is to track changes and know who they impact, and be able to prioritise changes. If Release and Change roles are separate, CM is closest to the business. Hope that helps.
I work for a company that provides software for deployment/configuration management http://www.certalertsoftware.com/. Whenever we engage with a new company we always have to do the dance for who owns what and where the line is for ownership. What we see more often then not is that IT owns the deployemnt - until it is broken. Then it goes back to Dev.
;) However, organizations that are EARLY in the Agile adoption or change are seriously challenging to deal with. There is still tons of turf wars and property disputes.
Some of the other comments that talk about the Agile approaches with rapid iterative builds and deployments are interesting. We have run into a few larger companies (F500) that are using Agile or derivitives. In these cases the dev and IT guys seem to have made peace with each other. I would guess - sometime after Agile was forced on them
I think it depends. Where I work we do multiple deployments: into the common development environment, into the QA system, into an internal user-test system, into a customer-test system and finally into production. The rule is that devs do the deployment into the CDE to work out the kinks. Devs and QA do the deployment into the QA so QA knows what's going on. Devs work with the IT staff doing the deployments into the two test systems so that IT knows how it goes and devs can work out any final issues that come up or that IT may have with the procedures. IT does the final deployment into production since they're responsible for those machines.
You can't just throw deployment on IT, it's the devs who wrote the software and know what needs done and who need to see what they missed so they can add procedures to cover it or fix whatever problem in the software caused the deployment glitch. You can't leave it to the devs to do production deployments, it's production fercryinoutloud, devs shouldn't be messing around directly in production except in dire emergencies. It's got to be a combination.
In my company QA is the bridge between development and production. I'm a team lead (dev) in a company which has a suite of web applications. Each application has a lead assigned to it, who handles the development and documentation of a product through their team. We do several deployments of software each week, and if our leads had to hand-hold through each of them we'd be hamstrung for time and working more night hours than we'd like.
When we have a RC I'll branch the trunk, and request that QA perform a Pre-Production build. Developers will work with ops to get this running properly on the pre hardware, as this can be done outside of maint hours. We'll then do several builds of the branch until it's gold, and then tag off the branch as X.vv.zz.
While a major release is in QA the lead focuses on creating/updating the operations document which addresses day-to-day maintainence issues and tells operations how to troubleshoot the app in the case of a problem. They also produce an implementation plan which identifies the groups/persons needed to deploy the application, and the steps needed to be taken, using what they've learned from the initial pre deployment. Once this is done, and QA has promoted the app, a dry-run is performed to try to catch any missing steps. The implementation plan is handed to QA, who coordinates with IT/Ops to resolve any conflicts and schedule the deployment. Ops/DBA's then physically performs the deployment following the steps given in the plan. In a major release situation, you may have a team lead or platform manager coordinating the actual steps on a conference bridge. But for minor releases we've been able to just have our operations teams do the full deployment with verification by QA and the product's customer service group.
We also have a twice weekly meeting where any upcoming production changes are discussed between IT/Ops, QA and Dev. Release documents are put on a calendar, so if an issue comes up on another product we can go to this and see what may have caused it.
Dev and QA also meet weekly to discuss the progression of products through or into QA. Any issues with testing or problems with builds not being stable can be addressed.
It took us a while to get to this point. We had previously been in a situation where dev would handle the build and deployment process, and it was had for many of the leads to let their projects go, but now we can see the benefit, not only for the company, but also in the fact we don't have to be doing releases at 12AM on tuesdays anymore. It takes a lot of work across departments, and definitely is a long road, but something that needs to be done.
There is no reason for developers to have hands-on involvement in operations. They may be involved in some sort of advisory capacity, but they certainly should not be doing installation and configuration of systems they developed. If there is a problem with the deployment, the devs should be working it out on non-production systems, preferably without live data. If, for some reason, there is an absolute need for debugging on live systems, someone from IT and the security team (you do have a seperate security team, don't you?) throughout the process.
SA's aren't developers, so they shouldn't be taking on any of their responsibilities. It's the dev's responsibility to deploy their code properly and ensure it won't cause any conflicts. If there's a problem as a result, it's the dev's fault not the SA's. You can tell i'm biased, but the end result is devs get to control things and SA's can continue to work on important things until the devs break deployments again and the SA's have to muddle through a lot of bad undocumented procedure to figure out wtf went wrong.
I work for a pretty big place and while I agree with you, we've found that it's nice to have some IT folks who work closely with dev to usher new products to the field. We have a testing lab where dev hands off their work to a mixed IT/dev team (mostly IT) who then upgrade a sample network with the new product. They test against every expected configuration of hardware and software the field is likely to be using. If there are no problems, there's a joint sign-off, the testing team writes deployment instructions (down to the "click here" level) for field techs, and the software is put on a download server and/or pushed. If there are problems, the devs get pulled in for a quick fix and update to the docs. If they can't pull that off, the Lab then sends the software back to the devs and through channels informs the customers of the delay.
It's my impression that this is different from many other large organizations where pre-deployment testing is strictly a dev thing. Technically, that's true of us but the reality is that the testing is done by frontline folks and others who are rotated into the lab and charged with thinking up new and innovative ways to break everything. I hear it's way too much work and a lot of fun at the same time.
One extra feature - every two weeks the IT and dev folks from the testing lab hold a conference call with the front-line techs. They very frankly tell us what's coming, what they're working on, what fixes they've developed recently, etc. The front-line techs are allowed and encouraged (and they take advantage of the opportunity) to vent, complain, suggest, ask for help, etc., directly from the devs, the testers, the customer reps who spec'd the project, the technical analysts, and each other. All this happens in an open, honest forum where there are no reprisals for plainly saying "This stinks and here's why..." even when the executive in charge of said stinky project is on the call.
For a big place (well over 100,000K employees), this system works pretty well.
It scares me to think of the sheer brainpower that would have to be possessed by the candidate they'd hire to do deployments with no prior dev knowledge of the apps... hey wait, that's me! I'm downstream from all the devs, who document nothing. Speaking as a former dev, to document everything to the extent required for deployment personnel is very costly and the antithesis of agile. I wouldn't want to read your stinking docs. Go ahead, jar everything to oblivion and hand me the keys.
Root. Root owns it. Root keeps everything that doesn't come from us running. Root, Chief Senior Root, Senior Root and 2nd Root keep a Binder and a Booklett with all procedures and Accounts written down, in case they all die in a planecrash. Dev builds the stuff deployable and is assitant to Root on deployment and irons out the glitches as Root orders. During Developement it's the other way around. Dev tells Root what it needs, Root delivers. If the System goes down, Root takes blame. Dev owns Root, but Root owns everything. That includes exclusice Root access (hence the name) ... That thing standing on your desk is called a monitor. You use it by looking into the glowing end. That thing with all the buttons is a keyboard. That round thing with the few buttons is called a mouse. Pick up "Idiots Guide to Managing more that 2 Computers in a Work Enviroment" for further details.
... Are you? .... That'll be 200$ geek-buddy-price for consulting services. Thank you. ... :-)
BTW: You asking that question here means only one thing: Your companies IT-frastructure needs a complete reedo. Trust me, I've seen it a million times. Stop right now, no need going an further. Fire anybody who doesn't basically understand the above (including yourself, if needed) and have somebody come in and do a setup for you. OSS geeks are OK if they are articulate and competent. Check their clothes - if they are unkempt and less than quality casual don't deal with them. They should have a few years of admin experience up their sleave and be good workers. Which means they should be able to give precise estimates of all factors and need no other than 8 hour days in a classic 5 day workweek to fullfill their estimates. If they can't do that, they are not professionals.
Now how come I strongly suspect you guys are Windows (Junk)shop?
We suffer more in our imagination than in reality. - Seneca
The CIO or whomever makes the IT policies should be the one to say who "owns" the deployments. That is what management is for, making these kinds of decisions. In a way, the person who should "own" the deployment is the Project Manager.
This decision should be enforced by the Project Managers (PMs) overseeing the projects. If your company doesn't have PMs overseeing the projects, then it may want to start assigning them.
The PMs are the ones who are supposed to be keeping the changes on track, on time, and on budget and that the application does not experience problems. They are the ones that are supposed to be letting people know about the changes. They should also be making sure there is proper documentation for the deployment and the software. They should make sure that IT and Admin can understand the documentation and that what is in the documentation can be implemented. The PMs should be meeting and let each other know the status of the various projects so that information can be passed down to whomever needs.
For the who needs to know problem, go to those who think they need to know and get a list when they think they should be notified about something and make them justify it. And, "Because I want to know" is only valid if that person is one's superior.
By the way, I work as production support in my company and I own the deployments for my application. My job is making sure the application stays up. That is my primary concern. Next comes what the users want and then what the developers want. Unless I approve, or a director demands it, the deployment does not happen.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
don't reinvent the wheel, use ITIL or some other tried and proven framework / methodology.
Unless your IT department is under-educated, you should have a number of IT guys who are moderately capable software developers themselves, and at least some of your developers should have some system administration skills as well.
Given that, one good solution is obvious: For each development project, put someone from IT on the development team. The development organization is free to assign that person some development tasks, but their *primary* job is to ensure that development is taking IT's concerns into account in the design and implementation of the application, documentation and maintenance and support planning. Obviously, it needs to be a fairly senior IT person who understands how things work and knows who to talk to whenever any IT-related questions come up.
Then, for each deployment, the development organization should provide one of the developers to IT to support the testing, deployment and cutover to production, plus the first few weeks of operations. This developer should be one who understands the code fairly well, and knows who to talk to whenever there's anything he/she doesn't understand.
Later, rinse, repeat. Over the course of a few years, assuming you can retain your staff, you should end up with IT and development organizations who both have a deep understanding of one another's jobs and work together very smoothly.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
From a security standpoint, developers should not deploy, install, manage, or administer any production servers. It is a conflict of interests and is covered by separation of duties.
That being said, I've worked in many environments. In my current position we have four teams.
1) Development
2) System Administrators
3) Application Administrators
4) Database Administrators
Development develops the applications, System administrators manage and maintain the hardware/os, app admins manage and maintain the applications, and the dbas manage and maintain the databases.
Development writes the applications and hands them off to the application administrators. These guys install the app in our QA environment and go through some testing. Once it's tested internally, we install it in UAT and give our customers a chance to test. If there are any problems during testing, the release goes back to development. Once everything has tested clean, we have a change control meeting with the DBAs, App admins, and System admins to discuss all of the moving parts and work together to put everything into production.
I think we have the best of both worlds. Development is out of the loop after they hand off the product, but we also have Application support folks which work closely with development and know the applications and their requirements.
It should be IT deploying to production period (I'm the Sr. System Admin and deploy all apps to production). Yes we know our own systems and we know the application to an extent. If done right "code monkeys" have done their job and the application passed QA'ed and testing, the application then belongs to IT. In my experience the vast majority of DEV's don't understand System Administration (there are exceptions to the rule of course) and no it's not about us loosing control but boils down to accountability. Just yesterday I had to fix and (try) explain to a dev why they could not deploy the code to a QA box because they kept on getting a CRC error in thier package. Now would you trust a production deployment to that person? What would happen when you have to account for service levels for a customer and the dev's kept scratching their heads trying to figure out what went wrong and then call a system admin into the frey.
Sig
The answer is: "neither". This is why you have a project manager (I hope.) Someone neutral who tracks progress and assigns tasks to dev/IT/whatever.
At least I hope you have a project manager. If not, assign one who's capable of working outside his normal role, either dev or IT, but they'd better know what they're doing (cost tracking, that sort of thing.)
Cole's Law: Thinly sliced cabbage
Software going into production systems belongs to IT. End of story.
IT should treat Development just like any other software vendor.
If vendor representatives are there at the production site, it is only to offer help at the behest of the customer
(which in this case is IT). Nothing goes into the systems directly from the vendor without being vetted by the IT guys.
If the product has bugs, it needs to be treated by development at least as seriously as with any other important customer.
What this means therefore, is that IT has to have enough people on staff that can understand and maintain the deployed system.
The usual problems here are that
1) a lot of IT staff don't want to learn a new system (specially one that is in-house and doesn't give transferable skills) -> The answer here is to suck it up and replace them with someone more willing to learn the required systems.
2) development doesn't treat IT as a real customer (it's only IT) and releases shoddy code that needs frequent updating -> The answer here is for IT to do a formal acceptance test and reject the software if it is not ready for prime time (with high visibility to upper management). Ideally, these formal acceptance tests should not be scheduled too often, so that deployments can be better spaced out to force development to actually plan and design what is going to be in each release before they toss it over the wall.
3) IT doesn't see itself as having a stake in the deployed software. Since the software comes from development, and IT had no choice in its selection, any success makes development look good and does nothing to make IT look good. IT treats the situation as forced babysitting. -> the answer here is for IT to treat it like any other deployed system upon which they are dependent for success. Any bugs, or problems in the system should be reported to the "vendor". Anything which IT thinks could be done better should be sent back to the vendor as a requirement (to be assessed by the product manager to be sure).
If the company is too small to have very many people, then you can dual-role some people with the required skills,
but the job roles have got to be kept separate even if in the same person. You need people who can change hats and role-play effectively here so that oversight boundaries are not erased.
In software, as in so many other venues, good fences do make good neighbours.
Here at our shop, there is a gradual transition of ownership. I'll replace IT with Operations for us, because we're actually a tech company, and our IT department is the internal computers-on-desks group, where as the Operations team is the group that manages all fo the customer-facing money-making production sites.
Development owns their development environments. We provide as-can-do support for them, during business hours, when nothing else is going on. From there, a build manager (development does not do code builds eventually destined for Production) starts building code, and it deploying it with our pre-production team. In the first QA environment, Development and Pre-Prod work to make sure the release notes are complete, that all environmental and configuration changes are documented, so that the code will deploy to a downversioned environment. Development has significant, but not full, access to this environment. There is at least 1, and in some cases as many as 3, additional QA environments for each customer we have. Dev loses access at each step, until we get to Production, where they have a normal user-type account that isn't in any of the group IDs that the application install runs as, but still has access to view logfiles and such. In Production, Operations owns the install and the application.
There are a few assumptions that lead to "Dev owns Dev, Ops owns Prod". The biggest one is that, for each new release, the Release Notes HAVE to be complete. If we can't bring in a monkey off the street (OK, a technically oriented, slightly trained monkey...) to install the new code from the directions, then it doesn't get installed.
In addition, for the customer teams which run smoothest, we have several meetings each week. Just an hour, but we get people from Dev, QA, Operations, DBAs, Project Management, and Client Services in a room, and just go over the current issues. Sometimes the meeting is fast and over in 30 minutes, sometimes it runs long, but it gives everyone a chance to talk about what's going on and keep in the loop of current and pending releases. It seems that projects that have more problems don't have as much communication between teams have more problems in the transition from Development to QA to Production.
So, the answer you're probably lokoing for is that there needs to be a transition from Dev to Production. Dev needs to document, document, document. Dev can't be involved in the actual release of code. And the more often you can sit down with everyone involved, the more smoothly things will run.
This space for rent. Call 1-800-STEAK4U
Where I work, there are nine of us in the server team, looking after 350 production servers. Only we do any releases. We are all skilled programmers. Only we are on call.
Politics is often a problem, but our team refuses to release important software that hasn't been tested properly, since otherwise we have to support the mess at 2:00 AM.
The developers need to own the entire product
No, whoever supports it and interacts with the customers should own it. If it is an internal product then that may be IT, if it is a product for outsiders then that may be Tech Support. I've worked in both environments, and while my ego as a developer says I am at the top of the food chain and should be in charge of everything, I realize that things worked pretty well when others owned the deployment and interacted with customers. It was an interesting environment when Tech Support owned the distribution. We would create a build, do our own testing, and then deliver a text file with the CRCs of files. QA would take a wiped system and check out the source and all development tools from version control, build, and verify CRCs matched. An unexpected high number of problems were detected by this simple step. Putting barriers between developments and customers is probably a good idea.
It may sound highly bureaucratic to you, and yes, for a small change is like 5x times more slow but it works if you establish some policies.
I work for a fairly sized enterprise. There are 20+ engineers in development and only 6 of us in "infrastructure" (sys admins). Development teams simply don't touch the Test or Production environments, they have their own domain and servers which we help them to run on a you-broke-it-you-fix-it policy. All code, db scripts, catalog data, etc is managed by a Change Management Application, there are 3 people in change management that do the compilation of code and only them are allowed to upload executables (dlls, exes, jars, ears).
When the test environment admins receive a Change Order this package has already been checked for completeness (including deployment documentation and End User Manual deltas). If the test env. admins are not happy with the deployment documentation or if any script fails for something that is not infrastructure related we simply reject the package to development. Of course development has a chance to rebate the rejection but we usually have evidence for the failure prepared by then.
16,777,216 comments ought to be enough for any forum!
The arrangement I tend to advocate revolves around four teams:
Development The guys producing software. They shouldn't need to know anything about the servers, though potential server configurations need to be part of their requirements, so they don't do something stupid that prevents them from running. Software defects go to this team to be resolved. Things like data sources, remote services, usernames and passwords, etc., are specified through configuration, not hard-coded. Deployment The deployment team takes the software and deploys it on test/stage servers, and later on production servers. They are responsible for obtaining access to servers, data sources and other services and for configuring the application to use them. They may set up usernames and passwords, etc. This may be independent of, or a sub-team of either the development team or the Production Support team. Just pick one, define the lines of responsibility and stick to it. QA The QA team looks at the original requirements and verifies that the application meets those requirements. This could include functional testing and performance testing. This needs to be a team completely independent of the development team. Production Support This is the team that maintains the servers and the health of the application. They need to monitor the functionality of the application, analyze logs, and troubleshoot problems. If a software defect is likely, they submit it to the development team, but their job is to keep service going even in the face of defects, and to like it. (It may be cheaper to dedicate someone to nursing an application than it is to fix the defects that make that necessary. Suck it up.)If you have a hard time justifying this type of arrangement to your management, tell them that separating roles and responsibilities this way, and formalizing the interface between these teams, makes it pretty easy to outsource the Development role to India, because you'd have to do it that way anyway. :)
The process needs to be headed up by the head of development.
The head of development needs to tell the IT folks what to deploy and when.
The IT folks just do the move, they head up the systems and networks so they need to send out notices on unavailability. Once IT is done with the move they report back to the head of development.
Then the head of development asks a few key users to make sure the system is working.
After that the key users tell the rest of the users that the system is ready for prime time.
Whenever the responsibilities of designing, coding, testing, implementing, and maintaining internal software are split across two or more groups you end up with no one person who understands the entire system top to bottom as well as the entire set of requirements for all phases. IT should be there to keep user's desktops running, run internal email and ticketing systems, etc. Dev should be there to gather requirements, architect (both software and hardware), and implement internal software. They should also then own all of the deployment maintenance and management of all production systems running the software they have written. This forces developers to think about real production needs, scalability of processes during deployment, etc. It also ensures that the people with the most technical knowledge and understanding of the system are the ones responsible for its maintenance and that they have a large stake in making it reliable as they will be the ones getting paged at 2am when the system crashes. Finally, as to the actual deployment, it shouldn't really much matter who does it as it should be a completely automated task which takes no more than a few minutes and any monkey could do. Rollback to previous versions should be equally automated. Many comments on this thread are from IT guy's saying 'oh IT should own it, of course!' but please think things through and don't blindly accept the conventional wisdom. The only reason IT needs to own it in most shops is because of the broken separation of responsibilities that is already setup. If IT are the people who are getting fired when servers are down and who are getting paged and woken in the middle of the night then yes, of course they should be the only ones mucking with the bits on the servers as it's their neck on the line. However, the reality is things would be much smoother for both teams if dev's were the ones with their necks on the line and they were responsible for the systems they are coding for. As a still small company you have a chance to get this right. The final thing to consider is that good Dev's hate working for a company where they can't get work done and can't make unnecessary changes because of too much process and too many political barriers where the all too often less skilled workers in IT are trying to control everything. If you want to recruit top dev talent then you need to let your dev's control their own destiny and take responsibility for their own systems.
IT is strictly responsible for low level infrastructure (OS, hardware, physical network, power). Development teams own services and are responsible for their fleets in both a development and operational sense, and is responsible for notifying their upstream and downstream dependancies of changes in advance. Actual deployment (which, if it requires documentation, is not being supported by a sufficiently advanced deployment management system) to production is gated by Development's QA teams, who are responsible for testing on non production systems.
We used to have dedicated deployment engineers, but that just added friction, and guarenteed that the person doing the push to prod didn't know the full contents of what they were pushing.
Realities just a bunch of bits.
At my company we have a clear de-markation Dev does dev and IT looks after the env, including OS and all installs.
IT has it easy beacuse we have an automated install process which includes roll backs etc. The only time we have any sort of issue is when the automated install software needs to be re-configured. Then two people one from IT and one from Dev get given the task of developing and implementing the change.
Basically, promotion to production is a non-isuue due to automation.
Now if I could just get the stupid devs to automate unit testing and the stupid users to automate regression testing my life would be a lot easier.
As an IT Auditor, here's the ideal.
First, the department with the business need being filled owns the system. They make all important calls on what happens with the system. They also determine who gets what access to the system although they usually defer to the IT department to do their bidding.
For large scale projects still under development there are several entities involved. Developers, Quality Assurance, Users(represented by a user advocate), Project Management, Upper Management(represented by the project steering comittee) and IT staff.
The developers' job is to write code and communicate all relevant details of that code to the other parties (usually in the form of documentation). The QA person takes the code, tests it against the requirements and either sends it back to be developed some more or approves the code. IT's job is to maintain least priviledge access to the development environment, the test environment and the production environment while also maintaining the infrastructure. The user's job is to dictate requirements, give approval of functionality at milestones and do acceptance testing before deployment into production. Project Management coordinates all the entities in this particular project. Upper Management makes the business decisions about what resources will be allocated to the project. For large projects, there is also a project steering committee which ensures that the project goals are kept in sight, approves resource allocation within the project and handles escalated issue resolution.
As an IT auditor, here's what I look for concerning projects. First, there should be an established change management process in place which requires changes be introduced and approved by at least the user and often the user and the project manager. Ideally, the change process involves cost/benefit analysis and the costs are approved. Changes are then made to code and initially tested by the developers in the dev environment. The developers then request that changed code and object files are moved to the QA environment. Coders do not have access to code in QA, this is inadequate separation of duties as coders could make changes to the code slated for deployment after it has been tested. The QA environment should be as close to the production environment as possible and should not have compilers or other dev tools. If dev tools are present, the integrity of the process and of the QA people is questionable this is especially sticky in the case of malicious acts. The code is moved back and forth between Dev and QA until QA and the users approve. The moves are done by IT. When the code passes muster, it is moved into production by IT. Neither Dev or QA get anything more than read access to production. It is important that the code be moved alongside the objects that way, the production environment is in sync with te production code.
In this process, we trust IT the most since they move code and object from place to place. Additional controls should be in place to ensure that the IT person doing the move does not commit malicious acts. This could be anything from having one person move and another verify key aspects of the move to having the entire move process completely automated.
One final thing to consider, if the IT department is the user (project requestor/initiator) care must be taken to ensure that the IT staff doing all of the moving is objective. Additional controls may be required depending on the importance of the project.
We have fielded an enterprise solution (3 sites - Primary, Failover, DDR) with the requisite goodies(fibrechannel SANS, Oracle RAC, Sun Clusters,Windows and SUN servers, Windows clients.
We deploy new code releases, schema changes, etc., to the fielded system. We have sets of test servers in similar configurations to the fielded systems.
We have a deployment team that creates all the necessary patch and unpatch scripts and test them on the test systems to ensure that all paths work.
These scripts are created off of the deployment notes in the Clearquest item number for updating the system that the developer created when they made the change. So, it is the developer's responsibility to know what types of changes will be required on the live system to make the their change.
Now, the change developer may well work with a deployment developer in creating the required scripts. The modular package of those scripts, with dependencies cross checked - is deployed on a test system, probably multiple times. The developers are often called down to such tests to determine if th eissue is known or if a bug report should be filed. Systems engineers are responsible for checking the requirements that are supposed to be fufilled by what they negotiated with the customer.
We run full systems tests in the lab with real world feeds on a mostly continuous basis. Reconfiguring the clusters to different configurations for tests of different baselines.
Then, when the window is finalized with the customer we deploy the patch in that window. If after everything comes back up something is not right - depending on the severity the problem will either debugged on the spot for a period, logged for debugging or the patch will be removed.
IT's job is outside of all of this, including software deployment, except in such cases where the SW drop requires new IT support - a VPN stetup, more bandwidth, end-to-end ecryption (although could be done is sw), hell one of those new Quantum cryptography setups...
The sw engineering team also contains the infrastructure group, which works with IT on many details but deals with the deployment interface to IT.
If there are going to be IT infrastructure changes, the sw infrastructure team should know about any that could affect the sw functioning on that level (this rarely seems to happen for us), just as Operations should be notified if there is going to be a scheduled outage to operations.
The closest this gets generally are things like OS upgrades (major),java (now that took a little work!), DB upgrades (suprisingly mostly trouble free these days, for us), WebSphere, BMC, , etc. But many of these bigger ones are joint exercises. You know, everyone working together to solve a problem. We also test the hell out of it before ever fielding anything. Even then, there are conditions in the field that take your sw down old dusty rarely used paths every day, so it's not to say everything is perfect.
So, there's how we handle some of those interfaces.
So, how should one deploy changes?
...
1. Dev completes their changes and makes a release including operational details as needed.
2. QA/Testing roll the package to their staging environment and complete their testing. Pass goto 3 fail goto 1
3. Configuration Management (usually part of QA) releases the package with installation instructions
4. IT follows the instructions and rolls the application to the live environment
5. QA tests the operation in live and reports the status for a go/no-go on the changes
DBAs should package their changes in the form of repeatable scripts that are used to move the code and data to Staging, and Live. That reduces variability. Most DBAs already know the impact of their actions so they can perform the moves as requested by QA.
Now, before you ready the tar and feathers, it is possible to plan orderly releases that follow that process and it produces near zero failures in production. QA's job is to be the interface between the development activity and the real world. They have the discipline and skills to follow processes and keep bad things from happening. But QA has to have the power to call the shots.
If you do this
* Developers win because they no longer hold the bag for consequences of bad changes.
* IT wins because they know precisely what is going on and they are empowered to fix or restore stuff because they know exactly how to install the code without breaking something.
* Project managers who carefully orchestrate the whole process earn their keep.
* Micromanagers and others who like to call for quick hit changes to cover up for disorder and disarray somehow find their habits have no place in the organization.
Customers will be much happier and willing to accept slower and more orderly propagation of changes when they realize that they get better quality and uptime. Most of the pressure on development comes from emergency recovery from avoidable errors rather than actual work to be completed. One could argue that if the time from a request to acceptable code is measured, the prcess saves time overall.
I've often seen people try to split it up this way: dev creates the software, sysadmins (or IT or operations or whatever your company calls it) owns it after deployment. It seems logical to have people who are responsible for overseeing the network and machines, and having them install and run all the software on those machines sounds reasonable too.
In practice though, this has always led to a very adversarial relationship. The sysadmins probably don't know much about how the software works, so they end up having trouble with it and cursing the developers for making such impossible software. Very few dev shops have the resources to make software so polished that a person can run it without bothering to learn how it works. It costs tons of money to make software that easy to run.
And maybe that's the crux of the problem: handing off a technology to a different group is hard. It's easier for a group to own a piece of software through its whole lifecycle than to try to throw it over a wall at some point.
Lately, all of the interviews I read that talk about how things are run at the big Internet companies -- Google, Yahoo, Amazon -- talk about how they make the developers responsible for maintaining their software in production. The developers have a lot of personal investment in seeing their software work, so having them maintain it makes sense. The sysadmins take care of the network, the OS-level maintenance, and the hardware.
The only company where I saw this tried was also the most successful at keeping a solid and well-maintained production setup. It still had some friction between groups (DBAs vs. programmers, etc.) but a lot less than in other places.
If anyone has pointers to more reading about the production/maintenance practices of successful software companies, I'd certainly like to see them.
As someone who does development and IT, I'd say that this approach leaves out a lot. The development guys should be involved with the deployment. The IT guys should be involved in understanding the code better. Compartmentalizing things this way is guaranteed to miss an opportunity to educate both groups, and it WILL result in problems for your customers.
The Dev guys should be in charge of their own servers. Let IT handle the rest of the company; but make certain the developers are directly eating their own dogfood. You'll get a lot faster turn around time that way.
The single best IT infrastructure that I've ever seen was at Sun. There, you didn't originally have the concept of "IT". The developers built everything and rolled it out. Later, formal IT groups were established. But still, for the main machines, the developers still ran things.
Here's another example, that's guaranteed to piss some off, I'm afraid. The single silliest thing I've seen is a kernel hacker who can't admin their own machine. I'm sorry, but such folks just miss out on a wealth of knowledge about how Operating Systems work in practice.
In our company, everyone is on the same team. That is to say, we avoid the split between developer and IT staff, and instead consider us all as part of the team working on a particular software solution.
Just fixing this psychological schism helps quite a bit, as now we all work together to solve all those problems that always arise when you face a new deployment.
Of course, it also helps that we are a small enough company that you don't have to compete with anyone else in order to brown nose with management, which is something that seems to happen in bigger companies. Of course, once the competition starts, no one plays well together anymore, and then the company falls apart. That's why we fire the "children" who can't play nicely with all the other employees.
Anyway, the point is that there is no silver bullet, but communication and cooperation go a long way.
If you're asking the question, you have already gone wrong. Seriously, if the organization divides the functions and has fiefdom wars going on like this, you're sunk and someone higher than you will have to fix things.
In places where I've worked where this divide exists, I just don't get into the dispute. I do my work and let the office politicians worry about the next step, unless I'm specifically asked for my input.
By 'do my work', I mean do it right. In one environment where IT did the deployments, I got a vote of confidence from everyone who said "we don't worry about deployments with you, your shit just works."
You need to get your DEV to generate a package of software that's installable against their last release. That package goes to your QA to install in the QA environment as per the installation instructions that go with the package. If QA says the package is good send it up to your production/application/sys admins to install on the live system.
The key is to keep the process simple but never, ever let your DEV have access to your live production system unless it's a break/fix scenario and the admin is looking over their shoulder to see what's changing. My experience is that when DEV has access to the production system unaccounted changes will crop up like crazy and weird subsystems start to form in DEV owned areas. I saw a full application being run out of a programmers personal schema in a database; which of course crapped out while they were away. Not good.
Oops, how did this get here?
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
For a one-installation in house application, building a shrinkwrap-strength administrator's guide and installation system is a big expense. Better to have high bandwidth communication between IT and development.
...when you're writing a game...tweak the difficulty of "Easy" to something [your mother] can cope with. -- onion2k
As others have said, DEV doesn't belong in the deployment game, they belong in the development game. They take requirements and turn them into functioning products. IT takes functioning products and melds them into a "production infrastructure" of applications, networks, security access, servers & databases.
IT owns all the deployments, because they own the responsibility for keeping the production infrastructure functioning.
Reason why there is hope for the future generation #364:
"I wish my grass was emo so it could cut itself."
The first rule of operations is, Development does not have any access to the production servers.
The second rule of operations is, you can always roll back to the previous version without losing data.
The third rule of operations is, you always, always write an installer.
After that, the rules depend on how much money you have. In an ideal world, you've got 4 environments:
1) Dev. Minimal number of servers to build and test the app. Ops has no logins here, dev runs the show, they take care of sysadmin here and they can tweak everything how they want. CVS DOES NOT LIVE HERE. CVS lives on IT-controlled systems.
2) Sandbox (staging1): A staging mirror environment. Dev has logins here, and they handle deployments. Code tweaks to be rolled into the branch can be made here by dev. IT / OPS have logins here. (aka. alpha)
3) QA staging (staging2): A second staging mirror environment. Dev has no login here. IT can log in. Deployments are practiced here. Automated regression testing, bot-based QA, etc. all happens here. (aka. beta)
4) Production: Dev never touches production. Releases are packaged into an installer or an ear file or something similar. Changes are never made in such a way that they can't be backed out without losing customer data.
In an ideal world, the gap between a release leaving dev (step 1) and reaching production (step 4) is 3-4 weeks.
All that said, if you can show me a mid-sized organization that actually follows these rules, then for your next trick, I'd also like a pony and a supermodel to go with it.
Even Jesus hates listening to Creed.
> Better to have high bandwidth communication between IT and development.
You just managed to use more than one word to describe "synergy". Are you in marketing?
----
Now that we have that out of the way, let's look at who says when something is ready to advance to the next environment, shall we.?
First, what model do you use? Do you develop in one environment, test in the next, and run in a production environment? Or do you develop and test in one, and run in production in another? Or something different to either of the above?
If you are using the three environment approach, then you need to get sign off from the developers before you move the code to test, and signb off from the testers before you move the code to production - but it doesn't start there, it starts when you are planning the changes.
Two environment development is more difficult, as the testers and the developers need to give each other space (time) for each to do their jobs properly.
Whichever you are using, don't ever give undercooked code to the tester(s) to work with - "oh, here's the code I want you to sign off on - I'll be updating it again tonight, and tomorrow night, and every night for the next week" is a sure way to lose those testers before you even start. "This is ready for test, and unless you find bugs in it, I don't intend to change it" is much better.
Changes come about for many reasons, but let's look at a typical application enhancement. Even in the smallest shops, there is some sort of decision made that the enhancement is wanted. That decision needs to include all the stakeholders - the users, possibly outside clients, the developments, any architecture team you may have, the Data base guru, support, testers, and operations. Get them involved up front, and you'll get less surprises later on.
Then there is the matter of pushing your code etc from Dev to Test. If the people I listed as being involved in the concept end are kept up to date, there should be no surprises when you come to moving the code to test (if there are, someone isn't communicating, or is ignoring, for whatever reason, communications that they have received).
The same applies to pushing from Test toe Production, only this time you MUST have the go ahead from users, because they are the ones who are possibly going to have to change their practices because of the changes.
I haven't even mentioned Configuration Management yet. Even if you don't use a configuration management tool ("we're too small and we don't make mistakes when advancing software etc, so we don't need it"(famous last words!)), you should consider a Config Management Board 9or if you don't like that term, a Config Management team, consisting of the same group that gave the go ahead to the development in the first place. That way, the people who need to communicate are also the people who have to make decisions, and the people who should have a say so in the go/no go decision before the code goes live.
Bug fixes should be handled similarly, but on a much shorter (typically hours not weeks) time scale. Involve all the players early,at least by email, and get buy in early, then hold them to their decision when time comes to move the change into production. No "one man security", either, if it is currently possible. The person who builds the code, the person who tests the code, and the person who moves the code to production should always be separate people - not one person carrying out three roles.
What next? Mainly communicate, and get acceptance / buy in from ALL parties early, and make sure that the buy in is kept current by alerting all parties of any changes that might affect their acceptance of the solution. No surprises at any stage is a real key here.
Will those of you who think that you know what you are doing, get out of the way of those of us who know what we are doi
Who owns any part of an application? The group who is paying fopr it, of course!
The analysts don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to develop good, solid specifications.
The programmers don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to develop good, solid code.
The DBAs don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to maintain a sound data base structure, that meets the needs of the owner.
The testers don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to identify and eliminate defects in the product BEFORE it hits production.
The QA person / people don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to make sure that the development has been carried out using all the mandated procedures, processes, forms and templates.
The operations guys don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to keep the application running properly.
The IT Manager doesn't own it - he/she isn't paying for it. He/she has RESPONSIBILITIES towards it, of course, because he/she is being PAID by the owners to run the department that provides and supports the application.
The users don't own it - they aren't paying for it. They have RESPONSIBILITIES towards it, of course, because they are being PAID by the owners to work with the application, to the benefit of the owners.
So, we should be asking who is RESPONSIBLE for a particulr part of the development and deployment of new code, and I would answer ALL OF THE ABOVE!
Unless all of the above are involved in the decision making, they are put in a situation of responsibility without authority, which is totally unacceptable, and not tolerable by any of the groups involved.
So, communicate, communicate, communicate, and commit, commit, commit, and without commitment, don't proceed! And it, my friend, is the IT Manager's main responsibility to make sure is happening, efficiently, and effectively.
Will those of you who think that you know what you are doing, get out of the way of those of us who know what we are doi
It sounds like you have no staging environment for QA. Fix that problem, and the issue of who deploys the code will become moot.
Put another way: your problem isn't who is actually running the deployment script. Your problem is that the deployment script is wrong to begin with. It doesn't matter if Dev runs it, an admin runs it, or the President and CEO of the company runs it when it's wrong.
The two letters that are missing from your shop are 'Q' and 'A'. Fix your QA, and it don't matter who deploys the code.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
Please
:) Thx in advance!
get
your
Enter
key
fixed!!! Makes reading long posts much easier.