Slashdot Mirror
Ask Slashdot: Should Developers Install Their Software Themselves?
Paul Carver writes "Should developers be responsible for installing the software they develop into production environments? What about System Test environments? I'm not a developer and I'm not all that familiar with Agile or DevOps, but it seems unhealthy to me to have software installs done by developers. I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code. I'd like to hear opinions from developers. Do you prefer a workplace where you hand off packaged software to other teams to deploy or do you prefer to personally install your software into System Test and then personally install it into production once the System Testers have certified it? For context, I'm talking about enterprise grade, Internet facing web services sold to end users as well as large companies on either credit card billing or contractual basis with service level agreements and 24x7 Operations support. I'm not talking about little one (wo)man shops and free or Google style years long beta services."
My answer is mostly 'no'. The only benefit I see is that developers will become more motivated to have a simpler and better installation process. And that's a pretty nifty benefit.
But I've been on both sides of the fence. I've rarely been a developer who did anything less than thorough testing before declaring something 'done'. But I know that I'm an incredible rarity in that way. And on the devops side of things, the less ability developers have to push things, the more likely decent QA will get done before stuff ends up in production. But developers frequently also give you installation instructions that are unrepeatable special case installs with rollback instructions that make no sense.
I think that one good way to balance this is to have a preliminary test environment into which developers are allowed to push things. They are given limited rights to this environment, basically just the ability to upload some software and run a deployment script. This encourages them to write functioning deployment scripts. But it prevents them from shoving things into production because it just 'has' to go out today and it's such a small, low-risk thing. Of course it'll work!
Need a Python, C++, Unix, Linux develop
This question was not directed as consumer application. It is direct at Enterprise applications. InstallShield just won't do it.
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
This is to the developers benefit. When new environments are set up, they shouldn't have to contact the developer to deploy the application to those new environments.
And as always. Titles with questions are typically answered with a "no".
As a systems administrator, nothing frustrates me more than when a developer sends me an e-mail that says "install this".
First, they do not always say what the software is supposed to do, so I cannot prepare for any security requirements. I am rarely told if it needs a port opened, I have to check the security logs to see if the software is trying to communicate through the network.
Second, while I may have the ability to fix their software, I prefer not to mess with their code or configuration. Since I may not know what their software is supposed to do, I may get it running I do not know if it is operating properly.
Third, if you are asking me to install alpha or beta versions on a live system, it's usually a bad idea. I have no problem installing it on a test server or a VM, but I hate putting it on a production box.
sudo make me a sandwich
Title says it all - giving a developer access means they can deploy undocumented "hacks" and "quick fixes", usually meaning to document and normalise them later on.
Forcing the, to hand off installation and maintenance to a second team means documentation is enforced, standards are enforced and quick fixes are better vetted.
For the record, I wear both hats in different situations for different clients - as a developer I don't care about the production environment, and I like it that way. I care about the bugs the production environment uncovers, so the UAT environment should be identical, but even then the developer shouldn't have to care about it - that's the systems admin teams job.
There was a time in my career that I'd have said yes, the developers make the most sense as they are the only ones that really understand the process. But now I know that's exactly the problem with having developers doing the installs. For a production system you need to have a well defined process that produces repeatable results. The only way to ensure that is to have a separation of duties, whether it's an administrator that's being intelligent hands for a human-readable script, or is simply kicking off a developer provided computer readable script.
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
As a (web) developper, I strongly agree.
I'll just add that the Configuration Management team should have some knowledge about the software and the environment they manage.
I've often seen software come back because sombody did'nt have a clue what their job was. ("prerequisite: apache 2.x" should be enough for anyone: I don't have time to write a doc about how to install standard software, especially when I don't know the target server configuration)
You should have at least 3 environments beyond the personal ones the developers use to develop and unit-test code: development common, QA and production. Development common should be where dev does integration tests, including installation. If developers are responsible for creating the installation tools, they should be doing the installations there so they can debug those tools. If someone else is responsible for writing them, the devs should be working with them to make sure the tools do what the software needs to install correctly. You can't get the installation tools right if you don't test and debug them, and when they interact with the software the developers are the best ones to figure out what's wrong and what's needed to fix it.
Developers should not be doing the installation into the QA environment. They should be handing the installation tools over to QA and letting them run then according to the deployment instructions. That's the only way to confirm the instructions were really complete and that everything works per the documentation, by putting it in the hands of people who didn't write it and letting them deploy it. That way when it comes time to deploy in production you've got some assurance that the deployment will work because it has worked before.
Now, if things do go wrong you need dev involvement. They wrote the software and the tools, they'll often recognize exactly what's going wrong where Ops and QA won't. They're also the ones most likely to be able to give you a quick fix to the problem that'll get production up and running without having to back out and try another day. If you've already invested the time bringing the systems down and deploying the new version, it makes no sense to revert to the old version and waste more downtime tomorrow re-doing the deployment if the only problem is a path in a config file having the version number in it in QA but not in production and a quick edit of the newly-deployed copy of the config file will clearly fix the problem.
Of course the devs need to install it into their test environment. And QA needs to install it into their test environment, but Ops needs to install it on the production servers.
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
I'm a build and configuration manager (and a LONG long time software developer), and I completely agree.
If developers want to employ new (as in industry-wide, or new as in company-wide) technology, they are free to install, configure, test, and prove out the functionality and feasibility of the software on their own machine. But that should be the limit of their reach.
Any shared / integrated environment from QA on up to Pre-Prod and Prod, should be off-limits from a developers' sticky fingers. If there is an installation and / or configuration problem that arises with the application, AND a developer is needed for support, then they should absolutely be called in, but from a CONSULTATIVE role only. I.e., their fingers never touch a mouse or keyboard.
It's common (if not expected) for developers to build an intimate working relationship with the technology they're using. In the world of "familiarity breeds contempt", it means that a developer can overlook something that occurs "obvious" to them, but not to everyone else. How many times has a developer been "called in" to fix an installation problem, and the fix wasn't documented or proved out? (not to say that integrators are saints in this area, but they damn well SHOULD be). Or a developer has access to a test region, and just "hops onto the machine" to tweak a parameter that caused a test suite to fail?
QA (and QC Testers) need to count on the stability of a machine, it's known state. If a test is failing 100% of the time, it should keep failing. Or if it's passing 100% of the time, it should keep passing. Having the parameters of an integration machine change without the knowledge of the QA team (e.g. so that changes can be scheduled in, and updates to the testing suites can happen), then the validity of test runs is nullified, testing costs go through the roof, thus adding to the pressure to "skip the tests" and ship / deploy. Ick.
The instructions / script for installing a package on a machine should be EASILY understood by an installer who is skilled in the practice of software installation, and no more. (ie, not written for a senior engineer, not written for the janitor). Enough information to have it properly installed and configured, some basic troubleshooting, and a clear escalation path should issues not get resolved. Skip the 65 pages of configurable parameters if all the installer needs to alter are 12 parameters on the target machines. but don't skip ANY of those 12. If one is missed, find out why. If 10 extra are there, see which ones are needed for the different regions and skip the rest.
My personal line in the sand is the Developer integration area; that stays with the code monkeys. It's important to be able to test out package installs, and this is the type of machine upon which to do it (which is not to say there is a single DIT - have multiple, including one just to test out package installs if need be). QA regions and beyond are under tight control. I work in banks a lot, so Pre-prod and Prod are under a metaphorical armed guard.
Once the installation and config documentation is tested by the developers, the docs get thrown over the wall to the integration team (optimally, QA should be involved in a doc review to make sure that what's in the Doc is what is required, no more and no less). For a Waterfall / SDLC methodology, this documentation review and handover is one of the gating steps. For an Agile / Scrum / XP methology, this can be considered a single story, where the success condition of the task (story, etc) is working installation (works) and usable documentation (has been tested).
The key is not to go bat-crap crazy on it, but to ensure repeatability and workability. It would be GREAT if the install could be automated (or run unattended, or have little or no intermediate steps requiring human intervention) so as to reduce integration errors, but that is dependent on the requirements of those managing the QA regions and above.
Sorry, no. I couldn't disagree more. I've worked in places like that, were developers were unable to get close to the production servers, things wouldn't work in production (but worked fine in dev) and we were unable to do anything except send builds with more and more debug info, working late nights to get things done, with a client more and more pissed each day.
Then it turns out that contrary to what they said, dev and prod wasn't identical, in a number of important things such as library versions.
If the install team is unable to install it by fuck's sake *get a developer see the installation process by himself* so he can come back just once with all the data he needs.
Then your process is screwed up. The solution isn't to open production to developers but to get your process fixed. Part of the process must be ensuring that the production environment is regularly and consistently copied over to development to ensure the problem you describe can't happen. That's the solution.
I have no problem with giving developers read-access so they can look at the production environment to "gather data" but developers must not be able to ever install or change anything in production. That's just good sense.
You didn't understand the beginning of my post. Having the developer watch is not the same as having the developer do the install. Read-only vs. read-write. I'm OK with dev being able to read-only. They can't "infect" production with their assumptions if they can't change anything. If they find something that needs adjusting then they should be adjusting their own environment to replicate the problem (or simulate it if necessary), construct the fix, then communicate the fix to those who should be doing changes to Test and Prod. If Dev and Prod aren't the same for whatever reasons, then the powers-that-be have to understand the costs that are incurred by not having them the same. You save the $5k on using cheaper hardware in Dev, but cost them $50k in downtime because that difference causes a bug to be exposed in Prod. (Picking numbers out of thin air). And yep, sometimes that $5k savings does end up really being $5k in savings as the difference in hardware had no impact on the environment. It's something that needs to be considered. (And yes, I'm very firmly on the development side of the fence, and my software gets installed into large production environments that I will never see, and in some cases am not legally allowed to see. Something about foreign nationals not allowed to touch or see any hardware that controls satellites...)
I have often worked as a contractor for major corporations, and I have worked with contractors as a system administrator. I was not accusing you of anything but I was criticizing the environment you were forced to operate in. If the development environment gets so out of sync with production that installs succeed in development and fail in production, the solution is not to allow developers to change production but to correct the environment.
If production doesn't match development (and/or test) then the whole process is broken! Development is invalid and testing is invalid. Both development and test must duplicate production or absolutely nothing done in development or tested in test can be trusted.
As a developer, years ago, I actually modified a running production environment and I was very lucky that nothing went wrong. I understand that it happens and, in broken environments, sometimes it must be done, but that doesn't make it safe or right.