Slashdot Mirror
Ask Slashdot: Should Developers Install Their Software Themselves?
Paul Carver writes "Should developers be responsible for installing the software they develop into production environments? What about System Test environments? I'm not a developer and I'm not all that familiar with Agile or DevOps, but it seems unhealthy to me to have software installs done by developers. I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code. I'd like to hear opinions from developers. Do you prefer a workplace where you hand off packaged software to other teams to deploy or do you prefer to personally install your software into System Test and then personally install it into production once the System Testers have certified it? For context, I'm talking about enterprise grade, Internet facing web services sold to end users as well as large companies on either credit card billing or contractual basis with service level agreements and 24x7 Operations support. I'm not talking about little one (wo)man shops and free or Google style years long beta services."
Developers should concentrate on creating software. There are already tons of tools that help with the install and configuration state of software. Use InstallShield and the various Visual Studio install and config helpers. Visual Studio itself has many debugger functions available, and there are tons of extra helper plugins if required.
Developers should use those and make sure users can install their software themselves.
and I'm not all that familiar with Agile
I'm jealous.
My answer is mostly 'no'. The only benefit I see is that developers will become more motivated to have a simpler and better installation process. And that's a pretty nifty benefit.
But I've been on both sides of the fence. I've rarely been a developer who did anything less than thorough testing before declaring something 'done'. But I know that I'm an incredible rarity in that way. And on the devops side of things, the less ability developers have to push things, the more likely decent QA will get done before stuff ends up in production. But developers frequently also give you installation instructions that are unrepeatable special case installs with rollback instructions that make no sense.
I think that one good way to balance this is to have a preliminary test environment into which developers are allowed to push things. They are given limited rights to this environment, basically just the ability to upload some software and run a deployment script. This encourages them to write functioning deployment scripts. But it prevents them from shoving things into production because it just 'has' to go out today and it's such a small, low-risk thing. Of course it'll work!
Need a Python, C++, Unix, Linux develop
If you're following anything remotely like ITIL, your service transition phase should have an output that includes the processes for use in the operational phase of the service, including installation and upgrade procedures.
As a systems administrator, nothing frustrates me more than when a developer sends me an e-mail that says "install this".
First, they do not always say what the software is supposed to do, so I cannot prepare for any security requirements. I am rarely told if it needs a port opened, I have to check the security logs to see if the software is trying to communicate through the network.
Second, while I may have the ability to fix their software, I prefer not to mess with their code or configuration. Since I may not know what their software is supposed to do, I may get it running I do not know if it is operating properly.
Third, if you are asking me to install alpha or beta versions on a live system, it's usually a bad idea. I have no problem installing it on a test server or a VM, but I hate putting it on a production box.
sudo make me a sandwich
Title says it all - giving a developer access means they can deploy undocumented "hacks" and "quick fixes", usually meaning to document and normalise them later on.
Forcing the, to hand off installation and maintenance to a second team means documentation is enforced, standards are enforced and quick fixes are better vetted.
For the record, I wear both hats in different situations for different clients - as a developer I don't care about the production environment, and I like it that way. I care about the bugs the production environment uncovers, so the UAT environment should be identical, but even then the developer shouldn't have to care about it - that's the systems admin teams job.
Speaking from personal experience, the last person you want randomly installing software is a developer. You would think that because they work in IT they would know what they're doing, but you would be seriously wrong.
Time and time again I, or the folks I work with, have had to figure out what the developer did when we gave them admin rights to install software and now have a problem. It's tedious, time consuming and generally fruitless as the developer doesn't remember what they did. In the end, we end up having to remove the software anyway and start from scratch, now having wasted two to three times as much effort to do it right than letting a developer do it themselves.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
If it's enterprise class, you have a team responsible for the stability of the servers (support) that is not the same as the developers. Sometimes, you even have a team responsbile for just deployments (depends on size of the org). The developers should have access to install in DEV, but not to TEST. This allows you to test your deployment process (and backout process) before you touch your PROD environment.
http://en.wikipedia.org/wiki/Information_technology_controls
Even if not financial in nature, one of my former employers lumped it all unde SOX compliance.
Developers should concentrate on creating software.
Totally agreed, environments getting screwed up has lead to a lot of sacrificed man hours.
There are already tons of tools that help with the install and configuration state of software. Use InstallShield and the various Visual Studio install and config helpers. Visual Studio itself has many debugger functions available, and there are tons of extra helper plugins if required.
I think that's a bit overkill. Where I work we concentrate on having a unified development environment across boxes. Note that I said environment and not integrated development environment. While the IDE is important, we instead concentrate on maintaining a shell script that points to where things are installed so that there is a commonality in environment variables across boxes. We also like to zip up things that can be just zipped and unzipped and avoid the whole InstallShield mess altogether. So if we're using an agreed upon JDK, we put it in some directory of the zip (like dev/tools/jdkx.x.x) and then in that zip's environment scripts we point at that for JAVA_HOME. Then in Eclipse or Visual Studio or whatever you can tell it to find the preferred java runtime by pointing it at that environment string. In this way, we've managed to keep our development environment diverse with a large toolbox as well as possible to run in Linux, Windows, Cygwin and sometimes OSX (okay, we don't have OSX machines here but theoretically it'd be possible).
Nothing sucks more than sitting down at some coworkers box to help them and saying "What? Why doesn't this command work?" oh, "I guess I don't have that alias" or "I must have a different version of maven" or "I think I'm missing that Ruby gem" or "I don't know, I messed with Visual Studio a bunch and it hasn't worked since." Those are your nightmare scenarios and we try to make our dev box setup wiki page to avoid that at all costs. Two big things to focus on are a common environment and a diverse toolbox.
My work here is dung.
There was a time in my career that I'd have said yes, the developers make the most sense as they are the only ones that really understand the process. But now I know that's exactly the problem with having developers doing the installs. For a production system you need to have a well defined process that produces repeatable results. The only way to ensure that is to have a separation of duties, whether it's an administrator that's being intelligent hands for a human-readable script, or is simply kicking off a developer provided computer readable script.
What we do at my company is allow the developers to work with the project managers and deploy their applications out to a test environment for client facing review and acceptance as often as they like. This lets us do new test deployments quickly and easily with no red-tape. Once the project is a go for production then a formal request is made to move to the production server farm. The main guys in Ops, Dev, and the PM are brought into a meeting and make sure everything is taken care of ( SSL certs, DNS, monitoring, load balancing, number of nodes, etc ) then a go, no-go decision is made on the deployment. Once it's been decided that a production deployment is ready then the actual task of deploying the application is assigned to whoever wants it (usually the team lead) since the process of deploying to production is identical to deploying to test in our environments. Also, we use our continuous build server (Hudson) with a production maven profile for actually retrieving the war that is going to the server farm (i do Java web apps).
My personal preference as a sr.dev is to have other people do the deployments and verification as much as possible. It never ceases to amaze me how often over looked issues are found just because someone other than the person married to the code is doing things.
My best advice is, regardless of the size of your organization, map out a process on paper and follow it all day every day. You will appreciate the consistency when you get in those situations where a lot of balls are in the air at once.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
When the developers leave and their is no documentation and the thing blows up... No one will know how it works. With handing the product and the documentation off to someone else this provides a final check on the documentation to ensure that the documentation doesn't suck. Developers tend to intimately know their product well and therefore will be likely to leave out steps in the documentation, because they know how to do it anyway. I have seen this a number of times. When they leave it takes reverse engineering to figure out what was done. I am a big proponent of documentation. Here is how I think it should be done:
-Development happens where they are able to test using a test environment
-Developers hand off everything to the System Admin (SA) who will install it. They then install it on a test environment as well.. If there are issues found work with the developers to solve the issue, correct the documentation and proceed to step 3.
-Install in production.
The only issue with this is step 1 and 2 can sometimes become filled with accusations. SAs think the product sucks and Developers think that the SAs are idiots who need everything spelled out for them. It becomes a lot worst when the developers are contracted out (which is common). This needs to be avoided, both parties should see themselves as working together to create a better product.
Creating the flawless installers costs time.
Time means money. If the software is only installed twice (on test system and production), a good flawless installer is very expensive. If it is installed a lot of times (more than 5 systems), automation of the installation will pay itself.
Generally it is good practice to keep the developers out of the production environment.
However there must be exceptions.
-Emergency fixes.
-If there is no good team of maintainers, you can actually look at the installation logs, and understands them, the developer might be the better option. A good maintainer, who only blindly runs a script is not the best option.
-Uptime might be more important that the principle.
I work for a (mostly) US based Fortune 500 company who for various reasons I would prefer not to name. Short story - if you are a big enough company that external auditors come to visit you (we are) and your Dev people install code, even in test environments, you may fail an audit in the USA. Trust me, it is bad for business to fail this kind of audit. I'm in a system admin type job and a bit isolated from the workings of our Dev group, but it's my understanding that we run Agile or some variation of it and we absolutely do not allow Devs to ever install code in test or production environments.
You should have at least 3 environments beyond the personal ones the developers use to develop and unit-test code: development common, QA and production. Development common should be where dev does integration tests, including installation. If developers are responsible for creating the installation tools, they should be doing the installations there so they can debug those tools. If someone else is responsible for writing them, the devs should be working with them to make sure the tools do what the software needs to install correctly. You can't get the installation tools right if you don't test and debug them, and when they interact with the software the developers are the best ones to figure out what's wrong and what's needed to fix it.
Developers should not be doing the installation into the QA environment. They should be handing the installation tools over to QA and letting them run then according to the deployment instructions. That's the only way to confirm the instructions were really complete and that everything works per the documentation, by putting it in the hands of people who didn't write it and letting them deploy it. That way when it comes time to deploy in production you've got some assurance that the deployment will work because it has worked before.
Now, if things do go wrong you need dev involvement. They wrote the software and the tools, they'll often recognize exactly what's going wrong where Ops and QA won't. They're also the ones most likely to be able to give you a quick fix to the problem that'll get production up and running without having to back out and try another day. If you've already invested the time bringing the systems down and deploying the new version, it makes no sense to revert to the old version and waste more downtime tomorrow re-doing the deployment if the only problem is a path in a config file having the version number in it in QA but not in production and a quick edit of the newly-deployed copy of the config file will clearly fix the problem.
M.
it seems unhealthy to me to have software installs done by developers
"unhealthy" is putting it very nicely. It's insane.
The developers should never have to touch the production system. It is very, very important that the guys running the production system know how to install the software on it and get it up and running. That is exactly what they will have to do if everything burns down one Friday night.
Plus having the developers install it is an open invitation for all kinds of hacks and shortcuts instead of a proper deployment process.
Assorted stuff I do sometimes: Lemuria.org
I'm not sure a human should be installing anything manually on production environments. Upgrades should be done automatically.
"First they came for the slanderers and i said nothing."
NEVER. Never. never, never, never.
NEVER.
Signed,
Platform and Information Security Architect.
"Flyin' in just a sweet place,
Never been known to fail..."
I've seen software that gets deployed by the developers ... and it usually comes down to a bunch of manual steps and voodoo they don't share with other people. If you need a dev to install it, you have a horrid install process.
And, in my experience, a lot of developers can still be a little cavalier in dealing with Production machines. Sometimes just simply because they haven't in all cases figured out that Production outages are expensive and Really Bad Things. I've seen developers making manual changes in live environments which have caused outages. But far too many developers still have the mindset of "oh, we'll just do this" and don't always see the world outside of their little bubble and the possible impacts.
If I'd bought software, and was told it took a developer to install it -- that would be raising huge red flags for me.
As you say, for Enterprise software ... the change process should be much better thought out than having a dev just go in and make changes. Having been a developer, and having been now on the other side of things, in my experience, sometimes you need to wrangle the devs a little more to keep them on track.
To me, a dev largely shouldn't be anywhere near your prod environment unless you're 100% sure they're going to do absolutely everything by the book.
Lost at C:>. Found at C.
I would also insert a second layer of QA consisting of Project Managers and Product Managers. The primary QA guys are running regression tests and unit tests etc. You need to have someone actual sit down and drive the software too because Mercury and the like don't catch everything, especially logic and common sense things.
Automated tests are good and I prefer to write automated tests for all the software created by the developers with whom I work. But sometimes interactive tests are useful and/or necessary. Making sure that the core processing routine for your application does what it's supposed to do with known input? Automate that. Making sure that the interface (GUI, web page, etc.) passes the right stuff to the core processing routine when the user clicks on the nice shiny buttons? Harder to automate; it may be more cost-effective to create an interactive test for that.
On developers never having access to production:
In many cases, developers are the only people who understand the full application, and in many cases are the only people who can actually troubleshoot a botched install or figure out why things aren't working right in production. Yes, you are suposed to have some kind of QA or staging environment and you are not supposed to deploy bad code, but sometimes things go sideways. In these cases, only a developer who knows the code and any integration issues will be able to figure out what went wrong. Acting like developers should *never* have access to production is a lot like saying "the mechanic should never have access to my car's engine, ever". It makes sense 99.9% of the time, but there is a .1% where your engine is broken and the mechanic can't fix it without getting under the hood. Yes, Mr. System Administrator you can change your oil, rotate tires, and even change wiper blades but fixing a spun road bearing or smoked transmission solenoid is flat out.
On Developers and Access Rights:
There are a lot of developers who don't understand the computer they are developing software on. Usually, they are very BAD developers. Take for instance, a webdev who doesn't know Apache. Instead of using built in tools like mod_rewrite, the developer will build their own tools to do what is built in to apache. Good developers know their platform, often at a level that is much deeper because they take time to read code or API and config documentation so they understand the toolbox they are working with. Often a single line of configuration is more powerful than 1000's of line of code. Developers need to be administrators on at least their developement environments... usually extended to staging there is a large difference in scale between development (a VM on my laptop) to staging (multiple servers) and production (hundreds of servers).
On installer driven software:
It doesn't matter if you use installshield, roll your own RPMs or use Salt, Chef or Puppet. Any way you go you should do everything you can to automate installation. When you automate you reduce the chance of human mistakes in installation process. If you do installation automation right, then a deploy to production can be triggered by anyone with appropriate authority or any automated process with appropriate authority. Having people sit at the console and install software manually should be a red flag that the software you are buying sucks or is incomplete.
In Enterprise-Grade software:
Installatioin should be automated to the maximum extent possible, using the appropriate operating system installation tools. Documentation for the upgrade and install should be clear enough that a non-developer can successfully install and test the installation. Install activity should be logged, so that if something does go wrong, it can be figured out later.
-- $G