Slashdot Mirror
Ask Slashdot: Should Developers Install Their Software Themselves?
Paul Carver writes "Should developers be responsible for installing the software they develop into production environments? What about System Test environments? I'm not a developer and I'm not all that familiar with Agile or DevOps, but it seems unhealthy to me to have software installs done by developers. I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code. I'd like to hear opinions from developers. Do you prefer a workplace where you hand off packaged software to other teams to deploy or do you prefer to personally install your software into System Test and then personally install it into production once the System Testers have certified it? For context, I'm talking about enterprise grade, Internet facing web services sold to end users as well as large companies on either credit card billing or contractual basis with service level agreements and 24x7 Operations support. I'm not talking about little one (wo)man shops and free or Google style years long beta services."
and I'm not all that familiar with Agile
I'm jealous.
My answer is mostly 'no'. The only benefit I see is that developers will become more motivated to have a simpler and better installation process. And that's a pretty nifty benefit.
But I've been on both sides of the fence. I've rarely been a developer who did anything less than thorough testing before declaring something 'done'. But I know that I'm an incredible rarity in that way. And on the devops side of things, the less ability developers have to push things, the more likely decent QA will get done before stuff ends up in production. But developers frequently also give you installation instructions that are unrepeatable special case installs with rollback instructions that make no sense.
I think that one good way to balance this is to have a preliminary test environment into which developers are allowed to push things. They are given limited rights to this environment, basically just the ability to upload some software and run a deployment script. This encourages them to write functioning deployment scripts. But it prevents them from shoving things into production because it just 'has' to go out today and it's such a small, low-risk thing. Of course it'll work!
Need a Python, C++, Unix, Linux develop
This question was not directed as consumer application. It is direct at Enterprise applications. InstallShield just won't do it.
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
This is to the developers benefit. When new environments are set up, they shouldn't have to contact the developer to deploy the application to those new environments.
And as always. Titles with questions are typically answered with a "no".
As a systems administrator, nothing frustrates me more than when a developer sends me an e-mail that says "install this".
First, they do not always say what the software is supposed to do, so I cannot prepare for any security requirements. I am rarely told if it needs a port opened, I have to check the security logs to see if the software is trying to communicate through the network.
Second, while I may have the ability to fix their software, I prefer not to mess with their code or configuration. Since I may not know what their software is supposed to do, I may get it running I do not know if it is operating properly.
Third, if you are asking me to install alpha or beta versions on a live system, it's usually a bad idea. I have no problem installing it on a test server or a VM, but I hate putting it on a production box.
sudo make me a sandwich
Title says it all - giving a developer access means they can deploy undocumented "hacks" and "quick fixes", usually meaning to document and normalise them later on.
Forcing the, to hand off installation and maintenance to a second team means documentation is enforced, standards are enforced and quick fixes are better vetted.
For the record, I wear both hats in different situations for different clients - as a developer I don't care about the production environment, and I like it that way. I care about the bugs the production environment uncovers, so the UAT environment should be identical, but even then the developer shouldn't have to care about it - that's the systems admin teams job.
If it's enterprise class, you have a team responsible for the stability of the servers (support) that is not the same as the developers. Sometimes, you even have a team responsbile for just deployments (depends on size of the org). The developers should have access to install in DEV, but not to TEST. This allows you to test your deployment process (and backout process) before you touch your PROD environment.
http://en.wikipedia.org/wiki/Information_technology_controls
Even if not financial in nature, one of my former employers lumped it all unde SOX compliance.
Developers should concentrate on creating software.
Totally agreed, environments getting screwed up has lead to a lot of sacrificed man hours.
There are already tons of tools that help with the install and configuration state of software. Use InstallShield and the various Visual Studio install and config helpers. Visual Studio itself has many debugger functions available, and there are tons of extra helper plugins if required.
I think that's a bit overkill. Where I work we concentrate on having a unified development environment across boxes. Note that I said environment and not integrated development environment. While the IDE is important, we instead concentrate on maintaining a shell script that points to where things are installed so that there is a commonality in environment variables across boxes. We also like to zip up things that can be just zipped and unzipped and avoid the whole InstallShield mess altogether. So if we're using an agreed upon JDK, we put it in some directory of the zip (like dev/tools/jdkx.x.x) and then in that zip's environment scripts we point at that for JAVA_HOME. Then in Eclipse or Visual Studio or whatever you can tell it to find the preferred java runtime by pointing it at that environment string. In this way, we've managed to keep our development environment diverse with a large toolbox as well as possible to run in Linux, Windows, Cygwin and sometimes OSX (okay, we don't have OSX machines here but theoretically it'd be possible).
Nothing sucks more than sitting down at some coworkers box to help them and saying "What? Why doesn't this command work?" oh, "I guess I don't have that alias" or "I must have a different version of maven" or "I think I'm missing that Ruby gem" or "I don't know, I messed with Visual Studio a bunch and it hasn't worked since." Those are your nightmare scenarios and we try to make our dev box setup wiki page to avoid that at all costs. Two big things to focus on are a common environment and a diverse toolbox.
My work here is dung.
::Applause::
Sorry, I just do not have the mod points at the moment.
The exactly, correct response.
As a developer, there is only so much I can do. And I really do not want calls in the middle of the night. If I can describe how to install my software to a production support team, then the software release isn't ready for production.
There was a time in my career that I'd have said yes, the developers make the most sense as they are the only ones that really understand the process. But now I know that's exactly the problem with having developers doing the installs. For a production system you need to have a well defined process that produces repeatable results. The only way to ensure that is to have a separation of duties, whether it's an administrator that's being intelligent hands for a human-readable script, or is simply kicking off a developer provided computer readable script.
What we do at my company is allow the developers to work with the project managers and deploy their applications out to a test environment for client facing review and acceptance as often as they like. This lets us do new test deployments quickly and easily with no red-tape. Once the project is a go for production then a formal request is made to move to the production server farm. The main guys in Ops, Dev, and the PM are brought into a meeting and make sure everything is taken care of ( SSL certs, DNS, monitoring, load balancing, number of nodes, etc ) then a go, no-go decision is made on the deployment. Once it's been decided that a production deployment is ready then the actual task of deploying the application is assigned to whoever wants it (usually the team lead) since the process of deploying to production is identical to deploying to test in our environments. Also, we use our continuous build server (Hudson) with a production maven profile for actually retrieving the war that is going to the server farm (i do Java web apps).
My personal preference as a sr.dev is to have other people do the deployments and verification as much as possible. It never ceases to amaze me how often over looked issues are found just because someone other than the person married to the code is doing things.
My best advice is, regardless of the size of your organization, map out a process on paper and follow it all day every day. You will appreciate the consistency when you get in those situations where a lot of balls are in the air at once.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
When the developers leave and their is no documentation and the thing blows up... No one will know how it works. With handing the product and the documentation off to someone else this provides a final check on the documentation to ensure that the documentation doesn't suck. Developers tend to intimately know their product well and therefore will be likely to leave out steps in the documentation, because they know how to do it anyway. I have seen this a number of times. When they leave it takes reverse engineering to figure out what was done. I am a big proponent of documentation. Here is how I think it should be done:
-Development happens where they are able to test using a test environment
-Developers hand off everything to the System Admin (SA) who will install it. They then install it on a test environment as well.. If there are issues found work with the developers to solve the issue, correct the documentation and proceed to step 3.
-Install in production.
The only issue with this is step 1 and 2 can sometimes become filled with accusations. SAs think the product sucks and Developers think that the SAs are idiots who need everything spelled out for them. It becomes a lot worst when the developers are contracted out (which is common). This needs to be avoided, both parties should see themselves as working together to create a better product.
Creating the flawless installers costs time.
Time means money. If the software is only installed twice (on test system and production), a good flawless installer is very expensive. If it is installed a lot of times (more than 5 systems), automation of the installation will pay itself.
Generally it is good practice to keep the developers out of the production environment.
However there must be exceptions.
-Emergency fixes.
-If there is no good team of maintainers, you can actually look at the installation logs, and understands them, the developer might be the better option. A good maintainer, who only blindly runs a script is not the best option.
-Uptime might be more important that the principle.
I work for a (mostly) US based Fortune 500 company who for various reasons I would prefer not to name. Short story - if you are a big enough company that external auditors come to visit you (we are) and your Dev people install code, even in test environments, you may fail an audit in the USA. Trust me, it is bad for business to fail this kind of audit. I'm in a system admin type job and a bit isolated from the workings of our Dev group, but it's my understanding that we run Agile or some variation of it and we absolutely do not allow Devs to ever install code in test or production environments.
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
As a (web) developper, I strongly agree.
I'll just add that the Configuration Management team should have some knowledge about the software and the environment they manage.
I've often seen software come back because sombody did'nt have a clue what their job was. ("prerequisite: apache 2.x" should be enough for anyone: I don't have time to write a doc about how to install standard software, especially when I don't know the target server configuration)
You should have at least 3 environments beyond the personal ones the developers use to develop and unit-test code: development common, QA and production. Development common should be where dev does integration tests, including installation. If developers are responsible for creating the installation tools, they should be doing the installations there so they can debug those tools. If someone else is responsible for writing them, the devs should be working with them to make sure the tools do what the software needs to install correctly. You can't get the installation tools right if you don't test and debug them, and when they interact with the software the developers are the best ones to figure out what's wrong and what's needed to fix it.
Developers should not be doing the installation into the QA environment. They should be handing the installation tools over to QA and letting them run then according to the deployment instructions. That's the only way to confirm the instructions were really complete and that everything works per the documentation, by putting it in the hands of people who didn't write it and letting them deploy it. That way when it comes time to deploy in production you've got some assurance that the deployment will work because it has worked before.
Now, if things do go wrong you need dev involvement. They wrote the software and the tools, they'll often recognize exactly what's going wrong where Ops and QA won't. They're also the ones most likely to be able to give you a quick fix to the problem that'll get production up and running without having to back out and try another day. If you've already invested the time bringing the systems down and deploying the new version, it makes no sense to revert to the old version and waste more downtime tomorrow re-doing the deployment if the only problem is a path in a config file having the version number in it in QA but not in production and a quick edit of the newly-deployed copy of the config file will clearly fix the problem.
Of course the devs need to install it into their test environment. And QA needs to install it into their test environment, but Ops needs to install it on the production servers.
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
Sorry, no. I couldn't disagree more. I've worked in places like that, were developers were unable to get close to the production servers, things wouldn't work in production (but worked fine in dev) and we were unable to do anything except send builds with more and more debug info, working late nights to get things done, with a client more and more pissed each day.
Then it turns out that contrary to what they said, dev and prod wasn't identical, in a number of important things such as library versions.
If the install team is unable to install it by fuck's sake *get a developer see the installation process by himself* so he can come back just once with all the data he needs.
I'm a build and configuration manager (and a LONG long time software developer), and I completely agree.
If developers want to employ new (as in industry-wide, or new as in company-wide) technology, they are free to install, configure, test, and prove out the functionality and feasibility of the software on their own machine. But that should be the limit of their reach.
Any shared / integrated environment from QA on up to Pre-Prod and Prod, should be off-limits from a developers' sticky fingers. If there is an installation and / or configuration problem that arises with the application, AND a developer is needed for support, then they should absolutely be called in, but from a CONSULTATIVE role only. I.e., their fingers never touch a mouse or keyboard.
It's common (if not expected) for developers to build an intimate working relationship with the technology they're using. In the world of "familiarity breeds contempt", it means that a developer can overlook something that occurs "obvious" to them, but not to everyone else. How many times has a developer been "called in" to fix an installation problem, and the fix wasn't documented or proved out? (not to say that integrators are saints in this area, but they damn well SHOULD be). Or a developer has access to a test region, and just "hops onto the machine" to tweak a parameter that caused a test suite to fail?
QA (and QC Testers) need to count on the stability of a machine, it's known state. If a test is failing 100% of the time, it should keep failing. Or if it's passing 100% of the time, it should keep passing. Having the parameters of an integration machine change without the knowledge of the QA team (e.g. so that changes can be scheduled in, and updates to the testing suites can happen), then the validity of test runs is nullified, testing costs go through the roof, thus adding to the pressure to "skip the tests" and ship / deploy. Ick.
The instructions / script for installing a package on a machine should be EASILY understood by an installer who is skilled in the practice of software installation, and no more. (ie, not written for a senior engineer, not written for the janitor). Enough information to have it properly installed and configured, some basic troubleshooting, and a clear escalation path should issues not get resolved. Skip the 65 pages of configurable parameters if all the installer needs to alter are 12 parameters on the target machines. but don't skip ANY of those 12. If one is missed, find out why. If 10 extra are there, see which ones are needed for the different regions and skip the rest.
My personal line in the sand is the Developer integration area; that stays with the code monkeys. It's important to be able to test out package installs, and this is the type of machine upon which to do it (which is not to say there is a single DIT - have multiple, including one just to test out package installs if need be). QA regions and beyond are under tight control. I work in banks a lot, so Pre-prod and Prod are under a metaphorical armed guard.
Once the installation and config documentation is tested by the developers, the docs get thrown over the wall to the integration team (optimally, QA should be involved in a doc review to make sure that what's in the Doc is what is required, no more and no less). For a Waterfall / SDLC methodology, this documentation review and handover is one of the gating steps. For an Agile / Scrum / XP methology, this can be considered a single story, where the success condition of the task (story, etc) is working installation (works) and usable documentation (has been tested).
The key is not to go bat-crap crazy on it, but to ensure repeatability and workability. It would be GREAT if the install could be automated (or run unattended, or have little or no intermediate steps requiring human intervention) so as to reduce integration errors, but that is dependent on the requirements of those managing the QA regions and above.
I completely disagree. Developers should absolutely be involved with software installs. Rarely should they have the final say, but both operations and development staff benefit from working together on software installs.
The best example I can give for this is database installs. Working with the operations staff on installs helps developers better understand engine performance. They learn about things like prepared queries, connection pools, what tables remain paged into memory, etc. These are things that help the developers write better code. Similarly, the operations staff can learn what the application focuses are. They can optimize performance through VM provisioning, tablespace layout, memory pool size, etc. They can also understand the usage goals better, which lets them keep developers informed of important changes.
I've been running IT departments for over 10 years, and my experience has shown me that there is a definite benefit to having development and ops work together on installs.
Getting a developer to see the installation process is not the same as having the developer do the install as a matter of course. One is a debugging process, the other is development. If Dev and Prod aren't identical, then their Configuration Management team has failed. They need to learn that Test (and to an extent Dev) are just as important as Prod.
If Dev and Prod aren't identical, then their Configuration Management team has failed.
They usually aren't hardware wise for budget reasons, hardly ever are configuration and environment wise (considering other running programs as part the environment) and they definitely aren't identical user base wise.
While I understand devs shouldn't modify anything in prod, denying access to the system to see in real time what the fuck is going on when something doesn't work is just asking for the solution to take 10x, frustration, finger pointing and longer hours (usually the developers and often unpaid).
Seriously, nothing wrong in telling the developer "come down with me, I'll start the damn thing in front of you, you can check a bit of the system to you go back with a clue".
Sorry, no. I couldn't disagree more. I've worked in places like that, were developers were unable to get close to the production servers, things wouldn't work in production (but worked fine in dev) and we were unable to do anything except send builds with more and more debug info, working late nights to get things done, with a client more and more pissed each day.
Then it turns out that contrary to what they said, dev and prod wasn't identical, in a number of important things such as library versions.
If the install team is unable to install it by fuck's sake *get a developer see the installation process by himself* so he can come back just once with all the data he needs.
Then your process is screwed up. The solution isn't to open production to developers but to get your process fixed. Part of the process must be ensuring that the production environment is regularly and consistently copied over to development to ensure the problem you describe can't happen. That's the solution.
I have no problem with giving developers read-access so they can look at the production environment to "gather data" but developers must not be able to ever install or change anything in production. That's just good sense.
You didn't understand the beginning of my post. Having the developer watch is not the same as having the developer do the install. Read-only vs. read-write. I'm OK with dev being able to read-only. They can't "infect" production with their assumptions if they can't change anything. If they find something that needs adjusting then they should be adjusting their own environment to replicate the problem (or simulate it if necessary), construct the fix, then communicate the fix to those who should be doing changes to Test and Prod. If Dev and Prod aren't the same for whatever reasons, then the powers-that-be have to understand the costs that are incurred by not having them the same. You save the $5k on using cheaper hardware in Dev, but cost them $50k in downtime because that difference causes a bug to be exposed in Prod. (Picking numbers out of thin air). And yep, sometimes that $5k savings does end up really being $5k in savings as the difference in hardware had no impact on the environment. It's something that needs to be considered. (And yes, I'm very firmly on the development side of the fence, and my software gets installed into large production environments that I will never see, and in some cases am not legally allowed to see. Something about foreign nationals not allowed to touch or see any hardware that controls satellites...)
I'm sorry, I have a real problem with the underlying assumptions your answer makes about the process. There should not be a high wall between groups. Developers should not install it, but that doesn't mean the developers are not there when QA or Configuration Managment or whoever installs it. As a long-time developer, I learned more from the struggling of other people with my software than all the scaffolding and test-bedding done in isolation. Back when I was doing embedded programming, I made it a point to spend time in System Test to see how my software was being used...and misused. Next to me were the Documentation people, watching out for mistakes or head-scratching -- between us, we would see the holes that needed to be plugged so that the downstream processes would go more smoothly. And I would go out into the field, to customer sites, from time to time, particularly if a customer was reporting problems. This was particularly true of first launches, because sometimes the devils aren't seen until the customer hits them.
This was true for newspaper composition systems, newpaper press controls, bank check processing systems, key-entry systems, even a technical support group application.
I relate this story about the fallacy of compartmentalization: the General Manager gathers all the employees one Friday. Everyone had just been paid, the weeklies and the monthies. GM: "I'm not happy with the 'us versus them' attitude that seems to permeate this company. It's affecting our ability to get product customers want into their hands, so we all can get paid. So, tell you what: everyone pull out their paychecks, and fold them so the signature at the bottom is visable. See? All are signed by the same person. That should tell you something: that we should be working for the same goals, so we all continue to get paid." The change in that company was dramatic: instead of silos, it was more like an open-plan office writ large, with people talking with one another. One side benefit: sales stopped selling what we didn't have, and PARTICIPATED in the creation of new products. That company went from one step from closing it doors to being a booming business. My stock went from $1 to $65 a share. In six months. And the company was in the mid-West, not Silicon Valley
The softball team started doing a lot better, too.
NEVER. Never. never, never, never.
NEVER.
Signed,
Platform and Information Security Architect.
"Flyin' in just a sweet place,
Never been known to fail..."
depends on the software....
In terms of, for example, the web database etc. mentioned in the question you're into a lot of installing an SQL server (that's separate from a developer team generally, but it's a 30 minute job if you know what you're doing), and then building the database. Once you built the database you don't necessarily have an 'installer', you just send them an image of the whole installation (or the whole physical machine).
If you're a company that specializes in making billing software databases then sure, you might have an install script and setup parameters than your installer guys can execute.
There's sort of a hierarchy of techie people. Not everyone at a higher level can do the low level jobs, but one person at a higher level might be able to do all of the lower level stuff in 20 minutes that would take them weeks. This isn't 'consumer' product, in enterprise you might send out a 'linux guy' (who doesn't know linux scripting, that's an advanced linux guy) to do a linux install, but he can be given detailed, line by line instructions to follow from developers. In enterprise 'you' as the development firm can be responsible for getting it installed and setup. And you have co-op students who are basically high school grads following instructions for installations, or you can have a partially automated lab with software engineers running deployment tools to hundreds of machines at a time, it just depends on how big an outfit you are and the sort of customers you have.
One nice thing about virtualization. It's a lot easier to take a copy of a VM and spin it up somewhere so that the configuration *IS* the same and you can debug something without affecting the live systems. (Ok, technically there is a delta -- security permissions -- but that's the only thing that should be different, although it does sometimes make a difference.)
I have often worked as a contractor for major corporations, and I have worked with contractors as a system administrator. I was not accusing you of anything but I was criticizing the environment you were forced to operate in. If the development environment gets so out of sync with production that installs succeed in development and fail in production, the solution is not to allow developers to change production but to correct the environment.
If production doesn't match development (and/or test) then the whole process is broken! Development is invalid and testing is invalid. Both development and test must duplicate production or absolutely nothing done in development or tested in test can be trusted.
As a developer, years ago, I actually modified a running production environment and I was very lucky that nothing went wrong. I understand that it happens and, in broken environments, sometimes it must be done, but that doesn't make it safe or right.
You Sir, are absolutely correct.
Database changes are a tricky matter though. Despite testing our upgrades on a copy, we feel its safest to
one or two developers on site during software upgrades, in the event that something goes pear shaped.
I also think if developers can be involved in running and monitoring the actual system, you will get
better stability, better diagnostics and simpler handling.
(Banking systems, inhouse "enterprisey" applications on unix servers)
If I can describe how to install my software to a production support team, then the software release isn't ready for production
- BOFH, is that you?
MY OTHER COMMENTS