Slashdot Mirror
Ask Slashdot: Should Developers Install Their Software Themselves?
Paul Carver writes "Should developers be responsible for installing the software they develop into production environments? What about System Test environments? I'm not a developer and I'm not all that familiar with Agile or DevOps, but it seems unhealthy to me to have software installs done by developers. I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code. I'd like to hear opinions from developers. Do you prefer a workplace where you hand off packaged software to other teams to deploy or do you prefer to personally install your software into System Test and then personally install it into production once the System Testers have certified it? For context, I'm talking about enterprise grade, Internet facing web services sold to end users as well as large companies on either credit card billing or contractual basis with service level agreements and 24x7 Operations support. I'm not talking about little one (wo)man shops and free or Google style years long beta services."
Developers should concentrate on creating software. There are already tons of tools that help with the install and configuration state of software. Use InstallShield and the various Visual Studio install and config helpers. Visual Studio itself has many debugger functions available, and there are tons of extra helper plugins if required.
Developers should use those and make sure users can install their software themselves.
and I'm not all that familiar with Agile
I'm jealous.
"I'm talking about enterprise grade, Internet facing web services sold to end users"
If it is a web application, and if they are internal customers - if the software is being written by one department of the company for other departments to use - and if the software is properly documented by the developers, I don't have a problem with the developers deploying it.
Otherwise, um, yeah, the deployment/installation manual should document it well enough for an outsider to be able to install it.
My answer is mostly 'no'. The only benefit I see is that developers will become more motivated to have a simpler and better installation process. And that's a pretty nifty benefit.
But I've been on both sides of the fence. I've rarely been a developer who did anything less than thorough testing before declaring something 'done'. But I know that I'm an incredible rarity in that way. And on the devops side of things, the less ability developers have to push things, the more likely decent QA will get done before stuff ends up in production. But developers frequently also give you installation instructions that are unrepeatable special case installs with rollback instructions that make no sense.
I think that one good way to balance this is to have a preliminary test environment into which developers are allowed to push things. They are given limited rights to this environment, basically just the ability to upload some software and run a deployment script. This encourages them to write functioning deployment scripts. But it prevents them from shoving things into production because it just 'has' to go out today and it's such a small, low-risk thing. Of course it'll work!
Need a Python, C++, Unix, Linux develop
If you're following anything remotely like ITIL, your service transition phase should have an output that includes the processes for use in the operational phase of the service, including installation and upgrade procedures.
As a systems administrator, nothing frustrates me more than when a developer sends me an e-mail that says "install this".
First, they do not always say what the software is supposed to do, so I cannot prepare for any security requirements. I am rarely told if it needs a port opened, I have to check the security logs to see if the software is trying to communicate through the network.
Second, while I may have the ability to fix their software, I prefer not to mess with their code or configuration. Since I may not know what their software is supposed to do, I may get it running I do not know if it is operating properly.
Third, if you are asking me to install alpha or beta versions on a live system, it's usually a bad idea. I have no problem installing it on a test server or a VM, but I hate putting it on a production box.
sudo make me a sandwich
In the straightforward case a distribution package should be made and that could be used by just about anyone to install the software (SW). In other cases, depending on the target system, it may be more technical. If the SW interacts with other SW (like a database or some data collection app) quality checks must be completed to ensure the overall systems stability. There are also platform specific issues to be concerned with. It's also a very good idea in some situations to have the developer install the application as they should be performing other quality control steps (like cleaning temp files and other old log files).
Let's face it, some developers lack IT skills. Most of us have the understanding and appreciation of operating systems, security, and software to manage a safe environment, but some don't.
We also have peculiar environment requirements from time to time, and great need to experiment with different software setups in the course of our work. Engineers will make problems for IT, no matter what. The natural conclusion is to standardize as much as you can, grant admin access where you can't, and be prepared to put out some fires.
On Big Projects intalliation is team effort that takes several hours over a weekend to install at one location. On some projects building the application is a full time job for a "build" lead.
Title says it all - giving a developer access means they can deploy undocumented "hacks" and "quick fixes", usually meaning to document and normalise them later on.
Forcing the, to hand off installation and maintenance to a second team means documentation is enforced, standards are enforced and quick fixes are better vetted.
For the record, I wear both hats in different situations for different clients - as a developer I don't care about the production environment, and I like it that way. I care about the bugs the production environment uncovers, so the UAT environment should be identical, but even then the developer shouldn't have to care about it - that's the systems admin teams job.
Separation of duties is important. Developers should be developing. Release people should be releasing, and there should be controls around that.
I've no experience of a company large enough to have a whole separate set of people to do installations, so my comment is from the small company experience.
Problems will get fixed faster if the developer has access to the installation and is familiar with the environment. Live environments are never the same as test environments, so for software above a certain level of complexity there are likely to be site-specific issues.
But it takes a good developer to tread carefully around a live system and to then retro any fixes back into the code base and test it. A methodical release/report/debug/fix/rerelease cycle -- whether or not it's the developer doing the installing -- might be slow and tedious but will have a lower chance of regression errors. It depends very much on whether the customer can afford/tolerate the time and disruption to live systems involved.
If the dev team did their job right, installation of the software should be easy enough to have no more than a one page instruction sheet with some basic guidelines. Even horribly complicated apps, like a client/server AV, can be deployed on a network with minimum fuss when the software is built right. If I'm having to go in and adjust registry settings manually for your software, your development team is doing it wrong.
Occasionally living proof of the Ballmer peak.
I work as a Business Analyst in a large insurance centric company. We have developers that write or supervise other developers, DBAs that manage a large database and a staff of developers responsible for installing everything from Microsoft updates to custom code for both client facing (web based applications) to internal locally installed applications. I have seen that each silo can be either a help or hinderance to the process. I invite developer or deploy specialist to come into my area during the deploy process, this has immensly helped their understanding of the unique situation faced by each business area. Smaller operations would surely have different needs.
No.
Speaking from personal experience, the last person you want randomly installing software is a developer. You would think that because they work in IT they would know what they're doing, but you would be seriously wrong.
Time and time again I, or the folks I work with, have had to figure out what the developer did when we gave them admin rights to install software and now have a problem. It's tedious, time consuming and generally fruitless as the developer doesn't remember what they did. In the end, we end up having to remove the software anyway and start from scratch, now having wasted two to three times as much effort to do it right than letting a developer do it themselves.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
If it's enterprise class, you have a team responsible for the stability of the servers (support) that is not the same as the developers. Sometimes, you even have a team responsbile for just deployments (depends on size of the org). The developers should have access to install in DEV, but not to TEST. This allows you to test your deployment process (and backout process) before you touch your PROD environment.
http://en.wikipedia.org/wiki/Information_technology_controls
Even if not financial in nature, one of my former employers lumped it all unde SOX compliance.
Developers should concentrate on creating software.
Totally agreed, environments getting screwed up has lead to a lot of sacrificed man hours.
There are already tons of tools that help with the install and configuration state of software. Use InstallShield and the various Visual Studio install and config helpers. Visual Studio itself has many debugger functions available, and there are tons of extra helper plugins if required.
I think that's a bit overkill. Where I work we concentrate on having a unified development environment across boxes. Note that I said environment and not integrated development environment. While the IDE is important, we instead concentrate on maintaining a shell script that points to where things are installed so that there is a commonality in environment variables across boxes. We also like to zip up things that can be just zipped and unzipped and avoid the whole InstallShield mess altogether. So if we're using an agreed upon JDK, we put it in some directory of the zip (like dev/tools/jdkx.x.x) and then in that zip's environment scripts we point at that for JAVA_HOME. Then in Eclipse or Visual Studio or whatever you can tell it to find the preferred java runtime by pointing it at that environment string. In this way, we've managed to keep our development environment diverse with a large toolbox as well as possible to run in Linux, Windows, Cygwin and sometimes OSX (okay, we don't have OSX machines here but theoretically it'd be possible).
Nothing sucks more than sitting down at some coworkers box to help them and saying "What? Why doesn't this command work?" oh, "I guess I don't have that alias" or "I must have a different version of maven" or "I think I'm missing that Ruby gem" or "I don't know, I messed with Visual Studio a bunch and it hasn't worked since." Those are your nightmare scenarios and we try to make our dev box setup wiki page to avoid that at all costs. Two big things to focus on are a common environment and a diverse toolbox.
My work here is dung.
There was a time in my career that I'd have said yes, the developers make the most sense as they are the only ones that really understand the process. But now I know that's exactly the problem with having developers doing the installs. For a production system you need to have a well defined process that produces repeatable results. The only way to ensure that is to have a separation of duties, whether it's an administrator that's being intelligent hands for a human-readable script, or is simply kicking off a developer provided computer readable script.
What we do at my company is allow the developers to work with the project managers and deploy their applications out to a test environment for client facing review and acceptance as often as they like. This lets us do new test deployments quickly and easily with no red-tape. Once the project is a go for production then a formal request is made to move to the production server farm. The main guys in Ops, Dev, and the PM are brought into a meeting and make sure everything is taken care of ( SSL certs, DNS, monitoring, load balancing, number of nodes, etc ) then a go, no-go decision is made on the deployment. Once it's been decided that a production deployment is ready then the actual task of deploying the application is assigned to whoever wants it (usually the team lead) since the process of deploying to production is identical to deploying to test in our environments. Also, we use our continuous build server (Hudson) with a production maven profile for actually retrieving the war that is going to the server farm (i do Java web apps).
My personal preference as a sr.dev is to have other people do the deployments and verification as much as possible. It never ceases to amaze me how often over looked issues are found just because someone other than the person married to the code is doing things.
My best advice is, regardless of the size of your organization, map out a process on paper and follow it all day every day. You will appreciate the consistency when you get in those situations where a lot of balls are in the air at once.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
You'd fail an audit if the developers installed the software.
When the developers leave and their is no documentation and the thing blows up... No one will know how it works. With handing the product and the documentation off to someone else this provides a final check on the documentation to ensure that the documentation doesn't suck. Developers tend to intimately know their product well and therefore will be likely to leave out steps in the documentation, because they know how to do it anyway. I have seen this a number of times. When they leave it takes reverse engineering to figure out what was done. I am a big proponent of documentation. Here is how I think it should be done:
-Development happens where they are able to test using a test environment
-Developers hand off everything to the System Admin (SA) who will install it. They then install it on a test environment as well.. If there are issues found work with the developers to solve the issue, correct the documentation and proceed to step 3.
-Install in production.
The only issue with this is step 1 and 2 can sometimes become filled with accusations. SAs think the product sucks and Developers think that the SAs are idiots who need everything spelled out for them. It becomes a lot worst when the developers are contracted out (which is common). This needs to be avoided, both parties should see themselves as working together to create a better product.
Software installs need to be repeatable, and need to be something which can be done by someone other than the devs.
Even for testing, unless you're talking about fully automated testing, it is often good to have someone other than the developer doing the tests. Because in QA if you do something random the developer hasn't encountered or thought of, you can sometimes get very interesting results.
I can't even begin to think of the number of times where a user or QA does something, files an issue, and the developer says "well why would you do that?" thinking it absurd someone would have deviated from their carefully plotted out use case.
People develop their own blind-spots to the software they work on, and tend to use it the way you're "supposed" to, and then they miss some corner cases where you do something totally unexpected.
IMO, if you need your developers to install in Prod, your software is a little on the creaky side and it isn't going to work well for customers. If you need a developer to install it, either you're going to be deploying it in a small number of environments (making it ridiculously annoying to maintain), or there's something broken about how you install.
Lost at C:>. Found at C.
With the somewhat bigger, more enterprise-y applications hardware and application simply depend on each other in terms of availability, performance and scalability. Therefore, placing this responsibility at just one team (be it developers, be it administrators) will never work. What we've done is use Puppet to automate all server installs (Chef could be an alternative). Every system from development (relying on VirtualBox and Vagrant) to Production uses puppet. A developer simply writes the puppet manifests he needs to get the software working. Then, that is reviewed by the administrator (or devop, so you wish) who may also optimize it here and there for the relevant environments. Once both sides of the game agree, the new puppet manifests are deployed across the relevant environments together with the application itself.
This also ensures that on development the same configs are used as they are on production (although on development you'll hardly ever replicate an e.g. load balanced setup).
I have been in various enterprise environments as a developer and I would have to say the best environments had a deployment team who were responsible for this.
By having a team/person responsible for application installation/upgrading they can focus on the responsibility of ensuring things are properly backed up and that installations can be timed properly. By knowing the state of the production servers they can make informed choices about what is going on and typically have the permissions to view logs/set permissions/make server configuration changes that a developer should not have permissions to. This leads to less problematic installations/upgrades I have found.
By forcing having a documented installation procedure (and post installation configuration too) the deployment is repeatable. If the developer is gone and the application needs to be installed on another server how does someone know what to do? By having that documented and packaged as one unit it should be relatively easy to install ten years down the road if need be.
Another thing I have found with developers deploying is that they may shoehorn a project in. Some examples are turning on debug messages to diagnose a problem and never turning it back off which can be a security issue. They may also patch things on the fly without documenting that, the next person to deploy may not know that step was done and overwrite a change or run into the exact same issue without knowing what was done to solve it the first time. This can end up being very wasteful.
Normally I'd much prefer the people actually maintaining and supporting the software to install it in QA and promote it to Production.
This gives them some more knowledge about what they're supporting, and also avoids the 'Just gotta change this' mentality so prevalent in developers :)
Coz eternity my friend, is a long *ing time.
We'll assume for many reasons that this is not a Windows-based package, but it can still be bundled into versioned installable packages appropriate for the operating system, or at the very least use WAR packages to deploy web app updates.
If the standard production deployment technique is too complex, it becomes its own significant risk area in the product. It therefore requires that QA test the Dev-provided deployment, upgrade, and rollback procedures in test or staging environments.
Or you can just Cowboy it and take your chances.
Developers (myself included) are self-assured of their work to the point where they can't be expected to test things thoroughly between the critical points in the timeline when (a) it's done, and (b) it's being used by customers.
The exception to this rule is when a customer has discovered a problem and it requesting a fix because the service "is broken". In cases like this there is a strong desire to minimize the timeline between (a) and (b) so having a short-list of developers who can circumvent/expedite the formal test/release process in these cases is a good thing.
But yeah... for scheduled upgrades of software that "must work" you've got to put QA barriers between the development team and the team who takes care of the systems that customers use.
Creating the flawless installers costs time.
Time means money. If the software is only installed twice (on test system and production), a good flawless installer is very expensive. If it is installed a lot of times (more than 5 systems), automation of the installation will pay itself.
Generally it is good practice to keep the developers out of the production environment.
However there must be exceptions.
-Emergency fixes.
-If there is no good team of maintainers, you can actually look at the installation logs, and understands them, the developer might be the better option. A good maintainer, who only blindly runs a script is not the best option.
-Uptime might be more important that the principle.
I think this is one of those situations that is really very context dependent... it will depend on things like how isolated the two teams are from each other, how frequent updates are rolled out, etc. You want a good process in place, but where individuals or groups fall across the process really comes down to 'what will work best in the situation as it is'. Even large(ish) scale operations you might have people wearing multiple hats and strong dividing lines would only make things less efficient.
If code is being developed for in house use there should be three separate but identical systems:
Dev - development is done here - builds etc. with version control software used to track history and allow for roll back etc.
Test - once the developer has finished with the code - code is installed here for testing by QA - NOT by the developers, if critical bugs are found, back to Dev.
Prod - once full passed but QA, then and only then should code be install to production
I work for a (mostly) US based Fortune 500 company who for various reasons I would prefer not to name. Short story - if you are a big enough company that external auditors come to visit you (we are) and your Dev people install code, even in test environments, you may fail an audit in the USA. Trust me, it is bad for business to fail this kind of audit. I'm in a system admin type job and a bit isolated from the workings of our Dev group, but it's my understanding that we run Agile or some variation of it and we absolutely do not allow Devs to ever install code in test or production environments.
If you're going to lock me out of the production environment (which I prefer, thank you), don't call me on the weekend if something isn't working! On the other hand, if you need me to install, get out of my way and let me work. I'll just build a tool that does the job so I can go home. It should be a repeatable process, anyways.
:wq
When done right, deployments shouldn't be a big deal. If deployment is a fully automated, no-brainer, one-click operation, it also shouldn't matter (much)who deploys. But prior to deployment to a live environment, make sure the package deploys correctly both to a clean test environment and a backup-from-live test environment. If you haven't already, set up nightly builds to do this.
I do all our production deploys for a Big 3 dealer application. We do it this way so that I understand the app and can better trouble shoot it at 3am if It takes a dump. Because we all know getting a hold of the dev team at 3am is a crap shoot at best.
You should have at least 3 environments beyond the personal ones the developers use to develop and unit-test code: development common, QA and production. Development common should be where dev does integration tests, including installation. If developers are responsible for creating the installation tools, they should be doing the installations there so they can debug those tools. If someone else is responsible for writing them, the devs should be working with them to make sure the tools do what the software needs to install correctly. You can't get the installation tools right if you don't test and debug them, and when they interact with the software the developers are the best ones to figure out what's wrong and what's needed to fix it.
Developers should not be doing the installation into the QA environment. They should be handing the installation tools over to QA and letting them run then according to the deployment instructions. That's the only way to confirm the instructions were really complete and that everything works per the documentation, by putting it in the hands of people who didn't write it and letting them deploy it. That way when it comes time to deploy in production you've got some assurance that the deployment will work because it has worked before.
Now, if things do go wrong you need dev involvement. They wrote the software and the tools, they'll often recognize exactly what's going wrong where Ops and QA won't. They're also the ones most likely to be able to give you a quick fix to the problem that'll get production up and running without having to back out and try another day. If you've already invested the time bringing the systems down and deploying the new version, it makes no sense to revert to the old version and waste more downtime tomorrow re-doing the deployment if the only problem is a path in a config file having the version number in it in QA but not in production and a quick edit of the newly-deployed copy of the config file will clearly fix the problem.
I've been in a similar situation, where developers were actually stopping a production service, checking code out onto the prod box, compiling it in place, and troubleshooting it for hours after that. Our service support team hated it, and it caused a lot of penalties with our clients (which, of course upset the CFO and the investors). We went to a model where the dev team would give us a package, we would get it running on a parallel server, test it to make sure it was working, then swap the old and servers' addresses.
If the package didn't work, there was no negotiation- we just didn't deploy the new code and the dev team went back to work the kinks out. A side benefit was that the prod support team didn't catch hell for an outage; the dev team absorbed the full impact of their lack of attention to detail and quality. Things got dramatically better with the products after that.
It ended up making everyone's life easier. The whole dev team didn't have to be awake all night for the change, the time to make the change shrank from 7 hours to about 30 minutes, and the customers started seeing what was good about our product rather than the monthly multi-hour outages.
I'm a developer and a manager at a small company and I can unequivocally say "no" here.
Developers installing software inevitably causes issues - even when you're talking about QA/Test. It invites short cuts and quick fixes that never get documented so six months down the line, no one remembers what was done or how to reproduce it. That's if the original dev is even still around, which they often aren't.
If deployments aren't easily reproducible by another party than you can't guarantee what's running or when it was running. This invites all kinds of issues - especially if you deal with monetary transactions. There's a reason why stuff like PCI requires separation of roles.
Note that I haven't even touched on the potential damage that can be caused if there is malicious intent.
M.
If the developer is going to get woken up in a mad panic to get customers back on-line nights and weekends, then, yes, she should be able to install software into production. And in this scenario the developer _is_ also the operations person. These scenarios usually don't last long.
I'm a DevOps person. I value developers getting close to production to better understand how customers are being impacted by their software, but I do not want them in a position to break things. My job is to keep customers up and happy and to never know I exist.
Good operations people are conservative (not as in being a Republican, of course) and only change production in a safe way. If the organization is healthy and prioritizes what's important, then changes can be made rapidly and often, but this doesn't mean developers get to do what ever they want.
I refer the honourable gentleman to the ITIL guidelines which cover this sort of thing. In brief: developers do not even get access to live systems, much less carry out updates on them.
(Much of this is based on somewhat rusty memories; please correct any mistakes I've made)
Developers package their applications according to an agreed process and hand the packaged application over to an operations team whose job it is to update the live systems.
This update must not be acted upon unless and until the update has been carried out successfully in on a test environment - which is again separate to the developers' environment and is meant to mimic the live environment as closely as possible. ("Successfully" in this case means it installs cleanly and the updated version passes an agreed set of tests, including user acceptance tests). Ideally you'd have a separate test team who vouch for the build.
Yes it's slow and cumbersome. But it's not intended to be quick, it's intended to ensure that the IT department is not at home to Mr. Cock Up.
Having worked at several startups with large environments where development routinely was leaned upon to jump into production I firmly believe that ideally developers should not install their software (or even touch it) once it is released. Allowing them to do so leads to band-aids, hacks, lack of documentation, short-cuts, and all other manners of badness.
Strictly maintaining that division between development and IT/end users helps ensure that development maintains a complete package. Incumbent in that is that the appropriate feedback loops into development must be established, implemented, and acted upon. Bug reporting, issue tracking, customer feedback, and the like are critical bits of information that cannot be ignored by development.
Developers shouldn't even build software for customers/production.
make is 35 years old!
make
make install
Not having an automated build and install process is a waste of everyone's time, but especially the developers' time.
Use of the words "good", "bad" or "evil" is almost invariably the result of oversimplification.
Enterprise systems come with package management systems. Use them. Developers should be required to develop in one environment, package it and test in a testing environment, then scheduled installations into production with some means of management approval and auditing. Any backouts required involve simply removing the bad package and installing the previous working one. But I speak from a sys admin perspective.
- It's the build manager's job
- It's the deployment manager's job.
Really -- this question completely depends on the size of the shop.
I've had sysadmins tell me that a platform that only ever runs on linux, they'd rather we didn't require a single cron entry to run -- even if that would require importing half a dozen new libraries.
And I think that's a potentially reasonable request to make... depending on /why/ they make it.
Unfortunately, in this case it was "I don't want to have to maintain the program when installing it". Tough shit, princess -- the time that job runs is important, and the manual and ticket system says why.
If you're building up to an MSI, you should have a person in charge of it depending on the build. Same with RPM or .deb.
For the past few years my builds have consisted of python egg files, and the sysadmin deploys it with two lines of config I supply.
Making devs screw with the install depends on... whether or not the dev's are responsible for the *whole* install. If they aren't, then no, don't have them use the instructions, you'll just interfere with testing.
Really, you're talking to the wrong group about it. When my install instructions for the magic-black-box ommitted that we use apache tomcat and two warfiles in it and I got an earful from a project manager, I gave it right back to her. Not my fault I consider them external libraries that speed things up. The system works just fine on a brand new install without them. They were previously installed on the platform, listed as "highly recommended dependencies" and our config file had comments on how to point to them with appropriate defaults for a default install.
But if you convert or upgrade the server and roll an existing install into the new one -- while following install instructions that helpfully detect an upgrade instead-- then yes, our install depends on the old services being on the new setup. And yes, we give a 'cryptic' error message .
Not my fault your sysadmin -- (the PM) is an idiot.
Yes, I can write a test for any given error -- but that tends to cause bloated, unmaintainable heavy code. At a certain point, it's time to assume competence. That means if you get a message saying I can't find tomcat, you man up and demonstrate that your degree made you at least as literate as someone with a decent background in liberal arts, much less the 20 years of experience you show on the resume.
caveat - this is talking about as the poster requested, production deployment of web services from big companies.
The software should be able to be deployed by nearly anyone familiar. Any changes to deployment are modifications of install scripts or applications.
Actual deployment to a production environment should be able to be done by a lot of different roles.
I prefer not to have developers deploy because they have access to a lot of different possible packages.
Build should be produced by CM.
Test should be able to install on their test systems, as checking the install is part of the test process.
Once a gold build is produced, the people administering the production application servers or ESBs should probably manage the package deployment.
Please don't tell me the developers are also administering the production boxes!
But really, test is familiar, development is familiar, systems should even be familiar... It could really be any of them.
"Should" then just becomes a choice made, and someone puts it down on their time card.
If your developers have time to spare and are salaried then it might make sense to have it be part of their jobs. If it can be done by someone cheaper or less critical to tight developer deadlines then you might have it be a job or a role in some other job.
I don't see anything inherently wrong with having some developers in that role though.
Ideally, the systems management team should handle install into staging and production environments. They have worked with the dev team to establish a standard environment. The dev team should take time to understand the environment where the application will be deployed, and the systems team should understand the application well enough to diagnose basic problems.
Unfortunately, in my experience, cost cutting means that the installation and maintenance is done by an IT contractor pulled from a rotating pool in Bangalore who doesn't know how the application works. They usually can't tell a software bug in our code, from a missed firewall configuration step, from bad permissions in the filesystem, from a bad load balancer config... the list goes on. That's when I get pulled in. Of course I don't have permissions to do any useful diagnosis, and they are so fresh out of training they don't know how to execute a TCP dump. ...and that's why I don't do J2EE development anymore :)
The folks above have commented on why it's a best practice for coders NOT to push their own code to PROD. Peer review, undocumented fixes, hacks, security holes, malicious insiders, it's all covered up there.
Which is why SOX auditors, PCI auditors, HIPAA....etc all look for separation of duties. It's a case where Compliance actually has Security's interest in mind.
Where I work developers do not have any access to production. We have a process and it tries to ensure that what gets to production has been tested by the dev team, reviewed by the support team, and finally tested by the client before it even gets near the production machine. The devellopment team defines what is to be added/changed/deleted (specs). The support team and client approves the changes before we start. The code is then tested and signed off by everyone. Once that code is ready the development team writes the installation document that defines every change required to release to production (and must match the modules in the specs) and the computer operations team follows that list of instructions. The list is also then checked against the change list and only the changes identified get released, anything else will not be released unless authorized by the support team and the client.
There are exceptions for emergencies, but, all code releases for any reason has to be signed off by the support team and client before it even hits the production machine.
The upside of this is that every change made has been documented and verified. All code released to the production machine has been documented and an audit trail is available for review. This way a developer will have a very tough time trying to sneak in code that should not be in a production machine (it still can happen, but, it is very hard).
Panic now, beat the rush!
I have been with this company since we were very small, pre IPO days. Those days, we did not have a separate QA dept and we devs pitched in as QA and documentation. Many routine stupid things to do QA kept getting in the way of what we believed as our true job, develop code. So we wrote stuff in the product to ease our own QA.
For example Win98 introduced a "preview" pane on its file explorer. We had our own file explorer, and we had a preview since 1992. Why? Because we had tough time telling which project name corresponds to which project/design we had to test.
We wrote a pseudo "non rendering" window whose only job is to accept and dispatch mouse clicks, in 1993. We had an env switch to dump out mouse clicks during the normal run and then we use this non-graphical-non-rendering window mode to run jobs in a batch. That pseudo event dispatch window eventually grew into our record and playback feature. Then it grew into our macro play back feature. Eventually it grew into our "Command Langugage".
We wrote our own "process manager" to run the jobs in queue, run queues in different machines (only in unix, we used rsh).
We grew big, and we have a separate QA dept. They have bought expensive mouse-click-recording and playback software. They have an expensive bought some queue manager, remote job manager etc etc. But when we were doing our own QA, we felt the pain of the customer and added tons and tons of things to make our jobs simpler, and the customer benefited.
So, yes, eating your own dog food helps.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
My huge company with many brands has 100s of web sites. We have a few classifications of sites, and each classification has it's own process. In some cases (e.g. credit cards involved), it MUST be deployed & operated by a 3rd party vendor (different than the developer). In other cases, a single (approved) developer can do everything him/herself, as long as it's coordinated with the brand manager and corporate PR and has no features corporate privacy or security teams would object to and is hosted at an approved provider.
In some cases, a seemingly simple new facebook page needs LOTS of input from LOTS of groups, and needs to follow a pretty complex process of who does what. In other cases, it's a trivial task that can be done in a morning with emails between 3 people.
The key is to think up front about the different classifications. What are the legal/privacy/security/pr risks? In our case, risks not just for the brand, but all brands in the company. What can/can't be done in the different classifications? What processes need to be followed for each? What common components/services can be used/deployed without a security or privacy risk? After that, each site can be managed with the quickest and cheapest process? We would have 1/2 the sites and 1/4 of the facebook presence if every site had to follow the same processes (or we'd have lots of privacy, security, and operations issues).
While credit card sites are operated differently, all sites and facebook pages are operated by the same 2 vendors with the same SLAs. We worked very closely with the operations companies to come up with the different deployment processes so that we can have both the quick/cheap and more complex types of sites with the same 24x7 SLAs. We also worked with our legal, privacy, security, and PR teams to make sure they are all satisfied that the processes are acceptable.
The best way to do this is to have automation. That serves as atleast some kind of documentation, it makes stuff repeatable , it makes stuff easier to replicate etc. Preferred method would be to have the automation in version control and a build machine which checks if a tag changes, then pulls the automation, executes it, it will install whatever is tagged, tag it as installed and shut down. Preferably this system is built so that you can only "please install this" tag stuff that is properly staged and autotested.
I work with Industrial Automation Software. I have developed the software and installed it a customer sites. Any customer that depends on the vendor to always install the software is courting disaster. What if your critical enterprise software is custom installed, runs for five years and then due to a hardware failure you need to reinstall from scratch? Will the original person who did the install still be available? Is the software still being supported? All end users should have explicit written instructions for how to install the software. The industrial automation software I use can kill, cause fires, and do all manner of damage if not installed and configured properly. It is just good practice to have explicit detailed installation instructions, that someone with a modicum of knowledge of the system can follow to reinstall from scratch. I've seen plenty of customers who are using software for which they have no idea how it is configured or installed. The original engineers are no longer available. They were laid off in cost savings events. Then I have to come in to figure out what they have so that I can determine how to upgrade them to the latest version while preserving years of data because once the hardware failed you can't get it to use the old software and the old software won't run on newer hardware.
Application quality/reliability is inversely proportional to developer access. 'Nuff said...
.. stop asking us to wipe their asses, and do it themselves.
Seriously? What kind of technical people are you hiring that can't take care of this?
I agree 100% with no. One of my clients bought a payroll system without asking anyone in IT to look at it. The developer required that they install it themselves. There were warring signs all over that anyone from IT would have seen a mile away. The developer install requirement was just the first clue. My thoughts are that if a company has not had time to build a decent install package yet, the rest of the product will also suffer from lack of refinement. In this case I was correct, and after less than three years we migrated to a different payroll product. I am continually surprised about how ignorant developers are about the operating systems they develop for. .. ESPECIALLY SECURITY! If politics prevail watch them like a hawk and do not grant them unsupervised access to you network.
Here is another issue to think about. We also had problems when we need to do any type of infrastructure work. We will work after business hours to complete a project as to not disrupt business operations. Will the developer do the same? Probably not. We had a great deal of trouble coordinating support from the developer to schedule work on the payroll system. After it was installed and the check was signed they kind of wash their hands of supporting anything other than trivial end user issues. It took 2 months to get them to coordinate configuring their system even for an IP change. Any problems they wanted to blame on IT; luckily we gave them a dedicated server so that excuse did not fly.
I would suggest that some products are very complicated and may require developer support to install and configure correctly; especially for some lame IT departments that I have seen. But if they cannot even write an installation procedure for their product, the install issue will probably not be the only issue you’re going to have with this product.
I work in a company where the Datacenter , not developers deploys to test and production and they get it wrong a high % of the time. So yes. The datacenter employees don't even understand the server software or database involved , yet they control deployments. Its a nightmare
For very large corporations, backend deployments are few. Frontend deployments are many. I'd say developers who know their backend environment, which changes little, can be trusted in deploying what they develop - with supervision and prior auditing by those entrusted with maintaining the environment. Absolutely not in the case of frontend deployments, where you may see many different configurations hardware-wise. Something which works on one box might bring another box down - workstation administrators are the people to trust with such deployments, who can then carry out diagnostics when something goes wrong. In this case documentation and audits are important.
As a former SQL developer, in a very small company, I had complete access to everything. For me, it made life easier but more complicated, to be paradoxical. Something goes wrong, I have to stop developing and fix it myself. Therefore I'm incentivised to get it right first time, which means careful coding and careful testbedding and careful debugging before the product even sees the production machines. Then I can move on to the next job safe in the knowledge that the chances of me getting a call at 3am because of some bit of rogue syntax brought a few thousand £ worth of gear to its knees is next to zero.
closure: if the developer is coding for production where he is not likely to be called upon to drop everything and fix a problem on a production box, such task reserved for someone else, then full documentation of every call and function is an absolute must.
Operation Guillotine is in effect.
it seems unhealthy to me to have software installs done by developers
"unhealthy" is putting it very nicely. It's insane.
The developers should never have to touch the production system. It is very, very important that the guys running the production system know how to install the software on it and get it up and running. That is exactly what they will have to do if everything burns down one Friday night.
Plus having the developers install it is an open invitation for all kinds of hacks and shortcuts instead of a proper deployment process.
Assorted stuff I do sometimes: Lemuria.org
And what do you think an installer is, other than *software*, written by developers, that contains instructions, written by developers, on how they would install the software!
Let alone something more sane, like ebuilds & co for package managers!
*facepalm*
I'm not sure a human should be installing anything manually on production environments. Upgrades should be done automatically.
"First they came for the slanderers and i said nothing."
If this is software only used internally, you may not have a choice. Unless the department is large enough to include an administration group, developers may not have any choice. But everywhere I've seen this, it's been a bad practice. Developers are good as developers. It is not necessarily in their skill set to be administrators and system architects also.
For software intended to be sold, it is *very* bad practice for developers to be responsible for installation on QA (especially QA!) and final test. Were I have knowledge that this practice is followed, I decline to buy from that company. Proper versioning, keeping the stack aligned, assuring a pristine QA environment, are all specialized skills that developers are not required to have, any more than race car drivers are required to be mechanics. (There. A car analogy.) Some may be, and a few may be good at it, but it is not their speciality and should not be a job requirement. (That said, developers should be on call for install problems that require dipping into the code, or may result in code changes.)
But again, it depends on the size of the operation. If you're developing a product with the laptop balanced on the ironing board, you do what you have to. But it becomes a lot more difficult to convince people that your product is enterprise ready.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Huh, another example of how almost any question that appears in a Slashdot title should be answered with "no". No, the developers should not install the software themselves. Not in test, not in production. (Assuming a sufficiently large company that there are people other than the developer...) It shouldn't be that hard to install and get the software up and running. Should something happen to the developer (quit, fired, Hit By A Bus, whatever), who's gonna install it then? At a minimum, there's a bunch of missing documentation.
Part of having a validation team is to approach the code without the biases of the developer. The developer is too intimate with their software, and on top of the two primary methods for code coverage (random and directed tests), there is still the human element that a validation team brings.
On the other hand, it is also useful for a developer to "eat their own dogfood". I wrote CAD tools for a decade, and when I finally became a designer and actually USED them I was like, "Dafuq did I code this stupid feature X for, when I really needed feature Y?!?!?"
There's a fine balance, but they definitely should be involved beyond kicking their latest drop over the fence to the validation team (or end users, if there is no team).
https://www.accountkiller.com/removal-requested
Whoever is accountable for the system should control deployment strategy.
If they want the developer to install and configure, fine.
If they prefer operations, that is okay too.
If reliability is important, it should be "hard" to deploy new code.
If features and turnaround are important, it should be "easy" to deploy new code.
Hard means more testing, documentation etc, easy means less.
developer: ... but it runs fine on my machine
boss: ok, then we ship your machine
Developers -> QA -> Operations (Am I dating myself here?), with the project manager (or Product Manager) driving the process.
I also think developers and QA personnel should be more or less interchangeable, but that's not likely.
I would also insert a second layer of QA consisting of Project Managers and Product Managers. The primary QA guys are running regression tests and unit tests etc. You need to have someone actual sit down and drive the software too because Mercury and the like don't catch everything, especially logic and common sense things.
But the reality is almost no one has the budget for this. So the developers are doing QA themselves and helping with installs, etc.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Yes, the installation process should be relatively automatic, and well documented so that another team can and should usually install it. However, such teams are often *too* capable, that is the Developer(s) should do some installations themselves so they see just what sort of nightmare they have created ... or how fragile it is in the context of a full production environment.
Development environments are no substitute for the RealWorld. Ignoring the RealWorld is a BadThing
There should always be an installation package. Not having instructions, or allowing the developer to hand-install stuff, leads to all kinds of problems (inconsistencies for instance, i.e. no two systems the same).
Another reason - simple scalability of a business. The installation must get to a point where it can be handed off to somebody else.
Installation and deployment generally doesn't get the respect it needs. Imagine buying an app for your phone - and somebody handed you a long text document detailing the 5 libraries that you needed to install first, and then they needed to make a few hand-patch configuration changes to the device? That wouldn't be acceptable - Enterprise software should be just as easy to install as consumer products. Customers see value in this and the installation tends to be the first encounter with your product and can set the tone for the relationship.
Somebody else posted something about dogfooding. Yes - engineers must know how "hard" it is to use the product (whether it be installation or usability). However, the engineers cannot be the ones responsible for installation.
Exception to the rule: brand new proof of concepts being delivered to the "partner" customer. By the time the "second" partner install comes up - there needs to be an installation package. I work for a large company and I generally cannot touch a customer system - so everything must be packaged and handed off.
In my line - the installation is easy. It is the configuration that is extremely complex. Although it doesn't require an engineer, it does require an "expert." But this too should be as easy as possible.
Should developers be responsible for installing the software they develop into production environments?
Absolutely not. This is horrible security practice and will not fly even an inch in many industry standard audits. I can give countless examples. A developer sneaks a change into the software to funnel personal information and credit card numbers off to a private database and then sells it on the black market.
The whole point behind the division of labor is a set of checks and balances and reduction in conflict of interest. Quality Assurance analysts validate the code does what it is supposed to. The developer doesn't do this. Developers typically don't care much about that level of quality of their code. They have a different set of objectives. They must complete objectives, they have deadlines. The customer performs user acceptance testing not the QA analysts for obvious reasons. Even the brightest of user proxies cannot replace the customer's direct feedback. The deployment specialists, IT or Operations specialists deploy the software. They are responsible for the production server infrastructure and the environment not the developers. They must exercise great care in what they install, whether its the company's own product, Microsoft updates, drivers or anything else. It's all the same.
I'm shocked this question even gets asked. There is no free lunch!
We'll make great pets
First, nothing smells disaster like having a developer install its own software ... And nothing smells disaster more than a developer that cannot have access to prod ...
The Lone Developer should have have a preprod environment. Do its bidding there. Then, a staging environment, where he should write a documentation on how to install it, and how to test if everything is all right.
Once that document is done, revert, and use The Intern to follow the document on Staging. Have The Intern write his thoughts, his findings. See if he was able to follow through.
Finally, have The Production Team and The Intern install it on Prod using the documentation, with the Lone Developer on standby for any issue.
From this point on, although The Lone Developer should have access to the servers, he should be instructed not to do any modification on the server. That's mostly a monitoring and debugging phase for him.
My final thoughts: being anal retentive on any process will make things break. So by being slightly lax under a tight grip, it gives the best of both worlds.
Given questions about Sarbanes/Oxley, security audits etc for large companies - the answer is no.
However, and this is the twist, I do believe developers should have a good working understanding of the issues associated with the installation and running of their applications in a production environment. To that end, you should have a test environment/network that mimics as much as possible the environment in which the application will live.
Ostensibly programmers are (mostly) computer science graduates - and as such should have an understanding of all aspects of computing - particularly with regard to how their application interacts with the environment in which it lives - yet too many times I've seen issues along these lines:
* Mismanagement of hardware resources.
* Mismanagement of network and pooled resources.
* Security failures.
* Operating system environmental dependencies / failures.
As a result I believe developers need more hands-on in test environments. Testing on their personal machines is not enough. They need to get their hands dirty doing what system admins will do when installing and managing the app, and should do automated testing to validate resource management, security and other dependencies can be handled properly. There can be a team that specialises in this - but the programmer should have some basic hands on in the test environment before they hand off to a test/packaging team.
This will give them an appreciation of what others have to deal with - but more importantly, it will make them a better programmer (assuming your measurement of success is working code; and I define working as complete[in terms of installation dependencies and what it is currently designed to do; I do realize that the codebase could be on an iterative life-cycle], secure, and either no resource impacts - or smart enough to have behaviour appropriate when more resources are needed).
I wouldn't trust a programmer who could not build his own machine, load the OS, and integrate that machine into a complex network. I've seen the bad results too many times when this is not the case. A program is not a thought experiment - it is a living/breathing entity existing in a complex environment - with interactions beyond the program logic; too many 'programmers' forget that.
That is my advice.
And more of my career still was as a programmer, oh, sorry, let's be politically correct, "developer", than as a sysadmin. And I've done configuration management.
Not ever.
For one, they know how their systems work, and what they need. Wait until they leave, and let the new person install it. Much hilarity will ensue. And after the 20 or 30 hour day and a half, management will want to know why, and as much as I disparage management, on this, they'll be completely and totally right.
Egoless programming, as it was called in the early nineties, had someone else look at your code before it went to test. Why should the same person move it from dev to test to prod?
mark
COBIT is a globally accepted framework for providing end-to-end governance of enterprise IT systems.
IT Auditors (internal and external) will test your systems against the COBIT best practices.
Here's the two areas you'd fail an IT audit if the developer could push to the production environment (you can google for the details on these controls):
Change Management Process
User Access Review
If you're a US-based company and you fail a SOX audit, you could be subject to a fine of $1 million and 20 years of mandatory audits. We also test this for PCI DSS and HIPAA compliance so you can see why from both a security and a business perspective why the developer role is separate.
For example, I had a project which required me to make some basic, rudimentary black & white icons. The ONLY graphic software I had was Microsoft Paint in Windows Vista. A dated and wholly incapable program.
I requisitioned ANY graphics package, even some free open source ones installed previously. And was turned down as it not being a necessity. So what should have been 10 minute job of icon creation turned into hours of needless painstaking pixel by pixel design in MS Paint.
Now there's stupidity.
***
Basically, a large environment should have a list of software package approved for installation. That list should cover a variety of utility types, (text, graphic, audio editors, etc). Including free and/or open source options that are approved for installation. It's not even that I don't have the ability to install. Just not the authorization.
And that's frustrating.
My last job was in the IT unit for the IS section of the orginization, supporting desktop IT for mostly devlopers, and a few other types as well. We provided the computer, with an image. The devlopement side had their own image, with Office and Visual Studio and the like pre-loaded, and everyone else got the company image with just office and whever else. Unless the devlopers aksed for help with their software installs, they were on their own after that. The devleopers sent their code on up their chain of command to people to test install packages and the like in the IS department. Where i work now, again the IT department provides the devloper with a computer and OS, and they take care of the rest, of course we are always willing to help them load whatever software they need, but most of the time they choose to do that work themselves. As for their product they are responsible for producing the installation (although most of it is web based), and we again help them deploy it.
NEVER. Never. never, never, never.
NEVER.
Signed,
Platform and Information Security Architect.
"Flyin' in just a sweet place,
Never been known to fail..."
Its a very easy NO! Production is production. It can not go down! All software installed there needs to only be installed by the people that get the calls at 0300.
In an ideal work, I want to leave the system test environment to the test people.
In an ideal world, a company cares about system and test engineers and think they should be just as capable as developers. They know how to automate labs and use whatever tools they use. Working in R&D, this is often not the mentality of management.
In an ideal world, software installation and configuration is given constant up-front importance. I've always push the that any piece of software should be easy to build and install from the beginning. It simply helps everyone start working on the project.
So while I'd love to leave the system to System/Test engineers, the reality is that I often do need to end up installing and doing systems work. Part of this is good as a develop who doesn't know anything about systems is going to have a lot of problems.
But that's all ideal. In reality, we don't have enough system/test engineers to really make it feasible. I have to build/setup my own labs half the time, so I know how to do it anyways. So it's not a stretch for me to manage that half the time.
How about you record "everything" in the session when installing production? This way devs aren't doing the process (which can be bad, because they can skip steps in documentation and god forbid they go or forget), and you make sure you know the problem is either a misstep, some different variable, or simply bad code. Either that, or have the developer be there watching what's happening WITHOUT interfering. (More costly)
Ideally there is a lot of communication before and after, and the process of deploying is pretty transparent (ensured by the recording of the process).
It all depends on the individuals involved. When problems happen, your best guy/gal needs to grab the problem, be that a developer or integrator. Have some flexibility and use your people according what best fits the situation.
Developers should never be allowed to install anything in a production environment, for the simple reason that a production environment needs to be well documented. When the Devs install their own stuff, they don't follow the documentation, if any even exists; they just install it the way that works, which probably bears no resemblance at all to the documentation.
On the other hand, Developers should be forced to watch while their software is installed, per the documentation, and then when nothing works, they should be given the documentation and made to fix it. Alternately, a separate person/team should watch the developers install their stuff on a clean system, and document every step they take, for reproducibility.
I know I'm posting late, but where I work, for internal apps, we "git flow" and have some naming conventions and if your "deployment instructions" as a dev are more than the single line "git checkout release/2012-09-blah" then you've failed as a dev, naughty dev, go away until that works. That's the official "demac point" or whatever you want to call it.
So... umm... yeah... a dev can log into a production server and type in "git checkout release/whatever". And the admins know it..
I'm sure you can make a horrifically complicated system involving all kinds of software and checks and balances and endless procedures and meetings, but if you can eliminate all that via technology and standardization, that's a win.
The biggest problems I've ever heard of with "git flow" is devs making releases work unidirectional (like they only support database schema upgrades not downgrades, add a line to a file every time it notices it was upgraded without checking to see if the line is already there, etc) and devs and admins getting into branch "revert" wars and people on both sides refusing to adhere to the demarc (well, yeah, git checkout and then also please manually edit file X and blah blah for 5 pages, um, no thats not the agreed upon standard, nothing but one git checkout line is allowed) All three are management level issues not dev / admin issues..
Needless to say this forces massive automation and logging and a real test environment the use of which is enforced. Which is good.
You can put a hook into git such that it emails when you change branches. Don't recall how, maybe its just a script that periodically looks at diffs of the output of "git status", but its obviously quite valuable in this situation, combined with a email list for the admins and devs, so everyone is instantly on the same page when something gets released.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
It's all coming to a head.
Individual developers just can't do it anymore. They didn't build that library. They can't deal with the combinatorial explosion over time of code in the wild.
We're just going to have to hire more bureaucrats to dance around the issue, throwing acronyms on PowerPoint slides at the problem.
Higher level code, called 'legislation', in the language called 'English' will be delivered by the ream.
This write-only code will trigger paralysis in all who read it, and, eventually, the economy the readers inhabit.
Legislation is become the destroyer of worlds, and a right jolly virus indeed!
After you collapse the economy, the problem of developers installing software enters a self-solving stage.
You're welcome.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
First off if your reason for not letting your developer touch the production server is that they might hack it up, I suggest getting a new developer. With that said I assume you are running a business which really makes this a resource issue. There may be a couple of reasons why developers would do the installation.
1. The cost in time for developing/maintaining an install system is more than is required for the developer to do the installs themselves.
2. The systems could be very unique and having installers with enough knowledge to install even with a good installer is not as cost effective as having a developer that does it part time.
I'm the only one in the office that knows how to use the JTAG
I've worked in shops that developers are not allowed to install on their desktops. Never seemed worth the weeks it would take to me, but at least I got paid for doing nothing.
If it works in production who cares who installed it. The idea is creating applications that work for the appliance they are built for. Unless your making a consumer product there is no need to even bring it up.
Do you want it installed today, or do you want to wait a week while I write up instructions and scripts and test them in a clean environment (that will probably need to be built from scratch)? And, of course, while I'm doing that I won't be working on the other projects everyone's been hounding me for. Is it OK to push the delivery for those back a week? Your call.
If the amount of developers in a company is below X they usually install their own software and decide on their own which tools and libraries to use. The millage will vary, but you can call them the teams with 100% productivity. (Yes one of those teams will beat the other one badly ... )
If you have a centralized policy that usually means: new developer, takes 2 weeks to even get his computer; now he has it, it takes some days until he has the base software (IDE, source code revision control software, CASE System - if needed - , Email, external Web access (yes sometimes developers need google or stackoverflow) ) finally he needs: credentials for the http proxy, to access a maven repository and subversion. After 4 weeks he has all the necessary software and credentials ... consider: the year has 12 month, 1 the developer is spending on vacation and one on getting his initial development environment up and running.
Allready 10% time is lost. Not to mention all the other hurdles because 1000 developers share the same SVN repository via a 100Mb ethernet.
I for my part prefer a situation where developers can decide themselves. Usually they just put their own local svn server on one of the dev boxes ... 100 times faster.
Or they install a groovy plugin or take another rules engine ...
Bottom line I would claim they are twice as effective as teams where the corporation tries to control everything.
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
For embedded development work, it's a requirement that the dev pubs their build onto the target hardware. We just couldn't do our work without it. But once the developed software has been tested on a test system, it should be left to QA/V&V to make installations...
Should Developers Install Their Software Themselves?
Should developers be responsible for installing the software they develop into production environments?
It depends on what the hell they are doing. Are the developers creating a medical device or mission critical system? Are they developing a shopping cart? Are they contractually bound to use a specific set of tools (or eat their own dog food if they are in the business of tool development)?
What about System Test environments?
What about them? Again, it depends? What's the size of the organization? What are their testing needs? Is the organization or project contractually bound to have a strict separation between development and testing (say as per DO-178B and/or other standards for avionics)?
I'm not a developer
Then shut up then. Seriously, I tell you why a few lines below.
and I'm not all that familiar with Agile or DevOps,
What does Agile/DevOps has to do with developers installing their own tools. That's unrelated to the development process of choice (agile, waterfall, whatever.)
but it seems unhealthy to me to have software installs done by developers
And this is why I tell you shut up. What the hell do you know about developers and development? Since you know nil, on what basis can you say this or that is unhealthy. Unsubstantiated, subjective opinions are not facts, and are a dime a dozen (and we have way too many of those already in the industry... and in life in general.)
Again, as I mentioned before: IT. DEPENDS. In some cases, it is ok (and even necessary). In other cases, it is not. Making assumptions about it without specifying a working context, that's just brain farting.
I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code.
So the end user should know how to install the compiler suite and editors I installed on my machine (plus dev-specific configurations) to use the system I wrote. Brilliant. Make users' live more difficult. Yay to productivity and increased ROI!!!!! That was sarcasm btw.
I'd like to hear opinions from developers.
You like pointless exercises for attention whoring sakes, do you?
Without a proper Move To Production strategy, mandated and followed, developers end up being sloppy about deployments.
The only time I have consistent success is when we have a Move Coordinator, who reviews all moves before deploying to Production.
The developer builds the move process, including rollback steps, and tests it against Dev. (They have a rollback step that they can use to reset the Development environment, and it had better be tested.) It is reviewed by another person, preferably the task lead or the actual Move Coordinator, who verifies it works in the Test region, including the rollback steps. (So what if you have to run it twice, better safe than sorry.) The results are verified by the End User / Client. If approved, it is scheduled for Production. Ride the end user, cause they will forget to test it, and then when Prod is refreshed down to Test, they can't find the changes and you have to redeploy again.
If the Move Coordinator is the developer or task lead, an alternate is designated to review and verify.
No matter how many developers and skill and experience levels I work with, someone inevitably makes a shortcut, saves it as a on off, and "I'll remember to plan this as part of my deployment." and forgets it. It does not matter on complexity or experience. Everyone does it.
Trust no one and assume the worst. The rollback is your safety blanket. Never forget it. Back up everything. The more data changed, the more important to have a recoverable methodology. The bigger the system, the bigger the exposure. Frankly, I like my a$$ as it is, still attached and not chewed off.
I run into this regularly. A full-scale system install costs $$$. However, we have a (very) limited number of them for the dev team to use when debugging issues that only show up on full-scale installations.
When your bug only shows up at 90% of rated operational load, you _need_ a system that can reproduce that scenario.
then pay for it. Writing and automatic installer and or/packaging software takes additional time. The more standardized and better working you like it the more work it takes.
In my experience, developers no very little about the OS, security or any of the technical nuts and bolts of how an OS works, and what bits should not be modified, they know just enough to be dangerous. I've actually had a developer go into security settings and deny access to Everyone at the root of C:, to keep virus scans and software updates off their system. Of course, they didn't realize that "Everyone" means them too, and the system account as well, and are surprised that the machine reacted badly to this. Also, too, they managed to do this on a Friday when they had CRITICAL WORK that MUST GET DONE by MONDAYS DEADLINE!.
Which makes me want to ask SO WHY ARE YOU F#$KING WITH YOUR PC INSTEAD OF WRITING CODE? Lets make a deal, I wont write C++ code and you don't try to administer my boxes. Never mind, I wasn't doing anything this weekend anyway.
Should developers be responsible for installing the software they develop into production environments?
Absolutely not.
I know there are developers who have a grasp on systems out there, but your average developer couldn't tune something as simple as Apache, for example, to save his or her life. They're often not aware of business requirements change management, sign-off, et cetera. (They don't need to be - they're developers.) And the number of developers I've had to personally teach, "Back up the database before running your fucking script that alters data," is legion.
I could go on, but I think the reverse question illustrates the stupidity: Should you let your sysadmins write critical application code? How about your salespeople? Surely the head salesperson can crank out hardcore C, right? No big deal.
What about System Test environments?
That depends. How lazy is your systems team? Test environments should be provided. Unless you're running a half-assed amateur hour business, you're going to need to be concerned with package version matching, configuration similarity, et cetera. with production.
'course, if your'e lazy, you let developers do whatever they want locally, and force integration/testing on a staging environment that mirrors production. This is often the best of both worlds, because developers can get anal about their local setup. (Rightfully so! As a systems person, I myself cringe whenever vim-enhanced isn't available on a remote box.) It does come with the potential for, "Works on m local system!" syndrome, but that's generally easy to remedy provided your developers aren't "rockstars".
Testers should do the testing.
Packagers should do the packaging.
Sysadmins should do installations.
Requiring devopers to know everything in the Debian Policy manual would be too much.
the funny part is if you put enough monkeys in aroom some day something wonderful will happen
Three words dude: Segregation of Duties
Read up: http://en.wikipedia.org/wiki/Separation_of_duties
Several free matrices exist on the Internet. If role A and role B together are done by same person and can lead to fraud or significant error, either these two roles cannot be done by same person, or you need acceptable compensating controls (mitigating factors). Typically these should be preventive, but corrective and detective controls may also be appropriate. YMMV.
If the production sys admins can't install (AND BACK OUT) software they should reject it,
since it doesn't meet their needs.
Going another step
If the Config Management people can't build it (given a 1 page max README) they should reject it.
They should be able to take that 1 page document, the SVN repository, and OS install disks
and I guess an internet connection and be able to build in half a day.
We had an installation department that I was a part of. All we did was install the software and then we had another person who stayed at the company to see that it got up and running. This was a major insurance system and we could not trust just anyone to install it. I was a software engineer (current day glorification term for computer programmer) so that someone was there to fix any problems that might pop up. Every main frame system is different and you don't know what you might run up against.
We agree one needs at the very least two environments. When you have lots of money -- or lots of VMs :-) you can have lots of environments. You need a developer free-for-all environment where developers can play, this could be their own machine but sometimes you'll need dedicated machines (hardware, expensive licences, whatever). Then you need testing/QA machines where normally you would not permit developers, and production machines where you do not permit anyone more than necessary.
The packing needs to be done because when you have few people authorized to do installs, then the people who do have that authorization must be able to do any and all installations, and it forces handoff to Operations. I work in a place where we we have so many different applications we need a database to keep track of who is responsible for which part of which application, and it would be (even more of) a nightmare if something had been installed in some unknown or non-standard way.
But there are two reasons for these rules (at least!).
Most people here are taking this from an engineering reliability aspect, and that is a valid concern, but in many companies the rules separating environments are also motivated by security and confidentiality, and are often even based in law and contract agreements. One might hire a team of contractors/temps to develop something, but not only must they have no chance of inserting malicious code anywhere, they must never ever even *see* production data, only dummy data! The classical example is a bank or a hospital, but this could/should also apply to ISPs (mail...), anything that stores SSNs or credit card numbers or passwords, etc.
Developers should not install software, especially stuff they wrote.
The *nix software available is proof of this. All the Linux folks say it is more secure than windoze 'because' you don't have to run everything as root.
Maybe not but most all the devs I've ever seen login as root just to work. And then when they give out some piece of new software, it doesn't install OR work unless you're logged in as root
TESTING should involve testing both the software and the instructions.
On developers never having access to production:
In many cases, developers are the only people who understand the full application, and in many cases are the only people who can actually troubleshoot a botched install or figure out why things aren't working right in production. Yes, you are suposed to have some kind of QA or staging environment and you are not supposed to deploy bad code, but sometimes things go sideways. In these cases, only a developer who knows the code and any integration issues will be able to figure out what went wrong. Acting like developers should *never* have access to production is a lot like saying "the mechanic should never have access to my car's engine, ever". It makes sense 99.9% of the time, but there is a .1% where your engine is broken and the mechanic can't fix it without getting under the hood. Yes, Mr. System Administrator you can change your oil, rotate tires, and even change wiper blades but fixing a spun road bearing or smoked transmission solenoid is flat out.
On Developers and Access Rights:
There are a lot of developers who don't understand the computer they are developing software on. Usually, they are very BAD developers. Take for instance, a webdev who doesn't know Apache. Instead of using built in tools like mod_rewrite, the developer will build their own tools to do what is built in to apache. Good developers know their platform, often at a level that is much deeper because they take time to read code or API and config documentation so they understand the toolbox they are working with. Often a single line of configuration is more powerful than 1000's of line of code. Developers need to be administrators on at least their developement environments... usually extended to staging there is a large difference in scale between development (a VM on my laptop) to staging (multiple servers) and production (hundreds of servers).
On installer driven software:
It doesn't matter if you use installshield, roll your own RPMs or use Salt, Chef or Puppet. Any way you go you should do everything you can to automate installation. When you automate you reduce the chance of human mistakes in installation process. If you do installation automation right, then a deploy to production can be triggered by anyone with appropriate authority or any automated process with appropriate authority. Having people sit at the console and install software manually should be a red flag that the software you are buying sucks or is incomplete.
In Enterprise-Grade software:
Installatioin should be automated to the maximum extent possible, using the appropriate operating system installation tools. Documentation for the upgrade and install should be clear enough that a non-developer can successfully install and test the installation. Install activity should be logged, so that if something does go wrong, it can be figured out later.
-- $G
Never, ever have your devs do ops. Developers need to think further out, work on projects, in a deterministic proactive fashion. Day-to-day Ops is reactive and if you get your devs involved in it it's going to be a huge distraction in the short term and a burn-out factor in the long term. The best devs are very good long-term proactive thinkers that will very quickly burn out when confronted with ops work, not only because it's counter to how they think but also because they still have their project responsibilities. And, the worst possible thing you can do is put your devs on call. They won't say anything, because they will recognize the necessity of someone responding, so they'll suck it up. Then they leave for a job elsewhere where they don't have to.
Looking over all (heh) of the posts already here, then adding my own $0.02, the answer can be summarized as follows:
IT DEPENDS
The tools Chef and Puppet take script files that define system configuration and installed software.
With these scripts you can define
* what packages are installed (effectively scripting apt or yum)
* what services are active
* what users exist
* many other useful things
Having production machines configured by Puppet or Chef has many great advantages:
* can deploy the same configuration to multiple servers (eg a cluster, or standby machines)
* can version-control the system configuration
* can apply a variant of the script for test/uat and be sure you are getting a very prod-like environment
And it solves the original question here - the developer can participate in "deployment" by committing changes to the scripts. Sysadmins can review these scripts, run them in UAT environments, and then merge the changes to the production branch in the VCS.
I worked in a large company that used this approach, and it functioned extremely well.
There is also a large open-source foundation that *open sources its sysadmin work* by having its scripts in a public repo. People can post patches to the scripts, which are reviewed/tested/committed just like commits to the sourcecode. Very cool. Sorry, forget exactly who that is...
It's good to have lend the developers into the delivery project team. They learn how their shortcuts or bad programming habbits actually affect the product that the customer receives.
"Never EVER mess with a jumper you don't know about, even if it's labeled 'sex and free beer'." - Dave Haynie
No. At any decent shop developers do not even have access to production.
You develop your X hours in the morning (or whenever your contract says so) at a development environment. At smaller shops then the developers or testing engineers move it to the testing environment. (with sufficient documentation on what should go where). At bigger shops it is "middleware ops" who do this after a change request is approved at the change meeting (if the company somewhat follows ITIL).
Then the product is tested by comparing the wannabe you developed to the specification. Testing can be by testers/qa/customer/whoever else. Really depends on the house.
If the product is functionally OK, it is moved to staging/performance testing (one or two environments). These environments are EXACTLY the same as production in an ideal case, the performance testing many times is an environment with both worse and better hardware then production (or something you can throttle/bottleneck artificially).
After that it is production. Production should never be touched by developers (only in extremely small shops where you design, develop, deploy maintain ... errr .. and answer the door and help customers ... and clean the floor sometimes)....
We developed a product that was released early, we had lots of bugs. Our "tech dept" (kinda like middleware, but not too clever) cannot support it, because it is "all too new complicated technologies" they refuse to learn. The owners do not understand these matters very well (being old and not too much into IT), so the problem escalated, that not only our entire environment (small: 5 DBs around 20-25 linux servers (Apache, MQ, APE, some JAVA console apps, and 10 client stations that run some of our web interfaces for intranet) had to be supported by my small team of 5 guys.
Voicing my opinion that our 5 developers should probably not be forced to start setting our clients' environments up too while still finishing the product - an argument you don't want to be part of - got me dizzy today morning, with a high heart rate, sweating and slightly slurred speech. I decided that no money or company is worth ruining your health and when someone expects you to do their job you should either suck it up and do it or stand up and walk. I stood up, packed my stuff and left. Now I am sitting home and wondering what now ....
If your company is starting to push your developers into doing tasks they should not do, they should be informed of the correct procedures and pointed to the dangers of letting a developer deal with systems. Yeah, I also did 10+ years of network and UNIX/Linux before got back into development, so there are some (a lot) of us who can do it - but we should not, unless it is an emergency of some sort.
FYI we have 22.000 + registered users with 5-10K concurrent. Not an environment 5 developers should cobble together then install their crap on.
Once you let the "emergencies" in, they become the norm and you might have to design, develop, deploy and then even troubleshoot, since now they know you can do it. And then there is no going back, you are doomed. Like me. Hopefully not shaking for 10 hours and not out of a job with a family where the 2nd baby is on the way......
OTOH ... it depends a little on the product. An .exe or .jar or .app can be easily given a 3 step install with roll-back. A PHP site with 10.000 files is a little different. For a long time I helped our sysadmins with a replication script I wrote in Curses/shell with rsync doing the heavy lifting. They demanded the source (I had a good reason to keep it back) so I gave it after agreeing, that this was just "nice of me" and that they should be doing the scripts while I develop with my team. They ran the first replication test in the middle of our afternoon rush. A new customer with 10 sites was down for 18 hours. It took 5 of my guys 5 hours to fi
Anyone from Knight Capital want to comment?
Developers should only have read access to production environments, never write access.
Production support people, the ones who do the actual software installs, need to be competent individuals able to understand the install scripts are supposed to do and be able to fix them when they break without calling the developer and saying "Wahh! your script broke!", because production environments always differ from dev and QA environments. That means your installers have to be more competent than trained monkeys.
I've seen apps that were down for days because of a typo in the installation instructions where the installers couldn't do shit about it, and couldn't give any feedback other than, its broke. The ignorant fucks were too stupid to see that the path was simply wrong, because production was not identical to QA or DEV. and the dev made a fucking typo.
And no making sure that dev/qa/integration/beta/prod identical is not a realistic solution I have never fucking seen it outside of a mainframe environment, ever.
I have no problem with developers installing in DEV, but they should not be pulling the trigger on any environment outside of development. The trick is to make sure that the process is consistent across all environments with different teams executing the release. I sell software that solves these problems, but its usually the process not the technology where the biggest problems reside.
As a developer, the last thing I want to deal with in ops issues. Here, I give you the install script, does it fail? Fine, let's fix it. Do I want to be on call at all hours because your monkeys can't run 'make install' properly? Fuck no. Half-assed attempts by ops staff and craptastic server clusterfucks that result are not my problem, and trying to install my software on a mis-provisioned system with a single-core 128MB instance when the system requirements clearly state a 4-core system with 4GB should result in immediate termination.
I'm a big fan of virtualisation, but not for copying whole VMs along with software configuration - that's just crazy talk. Virtualisation is a tool that, when combined with others like Puppet/Chef/Saltstack, allows you to quickly iterate and get new machines to the same working state every time. These configuration tools also allow you to plug in your own custom configuration data for a particular environment, but use the same "recipes" to deploy that configuration. You can version and release the recipes and the configuration, thereby treating them as code. The result is: (1) developers/ops jointly own the deployment recipes, (2) developers OR ops can jointly own the (now much simpler) configuration data, (3) everything is versioned so you can be sure you're deploying the same thing to production as you tested (4) "reinstall from scratch" is now possible for dev, test, and production environments with 100% consistent results every time.
Of course databases etc. have to be treated specially, but databases don't usually sit on the same servers as applications, at least not normally in organisations of such a size as to have deployment problems.
Devs need access to logs. In a trusted environment with trusted devs, give them access to the servers with strict instructions to not change anything except in emergency situations with prior approval. In less trusted environments, make sure an Ops guy can sit next to a developer (or vice-versa) to debug a problem, and for gods sake, give developers access to the application logs somehow. Look at Logstash or similar.
I work for one of the biggest websites in Australia and at heart an OPS guy with specific skills and im a strong believe in configuration as code and having an automated build pipeline. Once you are at that stage then giving full access to (their own specific servers in) production isnt a mind blowing exercise to 'operations'.
Our mantra is 'you build it, you run it, you own it'.
The key thing that we have gotten to is that we, for all new things anyway, deploy by trashing the servers and rebuilding them and installing an RPM package of the application. Re-installing a server to deploy sounds a little insane but if anything provides the assurance and environmental consistency that are the key to this issue.
If you re-install your box to deploy and your deployment is an RPM (or similar) package install that pulls in the required packages, then you have assurance that your test, preprod and prod boxes are exactly the same.
The difference is around configuration - ie talk to X database here, Y database there. To address that, we use DNS (short host names) and environment variables for sensitive information.
deploy by trashing the servers and rebuilding them and installing an RPM package of the application
What happens when your application is not the only thing on the server? What if there are 30 apps on a server? Even if no downtime were incurred due to HA configuration, that creates a lot of risk and extra testing effort in scenarios where you don't have dedicated servers for individual apps.
One of the greatest learning experiences for a developer is to see things from the customer/user perspective. For that reason, I'm going to buck the trend here and say that developers should HAVE to do at least 1 customer install, just so they know what hell they are putting them through. There is often a big disconnect between what the developer thinks he's producing and what the field people actually have to do to get it to work. The way to fix that is to make the developer go through the experience.
That being said, I do agree that anyone (properly authorized) should be able to install the software. If you can't install it without the developers help, then it isn't ready for prime-time.
All of our Oracle developers install their own software. They have no problem doing so.
On Production: Never On Test machine: No On Dev machine: Definitely yes and only they should install.