Slashdot Mirror
Should Developers Have Access To Production?
WHiTe VaMPiRe writes "Kyle Brandt recently wrote an editorial exploring the implications of providing developers access to the production servers of a Web site. He explores the risk introduced by providing higher level access as well as potential compromise solutions."
Whenever an error occurs that I can't replicate in a dev environment, I'm always SO tempted to hop into prod and start adding in some output statements.
Yeah, it's probably a good thing I don't have access to prod.
LOL! No.
No. It just encourages sloppy development practices.
Would you want to drive over a bridge that wasn't actually designed and engineered, but rather they just piled some stuff up and will fix it if it collapses? Or have a surgeon chopping you open with the idea that they'll figure it out as they go? So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?
It is my experience that giving development access to production gives you a production environment that looks like it has been vandalized. Although meaning well and trying to make the best application as possible; they need their own development lab, and their own staging / production lab.
If you want to have control over your production code, you need to have assurance that it is not changing in an uncontrolled fashion. Allowing developers to have access to production locations makes it all too easy for this to happen. Read-only access allows developers to see the running code and perform file comparisons which can be useful in troubleshooting. They should never need more than this.
And in some cases, even read access can be risky -- I've seen production web sites with resources linking back to development server URIs. It's a good idea to firewall your production servers in such a way that it is not possible for them to reach resources on development servers. This shouldn't prevent developers from being able to read the files on the production server, though.
You see? You see? Your stupid minds! Stupid! Stupid!
Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.
Its a good practice to keep them separated, but in the end its just a pissing contest. The server admins don't want some filthy dev messing with their stuff, and I can appreciate that.
However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.
In the end, its the best practice to have everyone work together sensibly, than throw down inflexible rules that cause more trouble than they prevent.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
So, yes.
This is why there is a change control process, and a testing environment.
If you're doing it wrong, you're asking for trouble.
The price is always right if someone else is paying.
There's no correct answer to this question. It depends on the size of the organization and the nature of the system. I've worked in different companies that have been on either sides of where I thought the line should be. The line is drawn in a very different place for a 20 employee company than where it is in a 20,000 employee company.
The day developers can write code that compiles the first time, then yes, otherwise, jesus, no.
I work as an Oracle DBA for a mid-size company, and I provide a day-old cleaned copy of production in a different environment/box, and it does the trick.
Hi, I Boris. Hear fix bear, yes?
I think it's helpful in analyzing real-world data and getting an idea about real system loads, testing issues to see if they are in the wild today, etc. For a good developer, it makes life much easier.
In a very healthy development ecosystem all this data is replicated and there is never any need for a developer to touch prod. In the development ecosystems that exist in the real world though, most are very unhealthy, frustrated by ham-fisted security, process flaws, red-tape, inconsistency, and incompetence ranging from scattered to mostly cloudy.
The answer is, do you have the class of developer that knows what not to do and desires to play nice, or do you have the usual.
As a developer I can tell you that it's impossible to test programs properly and thoroughly without access to production data. However, developers should NOT be granted access to production logins/sites - production data should be copied into development work areas so that developers have an appropriate "sandbox" in which to work/test.
I don't think he meant malicious backdoors. I read that as backdoors to allow debugging/etc.
And the concensus is ... NO
Who let this question through? It doesn't even seem controversial. I am not aware of any good reason to routinely give developers access to production.
Censorship is obscene. Patriotism is bigotry. Faith is a vice. Slashdot 2.0 sucks.
Biggest issue my cow-orkers and I have is that the sysadmin *claims* that the dev box and production box have the same packages, configuration, etc. but in reality, they don't. Most often we find out when we ask for production stuff to be copied over to the dev site to test errors, etc. and just loading it - which works on the live site - generates errors on the dev site.
Don't blame me, I voted for Kodos
If you are a small software shop then I can see reasons for allowing your small technical staff to have access to production. It's all well and good saying that only the admin of that server should have access and there's a full rollout procedure in place to be followed only on certain days, certain times; but even when I've seen that sort of structure in place there are times when it's useful for the developers to have access to production. Nothing is perfect and we'd all love to have multitude's of staging servers, replicating the typical load and uses of production but for a hell of a lot of (non critical I'd add) systems that just doesn't happen.
There simply is no one rule fits all. Sometimes I wish we had extremely rigorous rules & regulations in place - I'd probably get to go home a hell of a lot earlier. I'm not suggesting you start chucking exceptions all over your checkout code on live but I think you should asses your own situation (and staff for that matter).
jaymz
With some fore-thought and some discipline an application can be developed with very robust logging techniques. It takes development time, but there is nothing cooler than asking the production guys to turn the logging detail up for a few packages and seeing tons of data in the logs. It's not perfect as you can't log every variable at every moment but it certainly does help.
I understand some shops can't or won't modify the logging levels on production servers.
Blar.
I work in an environment where the devs fix bugs before adding features, so the code is stable almost all the time. I have less than 1 callout a week that's caused by something a dev has done to the code.
We hire the best devs, and work in an environment where fixing bugs is more important than adding features. The result is that our devs get full access to production, and even offer to provide support in order to ensure that they're the ones that are woken up if something they've broken falls over OOH.
I've been at my current company long enough that I'd forgotten there were places where devs and ops didn't trust each other.
Speaking as a developer, I want/need read-only access in production. All too often I need to dig out information while troubleshooting, and most commonly I don't know what all bits I'll need when I start. If it were easy to identify exactly what I'd need to find the problem, I usually already know what the problem is. The hard ones are the ones I can't replicate in development and I only have a starting point, something that won't identify the problem but might help me narrow down where to look next. In those cases the only place I can look is production (since I can't make it happen in a controlled development environment) and I can't give the admins a list of what I'll need (because I need to dig through logs and config files before I'll know what I need to look for next). And if we've gotten to this point, it's probably a priority problem impacting production so it needs to get fixed Right Bloody Now.
OTOH, while I may need to look at production, I don't need and don't want the ability to modify production except by going through the admins. This, of course, also requires admins who can follow basic instructions like "Look at config file FOO. Find the line in section X that starts with Y. It's value should be XYZZY followed by the number 1. Change that 1 to a unique number for that machine/instance. Repeat this for every machine/instance.". But all too often the response is "That's too complicated. Can you just give us config files to install?". And of course when I ask for the current config files, so I can be sure I'm not overwriting any other modifications to them (which may have happened since the admins control them and do modify them), I get "We can't do that, they've got production passwords in them.". Now all I can do is throw up my hands and go "Whatever.".
It's not necessarily a case of the admins versus the developers, its more of practicing good data governance.
Our developers used to have direct access to all of the production databases. This was bad enough, but because of this the organization permitted them to directly "clean up" databases (meaning they wrote to tables directly), we had data that was being changed without the ability to really know who did it. The DBAs hated it and the developers were extremely uncomfortable doing it but it happened anyway. We eventually had a real process audit and the auditors had a field day.
Needless to say we changed. I hope.
I am one of the few people who can run correct code the first time round. I am also proficient enough in OS matters to be able to circumvent access to locked down resources. So I don't care what this post says, I'm doing it myyyyy waaaaay.
I worked as an IT auditor for a very big public accounting firm. Reviewing IT controls was a key part of the financial audit (and more so now with Sarbannes Oxley).
If I found developers had access to production, it was automatically a "no reliance" finding.
This means the financial applications are inherently untrustworthy that the financial auditors would have to review original source documents for validation.
"No reliance" meant the audit became much more expensive as a result.
Also - if the auditors can't rely on the financial reports, should management?
If your children ever found out how lame you are, they'd murder you in your sleep
Even when the suggestion of "would you like root on this internal box?" was put to me, my answer was always "No". I write code. Others test it. Admins deploy it.
People specialize for a reason. If you want half-assed administration, give root to a developer. If you want half-assed code, let admins write software. If you want half-assed testing, have admins and/or developers do it.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
If I found out a developer changed something in a system I tested without it going through the proper process...
Let's just say I would be very interested to hear why they shouldn't go back and rerun everything again on THEIR dime. (at the very least) In fact, we DID do just that to someone who let a revision slip into their UUT because a developer felt it would fix something and make it perform better.
It wasn't too expensive of a mistake, just $250,000 to rerun that portion of the test. Although that was just the physical cost of performing the test. I don't even want to know how much it cost in labor especially considering it was a 22 day test.
Even if the change was removed, how do I know that without physically verifying checksums (do I even trust it anymore since their CM process is obviously flawed)
Out of modpoints but really liked a post? 1BDkF6TtmmeZ3yqXbz9yhdYVqRYnwFoXDj
Of course developers should have some level of access to the production environment. No matter how good your test environment is, it's not going to match the live server in load, or what's in cache, or the concurrent access to some resource, etc.
Our process was to have one person with access, investigating whatever problem via the SQL command line, or the Rails console (let the RoR jokes commence), with another person watching, to make sure they were doing select * and not update or delete. Even then we'd execute stuff in a transaction or sandbox so that we weren't making any permanent changes, although changes to memcache generally can't be rolled back so easily.
I've seen admins, who are adamant that dev not be allowed to change anything, change psql configurations at a whim, crippling DB performance. And then blame dev for poor response times. That's so not cool.
Comment removed based on user account deletion
If I screw up, people can't get the correct pills. It's fun to make other people live dangerously. :-p
FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.
I don't know if Blue Cross Blue Shield has fixed this but, as of a few weeks ago (and this probably has existed for a while), living in EST has made it impossible for scrips to be fulfilled via insurance between midnight and 3AM. This is because, according to the late night pharmacist who is familiar with the issue, the servers are in PST and won't allow fulfillment from the anything but the "current day" regardless of time zone. Too bad the devs there don't understand time zones adjustments / UTC/GMT. Yet again, non-profit environments don't tend to attract the swiftest of folk in general.
For a one man show the answer is self evident.
For a small web company developing "brochure-ware" - probably more efficient.
For a small team it's ideal to have individual sandboxes - with one sandbox listed as "staging". Assign the lead developer to turnover code to production. Individual developers have access but are told not to touch anything. They will typically sift through live environment making sure it matches what is in their sandbox, looking at logs, etc.
For a mid-size team you need one person for maintenance (which includes monitoring nightly builds, responding to code turnover requests, managing automated testing). Even more critical if the code you write is compiled, fragile, or highly sensitive. - Individual developers don't have access to the live box - maybe the team lead will.
For large teams or small team "units" part of a large production shop : Several layers of "staging and testing" will exist. Code turnovers are mostly automated. Developers don't have access. Automated rollbacks are possible from a robust code management system.
The key is discipline. If you find yourself modifying live code - you're not disciplined. It means you're not willing to insert logging code and would rather pollute the production environment. There should never be a need to copy from production back to a sandbox (that is what version control software is for!) And version control files should never live on the production server (i.e. in Subversion you never do a checkout of code on the production server - you do an export instead).
Even with controls in place, there may be a tendency to "develop on production by proxy". Which means instead of re-creating the problem in development, the developer is saying "here try this, here try this, here try this". The team lead should recognize this and put a stop to it.
-CF
... in a split decision, vi wins. Oh wait ... wrong holy war!
Que Deus te de em dobro o que me desejas
[May God give you double that which you wish for me]
The more developers work in production, the more they can ONLY work in production.
I'm all for read access (the more eyeballs the better), but actual access to change anything is a train wreck. The devs will forget to check the changes in to the source repo, or they'll check them in differently (bad copy/paste), or they'll check them into the wrong branch/tag. Regardless the next release that goes out silently adds the bug back into production.
And if developers think it's difficult to fully clone a prod environment configuration into dev now, wait until they try to do it after developers have been hacking on it directly for a while.
Pretty soon every release is a train wreck requiring tons of post-release tweaking and hammering to get it in place. Every release is a stressful mess as you're all crossing your fingers because you really have no idea what you are actually changing and no way to find out.
Just don't do it. Hire a good build engineer/release manager/software configuration manager that can sort out, automate, and track environment management well enough that yes, you can reliably clone an accurate representation of production in a matter of minutes. He'll cost you about as much as a good sr developer, but the savings across the board will easily dwarf his salary.
My
Long answer, it depends on the situation.
I (and a couple of my coworkers) have access to production servers, but we don't develop on prod. End of story. We have other devs who do not have access to prod. Dev is for dev, prod is for prod and don't let anyone without the discipline to keep that rule have access to prod.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
Reading a lot of comments it looks like there's a wide variety of definitions for some of the job titles and roles people are discussing here, so I'll list how I see them:
* System Admin - Person(s) responsible for the hardware and supporting (OS, Web service, code language and client libs, JVMs, etc) software. They do not in any way support the applications running on said system and would be incapable of debugging or supporting an *application* problem even with a gun to their head. Most can only describe 2-3 sentences of what the applications even do. They do not report to or answer directly to the application teams. They also do NOT install application code.
* Database Admin - Only want to address roles here. At every location, the actual application data stored in the database is NOT the role or responsibility of the admin. It belongs to the application team and any changes are their job and their accountability. The DBA only deals with schemas, packages, procedures, scripts, access roles and grants, etc. DBAs should NOT MANIPULATE DATA. Asking or allowing them to do so opens up a never ending blame game and is counterproductive. If you want to create some title and role within the application team where all data manipulation funnels through, that's the way to do it.
* Implementation Specialist (Code Migration) - Trained monkeys who are supposed to follow a set of pre-delivered instructions for deploying application changes. In my experience their technical knowledge is limited, they cannot verify copy/paste correctly, and screw up (transferring ZIP files in ASCII instead of binary) more than they succeed. I don't feel this position is even necessary. The PROCESS is necessary and it can be performed by anyone, even a developer, as long as they switch their role hats before starting and are held accountable for accurately following the deployment instructions given.
* Production support - They act both in a technical and relationship role, being the contact point between the customer (internal or external) and the application team when issues arise. Generally have read-only access to production. They are able to debug many problems and resolve a few, but definitely not all of them. They do not participate in any part of the development lifecycle processes.
* Developers - Not going to discuss or debate any pre-production roles here since it's irrelevant to the topic. Developers are the only ones I would be confident could debug ANY problem. They are going to need some reasonable level of access to production, logging, or information if you want to have an application that can maintain high availability and recover quickly from any type of outage.
If your definitions to these roles differ significantly, then my answer for your company's situation would change.
Depending on the size of the application and the team allocated to run it, I've performed up to 4/5 roles and was pretty much the 5th as well since the Sys Admin only could barely squeak by supporting Windows 2k and definitely had zero knowledge of any of the supporting software. Are you going to hire someone to do 6 hours of work per month just to separate the responsibilities? Of course not. So the OP's generalized question is open to a million different interpretations because of all the different variables that weren't specified.
My most recent application team recently went through our production lockdown after finally migrating over an application suite purchased from another company. Developers and Prod Support have read-only access. Database passwords used by the applications have been restricted down to just a couple individuals. When changes need to be made to either the application or data, an Emergency ID is checked out to a requesting individual with the appropriate access level,
Hell. No.
I'm a developer as well as a sysadmin and I NEVER tweak anything in production and I have full access to it.
I have an exact copy of my production environment for development and I do all my tweaking/test deployments there.
In fact nothing gets deployed to production until everything has been checked in development.
My previous job had dev/qa/prod environments where the devs had full access to development and it was so bad that we had to virtualize it for them just so we could revert back to a pristine snapshot whenever they jacked up the dev server.
No. Do not give developers access to the production machine ever, except me...just this once..
The Kruger Dunning explains most post on