Slashdot Mirror


Should Developers Have Access To Production?

WHiTe VaMPiRe writes "Kyle Brandt recently wrote an editorial exploring the implications of providing developers access to the production servers of a Web site. He explores the risk introduced by providing higher level access as well as potential compromise solutions."

19 of 402 comments (clear)

  1. For me by enderjsv · · Score: 5, Insightful

    Whenever an error occurs that I can't replicate in a dev environment, I'm always SO tempted to hop into prod and start adding in some output statements.

    Yeah, it's probably a good thing I don't have access to prod.

    1. Re:For me by IICV · · Score: 4, Insightful

      That's why you shouldn't have access to prod, but you should be able to either A. get a clone of prod made fairly quickly or B. already have one running so you can mutilate it however you want.

      Seriously, hardware is cheap and people are expensive. Minimizing person-time is worth a bit of hardware gluttony.

    2. Re:For me by stillpixel · · Score: 5, Funny

      User: There is an error on page X
      I tweak that page code on the production server after looking at the error log. Me back to User: An error really? Have you tried pressing F5?
      User: Oh.. hmmm I guess I must have done something wrong. Sorry for bugging you!
      Me: Hey, no problem.

    3. Re:For me by FatSean · · Score: 4, Insightful

      Test environments are essential, but they do require people-time to keep them matching production.

      --
      Blar.
    4. Re:For me by TheRaven64 · · Score: 4, Informative

      Depends on how your deployment system is designed. Mainline Xen now has support for brining up complete live duplicates of a running VM and keeping them in sync (or letting them diverge), and this support has been in high-availability hypervisors for decades. If you're using a system like this, it's just a couple of commands to clone the production VM and get something that you can break without impacting users.

      --
      I am TheRaven on Soylent News
    5. Re:For me by mwvdlee · · Score: 4, Interesting

      In the mainframe shop we used to have 5 stages; (production, shadow (with similar load to production), functional acceptance, system integration and development), next to that 2 well secured "emergency" stages linking to prod and shadow and a single "free for all" development area outside the control of the basic stages.

      Mainframe shops tend to be much more closed and mature than more modern environments, and hence much less goes wrong.

      I've also worked in a Java shop at the same company, where they had 3 stages and a locally for dev, but the stages were much less controlled and you could easily skip straight to production. Obviously only the most experienced of programmers did this and only when they were absolutely certain. Obviously quite some more fixes went wrong on production.

      Currently working in an environment without stages, I try to work on test copies as much as possible but the temptation of bugfixing directly in production is quite large.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
  2. Short answer by Issarlk · · Score: 5, Insightful

    LOL! No.

  3. Read-only, if that, and nothing more. by Junior+J.+Junior+III · · Score: 4, Informative

    If you want to have control over your production code, you need to have assurance that it is not changing in an uncontrolled fashion. Allowing developers to have access to production locations makes it all too easy for this to happen. Read-only access allows developers to see the running code and perform file comparisons which can be useful in troubleshooting. They should never need more than this.

    And in some cases, even read access can be risky -- I've seen production web sites with resources linking back to development server URIs. It's a good idea to firewall your production servers in such a way that it is not possible for them to reach resources on development servers. This shouldn't prevent developers from being able to read the files on the production server, though.

    --
    You see? You see? Your stupid minds! Stupid! Stupid!
    1. Re:Read-only, if that, and nothing more. by SatanicPuppy · · Score: 4, Interesting

      See, to me this is more an issue of devs not obeying the rules. They damn well shouldn't be changing production code, and they damn well shouldn't be linking code from other servers.

      Either your devs are a bunch of barely trained lunatics, or they're breaking the rules in a vain attempt to get things done in a timely manner.

      Most times, when I see devs screwing with production it's either a "hero" coder who is way too good to use best practices, or a situation in which the environment is so hostile that the "best" solution seems to be breaking the rules.

      I once did some contract work for a company where the Q&A and testing process took a minimum of two weeks for the most trivial changes, and where the admins on the production servers refused to deploy things like security patches without a testing period that ran close to a month. The devs there had a hundred tricks for sneaking their code into production, and linking production code to the development servers in an attempt to meet their productivity goals.

      Fucking nightmare. Once we ironed out the Q&A thing, and split the admins into two groups (one who maintained, and the other who upgraded and approved changes) the whole process evened out and the devs stopped screwing around on production.

      --
      ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  4. Everyone agrees... by SatanicPuppy · · Score: 5, Insightful

    Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

    Its a good practice to keep them separated, but in the end its just a pissing contest. The server admins don't want some filthy dev messing with their stuff, and I can appreciate that.

    However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.

    In the end, its the best practice to have everyone work together sensibly, than throw down inflexible rules that cause more trouble than they prevent.

    --
    ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  5. Uhm, no. by HerculesMO · · Score: 4, Insightful

    This is why there is a change control process, and a testing environment.

    If you're doing it wrong, you're asking for trouble.

    --
    The price is always right if someone else is paying.
  6. Re:What a silly question. by jameson71 · · Score: 5, Insightful

    On the other hand, I wouldn't want the surgeon to have to give instructions to a trained monkey on how to do the the surgery because the surgeon does not have access to the production patient.

  7. Re:What a silly question. by dkleinsc · · Score: 5, Insightful

    So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

    Because if there's a problem, there will be an expectation that they need to intervene to resolve their failures.

    To play Devil's Advocate here, there are some semi-legit reasons why developers might get production access:

    • If there's a serious production failure, developers are often called upon to assist the admins, because while they aren't admin experts they generally have some administration skills.
    • If there's a bug that makes it to production, the time it would take to fix the bug using proper procedures may cost more than doing a quick-and-dirty fix now and cleaning up using proper procedures later.
    • Diagnosing production-only bugs, which frequently require read-only access. For instance, developers may need read-only access to determine that their software didn't deploy correctly.
    • Helping admins properly configure their software.

    Now, none of this should be done willy-nilly. The basic rule at my workplace is that a developer can do nothing that could potentially alter behavior without managerial approval and admin approval where appropriate. At the same time, the primary enforcement of that rule is trusting our devs, so very little of that is actually enforced technologically.

    --
    I am officially gone from /. Long live http://www.soylentnews.com/
  8. Re:What a silly question. by SatanicPuppy · · Score: 4, Insightful

    I work for a big company where every significant business unit has it's own devs and own processes. And while I'm primarily an admin, I still deploy some dev-level stuff a few times a year.

    And nothing, nothing annoys me more than going to a shop where the admin doesn't understand the tech, doesn't read the project requirements, and will not, under any circumstances, let me see his production hardware before he tries to deploy.

    Because inevitably it fails and then he tries to throw me under a bus, and I have to get on a conference call and try to use my magic mind powers to debug a system I'm still not allowed to see.

    Back when I was primarily a java dev, I used to have to configure Tomcat all the time, and it was the sort of thing were you needed to have some experience. And time after time I'd end up dealing with some windows admin who knew absolutely nothing, but was utterly convinced that I could never know more than him about a server technology.

    In short, common sense and a reasonable respect for experience beats a hard and fast rule any day.

    --
    ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  9. Hope you are never audited! by TarPitt · · Score: 4, Interesting

    I worked as an IT auditor for a very big public accounting firm. Reviewing IT controls was a key part of the financial audit (and more so now with Sarbannes Oxley).

    If I found developers had access to production, it was automatically a "no reliance" finding.

    This means the financial applications are inherently untrustworthy that the financial auditors would have to review original source documents for validation.

    "No reliance" meant the audit became much more expensive as a result.

    Also - if the auditors can't rely on the financial reports, should management?

    --
    If your children ever found out how lame you are, they'd murder you in your sleep
  10. Re:What a silly question. by John+Hasler · · Score: 5, Funny

    > There's a difference?

    Sure. The monkey is trained.

    --
    Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
  11. Re:What a silly question. by vegiVamp · · Score: 5, Funny

    As a systems admin, I can assure you that there is definitely a difference.

    Trained monkeys get free bananas and are allowed to fondle their bits in public, to name but two.

    --
    What a depressingly stupid machine.
  12. Comment removed by account_deleted · · Score: 4, Insightful

    Comment removed based on user account deletion

  13. Blue Cross Blue Shield Anyone? by Bourdain · · Score: 4, Interesting

    If I screw up, people can't get the correct pills. It's fun to make other people live dangerously. :-p

    FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.

    I don't know if Blue Cross Blue Shield has fixed this but, as of a few weeks ago (and this probably has existed for a while), living in EST has made it impossible for scrips to be fulfilled via insurance between midnight and 3AM. This is because, according to the late night pharmacist who is familiar with the issue, the servers are in PST and won't allow fulfillment from the anything but the "current day" regardless of time zone. Too bad the devs there don't understand time zones adjustments / UTC/GMT. Yet again, non-profit environments don't tend to attract the swiftest of folk in general.