Slashdot Mirror

← Back to Stories (view on slashdot.org)

Should Developers Have Access To Production?

Posted by CmdrTaco on from the can-of-worms dept.

WHiTe VaMPiRe writes "Kyle Brandt recently wrote an editorial exploring the implications of providing developers access to the production servers of a Web site. He explores the risk introduced by providing higher level access as well as potential compromise solutions."

19 of 402 comments (clear)

  1. For me by enderjsv · · Score: 5, Insightful

    Whenever an error occurs that I can't replicate in a dev environment, I'm always SO tempted to hop into prod and start adding in some output statements.

    Yeah, it's probably a good thing I don't have access to prod.

    1. Re:For me by IICV · · Score: 4, Insightful

      That's why you shouldn't have access to prod, but you should be able to either A. get a clone of prod made fairly quickly or B. already have one running so you can mutilate it however you want.

      Seriously, hardware is cheap and people are expensive. Minimizing person-time is worth a bit of hardware gluttony.

    2. Re:For me by FatSean · · Score: 4, Insightful

      Test environments are essential, but they do require people-time to keep them matching production.

      --
      Blar.
    3. Re:For me by idontgno · · Score: 3, Insightful

      If I screw up, people can't get the correct pills.
      It's fun to make other people live dangerously. :-p

      FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.

      --
      Welcome to the Panopticon. Used to be a prison, now it's your home.
    4. Re:For me by Americano · · Score: 3, Insightful

      Deploying to an extra "prod-clone" test server should require pretty trivial amounts of people-time if it's set up properly - it's just another test server, if it takes ridiculous amounts of manual effort to deploy to a single extra test server, you're probably doing something remarkably inefficient.

    5. Re:For me by Gerzel · · Score: 3, Insightful

      This article also assumes that the developers (or often developer) is not also the server admin. Many shops have one or a few IT people wearing many hats.

    6. Re:For me by Anonymous Coward · · Score: 3, Insightful

      Yes, we call that staging.

      We work on dev.

      We promote changes to stage, which is a dup of the prod environment.

      When stage is approved we promote to prod.

    7. Re:For me by tomhudson · · Score: 3, Insightful

      You've obviously never had your boss direct live traffic to your "test" server at the datacenter, and not tell you for 2 weeks.

      I'm comfortable working without a net - but I'd like to know about it.

      Obviously, from that point on, any time we did a make, we made sure that we could install the new code within a few seconds, so as to avoid downtime (we could tolerate a delay of 15 secs once or twice a day without raising eyebrows).

      And sometimes a fix HAS to be done live to prevent bad data from propagating. If you don't trust yourself to get the command just right the first time, then step away from the keyboard and find someone who can type the right sql command the first time with 5 people hanging over their shoulder sh*tting bricks. In other words, you need one of the devs - fast. You certainly wouldn't let a sysadmin do it.

      In theory, you may be right. In theory, theory and practice are the same. In practice, they're not. There are ALWAYS exceptions. Not realizing this, and being flexible, is a mistake.

  2. Short answer by Issarlk · · Score: 5, Insightful

    LOL! No.

  3. What a silly question. by Score+Whore · · Score: 3, Insightful

    No. It just encourages sloppy development practices.

    Would you want to drive over a bridge that wasn't actually designed and engineered, but rather they just piled some stuff up and will fix it if it collapses? Or have a surgeon chopping you open with the idea that they'll figure it out as they go? So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

    1. Re:What a silly question. by jameson71 · · Score: 5, Insightful

      On the other hand, I wouldn't want the surgeon to have to give instructions to a trained monkey on how to do the the surgery because the surgeon does not have access to the production patient.

    2. Re:What a silly question. by dkleinsc · · Score: 5, Insightful

      So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

      Because if there's a problem, there will be an expectation that they need to intervene to resolve their failures.

      To play Devil's Advocate here, there are some semi-legit reasons why developers might get production access:

      • If there's a serious production failure, developers are often called upon to assist the admins, because while they aren't admin experts they generally have some administration skills.
      • If there's a bug that makes it to production, the time it would take to fix the bug using proper procedures may cost more than doing a quick-and-dirty fix now and cleaning up using proper procedures later.
      • Diagnosing production-only bugs, which frequently require read-only access. For instance, developers may need read-only access to determine that their software didn't deploy correctly.
      • Helping admins properly configure their software.

      Now, none of this should be done willy-nilly. The basic rule at my workplace is that a developer can do nothing that could potentially alter behavior without managerial approval and admin approval where appropriate. At the same time, the primary enforcement of that rule is trusting our devs, so very little of that is actually enforced technologically.

      --
      I am officially gone from /. Long live http://www.soylentnews.com/
    3. Re:What a silly question. by jellomizer · · Score: 3, Insightful

      Bad analogy.

      It is more like having a Cardiologist diagnosing the problem then telling the heart surgeon what to fix.
      We don't give our fixes to a trained monkey we give them to System Administers or implementation specialists.

      I would say that I am a good developer. However I am only an OK System Administrator and I often when my code is done and working well. Deploying the code offers a new set of issues to work threw.

      However If I give to people who are good at taking my code and implementing it, the process runs much smoother without much problems.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    4. Re:What a silly question. by SatanicPuppy · · Score: 4, Insightful

      I work for a big company where every significant business unit has it's own devs and own processes. And while I'm primarily an admin, I still deploy some dev-level stuff a few times a year.

      And nothing, nothing annoys me more than going to a shop where the admin doesn't understand the tech, doesn't read the project requirements, and will not, under any circumstances, let me see his production hardware before he tries to deploy.

      Because inevitably it fails and then he tries to throw me under a bus, and I have to get on a conference call and try to use my magic mind powers to debug a system I'm still not allowed to see.

      Back when I was primarily a java dev, I used to have to configure Tomcat all the time, and it was the sort of thing were you needed to have some experience. And time after time I'd end up dealing with some windows admin who knew absolutely nothing, but was utterly convinced that I could never know more than him about a server technology.

      In short, common sense and a reasonable respect for experience beats a hard and fast rule any day.

      --
      ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  4. Everyone agrees... by SatanicPuppy · · Score: 5, Insightful

    Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

    Its a good practice to keep them separated, but in the end its just a pissing contest. The server admins don't want some filthy dev messing with their stuff, and I can appreciate that.

    However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.

    In the end, its the best practice to have everyone work together sensibly, than throw down inflexible rules that cause more trouble than they prevent.

    --
    ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  5. Uhm, no. by HerculesMO · · Score: 4, Insightful

    This is why there is a change control process, and a testing environment.

    If you're doing it wrong, you're asking for trouble.

    --
    The price is always right if someone else is paying.
  6. No correct answer by Motard · · Score: 3, Insightful

    There's no correct answer to this question. It depends on the size of the organization and the nature of the system. I've worked in different companies that have been on either sides of where I thought the line should be. The line is drawn in a very different place for a 20 employee company than where it is in a 20,000 employee company.

  7. Depends on the size of your organization by jaymz2k4 · · Score: 3, Insightful

    If you are a small software shop then I can see reasons for allowing your small technical staff to have access to production. It's all well and good saying that only the admin of that server should have access and there's a full rollout procedure in place to be followed only on certain days, certain times; but even when I've seen that sort of structure in place there are times when it's useful for the developers to have access to production. Nothing is perfect and we'd all love to have multitude's of staging servers, replicating the typical load and uses of production but for a hell of a lot of (non critical I'd add) systems that just doesn't happen.

    There simply is no one rule fits all. Sometimes I wish we had extremely rigorous rules & regulations in place - I'd probably get to go home a hell of a lot earlier. I'm not suggesting you start chucking exceptions all over your checkout code on live but I think you should asses your own situation (and staff for that matter).

    --
    jaymz
  8. Comment removed by account_deleted · · Score: 4, Insightful

    Comment removed based on user account deletion