Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: How Would You Stop The Deployment Of Unapproved Code Changes?

Posted by EditorDavid on from the tested-in-production dept.

Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes: In theory users report bugs, the developers "fix" the bugs, the users test and accept the fix, and finally the "fix" gets released to production as part of a larger change-set. In practice, the bug is reported, the developers implement "a fix", no one else tests it (except for the developer(s) ), and the "fix" gets released with the larger code change set, to production.

We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?

29 of 324 comments (clear)

  1. permissions by Anonymous Coward · · Score: 5, Informative

    "How woud you stop un-approved code changes from being deployed?"

    Require approval from someone before changes are pushed out.

    1. Re: permissions by Anonymous Coward · · Score: 2, Interesting

      Yes. In addition to 100% unit code coverage and integration tests.

    2. Re: permissions by Anonymous Coward · · Score: 4, Insightful

      That doesn't work. I manage devs on five different continents, and my boss always wins. There is no way to beat bad management.

    3. Re: permissions by ooloorie · · Score: 3, Insightful

      Yes, or even more. How many people do you think usually look at each line of text or each line of music before it gets published? And there, the stakes are usually considerably lower than for code.

    4. Re: permissions by Anonymous Coward · · Score: 4, Insightful

      I had a boss that gave me some really good advice 15 years ago when I was getting chewed out by the owner (which was his boss), "get a backbone". If your boss is constantly reaching around you to your workers, they don't respect you and you should be fired for sucking or you leave because they're doing it wrong.

      If YOU are managing them, it's your job, not your boss's. Take responsibility or take off.

    5. Re: permissions by vtcodger · · Score: 3, Interesting

      "That doesn't work"

      It doesn't not work. More eyes do tend to catch out and out bugs. But some still slip through. And stuff that sounds good but busts user's workflow still gets through. And developers hate the delay. And more review costs more. And more testing costs even more than more review. So managers aren't a big fan of either.

      I suppose the solution is never to work with millions of lines of code. But "This sucker is way to big and too complicated" is not an easy sell.

      --
      You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
    6. Re: permissions by Anonymous Coward · · Score: 5, Interesting

      It's an old saying that a doctor who treats himself has fool for a patient and an ass for a physician.

      Yet this is precisely the way many IT shops treat testing.

      One of the biggest problems with this approach is that the developer "knows" where the weak spots are and test them, insofar as the schedule allows any real testing at all. An independent tester is not as prone to this sort of tunnel vision, especially when the tester isn't looking at code, but instead at the way the code works. Which, is after all, what the code is ultimately for.

      A second problem is that characteristics that make a good software tester are not necessarily those of a good software developer. A good tester has to be the sort of meticulous person who can go over items line-by-line over and over again and never take shortcuts. A good developer may be a good developer precisely because he/she can leap around the concepts and tie together seemingly unrelated points.

      Then there's the third problem, which is that contrary to whatever non-Euclidean world Management lives in, you cannot dump the jobs of developer and tester on the same person and rationally expect that they can inflate to handle both requirements optimally, Real employees have limits. Not that that matters when "right-sizing" the corporate personnel assets for the next quarter's executive bonuses.

    7. Re: permissions by sg_oneill · · Score: 2

      This. We have devs in the US and in South America, Eastern Europe, NA, and Asia. That doesn't stop my boss from merging bad codel

      Its the worst. where I work our CEO is a business guy who runs a multi state multi million $ company with large numbers of employees aaaand he likes to code. In fairness, he's actually not a bad coder , coming from a C++ background, but hes rarely over all the issues that the engineers are over, and as a result theres always a background noise of randomness coming into the code from whenever the big boss is bored with money and meetings and all the things rich guys do. And its *strictly* non optional code. If god wants a new screen on the app, and he;s made it, well its in, and we have to stop everything and make the fucking thing work. And he doesnt know our build processes, how we stage things, how issues management works or anything. More likely I get a call at 1am saying "Hey the websites acting up can you have a look at whats going on, and I log into the production server and he's in there with vim boredom progrmaming on the principal production server. Its a nightmare.

      --
      Excuse the Unicode crap in my posts. That's an apostrophe, and slashdot is busted.
    8. Re: permissions by Shoten · · Score: 3, Insightful

      so now you have two coders looking at every line of code?

      Yeah...because this is how it's done when it's done professionally. You have one coder...the guy who wrote the change...and then another coder...the one who tests it.

      This happens in non-code places too, like journalism. One person writes the article, and another proofreads it. (Due to the acceleration of the news cycle, this has been going away...with predictably-bad results.) Consulting? Yes, you have quality control (another person reading and checking the deliverable..every line of it) before it goes to the client. Engineering? One engineer builds the spec, and another has to approve it; this is actually mandated by law for a lot of things, in fact, where permitting is involved (like construction).

      Fundamentally, the question is "how to you keep code from being pushed to the public before it's tested." You seemed to miss that in your reply, because the very point of the question requires two people...people who must understand what their reading (and thus, are coders)...to look at the code. Also, your reply seems to imply that a code change requires reading ALL of the code, not just the new or changed code, and this is simply not true.

      --

      For your security, this post has been encrypted with ROT-13, twice.
    9. Re: permissions by gnasher719 · · Score: 4, Insightful

      This. We have devs in the US and in South America, Eastern Europe, NA, and Asia. That doesn't stop my boss from merging bad codel

      Where I work, when I do a pull request for the develop branch, I _must_ specify a reviewer and a tester, and until the reviewer has marked the code as fine, and the tester has marked it as fine, and a merge can be done with no conflicts, nobody can merge the code, including any boss. You can quite easily set this up in JIRA, for example.

    10. Re: permissions by rickb928 · · Score: 2

      Before forced reductions in anticipation of product decommissioning, we had a dedicated tester, who had good enough regression testing suites that he caught bugs in most releases. And these were most always forehead slappers, 'darn, I forgot that again!!!' type.

      Moving to the new version, new platform, they solved the testing problem by delaying releases interminably, denying features until forced, then complaining about schedules and arbitrary timelines, as in being forced to release code 8 years late.

      Our web team is magnificent. the usual response to release bugs is 'it didn't do that in test'.

      Let's pause for a moment and re-read that. For full effect. You've gotten the joke, I know.

      And the then 'Is it doing it for (fill in this blank with any criteria)?'.

      Followed closely by retracting the code, re-releasing it, and the same bug occurs. More recriminations and demands for objective reports from users. Yes, as their interface to users, I am regularly questioned as to the veracity of the bug reports.

      This is the Agile process in action. Sort of.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    11. Re: permissions by zifn4b · · Score: 2

      Some of that. But I made quite a good living for several decades back in the 1960s, 1970s, and 1980s integrating large systems. Did some development also, and have some sympathy for developers. Good unit tests help. and so would automated testing. But let's get real. Few actual deployed systems have meaningful specs, and even those that started with specs probably didn't maintain the specs and have bugs in the original specs -- omissions

      Hey I'm an old timer too but you apparently never caught up-to-date. Waterfall doesn't work. Never did. By the time a specification is written, it's already out of date and doesn't reflect what the customer actually wants. Other timers by the names of Ward Cunningham, Kent Beck, Martin Fowler and many others figured this out. Now we have Agile and Xtreme Programming and all of the variants.

      Here let's get real with you. First of all, the software projects of the 60's, 70's and 80's were comparatively minuscule to the projects we're trying to tackle today. Waterfall might have worked partially with much smaller less complex projects but it definitely doesn't hack it today. This is a fact. To dispute this, would be to dispute the large-scale empirical research done by the industry veterans I mentioned above. If you think they're wrong, you should be able to school them and make them look like idiots. Betcha can't.

      --
      We'll make great pets
    12. Re: permissions by Aighearach · · Score: 2

      Well, that's the thing about Dunning-Kruger; if you thought it through a little farther, you'd realize that a bunch of people glancing at my post and laughing are almost all at that early peak. ;) You seem rather sure of yourself too, and yet, you didn't even get far enough in to touch on the meat of my comment.

      Know going in that I'm an old-school software developer and that my comment was a universal truth. If you didn't understand it, or were too busy laughing at the nearest meme to try, that's fine; but it wasn't wrong. It can't be wrong, actually, because it was an observation of historical fact.

      Almost everything is waterfall, and waterfall was never the extreme absolutist caricature that you and other idiots laugh at. Yes, you're right, I am superior to you when it comes to reading comprehension of the term "waterfall development model." What people laugh at never even existed, so how much chance do they have to be correct, or have some insight? The author of the book people point at to support their weird claim that waterfall was absolutist even clarified later, in response to those criticisms, that it is not and never was a static system with no changes allowed, you simply try not to make late changes. There was never anything at the end that said, "and when problems come up, you don't even respond." That is just idiocy, and the target audience of the book were engineers for whom that is actually rather obvious.

      The vast majority of software was always made using waterfall process, and that is still true today.

  2. Users don't report bugs by Teancum · · Score: 3, Interesting

    That right there is a part of the problem. Software testers report bugs. Hints about potential bugs can come from end users, but end users are not software testers.

    There is no substitute to a really good (professional.... and paid) software tester that and reliably reproduce bugs that need to be removed. If anything, they are far more valuable than even code monkeys writing thousands of lines of code per month (something also largely irrelevant for quality software). If anything, I would pay the software testing team before the coders if you need to use some volunteer labor like in an open source project.

    End users can offer hints to a good software testing team as to what might be bugs, and end user reports should definitely be taken seriously since it is something that slipped past those testers as well. When the software testers are fired and/or it is presumed that unpaid volunteers are going to be doing that quality assurance process, especially for a commercial software product, you get what you pay for.

    1. Re: Users don't report bugs by Pikoro · · Score: 2

      Are you the owner of Ham Radio Deluxe?

      --
      "Freedom in the USA is not the ability to do what you want. It is the ability to stop others from doing what THEY want"
  3. Feature Toggles by louispq · · Score: 2

    In addition to the feature branching you could also look into feature toggles. A feature toggle is a variable used in a conditional statement to guard code blocks, with the aim of either enabling or disabling the feature code in those blocks for testing or release. While this approach would not prevent unapproved code from being in a release, it would act as a gatekeeper to determine if a specific feature should be exposed to the end user. I however am not sure how you could address your concern for the database

  4. isn't this pretty straightforward? by ooloorie · · Score: 4, Informative

    Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?

    - You set up the central repository to only accept code if it can be merged and results in all tests passing.

    - You make sure that there is defined code ownership and that people can only change code with a review and with the approval of the owners, also enforced by the source code control system.

    Long-term, there are two more things that should happen:

    - Developers need to learn how to break up large diffs into many small, individually testable diffs.

    - You need to break up your codebase so that it's not a single project with 1Mloc, but 50 small projects with 20kloc each.

    1. Re:isn't this pretty straightforward? by Excelcia · · Score: 4, Insightful

      In the 1960's was when you wrote software by punching cards that someone else fed in and where it had to work the first time. Every time. That kind of discipline is sorely needed by the original question submitter.

      The whole haphazard development model described in the question is absurd. First of all, what kind of single bug requires rifling through back end databases, business rules, web services and multiple front ends? That's not a bug in the software, that's a bug in the pre-design definitions phase. That is not a bug. Seriously... you can't just accept all the premises in the question without thought. That kind of change only happens when someone is is calling "the customer wants this feature changed" or "we misunderstood what the customer needed" a bug, which is wrong on its face.

      Secondly, multiple people making changes of that scope simultaneously is just wrong, whatever the cause. Distributed revision control systems were made able to handle multiple simultaneous branches in order to break bottlenecks on people working on different areas of a common source file. They were designed to accommodate merges that had occasional and minor overlaps. What is described here is a completely inappropriate use of that kind of environment. So to answer the question directly, when asked what tools can help, the answer is no tools can help you. The process is wrong. You are far better off reverting to a revision control system that enforces a single checkout of a source file if this is what is going on. Better yet, correct your development strategy.

      This can't be emphasized strongly or often enough. Code ownership is a good step forward in this scenario, but the only real fix for these problems is to completely refactor the way change is managed in this project. You wouldn't be wrong to Gantt chart these changes with their subsystem impacts so they can be scheduled on a non-interference basis. Better yet, if you are having to make multiple back-end through to UI changes, you need to go through a whole scope identification phase again.

      Your change system is hopelessly broken. Fix that, then the correct use of existing tools to assist you will become readily apparent.

  5. Gatekeeper(s) - Needed by brian.stinar · · Score: 2

    and the "fix" gets released with the larger code change set, to production...

    The problem here is that the fix "gets released." I agree, that it seems like releases should at least have the criteria that at least one other person has reviewed the code being released. Otherwise, they have the criteria that one person decided to release it (by definition.)

    I think you could create a system by which pull requests are approved by someone other than the person that created them, and then, after it's been approved, then the code is authorized to be merged into a release branch. Here's one such discussion of that. I'm not an expert on this, but I've heard of it, and I think this line of reasoning could help you.

    Good luck!

  6. Learn from the Rust project's developers. by Anonymous Coward · · Score: 2, Insightful

    A lot could be learned by observing what the developers of the Rust programming language project do when running their project.

    They're dealing with a large project that covers a complex domain, a huge amount of code, and many developers scattered across the globe.

    The first thing to do is to use git, and perhaps something like GitHub. This will allow your developers to collaborate using a free and open source version control system.

    Next, you need a Code of Conduct to prevent social injustice from negatively affecting the project. A Moderation Team is tasked with ensuring that everyone is tolerant, and any intolerance will be ruthlessly stamped out.

    Changes go through a request for comments process. This keeps everything organized and everybody on the same page. Bugs have detailed bug reports and discussion.

    GitHub pull requests are used to integrate the changes. All changes are reviewed by somebody else. If you're a lucky contributor, you may even have your contribution reviewed by none other than Steve Klabnik zerself! If the review passes then their automatic integration/merging bot will merge the pull request into the master branch.

    Your project doesn't have to follow the exact same approach that the Rust project follows, of course. But I think that there are a lot of things that any large software development project could learn from how the Rust developers work. Their approach has scaled to over 1,700 contributors!

  7. Step One by Anonymous Coward · · Score: 3, Insightful

    Step one is to get high-level management to understand and agree with the risks as well as to understand and agree with the costs of preventing them.

    You would think that this is a no-brainer, but its not. I've listened to a COO tell me from across a boardroom table that they have to be able to bypass deployment processes for business-critical hot fixes because time is of the essence in those situations, and that was the end of it. So what you've got is that in an "emergency" an "informal" approval from an "important" person is all that is needed. Feel free to define those words however you wish, naturally.

  8. Microservices by lucm · · Score: 2

    When people are worried about changes in "many layers of the stack", it's usually a good time to re-architect the system and build microservices. Basically, you get the entire stack in every microservice and you stop worrying about ripple effects; you upgrade or troubleshoot things at a much smaller scale.

    I highly recommend this book:
    https://www.amazon.com/Buildin...

    It explains how to achieve this, including how to deal with the tough parts like the database layer.

    --
    lucm, indeed.
  9. A piano by sjames · · Score: 2

    Next person to release an untested line of code will play the piano for us *SLAM*

  10. Re: Give all the users AIDS by anorectal insemena by Anonymous Coward · · Score: 2, Funny

    Somewhere, on some porn forum right now, someone is asking a coding question.

  11. Sarbanes Oxley/SOX ... by Billly+Gates · · Score: 3, Interesting

    ... hides

    --
    http://saveie6.com/
  12. Re:Yes it addresses the problem by phantomfive · · Score: 2

    Basically I'm saying that if you don't know how to write modular code, if you don't know how to define components in a way that corresponds to specific areas of the business domain (and that's not quite the right way to do it, but close enough), then it doesn't matter if you use microservices. Your microservice architecture will end up just as messy as your non-microservice architecture.

    The underlying problem isn't microservice vs non-microservice: both can be fine architectures. The underlying problem is not knowing how to divide up your system into components, not being able to recognize the natural lines in the problem.

    --
    "First they came for the slanderers and i said nothing."
  13. I'm somewhat in the same boat where I work. by aix+tom · · Score: 2

    From my experience I suspect the problem doesn't start with "The code being pushed into production" in wrong way, the problem starts with *Users* sending un-coordinated bug reports and feature request directly to the developers.

    Without some program/feature person "In charge" on the user side it feels pretty hopeless.

    The only solution I have come up with for me: Make sure that management KNOWS that without proper procedures in place the is an increased risk of bugs slipping through that might affect production code. It works pretty well so war. We have a few cases of bugs that affect production for a few hours each year, but as long as management understands they lose $X to bugs because they don't want to pay $Y for propper QA procedures, and it's *Their* decision to implement better QA when the cost of bugs increases too much for their tates, and don't blame the developers for it, then it works pretty OK.

  14. Code reviews: Just say yes by clay_buster · · Score: 4, Informative

    Good shops require at least tone reviewer for each code change. This paper describes the effectiveness of code reviews based on previous studies https://www.cs.umd.edu/project...

  15. So ... your users are the QA department? by Opportunist · · Score: 3, Insightful

    Hey, not that it's anything unusual, but if you cut corners, expect to be sued for patent infringement by Apple. Or something like that.

    In all seriousness, though, if your USERS report bugs, you have a fundamental problem here. Because this is what it should look like:

    User defines specification. Programmer codes to spec. QA tests if spec is implemented correctly. Program ships. User finds something he doesn't like? It obviously has to be a change request, because the program does what the user specc'd.

    Yes, it is that simple. And yes, I'm fully aware that users don't have the first clue what they actually want. But they will never learn if you keep treating their blunders and imprecise specifications as if it was YOUR fault!

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.