Slashdot Mirror
Ask Slashdot: How Would You Stop The Deployment Of Unapproved Code Changes?
Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes:
In theory users report bugs, the developers "fix" the bugs, the users test and accept the fix, and finally the "fix" gets released to production as part of a larger change-set. In practice, the bug is reported, the developers implement "a fix", no one else tests it (except for the developer(s) ), and the "fix" gets released with the larger code change set, to production.
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
That doesn't work. I manage devs on five different continents, and my boss always wins. There is no way to beat bad management.
Yes, or even more. How many people do you think usually look at each line of text or each line of music before it gets published? And there, the stakes are usually considerably lower than for code.
This. Bad management merges bad code, and there is ni overcoming that.
A lot could be learned by observing what the developers of the Rust programming language project do when running their project.
They're dealing with a large project that covers a complex domain, a huge amount of code, and many developers scattered across the globe.
The first thing to do is to use git, and perhaps something like GitHub. This will allow your developers to collaborate using a free and open source version control system.
Next, you need a Code of Conduct to prevent social injustice from negatively affecting the project. A Moderation Team is tasked with ensuring that everyone is tolerant, and any intolerance will be ruthlessly stamped out.
Changes go through a request for comments process. This keeps everything organized and everybody on the same page. Bugs have detailed bug reports and discussion.
GitHub pull requests are used to integrate the changes. All changes are reviewed by somebody else. If you're a lucky contributor, you may even have your contribution reviewed by none other than Steve Klabnik zerself! If the review passes then their automatic integration/merging bot will merge the pull request into the master branch.
Your project doesn't have to follow the exact same approach that the Rust project follows, of course. But I think that there are a lot of things that any large software development project could learn from how the Rust developers work. Their approach has scaled to over 1,700 contributors!
Step one is to get high-level management to understand and agree with the risks as well as to understand and agree with the costs of preventing them.
You would think that this is a no-brainer, but its not. I've listened to a COO tell me from across a boardroom table that they have to be able to bypass deployment processes for business-critical hot fixes because time is of the essence in those situations, and that was the end of it. So what you've got is that in an "emergency" an "informal" approval from an "important" person is all that is needed. Feel free to define those words however you wish, naturally.
I had a boss that gave me some really good advice 15 years ago when I was getting chewed out by the owner (which was his boss), "get a backbone". If your boss is constantly reaching around you to your workers, they don't respect you and you should be fired for sucking or you leave because they're doing it wrong.
If YOU are managing them, it's your job, not your boss's. Take responsibility or take off.
In the 1960's was when you wrote software by punching cards that someone else fed in and where it had to work the first time. Every time. That kind of discipline is sorely needed by the original question submitter.
The whole haphazard development model described in the question is absurd. First of all, what kind of single bug requires rifling through back end databases, business rules, web services and multiple front ends? That's not a bug in the software, that's a bug in the pre-design definitions phase. That is not a bug. Seriously... you can't just accept all the premises in the question without thought. That kind of change only happens when someone is is calling "the customer wants this feature changed" or "we misunderstood what the customer needed" a bug, which is wrong on its face.
Secondly, multiple people making changes of that scope simultaneously is just wrong, whatever the cause. Distributed revision control systems were made able to handle multiple simultaneous branches in order to break bottlenecks on people working on different areas of a common source file. They were designed to accommodate merges that had occasional and minor overlaps. What is described here is a completely inappropriate use of that kind of environment. So to answer the question directly, when asked what tools can help, the answer is no tools can help you. The process is wrong. You are far better off reverting to a revision control system that enforces a single checkout of a source file if this is what is going on. Better yet, correct your development strategy.
This can't be emphasized strongly or often enough. Code ownership is a good step forward in this scenario, but the only real fix for these problems is to completely refactor the way change is managed in this project. You wouldn't be wrong to Gantt chart these changes with their subsystem impacts so they can be scheduled on a non-interference basis. Better yet, if you are having to make multiple back-end through to UI changes, you need to go through a whole scope identification phase again.
Your change system is hopelessly broken. Fix that, then the correct use of existing tools to assist you will become readily apparent.
Hey, not that it's anything unusual, but if you cut corners, expect to be sued for patent infringement by Apple. Or something like that.
In all seriousness, though, if your USERS report bugs, you have a fundamental problem here. Because this is what it should look like:
User defines specification. Programmer codes to spec. QA tests if spec is implemented correctly. Program ships. User finds something he doesn't like? It obviously has to be a change request, because the program does what the user specc'd.
Yes, it is that simple. And yes, I'm fully aware that users don't have the first clue what they actually want. But they will never learn if you keep treating their blunders and imprecise specifications as if it was YOUR fault!
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
so now you have two coders looking at every line of code?
Yeah...because this is how it's done when it's done professionally. You have one coder...the guy who wrote the change...and then another coder...the one who tests it.
This happens in non-code places too, like journalism. One person writes the article, and another proofreads it. (Due to the acceleration of the news cycle, this has been going away...with predictably-bad results.) Consulting? Yes, you have quality control (another person reading and checking the deliverable..every line of it) before it goes to the client. Engineering? One engineer builds the spec, and another has to approve it; this is actually mandated by law for a lot of things, in fact, where permitting is involved (like construction).
Fundamentally, the question is "how to you keep code from being pushed to the public before it's tested." You seemed to miss that in your reply, because the very point of the question requires two people...people who must understand what their reading (and thus, are coders)...to look at the code. Also, your reply seems to imply that a code change requires reading ALL of the code, not just the new or changed code, and this is simply not true.
For your security, this post has been encrypted with ROT-13, twice.
This. We have devs in the US and in South America, Eastern Europe, NA, and Asia. That doesn't stop my boss from merging bad codel
Where I work, when I do a pull request for the develop branch, I _must_ specify a reviewer and a tester, and until the reviewer has marked the code as fine, and the tester has marked it as fine, and a merge can be done with no conflicts, nobody can merge the code, including any boss. You can quite easily set this up in JIRA, for example.