Ask Slashdot: How Would You Stop The Deployment Of Unapproved Code Changes?

Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes: In theory users report bugs, the developers "fix" the bugs, the users test and accept the fix, and finally the "fix" gets released to production as part of a larger change-set. In practice, the bug is reported, the developers implement "a fix", no one else tests it (except for the developer(s) ), and the "fix" gets released with the larger code change set, to production.

We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?

Comments

permissions

[Anonymous Coward]

"How woud you stop un-approved code changes from being deployed?"

Require approval from someone before changes are pushed out.

Re: permissions

[Anonymous Coward]

That doesn't work. I manage devs on five different continents, and my boss always wins. There is no way to beat bad management.

Re: permissions

[Anonymous Coward]

I had a boss that gave me some really good advice 15 years ago when I was getting chewed out by the owner (which was his boss), "get a backbone". If your boss is constantly reaching around you to your workers, they don't respect you and you should be fired for sucking or you leave because they're doing it wrong.

If YOU are managing them, it's your job, not your boss's. Take responsibility or take off.

Re: permissions

[Anonymous Coward]

It's an old saying that a doctor who treats himself has fool for a patient and an ass for a physician.

Yet this is precisely the way many IT shops treat testing.

One of the biggest problems with this approach is that the developer "knows" where the weak spots are and test them, insofar as the schedule allows any real testing at all. An independent tester is not as prone to this sort of tunnel vision, especially when the tester isn't looking at code, but instead at the way the code works. Which, is after all, what the code is ultimately for.

A second problem is that characteristics that make a good software tester are not necessarily those of a good software developer. A good tester has to be the sort of meticulous person who can go over items line-by-line over and over again and never take shortcuts. A good developer may be a good developer precisely because he/she can leap around the concepts and tie together seemingly unrelated points.

Then there's the third problem, which is that contrary to whatever non-Euclidean world Management lives in, you cannot dump the jobs of developer and tester on the same person and rationally expect that they can inflate to handle both requirements optimally, Real employees have limits. Not that that matters when "right-sizing" the corporate personnel assets for the next quarter's executive bonuses.

Re: permissions

[gnasher719]

This. We have devs in the US and in South America, Eastern Europe, NA, and Asia. That doesn't stop my boss from merging bad codel

Where I work, when I do a pull request for the develop branch, I _must_ specify a reviewer and a tester, and until the reviewer has marked the code as fine, and the tester has marked it as fine, and a merge can be done with no conflicts, nobody can merge the code, including any boss. You can quite easily set this up in JIRA, for example.

isn't this pretty straightforward?

[ooloorie]

Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?

- You set up the central repository to only accept code if it can be merged and results in all tests passing.

- You make sure that there is defined code ownership and that people can only change code with a review and with the approval of the owners, also enforced by the source code control system.

Long-term, there are two more things that should happen:

- Developers need to learn how to break up large diffs into many small, individually testable diffs.

- You need to break up your codebase so that it's not a single project with 1Mloc, but 50 small projects with 20kloc each.

Re:isn't this pretty straightforward?

[Excelcia]

In the 1960's was when you wrote software by punching cards that someone else fed in and where it had to work the first time. Every time. That kind of discipline is sorely needed by the original question submitter.

The whole haphazard development model described in the question is absurd. First of all, what kind of single bug requires rifling through back end databases, business rules, web services and multiple front ends? That's not a bug in the software, that's a bug in the pre-design definitions phase. That is not a bug. Seriously... you can't just accept all the premises in the question without thought. That kind of change only happens when someone is is calling "the customer wants this feature changed" or "we misunderstood what the customer needed" a bug, which is wrong on its face.

Secondly, multiple people making changes of that scope simultaneously is just wrong, whatever the cause. Distributed revision control systems were made able to handle multiple simultaneous branches in order to break bottlenecks on people working on different areas of a common source file. They were designed to accommodate merges that had occasional and minor overlaps. What is described here is a completely inappropriate use of that kind of environment. So to answer the question directly, when asked what tools can help, the answer is no tools can help you. The process is wrong. You are far better off reverting to a revision control system that enforces a single checkout of a source file if this is what is going on. Better yet, correct your development strategy.

This can't be emphasized strongly or often enough. Code ownership is a good step forward in this scenario, but the only real fix for these problems is to completely refactor the way change is managed in this project. You wouldn't be wrong to Gantt chart these changes with their subsystem impacts so they can be scheduled on a non-interference basis. Better yet, if you are having to make multiple back-end through to UI changes, you need to go through a whole scope identification phase again.

Your change system is hopelessly broken. Fix that, then the correct use of existing tools to assist you will become readily apparent.

Code reviews: Just say yes

[clay_buster]

Good shops require at least tone reviewer for each code change. This paper describes the effectiveness of code reviews based on previous studies https://www.cs.umd.edu/project...