Slashdot Mirror
Ask Slashdot: How Would You Stop The Deployment Of Unapproved Code Changes?
Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes:
In theory users report bugs, the developers "fix" the bugs, the users test and accept the fix, and finally the "fix" gets released to production as part of a larger change-set. In practice, the bug is reported, the developers implement "a fix", no one else tests it (except for the developer(s) ), and the "fix" gets released with the larger code change set, to production.
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
"How woud you stop un-approved code changes from being deployed?"
Require approval from someone before changes are pushed out.
Welcome to the concept of a qa department
make the devs pinky swear?
Fire anybody who deploys unapproved code.
That right there is a part of the problem. Software testers report bugs. Hints about potential bugs can come from end users, but end users are not software testers.
There is no substitute to a really good (professional.... and paid) software tester that and reliably reproduce bugs that need to be removed. If anything, they are far more valuable than even code monkeys writing thousands of lines of code per month (something also largely irrelevant for quality software). If anything, I would pay the software testing team before the coders if you need to use some volunteer labor like in an open source project.
End users can offer hints to a good software testing team as to what might be bugs, and end user reports should definitely be taken seriously since it is something that slipped past those testers as well. When the software testers are fired and/or it is presumed that unpaid volunteers are going to be doing that quality assurance process, especially for a commercial software product, you get what you pay for.
Fire the manager. This sounds like very bad management. This is a process to follow, not a tool to blame when developers fuck up.
I used to work as a QA manager. What I tested and accepted was generally committed and released on my OK and the senior Dev was rational and sane so we were almost always on the same page. The one time we really disagreed, he reminded me I wasn't his boss. I agreed, then repeated previous points made by our shared boss, and then he promptly gave in. Our boss was really good at making sure everyone was on the same page and priorities.
Our team worked so well together, we started our own company after the previous company went tits up. If you have members on your team who act against you by breaking process, let them go. They are worse than dead weight.
Don't be so cheap. The answer is well known and reliable enough just for some reason the organization can't be bothered with QA.
The best way to fix that is to ensure it's always so broken it can't be shipped. That will eventually happen anyway.
In addition to the feature branching you could also look into feature toggles. A feature toggle is a variable used in a conditional statement to guard code blocks, with the aim of either enabling or disabling the feature code in those blocks for testing or release. While this approach would not prevent unapproved code from being in a release, it would act as a gatekeeper to determine if a specific feature should be exposed to the end user. I however am not sure how you could address your concern for the database
More money, better computers, whatever it takes - get a subset of the users on board to test the new code before it's final.
Hulk smash!
Sleep your way to a whiter smile...date a dentist!
If it's not strictly a bug fix, feature flags are a good way to go. Keep a standard of adding any reasonably sized addition behind a feature flag. That way it's easy to turn on/off your new button, or swap between two methods of doing something.
If it's testable, have a code review and release process built around difference testing. Reviewers have to sign off on the externally visible differences in the code's effect. Release engineers can close bugs based on expected diffs and reject releases in the face of un-expected ones. These can be simple diffs of database files, or more complicated screenshot difference analysis. Either way, you have someone other than the dev having to click a checkbox that the change makes sense.
- You set up the central repository to only accept code if it can be merged and results in all tests passing.
- You make sure that there is defined code ownership and that people can only change code with a review and with the approval of the owners, also enforced by the source code control system.
Long-term, there are two more things that should happen:
- Developers need to learn how to break up large diffs into many small, individually testable diffs.
- You need to break up your codebase so that it's not a single project with 1Mloc, but 50 small projects with 20kloc each.
Presumably you tagged the sources that went into the build that went to your customers?
If you did, when you make bug fixes you need to check out against that tag, not to the bleeding edge code where new features are being added.
Depending on how many fixes there are and how complex and messy the source tree is, you can either try to merge the changes into your bleeding edge code base or make the changes twice. In general, if the bleeding edge is being vigorously refactored or otherwise aggressively reorganized, you'll probably end up making the changes twice. Something that isn't often brought up when refactoring is how you will handle merging bug fixes.
It is best to have one developer who drives the source control system and knows how to do merges.
If you didn't tag the sources you released to customers, you and your entire team needs to be shot.
and the "fix" gets released with the larger code change set, to production...
The problem here is that the fix "gets released." I agree, that it seems like releases should at least have the criteria that at least one other person has reviewed the code being released. Otherwise, they have the criteria that one person decided to release it (by definition.)
I think you could create a system by which pull requests are approved by someone other than the person that created them, and then, after it's been approved, then the code is authorized to be merged into a release branch. Here's one such discussion of that. I'm not an expert on this, but I've heard of it, and I think this line of reasoning could help you.
Good luck!
MODULE based languages help by isolating code changes to smaller chunks that are more easily tested.
DGDanforth
A lot could be learned by observing what the developers of the Rust programming language project do when running their project.
They're dealing with a large project that covers a complex domain, a huge amount of code, and many developers scattered across the globe.
The first thing to do is to use git, and perhaps something like GitHub. This will allow your developers to collaborate using a free and open source version control system.
Next, you need a Code of Conduct to prevent social injustice from negatively affecting the project. A Moderation Team is tasked with ensuring that everyone is tolerant, and any intolerance will be ruthlessly stamped out.
Changes go through a request for comments process. This keeps everything organized and everybody on the same page. Bugs have detailed bug reports and discussion.
GitHub pull requests are used to integrate the changes. All changes are reviewed by somebody else. If you're a lucky contributor, you may even have your contribution reviewed by none other than Steve Klabnik zerself! If the review passes then their automatic integration/merging bot will merge the pull request into the master branch.
Your project doesn't have to follow the exact same approach that the Rust project follows, of course. But I think that there are a lot of things that any large software development project could learn from how the Rust developers work. Their approach has scaled to over 1,700 contributors!
Two Johnny Jr coders should not be allowed to change code marked as mission critical by a senior developer or domain experts. Yes, its a pain to get changes approved by an expert in a time zone far, far away but youthful exuberance is no substitute for hard won experience. If you think you've found a simple, straightforward solution to longstanding problems you'd better be able to back up that assertion with a test plan that states more than 'Yup, this fixes my bug. SHIP IT!'.
Step one is to get high-level management to understand and agree with the risks as well as to understand and agree with the costs of preventing them.
You would think that this is a no-brainer, but its not. I've listened to a COO tell me from across a boardroom table that they have to be able to bypass deployment processes for business-critical hot fixes because time is of the essence in those situations, and that was the end of it. So what you've got is that in an "emergency" an "informal" approval from an "important" person is all that is needed. Feel free to define those words however you wish, naturally.
If I understand correctly; the poster is looking for a solution that covers the ENVIRONMENT; not JUST the code.
If you're looking to deploy features that require changes to multiple backend services; you NEED to start looking at something like configuration management; containers; or both. (in the following examples; I'm going to imply Puppet+Docker; as it's what I use every day)
As a precursor to this; if (you|your management team|the operations team) are NOT using configuration management to manage infrastructure; you REALLY going to want to take a long-hard-look at yourself in the mirror and decide if you want to continue with such an outdated company. ;)
If a feature needs user testing; Define the required infrastructure changes in either a new environment or a reserved environment and include them with your commit. Come up with a method to boot up a number of containers (the layout / number is going to be highly application dependant; this is why "ready made solutions" are do hard to find here) that are a full private implementation of your application; using the custom environment as defined earlier.
If there's a persistent backend database that requires changes; implement it through a proxy (effectively) where the (additions|changes|deletions) are virtually implemented to all calls made by the temporary application. (You could make the backend read-only; or restore a backup into your containers; I'd personally just bring up a read-only slave + begin replication and proxy writes into a private datastore to record them) At this point; redirect traffic (based on whatever criteria you determine is required) to this private environment (IE: point users who have opted into testing at it; HAProxy is great at this)
If you so choose; feel free to record changes they make either temporarily or permanently as part of the final commit (json blob or whatever format you prefer) or simply throw them away. (again; this is going to be highly dependant on your application / needs)
Once testing has been completed and you've determined the users have 'confirmed' the feature; merge the configuration management environment into your production branch and commit your change (to whatever master you decide is master). Push and let your stack do it's work.
This approach is highly recommended (personally) and works wonderfully; IF you're already in the position to utilize it. It needs to be approached from the get-go of the project however; else trying to migrate an existing application to this approach MAY consume more work than simply throwing the project out and starting over. (really depends on the project)
-jd
When people are worried about changes in "many layers of the stack", it's usually a good time to re-architect the system and build microservices. Basically, you get the entire stack in every microservice and you stop worrying about ripple effects; you upgrade or troubleshoot things at a much smaller scale.
I highly recommend this book:
https://www.amazon.com/Buildin...
It explains how to achieve this, including how to deal with the tough parts like the database layer.
lucm, indeed.
It's pretty straightforward.
You need a Release Engineer position (full time, not a repurposed Developer). They need to have the authority to say NO to a release.
Go read up on what REs do. That's exactly what your organization needs.
If your deployment path is: "dev > test > production" you are in for a world of hurt. Try adding another step or two like: "dev > test > acceptance > production" or even "dev > test > acceptance > staging > production". These extra steps are where code ferments for a few days and is subject or scripted tests or actions mirrored from production before it goes live live.
Developer's head on a pike as a warning to others. The hint will be taken, I promise.
It would work on paper.
... it will certainly stop the developers from deploying unapproved code changes for 2 minutes.
This link is great in relation to how Linus deals with a huge distributed codebase. It's actually what convinced me GIT was superior to SVN.
You lay down a CVS Tag and the build technician builds from it. Releases are clearly identified with an MD5 identifier.
If only one official binary exists, it's what people will run.
https://www.youtube.com/watch?...
Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
Next person to release an untested line of code will play the piano for us *SLAM*
for testing. That means full blown test environments as well as time in the user's day or a special team to test bug fixes. Also if you go with a team don't outsource that team to the lowest bidder. They'll just pass everything as "OK" and spend the rest of their time looking for a better job. Hire somebody, give them a decent salary and benefits.
Unless testing really isn't that important. Depending on your needs it might not be (despite all the indignation that engenders). One thing I've learned about writing software: Good enough is _always_ good enough. Just ask Microsoft (Word) and Corel (WordPerfect).
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
i have found in life that you cannot force your manager to do their job, which is to manage. like linus torvalds on the kernel. coordinate, communicate, and tie-break.
you can only implement small technical things that may prevent people from breaking things completely, and coordinate with other tiny islands of sanity in the company on your own. talk to them. if you can. if its allowed.
but overall, this is a fundamental problem in the management of the organization you work for, and has nothing to do with software really. if we were all on pen and paper, similar issues would happen - because its a management organization problem, not a technical problem.
You missed covering why you *want* to do this.
Unless you're working on human-life-critical software... does it matter?
Have a code review process. Make developers responsible for both what they write and what they approve. Call it a day.
If that's not enough? Improve your build process.
Build a canary process where only a small section of your users sees the code, watch it for errors, and when the error rate is good enough, push to 100%.
yes. code ownership is so 1960s. so is software that worked properly.
You're a judgemental control freak and you don't belong in the workforce. Quit now and give someone else your job.
Obviously, you'd allow the "community" to look at bugs and submit issues, but it seems as though in trying to prevent bad blood, there's no chain of command in these kinds of projects. The person with the highest bug fixing record gets the final word, followed by the "new features team" rather than adding features and then try to fix conflicts; this is where the original author of the project has the final say over the top bug fixer if they aren't one in the same. If multiple solutions exists to a bug, offer a community vote and an honest layout of any compromises that may result for each solution. Interface eye candy should always be last in a project.
That's how my company does it:
Git branches:
master
staging
dev branches, one for each fix/ticket
Each ticket gets a separate dev branch.
Each dev branch gets tested locally.
Each dev branch undergoes a separate code review.
Each dev branch gets deployed to an isolated dev machine.
Dev branches get tested independently, one by one in dev environments.
Once all the dev branches are signed off, they all get merged into staging.
When all tickets reach this point, each ticket gets re-tested in staging.
Regression testing kicks in and the whole system gets tested as a whole unit.
Staging gets merged into master, a release goes out the door.
Tools are one route, but they will take time & money to implement which you may not have and don't usually resolve the underlying problem of lack of responsibility or process. There are numerous integration tools that can bring together your source control, testing suites & feature tracking tools. In addition DevOps and release management software can provide automated workflows but again you are talking a rather large spend in time, money or both.
Communicate
Remove the stigma that developers can't "talk to the users". There is nothing written in stone that says you can't work closely with users and form lasting professional relationships with them. Before you go trying to enforce a formal process on them, simply communicate your concerns with them directly. You don't have to babble on about code merges, database scripts. KISS - focus on the cost to the company in time and money if something is missed in testing (or lack thereof). Their interest in making the system and release process go more smoothly should be fairly apparent based on their responses. If available resources are the concern then this is when you can bring in upper management to assist with bulking up your user base to improve the release testing and day to day operations. However, make sure you don't get stuck with just one person. Knowledge silos on the users end can be just as dangerous as they are on the development side.
Formalize your process and enforce it
If you got buy-in from your user communication this should go pretty smoothly, if not you'll need extra leverage from upper management. Make sure you've worked out a decent workflow with the users. It doesn't have to be laden with cool tools. The process should easily spell out what is changing and what needs testing for each release. Keep in mind the audience you'll be working with, don't design or purchase some crazy release management system if the users are only comfortable sending emails. Tools will come with time, the most important thing is that your process is easy to follow and enforce.
only on /.
I took away their password access to the production system, forcing them through test and quality assurance deployments first. (What's QA???)
They were incredulous. They cited the need to get software into production and "fix it later".
Management tended to side with them, as they promised profits.
You sound like an Uber manager assuming you don't kick any of the guys.
Have the guts to say it and then do it. PMs will get onboard when they get nervous that work won't get to clients.
As a long time SQA/HQA eng, this is an awesome question. I've been around a lot of blocks. New build. Broken. Here's another one. Broken. My Golden Rule? Stay until it works. I'd so much rather have a good build than something slapped together. It wastes everyone's time.
I've tried. And been fired for it. From the unemployment office, it is amusing to watch as WS ATG investigates a company due to bad testing practices. It is disturbing to watch as 'computer glitches' cause hundreds if not thousands of travelers to be impacted. It is disturbing to send an email that - due to bad management practices (just what this article describes) mean that I cannot guarantee or relate the quality of a product; then be escorted out the door the next day.
I have and always will stand up for quality, which in this 'we will not fix it next release' era is a difficult thing to do.
Best thing to do: Put in 'requirements' that there be a code review before a 'check-in' is accepted. Best of luck if no one want to accept your recommendation(s).
Random release chaos is no way to run a real product. A product with no coherent release process (including user notification) is a recipe for epic fail and gnashing of teeth. Grow up and do it methodically and consistently.
That doesn't address the problem, it just demands a complete rebuild to your current fad.
Stuff is rolled out that isn't tested by QA or user approved tested. Rolling it out in microservices just means some have issues and others not, making it more confusing and less cohesive.
This is typical of lazy management, if there is a problem, the blame will be assigned to the developers, but it's typical of Yahoo style development work. (Literally they do this crap in Yahoo, their CTO is a joke). It usually means management has become detached from the product and instead is managing the vague process that generates the product.
Yeh, it sounds like Yahoo's Warp Drive (link below). Where they sacked the QA. So all the code is merged and NOBODY tests it, the first the users get to accept it is when its live code.
The claim there is that developers do their own test tools to improve their code, i.e. they do an adhoc thing to fill in for the missing QA people. But obviously its a subset of QA, it doesn't involve users, and it doesn't span code outside the developers own reach. He would not be able to require another programmer fix a problem before release, since he is not QA.
Really sacking incompetent leadership is the only solution. Can you imagine a self driving car built by Yahoo? They wouldn't even be able to demonstrate to THEMSELVES that it won't crash, because they've simply never tested it.
https://developers.slashdot.org/story/15/12/12/0115226/no-more-qa-yahoos-tech-leaders-say-engineers-are-better-off-coding-with-no-net
The emergancy is now over. It is normal business hours. You now start the reviewing process for the emergency fix. Your peers evaluate your fix/workaround or the development of a proper fix is started. The fix might even be so dirty it becomes a new out of band fix.
Rolling it out in microservices just means some have issues and others not, making it more confusing and less cohesive.
I don't think you understand what a microservice is.
The point is not to randomly package web services; the point is to define components that correspond to specific areas of the business domain. That way, when you upgrade or deploy a specific microservice, you know exactly what part of the business you're impacting, and this makes governance a lot easier. Instead of having a handful of managers snoozing in CAB meetings and rubberstamping changes that are not clear, you can get the thumbs up from the right people and leave the rest alone.
This is by far the easiest way to avoid rogue or hidden commits making their way to production. And microservices are not a "fad", they're the latest iteration of a concept that's been around for quite a while (SOA) and that just needed a bit of further thinking.
You really should read that book, it's worth it.
lucm, indeed.
I have had this problem in several different projects with different teams.
I generally think of it as the "code hostage" or "cherry-picking" problem. You have the work of 15 issues reviewed and merged and loaded up and running on a test environment. For 14 of those issues, a "user" ( or whatever you call the non-developer issue-owner in this case ) checks in and says it is good. Time is passing and the 15th person is a no-show. It's worse than if they said it still wasn't fixed -- then you would immediately go to work reverting that code change and re-testing everything -- but instead everything is held up in limbo.
One way to address the majority of these issues, is to set up a separate testing environment or server per ticket, and put the pull request or branch on it, before it is merged to the main branch. Force users to test BEFORE it gets on the path that will end up in a deployment. This means that slow moving people won't block others.
This might seem expensive, but with APIs to cloud hosting and/or container technology, it can be automated, and is a lot cheaper than it would have been a few years ago.
You can still end up with a problem on the now integrated code, and still have to cherry-pick / revert some commits, and since the result has not yet been tested all together, you still have to re-test. However this happens much less often, because it is only happening in the cases where interaction between different work is causing the problem.
Somewhere, on some porn forum right now, someone is asking a coding question.
You have to come back from fantasyland. She was your sister, and you roofied her... And the "other cousin" was one of her stuffed dolls.
Sleep your way to a whiter smile...date a dentist!
as long as everyone buys in that Quality is important and fixing urgent bugs is also important.
When we go to production we have a 3 day code freeze during which only P0 bugs (which are rare at that point) can lead to the code being changed.
After production the code is tagged and new development start on a new branch.
Any Production bugs which cant wait till next release are fixed on the production branch, tested by QA team and released.
The developer makes sure to do the same fix in the new code branch (if the code has changed significantly sometimes the fix is no longer needed or you do an equivalent fix)
Code is built everyday. At the end of everyday we let the QA team know which stories are ready for testing. Developers work on their private branches and at 2 PM merge and build. A list of tickets fixed and stories completed is sent to QA team and they can start testin (what do they do till 2 PM? Work on automation) After 2 Pm developers will not check in code unless its for a fix to one of the stories they sent for testing and QA raised a bug. This is done if the build is broken and QA cannot work else it is fixed in private branch and available next days build. By the end of the sprint pretty much every story has been tested - we dont consider a story done if QA is not done and all P0 and P1 bugs are fixed. Automation may not be complete but at least manual QA is done for the effort to count in that scrums velocity.
Before every release we have a strenghtening Sprint where we are not doing new functionality rather fixing all leftover P2 and P3 bugs (we do fix these durings Sprints but a story is not considered blocked for P2 and P3). This also gives time for other activities like preparing runbooks and data dumps for the switchover.
Typically Business UAT team is running one cycle behind the dev and QA team so this also gives them a chance to catch up , do one final round verifying all stories and production tickets for the release and give a signoff. 3 days before we freeze. Typically by this time if we dont have a lot of bugs the team is actually pruning stories for the next release.
The only tricky part is when we tag and branch and try to start working on the new features during the strenghtening sprint. Then we have to be carefull about merging back the last minute fixes into the next release branch.
**Life is too short to be serious**
Hire better, more senior people. Hire more staff if you are understaffed. Because if you have enough developers, and they are seasoned, then together you'll come up with a process that works for YOUR company and YOUR team. Or - be like too many companies in a race to the bottom of underpaying, understaffing, and then complaining when it doesn't work out. Or worse - thinking some process or tool will save the day. It won't, and if it did - you wouldn't deserve it.
We use JIRA but there are many ways to do this. Implement a pipeline that must be followed.
* User creates a ticket in system (bug/issue, etc).
* Developer works on ticket (in new branch)
* Developer get's code review from peer
* Developer pushes changes to staging server
* user who filed ticket tests the change/signs off on fix
* code deployed to QA to test change
* QA signs off on case
* Developer merges code from branch to master (or build release engineer)
* code is released
I'm sure you've found that 6 weeks is long for a "sprint". 4 weeks should be the max length. Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes: In theory users report bugs,
Want to keep unapproved changes out and have a cross geo org? Easy, just make Gerrit code reviews mandatory.
idiots
... hides
http://saveie6.com/
Our setup is far from perfect, but what we do is:
Developer makes change (bugfix or feature), change is sent to code-review, if approved, sent to testing, when QA approves that the feature has been implemented or that the bug has been fixed, QA approves, and only at this point is the developer able to submit the code changes onto the project branches.
Fire all the developers but one. Implement an automated software patcher & deployer which uses a fully data-driven statistical model which correlates User-provided error reports to modifications to the resulting changes in machine code & software configuration to accomplish aforementioned task. Fire the remaining developer.
manager
Struggling to understand why team size and location would have any relevance to having a single gatekeeper ensuring code is tested and checked in properly? either the team is small enough that this can be handled easily or anything beyond half a dozen people even over different locations it is mandatory if you want to have any semblenance of quality in the end product. One of the projects I work on has over 100 developers across many locations (both different buildings and different cities in some cases). strict control of the check in and build process is the only way to ensure teams working across disparate functionality and components operate properly. If you have code getting to production that hasn't been tested then you have a broken build and test process and yes I know a lot of projects have this problem, that doesn't make it right.
Basically change processes are out of control, where trust and competency are missing.
I add there should be more than one proven change backout. Surprising how many change approvers approve anything - even without a backout plan.
Why? the process has been dumbed down so well paid know it alls, can be replaced with simps. You get what you pay for.
There's a wealth of excellent suggestion here already, but I'd like to take a different approach to answering your question. Put simply, start with you customer[s]. Your post does not mention but strongly suggests that this software is for in-house use. On that basis, identify your stakeholders and understand their appetite for risk and their desire to move forward quickly.
As the old adage [of project management] goes: "Do you want this quickly? do you want this with quality> do you want this at low cost? Pick any two..."
Once you have a clear picture of the risk appetite and operational demands of your client, you are going to be able to make some strategic recommendations about the way you go about developing software. However, one observation I would make is that whatever answer you finally come up with, one of the most important things to realise is that you can't "create" good software quality, it has to be controlled. What this means is that to drive up software quality, you have to apply the right controls to the right steps in the delivery sequence.
If we were to go just from the text of your post, then we might recommend that you deploy some form of software configuration management (SCM) solution, such as Rational Clearcase or one of the many closed or open source competitors. This will give you "hands off" deployment of code to your Production systems, which means that you can then remove Developer [update] access to Production systems. Of all the things you do, this might be the single most powerful, because now you have created a control that requires all software to go through a formal change process.
Once this is in place, you might then be able to layer additional requirements on to your process, such as "All changes to Production systems must be approved by the Business Owner of the application being changed". Or perhaps, "All change requests must be accompanied by a User Acceptance Testing Lead sign-off that the code has been tested and accepted." Or maybe, "All changes must be accompanied by a backout plan that has been tested in the UAT or QA environment". Sorry, this kinda assumes you have segregated environments. If you don't, well, good luck, I guess...
These are, of course, just scattergun suggestions as to things you might like to take a look at. Without conducting an end-to-end risk assessment of your SDLC practices, it's actually difficult to know where to start. However, if you have access to any Industry Standards, then you might find lots of useful "best practice" advice. Among publications I would recommend, you might like to look at:-
NIST Cyber Security Framework (CSF) Release 1.1 [Free]
NIST Security and Privacy Controls for Federal Information Systems and Organizations (NIST SP800-53r4) [Free]
COBIT 5 [Licensed]
ISO27002 [Licensed]
ITIL Service Management Framework [See Wikipedia reference Page, Axelos, etc][Licensed]
In short, there is no shortage of best practice guidance out there. Your challenge will be to look at the size and capability of your organisation, at the risk appetite and budgetary tolerances of your sponsors, then go from there.
Countries like North Korea would execute the deployment team.
From my experience I suspect the problem doesn't start with "The code being pushed into production" in wrong way, the problem starts with *Users* sending un-coordinated bug reports and feature request directly to the developers.
Without some program/feature person "In charge" on the user side it feels pretty hopeless.
The only solution I have come up with for me: Make sure that management KNOWS that without proper procedures in place the is an increased risk of bugs slipping through that might affect production code. It works pretty well so war. We have a few cases of bugs that affect production for a few hours each year, but as long as management understands they lose $X to bugs because they don't want to pay $Y for propper QA procedures, and it's *Their* decision to implement better QA when the cost of bugs increases too much for their tates, and don't blame the developers for it, then it works pretty OK.
It's a difficult concept I understand, but if you don't trust your engineers it might be time to consider getting new ones. Trust falls into 2 categories: They won't put the wrong thing live and when they do they will fix it quickly.
https://en.wikipedia.org/wiki/...
Let me guess, your codebase is written in Cobol?
Look, this really isn't up to the developers to fix. This is something the management of the company has to understand and put a stop to. All of us developers loathe heavy processes, but when it comes to releasing to production, it's absolutely necessary.
So my advice is to start working on the management chain of your company. You should tell them what happens right now, why this is bad, and explain that one day the production server will blow up.
And nothing will happen. For a while. Because if they were clueless enough to let this happen in the first place, they are not going to fix it.
But one day someone is going to push a fix that breaks everything. And then it's up to the developers to let management know that it's actually their fault, not the developer who pushed the fix. They were the ones who allowed this practice. But you have to be diplomatic about it, because just yelling "I told you so" won't help.
The question is if you think this is actually worth it - if you like your job enough to take this fight. Because I would be so scared of working in a company that allows such a sloppy process, that I would seriously consider getting another job.
Now I'm actually running my own company. And I can tell you that if this happens in my company, I wouldn't be mad at you for letting you know we are doing something wrong. But I would be mad at you if you didn't tell me. Sure, I know how to manage software, but in general there will be similar situations in other areas of the company where I'm not doing The Right Thing. A good employee tries to help by fixing things. A great employee helps the company to fix things.
Good shops require at least tone reviewer for each code change. This paper describes the effectiveness of code reviews based on previous studies https://www.cs.umd.edu/project...
what is your alternative?
Linus Torvalds disagrees. Your real problem is that you use Agile. Your process is broken by design. Study how the real professionals do it (Linux kernel) and you will learn a great deal.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
I don't really understand this development model "theory". Are users "end users" or are users in-house testers?
If they are end users, how would they test the fix before it reaches production code?
If they are testers and they don't test the fix, who the hell closes the bug report?
Since you worry about errors getting introduced by merges, it sounds like you are also missing regression testing.
If you don't do regression testing, and don't verify fixes on the release branch, what the hell is your QA department doing?
I think the theory you are looking for is this:
1: Users report bug.
2: QA verifies the bug on the released code, and also that it exist in latest development code.
3: Developer fix bug.
4: QA verifies fix and close the bug report.
5: Code Freeze. QA does regression testing, and verifies all new fixes still work as intended.
6: Release!
There is no need for a single gatekeeper to do merges. Your developers need to be competent enough that they don't routinely break stuff when merging. If they are not capable of getting a merge right most of the time, you need new developers.
GHE has a new mode where pull requests need explicit approval to be merged, and you can also prevent pushing to the main branch.
Hey, not that it's anything unusual, but if you cut corners, expect to be sued for patent infringement by Apple. Or something like that.
In all seriousness, though, if your USERS report bugs, you have a fundamental problem here. Because this is what it should look like:
User defines specification. Programmer codes to spec. QA tests if spec is implemented correctly. Program ships. User finds something he doesn't like? It obviously has to be a change request, because the program does what the user specc'd.
Yes, it is that simple. And yes, I'm fully aware that users don't have the first clue what they actually want. But they will never learn if you keep treating their blunders and imprecise specifications as if it was YOUR fault!
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
This wrong thinking is exactly why everything sucks today, and nothing works the way it should. Developers think it's the user's responsibility to QA their code, and that is simply not the case.
First, it is an epic failure on your part if one of your users HAS to report a bug. It means you did not do your job.
Second, if your users DO report a bug, you need to STOP EVERYTHING and get to root cause - not implement a bandaid "hotfix" that masks the symptoms while creating new bugs.
Finally, you need to do this BY YOURSELF. It is not our job to "here, test this for me, test that for me." We don't work for you. You developers need to get over that and grow up.
You don't want to make it even more cumbersome to change the code, as it sounds like you are already struggling with the 10 Mloc codebase. So forget about having humans "approve" the changes.
What you want to do is make it easy to submit good code and difficult to submit bad code. This means that you will need the capability to quickly assess the proposed patch, for some definition of "good" and "bad". Computers are fairly good at this. In other words: test-first development, with automated testing on several levels.
Bug comes in, you write an automated user acceptance test that FAILS. Then you try to find and fix the bug. When the newly added test passes, the bug is by definition fixed. Add a couple of unit/module tests, and a couple of UATs for good measure. Rinse and repeat.
There's plenty of tools for this. All major platforms have the capability for programmatic reflection. Use e.g. Cucumber for the acceptance tests, and the relevant unit test framework for module/unit tests. In addition to the functional tests, try static analysis.
Unit tests and pull requests. Use them.
And attack dogs. Angry attack dogs.
Don't allow developers to promote code - that should be done by a specific role responsible for change management, code review and deployment. Use file system monitoring software to alert and log all changes.
Greed is the root of all evil.
1. Development team has access only to Development and Testing servers
2. Testing team has access only to the application layer, not the servers
3. Deployment team has access only to Testing and Production servers
This way, developers can never deploy code changes to production servers. The deployment team should never deploy a code change without clearance from the testing team.
Produce a one-page "procedures" document - clearly, but simply lay out the process in moving code from the programmer's branch to the QA branch and into the production branch. Have everyone read and sign it.
The first time someone violates it, you give them an informal warning.
The second time they violate it, have them sit down with management and HR and tell them that if they violate the rules again, they'll be terminated.
The third time, you terminate them.
Easy...no automation required...you simply have to cause a change in the 'culture'. Programmers are very good at following rules - providing they are clearly stated and obviously put in place for a reason.
If you insist on there being an automated system...
When I've worked on setups like this, we give programmers the rights to do what they like in their own working branch(es) and into the QA branch - but deny them access to the production branch. QA get read-access to the QA branch and are the only people allowed to check stuff into the production branch. Generally programmers get their build together at the end of a sprint - then at the start of the next sprint, QA check it out and either release it or punt it back to the programmers...have two week sprints instead of six so that this doesn't add too much latency into the bug fix cycle. If you're spending more time fixing small bugs than adding large features, then you can temporarily drop the sprint cycle down to one week - if major features are being rolled out - then push the cycle up to at most 3 weeks.
www.sjbaker.org
I am dealing with similarly sized project for last 13 years. Our workflow is different though. We have continuous updates (no batch updates) when it comes to bug fixing. We use our in-house Project Management system.
- Client sends a request/bug report, our team creates new Task in our PM and
- assigns it to department/programmer
- programmer fixes it, tests it on Devel version puts the task to QC status
- QC team tests described problem and greenlights deployment
- programmer deploys/merges the fix into the live version and puts the task to Completed
- QC team re-tests live version and notifies client of completed task
Although there are few levels of fixes
- trivial - we skip the QC status/QC test in the middle, programmer fixes it, tests it and deploys it with setting it to Completed which trigers QC team test and client's notification (really super trivial fixes, text typos etc.)
- complex - when developing complex new features we freeze the deployment (blocking task) of other patches while the complex task is in QC status to avoid any conflicts during subsequent deployment - QC status ensures that the problem is tested by QC team in exact state it will be deployed and there will not be competing patches deployed before this one. But it blocks other patches. This one has high priority in our PM and gets fast-tracked before every other patch.
There are many draw backs of every scenario but we try to keep healthy balance between speed/flexibility and super-stability. This yields good results. But the PM (project management) with good prioritization and tracking features is essential. It solves the problems with deployment/testing...
Well, I've got to get back to work. When I stop rowing, the slave ship just goes in circles.
This may not apply for certain types of applications, but if you've got a million lines of code in an app, I'd consider breaking that bad boy up into a bunch of smaller, independent services. Instead of many fingers in one pie, with a finger-to-pie ratio of X, you would have the same number of fingers distributed across N pies, giving a finger-to-pie ratio of X/N. Would this be better for you? Fewer merge conflicts, independent (usually) release schedules, fewer interdependencies?
What Source Control Management (SCM) do you use? (Git, SVN, etc? Do you use one?) What Bug Tracking system (Jira, whatever) do you use? (Do you use one?) What Migration process do you use? Do you use tools? I must disagree with another poster. Agile itself is not the problem. However, if your programmers, etc. are not disciplined....they don't all follow the process that everyone agreed to....that is the problem.
An old system from IBM called CMVC (which is still in use by the way) was not just an SCM but a bug reporting system as well as a build system. A defect is opened. Code changes are tied to that defect. When that happens test records are opened and sent to the person who opened the defect. There is also a verify record created. The build group can not build that defect until the test records are complete. Then they build it. Now you have a defect that is linked to the code changes and linked to a specific build (or builds if you have multiple branches of development).
It does not prevent the problems mentioned above but it does clearly define who should do what when. Much like sbaker pointed out above. And then you manage the people involved to get them to do their part at the proper time.
When the complaint comes from a customer the support person takes on the role of the customer and is responsible for the testing. This usually causes support to be broken into pieces. One is solely facing the customer and the other(s) take larger roles in the testing including sometimes setting up tests for the developers to use.
Require devs to sign code, but also have a separate cert used to sign stuff that is PROD certified. Once QA signs off, and the RFC is approved, have a trusted non developer (an info sec engineer maybe?) sign the code with the prod cert - then set your rules to not let the dev cert signed code run outside of their systems.
you hire a person to be the gatekeeper that does exactly what you want.
Sorry, but there is no cheap and free way to do this, it's called a project manager and they need to be competent and detail oriented.
Devs sign off on the code and their own testing, then this person makes sure that the bet testers also tested and signed off on it, then they personally sign off that all is well and OK's publishing to a new gold release.
Not hard to do, and requires your management to be competent and understand they need to spend money on manpower.
Do not look at laser with remaining good eye.
My organization a while ago saw significant issues with untested fixes being deployed and similar bad practices (undocumented configurations, lack of integration testing, etc.). The thing that did it for us was seeing our up-time drop below 99% in production systems. It became downright embarrassing and started costing us real $.
So our then-CIO froze all production changes for 90 days. In that time, we instituted a change review board. They now approve all production changes. Without the culture change that came with the culture freeze, the new process might have failed. But we bought into it. And by putting them in as the gate-keeper, we now require documented QA approvals.
So in your case, a Change Management process like this would help. But you'll also need to change the way people care about your production environment. You'll need both, but take heart - it can be done.
Funtime Candy Wow! - my plan for eventually conquering Japan.
I think your difficulty is that your current work flow obfuscates two roles. You are using your Stakeholders (the users of your software) as the individuals in charge of acceptance. They're likely not motivated, or possibly not qualified, to do so really. Remember they're just interested in getting THEIR work done. If their workflow were "enlightened" like yours, then they might care about process... but probably not.
The missing piece of the puzzle here is the role of Product Owner. An individual that works with your team in close proximity that prioritizes your work and helps you determine what the acceptance criteria are before you start. Your work is not done until that person accepts the work according to the criteria.
How in the blue fuck is Agile the problem? Agile does not preclude have a single gatekeeper for code check-ins at all. Clearly you talk without knowing what you are talking about.
This is the part of C++ that you can't learn in 21 days.
Call Russia and ask them for some of those Gun-Shooting robots and install them behind your developers.
Developers code. Developers review code. Developers write code to test code.
Developers can't touch production.
Automated checks of ANYTHING are golden because you aren't relying on people, and you aren't pitting people vs people.
It is all about consistency when you boil down to the bones of a well-running team.
Start with those rules and actually follow them and you end up with a pretty awesome setup, because Developers will naturally gravitate to defensive, test-driven programming when those are the rules.
1) All code is reviewed before it is merged, this is easily enforcable in bitbucket or github, probably others, but use one of these two anyway. It won't stop all your problems, but it is great triage and it forces lone-wolf mitigation. I don't even care if the reviewer is an expert... this isn't a gate keeper activity as much as it is a sanity check. "hey why did this variable get set, then re-set before you checked it?" or "hey how come you deleted 90 source files?"
2) All your code goes through some kind of code-quality gate like Sonar Qube or some other Linting tool. This can find and highlight common mistakes, and get them fixed before they are even reviewed. Developers hate to modify a pull request, so give them as many free "oh if I fix this nobody will bother me" wins
3) All unit tests must pass before code is merged. This is slightly tricky but you are going to need Jenkins or some other tool that is not as widely used as Jenkins; and integrate it into your bitbucket/github.
4) All deployments are automated. Sure you can have someone push a button, but they shouldn't have to fiddle with anything. And whoever is pushing that button can't be a developer.
5) Since your deployments are automated, your QA system is exactly like your production system (but maybe smaller scale) this way you are actually testing the thing as it will be deployed. "Kinda sorta how it will work in prod" is never a good test. If your QA deployment is automated, and the deploy is broken in QA, then you have to fix it before it goes to prod. Ideally your developers can't touch QA either, but you can work toward that.
6) Since your developers aren't touching production, they will be forced to do things like: write actual helpful log messages, not make code changes that break the deploy, and reproduce production problems in Unit Tests so they stay fixed forever.
No, I'm saying that the comment "code ownership is so 1960's", which was obviously meant as a "get with the times grandpa" statement was improper. Good project management practices are timeless, and development strategies and disciplines from the past are not necessarily improperly applied to current projects. A certain amount of centralized command and control is necessary. Where that is applied depends on the technical and cultural aspects of a project. In projects with a strong central scope and well defined design, projects that have good change acceptance procedures and where developers are professional a more distributed and loose change creation process can be applied, a la Git. In projects where there is not a strong central change acceptance process and where the scope definition and design are loose, a much more rigid change creation process must be adhered to, like change management systems from the past (CVS, VSS). Like applying Gantt charts and other staging tools to the change scheduling process. Like anything except the "stick a bunch of developers together with a bunch of code together in a blender and set it to frappe" process that is being applied now.
In this project (if it's actually a real project and not just another Slash-question carefully crafted to incite discourse), there is no viable command and control on the management end and there is none on the revision control end. Which leads to a project mired in anarchy as we see is the case. There are no tools that can help at this point. There is no substitute for a disciplined methodology. What was depicted in the question was completely unprofessional.
To answer your question, reverting to punch cards would be a step up for this project.
You are missing a quality engineering or quality assurance group -- a group that's independent of the developers and specializes in developing appropriate tests, test harnesses, test automation and also provides another set of eyes on features and bug fixes.
Our process:
Bug is reported or feature request is approved for developer to work on.
Development cycle:
Developer implements feature / fixes it. Ideally based on design of fix/feature, QE develops test plan/tests. A lot of work can happen concurrently. Some people even write the tests first, then code second.
Code review code and fixes with other knowledgeable developers (and QE)
Repeat this cycle till feature/fix and tests are ready.
Fix is submitted to integration system with appropriate tests.
QE (most definitely NOT the developer) verifies 1) software works 2) regression test actually ran. Ideally this is a completely separate QE from the one who did the submission, but this depends on resources.
Software enters customer release process (dev and qe largely forget about it and move onto something else)
Customer release process runs tests (or some representative subset of) against a consolidated build of "everything" to check for integration issues.
The golden path should be:
1) User reports a problem to the service desk.
2) Service desk looks into the problem and either addresses it if it's user error or punts it to QA/testing.
3) QA/testing investigates and documents as much as possible about the bug - replication steps, affected screens, whatever. They would do this both in production and a staging environment to see if it's an environmental issue.
4) Developer takes the bug and figures out the issue, creates a fix, which is then sent back to QA/testing.
5) QA/testing tests the fix in a staging environment and signs off on it.
6) Fix gets deployed to pre-prod staging, QA tests it.
7) Fix gets deployed to production, QA tests it.
I get that in smaller shops devs tend to just push stuff out (I worked in a few of those) but that's really not a great way to do things.
Since I can't tell them apart, I treat all ACs as the same person.
Hire bug testers?
The correct solution is to add the source code to systemd
individually on a staging environment. It would even be possible to fire up stage instances for each branch automatically.
I don't see a big problem here.
Let me get this straight. You can't figure out how Agile is the problem and In the clueless one? ROTFLMAO. You can't take shortcuts and get good results. Agile is an attempt to do exactly that. Agile proponents are exactly the kind of clueless developers that actually believe you can't have a gatekeeper because of the reasons stated, and are completely unaware that it is being done every day and had been done for more than 20 years in the most successful software project on the planet.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
"How would you stop the deployment of unapproved code changes?"
"Russia wants to send a gun-shooting robot to the ISS."
If I remember right, someone even sells an automated nerf gun that will target and fire a dart at a person who breaks the build with his latest commit.
I use git for non-distributed development, and it works better than any other source control system I've used (and I've used quite a few). Has its quirks, but so do they all.
I do miss $Id$, though.
-- sigs cause cancer.
Last company I worked for had a very empowered group called "Change Control". No fixes to production without their approval. Emergency fixes could go in but were vetted and tracked. It's overhead, but if you truly care about the production environment and are measured on downtime with penalties, it's the only way I know of.
At the company where I work, we review each other code changes. If you are using Jira (or similar bug tracking system) it easy to introduce a review step. If you use a code management system (and you should) you can do development in branches and have the reviewer (who preferable also performs a test by checking if the scenario mentioned in the bug report now works correctly) merges the branch to development if no problems are found. Otherwise the issue is reverted to open. Code review is a very powerful technique for finding bugs. It also helps understanding the code base, because developers tend to be drawn to certain parts of the code, often to the point that they are considered the owners. Code reviewing code written by others also helps you get better at developing software, because often you will find solutions that you had not thought about yourself. And because you review each others code, the learning goes in both directions. I sometimes spend more time on reviewing code than the fixer spend on solving the issue. Often I walk over to the developer to ask some questions, why he or she has chosen a certain solution. Sometimes, they give a good answers, sometimes it leads to a short discussion, and it even happens that the developer him/herself realizes that there is another case that will go wrong as well, and that the bug needs to solved in a different way. If you are using Scrum or similar development methods with short cycles/sprints, make code review part of your definition of done, meaning that an issue is not demonstrated as fixed (or included in a release) if it has not been code reviewed.
Make it mandatory that the test for the error be part of the submission, then the developer has the code to test for the error/bug/problem etc. Yeah it might get problematic, but better to have the error actually checked than not.
You sound old. Maybe it's time to retire.
Problem solved
Most developers respond strongly to physical pain.
“Common sense is not so common.” — Voltaire
> 552 Fuck off -- the requested action was requested by a dweeb
Obvious much?
Each "change" (whether it is a fix or new feature) can be developed within its' own branch.
This branch is then deployed to one of your UAT environments for user testing (once it has passed QA, static analysis, code review etc.)
When the time comes for a release, you "roll up" your approved feature branches by merging them into a single release branch. On this branch you carry out extensive QA and regression testing. This is then the branch which is released (deployed).
Using this method any changes which have not been APPROVED as a feature branch never make it into your final deployment package. (though they still exist and can easily be added in future by merging).
(note if you have limited number of UAT environments this becomes complex and needs a schedule and conflicts can occur - or proper devops and more environments).
You make it so there is only one path to production. Use CI/CD with build promotion based on criteria. Don't give anyone the ability to install production code directly. You install code, configuration, and settings the same way -- though source control
If you have a process like this, it's simple and nothing gets into production that should not. This goes for configuration changes too.
- Vincit qui patitur.
So I did, despite the code changes being untested as the key tester was in India getting engaged. It broke production and I had to figure out how to "backtrack" the codebase before the last set of changes. Unfortunately, the release process had no 'revert to last working version'. I stayed late to get production back up sans the changes my boss wanted (because a customer was waiting for them). The code still didn't work until the weekend and my boss wanted me to push it then. But as a contractor, I don't work weekends, so I said no. Pushing code changes on weekends is also a Bad Idea(tm), IMHO.
Then I had a discussion with him about this mistake and new requirements for me to 'stay until it's fixed' because now we have paying customers instead of demos. I could have told my boss "You made this mess. You fix it.", packed my desk, and left, but I decided to stick around and be professional about it. I lasted a week until they had another emergency and "ended the contract". In my case, their needs changed from a build engineer to a customer support role, which I couldn't do.
How to engineer around this? Be ready to roll back changes, which we couldn't do easily. Have a Change Control policy for each scheduled outage and be ready with a contingency plan to revert to the previous version. Murphy's Law will always win.
Companies that don't do testing and code review spend 80% of their time running around putting out fires. There are other fires they don't even know about yet.
Companies who do code review spend 10% of their time on that, and their developers stressed out all the time trying to put out fires.
Doing it right is much cheaper than doing it over.
> a lot of risks are eliminated or realized before things go out the door; in practice, this means a fairly conservative and slow development process.
At one place I worked, we were very fast at getting new requests developed and out the door - because we weren't busy fixing the code we shipped six months earlier, and the code we shipped last week. So it may *look* slow if you watch a careful process in action from minute to minute, but it's much faster on a macro scale - features get released sooner when developers aren't running around putting out fires that they themselves kindled.
Face punch
Whole hand? Have fun with that huge cage. That's not good in the case you didn't know.
Before code is released to production, you need to have it properly tested. The tester must not be the same person as the one who modified the code as he simply is too likely to be blind to flaws in his code. Depending on what code was modified (complexity, where it resides, etc), you may need to have it reviewed by one or more area experts. Note that the review must not consist of the developer stopping by their desks and asking "could you take a quick look at this". Doing that guarantees that they will quickly read the code and say "that looks okay" without really considering the potential effects of the change.
There is no substitute for properly reviewing and testing code. If management is not willing to support that consistently, or is willing to skip it when it's inconvenient, there is no way to fix it. You can either live with the consequences or go work somewhere else.
Oh my, NO ONE has thought about the problem of un-approved code changes! Stop the presses! We need a Scrum meeting Right Now to address this emergency issue!
The OP does not have a technology problem, they have a political problem. If the OP were in charge of this problem (clearly they are not) then they would be able to fix it. Since the OP isn't in charge, they can try and convince those in charge that there is a problem. If Those In Charge do not agree then the OP needs to document the result and move on in life.
Is there a day that goes by that I'm not faced with a political problem, that I wished had a technological solution? Or even a political solution? Welcome to the real world.
The product owner is simply an individual responsible for the product.
Users report bugs to the product owner, not the dev.
The product owner is then responsible for any requirements updates, scheduling of developer(s), test(ers) and whatever else might be needed.
The only way to stay sane in an insane world, is to be mad yourself...
There were these things called software design engineers in test (a.k.a. QA) and they were an integral part of the SDLC, which went something like this:
1. Bug reported
2. Bug triaged
3. If triage-approved for the current release cycle, bug investigated, fixed, and unit tested
4. Bug-fix tested in local-builds by QA, including localized regression testing
5. Business/product owner approval sought, if needed
6. All bugs triage-approved for the release cycle integration tested by QA (including full system/product-wide regression testing)
7. Release milestone criteria scrutinized and, if met...
8. Release deployed
But, that was back in the dark ages. Now, in the age of enlightenment, we don't need testers or release cycles. Just fling the crap against the wall. If it sticks, it sticks. If it don't just roll it back.
Oh, and get off my lawn.
This was solved at least in the 1980s when we used to do it. The solution is to freeze the code. Pick a date and at that point the source code is frozen. Move it to a test machine if you have to, however no new changes are allowed. Build it and test it. If it's good (and therefore approved), release it.
I think we used to use sccs, however any revision control system will do. All of us developers had our own tree. In fact, the entire tree, linked file by file,dir by dir. This was back in the days when the largest drive you could get was a couple of gig, maybe not even that. I could check out my own version of code, update it and compile it all into my version of that program for testing. There were maintenance commands to resync all the links and such to stock, update the real source tree and such. Every night a new version would build. This was world wide.
If your asking about just some guy changing code on his own, a rogue guy, that's a daily management chore for someone (that isn't responsible for coding, a separation of duties). You'll have to rsync everything to someplace you trust to be the authority. Someplace the other people can't get to because some people will play games. Then check it all for a status to see what's changed. Do you have approvals for all the changes? If not, roll it back, find the offender and have a little talk with them about that. If everything is approved, check it all in. Make sure all of this is kept track of every day. I do this all the time. I had a case a couple of years ago that a manager swore his machine was set a certain way. I had subversion control over it. I told him how the machine was set up from day one to today. He was right, it was set up the way he thought years ago. Then about 2 weeks later he changed it to the way it is now. I had the change, date, even his e-mail and ticket to change it. So it was approved and he knew about it. He thanked me for my efforts and moved on. You have to be able to do this. Developers are great people, very creative, smart, etc. However sometimes they can be real assholes. Especially young ones that never make any mistakes and it could be years later and they don't remember. Once you show them, they're fine. No problem.
Good luck.