Slashdot Mirror
Ask Slashdot: How Would You Stop The Deployment Of Unapproved Code Changes?
Over a million lines of code -- in existence for over 10 years -- gets updates in six-week "sprints" using source control and bug-tracking systems. But now an anonymous reader writes:
In theory users report bugs, the developers "fix" the bugs, the users test and accept the fix, and finally the "fix" gets released to production as part of a larger change-set. In practice, the bug is reported, the developers implement "a fix", no one else tests it (except for the developer(s) ), and the "fix" gets released with the larger code change set, to production.
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
We (the developers) don't want to release "fixes" that users haven't accepted, but the code changes often include changes at all levels of the stack (database, DOAs, Business Rules, Webservices and multiple front-ends). Multiple code changes could be occurring in the same areas of code by different developers at the same time, making merges of branches very complex and error prone. Many fingers are in the same pie. Our team size, structure and locations prevent having a single gatekeeper for code check-ins... What tools and procedures do you use to prevent un-approved fixes from being deployed to production as part of the larger code change sets?
Fixes are included in a test build for users to test and accept -- but what if they never do? Leave your best answers in the comments. How woud you stop un-approved code changes from being deployed?
That right there is a part of the problem. Software testers report bugs. Hints about potential bugs can come from end users, but end users are not software testers.
There is no substitute to a really good (professional.... and paid) software tester that and reliably reproduce bugs that need to be removed. If anything, they are far more valuable than even code monkeys writing thousands of lines of code per month (something also largely irrelevant for quality software). If anything, I would pay the software testing team before the coders if you need to use some volunteer labor like in an open source project.
End users can offer hints to a good software testing team as to what might be bugs, and end user reports should definitely be taken seriously since it is something that slipped past those testers as well. When the software testers are fired and/or it is presumed that unpaid volunteers are going to be doing that quality assurance process, especially for a commercial software product, you get what you pay for.
Yes. In addition to 100% unit code coverage and integration tests.
I take it you don't work in the industry (called "Silicon Valley" by outsiders). Here, a senior developer is usually 21 or 22 years old, and has been with the company for about 2 months on average. Some of the more experienced senior developers might be 24 years old, and might have even been with the company for up to 4 months.
... hides
http://saveie6.com/
"That doesn't work"
It doesn't not work. More eyes do tend to catch out and out bugs. But some still slip through. And stuff that sounds good but busts user's workflow still gets through. And developers hate the delay. And more review costs more. And more testing costs even more than more review. So managers aren't a big fan of either.
I suppose the solution is never to work with millions of lines of code. But "This sucker is way to big and too complicated" is not an easy sell.
You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
Linus Torvalds disagrees. Your real problem is that you use Agile. Your process is broken by design. Study how the real professionals do it (Linux kernel) and you will learn a great deal.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
It's an old saying that a doctor who treats himself has fool for a patient and an ass for a physician.
Yet this is precisely the way many IT shops treat testing.
One of the biggest problems with this approach is that the developer "knows" where the weak spots are and test them, insofar as the schedule allows any real testing at all. An independent tester is not as prone to this sort of tunnel vision, especially when the tester isn't looking at code, but instead at the way the code works. Which, is after all, what the code is ultimately for.
A second problem is that characteristics that make a good software tester are not necessarily those of a good software developer. A good tester has to be the sort of meticulous person who can go over items line-by-line over and over again and never take shortcuts. A good developer may be a good developer precisely because he/she can leap around the concepts and tie together seemingly unrelated points.
Then there's the third problem, which is that contrary to whatever non-Euclidean world Management lives in, you cannot dump the jobs of developer and tester on the same person and rationally expect that they can inflate to handle both requirements optimally, Real employees have limits. Not that that matters when "right-sizing" the corporate personnel assets for the next quarter's executive bonuses.