Getting Development Group To Adopt New Practices?

maiden_taiwan asks: "At my software company, we occasionally need all engineers to adopt a new standard or 'best practice.' Some are small, like the use of Camel Case for function names, while others have tangible business value, such as 'every check-in must be accompanied by a unit test.' As you might guess, some new practices get ignored, not because people are evil or lazy, but because they're simply too busy to pay attention and change their work habits. So we are seeking creative ways to announce, roll out, and enforce a standard for 100+ engineers so they will actually follow it." What ways have you used to convince your developers and engineers to adopt a new set of practices that may or may not get in the way of their daily work habits? We already know to automate compliance when possible (e.g., the revision control system could reject check-ins without unit tests), and simple platitudes like 'tie compliance to their year-end bonuses' aren't helpful by themselves, as someone will still need to check compliance. The engineers here are smart people, so we want to spend less time on enforcement (having architects read the code and flag any non-standard practices) and more on evangelization (getting engineers to see the benefits of the standards and -want- to follow them). I'd welcome any advice on formal processes or just plain fun ways to get people's attention."

Comments

You don't ship test code

[BadAnalogyGuy]

One thing that really needs to be understood by all you "best practice" guys is that test code is not a shippable product. Requiring that all checkins be accompanied by unit test code is ridiculous because two developers working on the same code will need to update not only the code itself but also any test cases that rely on the behavior of the executing code. And if code A is supported by test code X, Y, Z, are you also going to require that any changes to A also be accompanied by changes to X, Y, Z? What happens if A is some fundamental architectural change (or maybe simply a refactoring) that affects all tests in the test suite? You can't seriously be talking about forcing the developer to go through the entire test suite looking for compilation errors and runtime errors just because those early tests don't make sense anymore with the new code.

You need developer buy-in to make a largescale standards effort work. You can do that by culling those developers who are realistic about development practices and by augmenting the remaining programmers with new hires who are just as standards-oriented as you are. Other than that, you'll be facing an uphill battle that will come to its conclusion the first time you approach a project milestone that is significantly behind schedule.

Do yourself a favor and get some test developers and testers. Let them worry about the test suite and let your developers worry about the product.

Re:You don't ship test code

[kfg]

This Ask Slashdot question just cries out for its companion question:

How do you get management not to implement a new standard or 'best practice.'

KFG

Re:You don't ship test code

[BadAnalogyGuy]

Lots of "clean up our practice" Ask Slashdots lately.

Here's another one that seemed a little bit hopeless: http://ask.slashdot.org/article.pl?sid=06/11/10/04 1257

I have a feeling that as word gets out that Microsoft is getting deep into the whole Agile phenomenon and using it as much as possible throughout its teams, that other companies want to emulate the dynamicism for themselves. However, what works for one company (especially one focused on building consumer applications) may not be a realistic or useful methodology for another company (like one trying to implement a system based on pre-packaged components).

That's not to say that Agile or XP or DP or Booch or Rumbaugh or Rational or Waterfall or any of the myriad development methodologies aren't useful. They are imminently useful *within their useful domains*. Once you start trying to apply a square methodology to a round development team, you're bound to run into problems. If you are deadset on implementing a set of standards, make sure you understand the methodology completely first, otherwise you'll end up with a half-assed implementation of piecemeal tactics without a comprehensive strategy.

Just look at Iraq for an onerous example of a project with capable team members and highly effective tactics led by incompetent managers with no overall strategy. You don't want to build that sort of company.

Re:You don't ship test code

[Osty]

One thing that really needs to be understood by all you "best practice" guys is that test code is not a shippable product.

Code without tests is not shippable code!

And if code A is supported by test code X, Y, Z, are you also going to require that any changes to A also be accompanied by changes to X, Y, Z? What happens if A is some fundamental architectural change (or maybe simply a refactoring) that affects all tests in the test suite? You can't seriously be talking about forcing the developer to go through the entire test suite looking for compilation errors and runtime errors just because those early tests don't make sense anymore with the new code.

You can, and you should. If your build is not organized enough that each developer can do a full build (or at least a build of modified components) prior to checkin then you have some work to do. As for architectural changes and refactorings, think of this as a small barrier to entry. It prevents frivalous re-architecting when your developers should be getting on with their real work. If a new architecture or a refactoring is important enough, then it's also important enough to fix or deprecate the test suite.

Do yourself a favor and get some test developers and testers. Let them worry about the test suite and let your developers worry about the product.

I agree with the idea, but not the sentiment. Testers and test developers are there ensure QA. They have their own job to do, and it's not mopping up after lazy developers who can't be bothered that they broke a build or checked in non-functional code because they didn't unit test. Bear in mind that the advocated "developer testing" really is minimum-bar stuff -- does your code work for the mainline scenarios? Maybe you have an obscure boundary condition or off-by-one error that you missed, and that's okay. What's not okay is just checking in a bunch of code and throwing it over the wall to the test department. Your quality will suffer and your testers will hate you. An adversarial test-dev relationship benefits nobody.

Re:You don't ship test code

[BadAnalogyGuy]

How fast does your codebase evolve? My money is on a glacial changerate.

As for "dumping" messes on others, I find it laughable that anyone would think that a developer whose mind is so geared towards developing a certain section of code would be able to objectively analyze and test their own code, much less write a test suite to cover the new code effectively. This is the same as asking an author to edit his own book. Sure, he may find the occasional misspelling or figure a better way to reword a chapter, but on the whole the work of editing is best left to editors.

The work of testing is best left to testers. An intermediate 'clear box' test team focused on the development and maintenance of unit tests is far more effective than the traditional dev/test organization that is so prevalent. By getting the development of tough tests out of the hands of the code creators, the tests are more likely to be fiercely objective and less prone to leniency.

As for creating a feedback loop, a developer who consistently puts out good code will consistently pass the unit tests and get to work on new code, but one that consistently puts out bad code will forever be stuck in the code-fix-code-fix cycle. If you were to leave the developers to their own devices, the first one would spend an inordinate amount of time writing good unit tests while the second one would write a test that was designed to pass. Or you'd have both writing the bare minimum unit tests which is worse than useless because you'd come to the incorrect assumption that the test was reaching some level of minimum quality when in reality the code is no less buggy than usual.

I don't doubt that unit tests are a good idea. I question whether it is a good idea to have the developers themselves write them, and whether it is a good idea to force the development of tests as a prerequisite to checking in code. Both sound great on paper, but reality shows that unless management is ready to step on the necks of the development team, that the practices get thrown out the window when projects hit the death march stage.

Re:You don't ship test code

[BadAnalogyGuy]

Code without tests is not shippable code!

Right on. However should the ultimate goal of a project be to develop a great test suite? Maybe so, if your product is the test suite.

Tests are necessary and automated tests are spiffy and should be part of any comprehensive testing effort. Having the developers create them themselves seems to be a waste of time. My other post to the AC whose code is flawless addresses this issue.

If your build is not organized enough that each developer can do a...build of modified components...prior to checkin then you have some work to do.

Amen. However compiling code is hardly the same thing as writing a test case. Clean builds should absolutely be a prerequisite to code checkin, no question. However, is that same sort of rigor necessary for behavior tests? Most developers don't code up something, build it, then toss it over the wall. At a minimum, the developer will typically put the new code through a few paces before sending it to test, so there is definitely some testing going on at the development level. Is more than that needed? Yes, sez you and many Agile enthusiasts. Personally, I don't see it.

I see a lot of time and effort going in to writing tests that will pass the new code, not tests that will fail it. Developers have a funny faculty of seeing their code through rose-tinted glasses, and their unit tests reflect that. Just because you have 100% of the codepaths covered by the unit tests doesn't mean that anything except for program flow through those codepaths is tested.

Testers and test developers are there ensure QA.

Yes, absolutely. This is why I think that there ought to be another team attached to the developers who have full access to the code whose responsibility it is to develop and maintain the unit tests. When a checkin breaks the unit tests, the developer responsible must either address the problem as a bug (ideally, the unit test team would be working from interface specs, but should be adaptable) or the unit test developer must update the test suite to accomodate the new feature.

IMO, the test team should be made up of people who are capable of programming, so that they are empowered to be test developers (SDETs) themselves. Too many times I've worked with test teams made up of people who took a community college Intro To Computers class and sat around all day pressing buttons and writing down results for tests created by the SDETs. What a waste! If you need button pushers, hire them as temporary contract workers, but get your core test team to be proactive in creating a solid test framework that can be automated as much as possible so that they don't themselves end up sitting around pushing buttons.

lazy developers who can't be bothered that they broke a build or checked in non-functional code because they didn't unit test.

Broken builds happen way too often to say 'It should never happen', but it should never happen. I'm surprised that uncompilable code could possibly enter the codebase, but I suppose those sorts of developers exist too. Weed them out of your company as soon as you can, is my only advice in that case. The occasional 'checked in all files but one' type of build break is bound to happen, though good source management tools can help alleviate this issue.

A good compiler with warnings or a tool like lint can make dead code easy to spot. If your strategy includes a restriction that all code must be warning-free, that's a very minimal requirement and one that is easily adapted to. If you say instead, "unit test all codepaths," then your developer is suddenly facing not only the bugfix of the original bad code (which would have been found by lint) but also the development of new code whose sole purpose is to exercise the bad code. It's a waste of effort and the developer will try to get away with the minimum amount of effort possible, which leaves you with the illusion of tested code but the reality of bad code + bad test code.

Bear in mind t

Re:You don't ship test code

[Coryoth]

Requiring that all checkins be accompanied by unit test code is ridiculous because two developers working on the same code will need to update not only the code itself but also any test cases that rely on the behavior of the executing code.

The solution to that, of course, is to integrate test definitions into the code itself so that it all gets updated together. As a bonus your API documentation is more precise and gets updated along with the tests and the code. You can then push a button, walk away, and have an automated testing system fully exercise your code. And yes you can have all of that for Java if Eiffel isn't to your taste (though it might be worth having a second look at Eiffel).

In other words, yes if you want software with lower defect rates you should expect developers to update their specification (and documentation) of how they intend code to be used and to work at the same time that they update the actual code. If you want to specify working code by having it pass a separately written unit test, fine; but that's not the only way, nor is it always the most efficient.

Re:You don't ship test code

[chromatic]

Having the developers create them themselves seems to be a waste of time.

In my experience, any developer worthy of the title "developer" already tests his code, though perhaps not in an automated fashion. I've seen plenty of developers (and I've done this myself) use debug statements to build up a function or method in pieces, checking the output manually each time.

Actual test-driven development (not after-the-fact writing a few half-hearted test cases) is a refinement of that process. It has three benefits. First, it captures those manual tests in a permanent format that I can run any time I want. Second, it gives immediate feedback as to the state of the code and allows me to work in much smaller increments of time. Third, it changes the way I approach code and low-level design. My code tends to be simpler and shorter and more cohesive and less coupled (and much, much easier to test) this way.

Having a poor developer create dozens of lousy tests does no one any good. However, my experience with doing TDD myself (and teaching other people how to do it and writing libraries to help them do it) makes me think that any decent developer who tries to do it seriously and with discipline will produce better code.

Re:You don't ship test code

[BadAnalogyGuy]

fifteen months for a decent mid-sized project.

I envy your team. You have a great PM who is able to pad a schedule quite ably. (Seriously, over a year? Where do you work?)

if they're such crummy developers that they are truly unable to determine which lines of code that a given unit test exercises, there are these nifty new tools to do code coverage analysis for them.

The point was that developers have a different view of their code than objective outsiders. They will not create as good of tests as someone not intimately associated with the code.

No, this is the same as reminding the carpenter to "measure twice, cut once". Most developers doen't even bother to measure once; they just saw off a chunk by eye and ask somebody else if it's the right length.

I don't think you'll find anyone is arguing that developers should be cowboys and just spit out code all day long. I'm certainly not advocating such a stupid thing.

What I am saying is that development time is finite and asking the developers to write unit tests is a waste of time since 1) it takes up actual time and 2) they have a vested interest in successful unit tests.

UNIT TESTING HAS NOTHING TO DO WITH QA

And if you had read past the first sentence you would have understood that I am not saying that QA should be writing unit tests but that the development team should have its own set of unit test developers who work in parallel with the code developers towards the goals of the specification. QA (the separate testing team) has nothing to do with what I am talking about.

Even if the QA at most shops weren't little better than monkeys, wasting your best testers on tracking down simple "Oops, oh yeah I forgot that case" bugs instead of actually doing what they do best is an idiotic waste of money

I am very disappointed to see this attitude. While I do agree that many QA departments are staffed badly with people who are ill-qualified, I find the attitude that they are "little better than monkeys" atrociously condescending. It is the persistence of this attitude that drives a wedge between dev and QA and makes many QA engineers view development as a step up rather than a step sideways. Some of the best engineers I have ever worked with have been QA engineers.

Unit testing code is similar enough to testing that repurposing a handful of development-oriented QA engineers for unit test development would allow extra breathing room for the code developers and the increased code quality would see to it that the QA cycle went smoothly. Are you going to argue that better unit tests are going to be worse for the product cycle?

Let the person who knows their code best (or one of his/her peers) double-check their own work.

I am in agreement about code reviews and informal walkthroughs before checkin. When done well, they have been more effective than any other development technique at sniffing out coding mistakes in my experience.

I do not agree that the person who wrote the code is the best person to judge the fitness of their own code, even if they take serious pains to double check their work well. It is because they have the code fresh in their mind that they cannot see the mistakes. It is because their ego is intimately tied to the code that they look for success rather than failure. An objective outsider (whether it be a peer developer or someone else) will be far more qualified to judge the fitness of the code than the coder himself.

We aren't talking about sawing pieces of wood. We're talking about the design and implementation of complicated machines that require intense focus and attention to detail. It forces the developers into a certain pattern of thinking, potentially blinding him to his own errors, and the difficulty and rewards of success bind the programmer's ego to the code such that he cannot judge its fitness objectively.

Re:You don't ship test code

[Stradivarius]

I would suggest that the original developer is the best person to develop the tests, because thinking about how you're going to test your code often exposes flaws in your design, interface, or implementation. Better for the developer to realize that early in his process, than after the fact when some QA or "test developer" comes along and says something doesn't work quite right. It's almost always faster/cheaper/more efficient to fix a problem early on in the lifecycle than at the end.

Besides, the developer is typically going to do *some* sort of unit testing to convince himself that his code works. It's often not a huge effort to formalize that already-planned testing into a repeatable/automated unit test.

You're right that there is the rose-tinted glasses effect, but you can overcome that by including unit test code as part of peer review that happens on the "regular" code. (This obviously assumes the company does some form of peer review; if they don't then who is picked to write unit tests is the least of their problems).

What I'd Do

[hahafaha]

> What ways have you used to convince your developers and engineers to adopt a new set of practices that may or may > not get in the way of their daily work habits?

Adopt the new practicer. Or you're fired.

Evangalize

[ubergnome]

It's been my experience that most good developers will adopt a standard practice because a) it makes sense and/or b) the new standard doesn't require any more effort than the old way, so it doesn't hurt to play the game.

So, make it easy to adopt and make an effort to educate.

Remember, though, that this is a two-way street; if it's hard to adopt or hard to argue for, then maybe management/architects should rethink their reasons for requiring the standard.

Also -- a training program might be a good carrot to help get the more junior devs onboard (at least with more complex standards like unit testing or patterns-based development).

Beat 'em over the head

[AKAImBatman]

At one of the jobs I worked at, we had a fellow who's sole role was to maintain the Version Control system, and manage the releases directly from that system. He was incredibly good at his job, to the point of politely beating the matter out of programmers who didn't comply. So if you just happened to forget to tag something for release (or otherwise tagged something that shouldn't have gone), he'd be over to let you know that you broke the build AND (here's the important part) work with you to get it resolved.

Honestly, having the guy around was the best thing that ever happened to our code tree. Suddenly, we developers didn't have to worry about handling all the minutia related to a test or production build, we didn't have to worry about pruning the tree, and we knew someone was watching our backs in case we screwed up. I know that my description probably sounds horrible, but it was honestly great! The whole process got a lot smoother after he came on board.

I think the key reason why it worked was because most developers wanted to follow good version control procedures; they just didn't have the spare bandwidth to manage it. By centralizing the handling, it offloaded a great deal of that duty and made everyone's lives easier. It also made clear the people who were intentionally keeping source control a few versions behind for "job security". :-/

Performance tracking with bonuses

[KU_Fletch]

At my software company, we have yearly bonuses and profit sharing. Both can be affected by what we call a "quality factor" or as some of our programmers deride it "quantitative quality." Basically, we have some set in stone rules set by our leads when it comes to workflow issues. For example, software compiles are scheduled every night at 8 (and a second at midnight if needed). Your work needs to be checked in to our tracking software by 8. If not, the compile will break. If you break a compile, your quality factor score goes down. If you remember and check your work back in by the midnight backup compile, your score ticks back up slightly. There are about a dozen of these rules, and they all center around interrupting other people's workflow. If you screw up a nightly compile, it means when people come to work in the morning, a new compile has to be kicked off and everybody has to wait. Every person whose workflow is interrupted gets a share of your quality factor points you lost. So at the end of the year, if you never screwed up, you get more money in your bonus. If you were constantly screwing up and making other people lose productivity, then you get less money and they get more to make up for your screw ups. If you screw up a normally expected amount of times, it ends up in a wash.

Before you ask, yes, there are certain people that hate that rule. But those people tend to be the ones losing out on money, and the ones screwing up a lot. Screwing up once or twice is no big deal. Screwing up on a weekly basis hurts you in the long urn, but it's really their fault for not correcting their problem. No, the entire bonus and profit sharing is not based around it, but it is a good chunk. One guy I shared an office with last year lost nearly $2000 out of his bonus. That ended up giving everybody about $50-75 more. Sure, that's not much of a gain for certain people, but it's a nice chunk of money to treat your wife/girlfriend to a nice holiday dinner at the end of the year. The important part is the loss hurts you big and requires that you stick to good practices for a long time to make it up.

Maybe involve them?

[seebs]

One thing that might help would be to involve them in the decision-making process. A lot of the "best practices" I've seen handed down were monumentally stupid ivory-tower junk.

Machiavelli

[Harmonious Botch]

Machiavelli said something to the effect that the prince should dole out the sweets in small ammounts on a regular basis but put all the bad news in one package and get it over quickly. ( Sorry. I don't have the time to look up the exact quote )

People are creatures of habit. They don't like changes. Every change that you make in their working environment is a small negative experience to them. Even if it is something that they might agree is a good thing, it is still a change, and thus a negative thing to them.
This, unfortunately, conflicts with the desires of a good manager, who wants to make improvements in his business every day. Even if they are GOOD changes, making changes every day is a sure way to alienate your employees.

So, be patient. Only make frequent changes if an emergency requires it. Otherwise, make a list of planned improvements, and keep it to yourself. Add to it weekly, or even daily. Then, once or twice a year, implement it all at once.
It sounds contradictory, but people will adjust to a bunch of changes better than a few, IF they know that the rest of the time their work will be relatively consistent. You can ask them to be in an absorb-the-changes mode once in a while, but not all the time.

Make sure the practices help rather than unhelp

[mrjb]

The main problem I see with adapting new practices is that project management relies on discipline as a replacement for motivation. This Does Not Work. Discipline alone is not enough- This is why for instance source control systems have been invented.

At some point at our company we started 'crosschecking'- which essentially means all developers have their code tested by another developer, mostly before things got to nightly build. (Other quality assurance practices are also already in place).

The result was twofold. First of all, incorrect code was communicated back the same day, or even several times the same day. This saved a huge amount of time in detecting problems. Second, because each developer knows their code is going to be reviewed by another developer, they try harder to avoid the bugs in the first place.

The first release after we introduced this practice, we noticed that the bugs being reported by the client mostly had to do with previous releases. The bug rate has steadily decreased since, and we're not constantly patching holes anymore. We catch most bugs before we deliver the product to the client- so the client is happier too. We spend more time doing 'fun' stuff now, rather than bug hunting. And it is quite an ego boost to ship a product and not hear of any defects. Would we stop crosschecking, we would have the feeling that we are shipping an inferior product.

By making sure the practices to adopt are insanely useful, time savers and make the work more fun, developers will adopt them without complaining.

Change Adoption

[KDan]

One of the most effective techniques I've used to encourage adoption of change is to get the people who will need to change to own the new process. How do you do that? You heavily involve them in designing the process, expose them to the reasons why the process is good, get them engaged in discussions about how to resolve the problems that you're trying to resolve. Sure, you're bright and everything and you know the solution already, but that doesn't mean everyone will do as you say.

Best way to do this is for you to create an imperfect solution then have one-on-ones with all the key stakeholders and get them to contribute to it. Expose them to the business requirements (expose them to the business if need be) so that they understand perfectly well why this is happening, and get them to own this process and thus commit to it. Then once everyone's agreed that this is the way to go, you can set up some sort of regular measurements to track the adoption of the processes. Make those measurements visible and the key stakeholders will get these processes adopted by their teams.

Daniel

Test-driven development

[Anonymous Brave Guy]

Let me be clear from the start that I have nothing against automated testing, nor against unit testing at a low level, nor against combining these two ideas.

However, I think it's important not to place inappropriate faith in them. I have heard senior programmers recite the mantra that you can refactor your code as much as you like, because as long as it still passes the unit tests, you know you didn't break anything. I look at those people and wonder how they got to those senior positions. Do they really believe that you can ever have an exhaustive set of unit tests, no matter what the context? I have seen cases where rewriting a pretty trivial arithmetic expression in a tidier form worked fine on one machine, but completely broke the optimiser with a different compiler on a different OS, resulting in a subtle change in numerical outputs.

IME the hard-core TDD advocates also have a false sense of security when it comes to design. Just because you have some tests in place, that does not mean you can design on-the-fly all the time, and suffer no overheads. There is a reason that good software engineers spend a lot of time designing and not much time writing code: it's because writing the code (and getting it right) is much easier with a clean architecture that's designed to be flexible and maintainable. Of course you can't plan everything up-front, but the whole "anything's OK as long as it passes our unit tests" approach is a recipe for endless headaches when your design starts to evolve, and can lead to weeks or months of wasted time later when a new feature can't just fit into the existing code because there's simply nowhere to fit it.