Is Process Killing the Software Industry?

blackbearnh writes "We all know by now that Test Driven Development is a best practice. And so is having 100% of your code reviewed. And 70% unit test coverage. And keeping your CCN complexity numbers below 20. And doing pre-sprint grooming of stories. And a hundred other industry 'best practices' that in isolation seem like a great idea. But at the end of the day, how much time does it leave for developers to be innovative and creative? A piece on O'Reilly Radar argues that excessive process in software development is sucking the life out of passionate developers, all in the name of making sure that 'good code' gets written. 'The underlying feedback loop making this progressively worse is that passionate programmers write great code, but process kills passion. Disaffected programmers write poor code, and poor code makes management add more process in an attempt to "make" their programmers write good code. That just makes morale worse, and so on.'"

Comments

"Creative"

[Anonymous Coward]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

Re:"Creative"

[Entrope]

This is dead on. Every software-developing business needs to decide its own process needs. Even if they are developing safety-critical code that has to pass rigorous certifications (FDA, DO-178B, etc), sane people do not order everything on the process menu. However, an organization does have to look at its market, its code base, its structure/culture, and its history to figure out what kinds of process it should use. Sure, having a defined process means developers spend less time throwing code at the wall, but the code they do throw usually sticks to the wall better and longer, and they usually feel good about that.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

Re:"Creative"

[heathen_01]

This is dead on. Every software-developing business needs to decide its own process needs.

This is just another cargo cult problem. Managers, seeing that another company/team use process X on a successful project, decide to implement the same process for their team. However no process will make a difference if the developers have been directed to build the wrong solution in the first place, even if the code is 100% bug free.

Re:"Creative"

[RyuuzakiTetsuya]

My thoughts exactly.

I worked basically in the skunkworks for an ISP's call center. We developed the CMS that handled all of our policy and procedure documentation, we developed neat tools that interacted with our IVR for outages, displayed coverage maps with said outage data, and various little tools here and there.

When we started, we weren't bound by our IT department's change management process. As our tools grew and as their usefulness permeated our business, we started to get noticed and before we knew it, we were being held to the same standards for process as teams who developed tools like our CRM and our internal ticketing system.

The whole point of our skunkworks was to be nimble, efficient, and effective on smaller projects those teams didn't have the ability to take on due to commitment to other projects. Yet, as IT's demands for compliance grew, all of our advantages went away. Soon, i found myself having to answer, "Why is late?" and invariably the answer would be, "Oh, it's held up by IT's Process. It'll be three weeks." Managers weren't happy. I REALLY wasn't happy and stressed and my user base wasn't happy either(Although my user base had a certain degree more sympathy than management did).

It sucked the life and effectiveness out of our team.

When layoff time came a month ago, on paper, we looked like were banging rocks together, and bunch of us were laid off, with those teams that didn't have the time or resources to build these tools in the first place having to absorb yet more work.

(note: I didn't mind the particular IT policies, they made sense, but, I felt like a sense of scale was lost when it came to our projects. Oh well.)

Re:"Creative"

[Omnifarious]

I rated you up, but I notice that most people seem to be missing the most important part of your post.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

I agree that software development can be 'over-processed'. And I think that management frequently decides to 'apply more process!' in lieu of actually carefully thinking through what would help. But usually, when developers complain about process and delivering things on time, I think "Oh, so you just want to slap that code in there without even really knowing if it works. Write code, compile, problem solved!". Good process is what keeps you on-track and disciplined to write quality software. Too much software in our industry is of extremely poor quality, and that's still the case. Developers should feel proud of writing software that works, not how much software they write.

Most of the responses I see to your post seem to be of the "You're darn right, all that process just kills development!" variety. No talking about how to balance things for quality results or anything of that nature. I'm kind of disappointed.

Re:"Creative"

[Archangel Michael]

Was gonna mod you up, but instead, it hit me. The problem isn't the process, it is where the process is engaged. And I think the Article and subsequent posts clearly indicate there is a time and place for process, but not everything needs to be stuck in a process.

In your case, you and your team were building tools for your own use and were doing it the quick and agile way. Dirty code that got stuff done quickly. You bypassed the systemic processes that were in place because they just got in the way.

The successful code base often starts out as UGLY spaghetti code, but eventually needs to be cleaned up.

What needs happen is your skunkworks program should be autonomous from the normal process. Only when something you've created gets noticed does it move from the skunkworks over to the normal development channel. To accomplish this, you have to be willing to hand off your code to someone who is, more than likely, going to "ruin" it by running it through the process of getting it clean and neat.

This way, you can still build fast efficient code that is messy and ugly, and the managment can get code that is functional, and everyone can work towards getting it all nice and pretty by process. Everyone wins.

Just my thought

Re:"Creative"

[Skapare]

And the certifications don't actually check any code, they just check that the processes chosen are the right ones, and can even miss if the processes don't get adhered to. Perfectly developed and certified software can still have a ton of bugs that will crash that 777 or shut off the heart monitor in a hospital.

Less on process and more on coder competency would help. Pick the best coders and double their pay.

Re:"Creative"

[lonelytrail]

I feel sorry for all you guys. I've worked in "no process" companies before and it sucks. No requirements, no process, no cohesion. Politics were rampant. It was just plain elementary.

These days I work in a company that's been CMMI level 3 certified for years and we have a process in place for each level of necessity; light, medium, heavy, manned-flight. All of our requirements are part of the contract, before any design even begins, and if they change the customer must pay for the change. It's a joy and my creativity is unleashed, not restrained.

Process is not your enemy. It's the lack of buyin. Everyone must participate and ensure the process is held. Requirements definition is part of process and if you don't have stable requirements, you aint got shit.

Again, I love working within our process, whether it's the light one or the manned-flight one. It's well defined and provides a schedule, but doesn't tell me how to write my design/code/test, just that I get it reviewed at all of the proper phases.

The one process to rule them all

[Tribaal_ch]

Karma be damned, this is relevant to TFA:

This is why I left development

[tripleevenfall]

This is why I left development. After 5 years working at a software company I found that only 10% of my time or so was spent writing new code, which is the only thing I really liked about the job. The rest of the time was spent in meetings, wading through red tape, reviewing others' code, doing maintenance on the (junkpile) legacy system, doing remedial training for the front-line tech support staff, and the 5 million small tasks that have nothing to do with why I went into the career field.

Re:This is why I left development

[mevets]

I used to deplore meetings; listening to some preening jackass and thinking about how far we were getting behind by this. It isn't just the time sitting in the room, but depending on how bad it was, focus could be lost for the rest of the day.

Then I became freelance. Meetings took on a whole new significance. These jackasses paying my rate because I'm good at a few things; but rather than have me do those things, I'm sitting in a meeting, bored, but well paid.

Look at your contractors in the next meeting; if they aren't in the scapegoat chair, they are the only happy ones in the room.

Mutual dislike between managers and coders

[140Mandak262Jamuna]

People who are passionate about coding, rarely make it to the management cadres, partly by choice, partly because they lack the other skills needed to be managers. So the management is filled with folks from sales, marketing and mediocre programmers.

I think may be the guru-disciple system of education practiced (allegedly) in ancient India might bear better fruits. Guru lead teams with disciples learning from them might turn out better quality code, but the system would be expensive in the short run and takes a while to take root. The quarterly bottom line obsessed corporate world is as far away from the system of stoic ascetic guru living in a hut in a jungle accepting princes as students romanticized in Indian mythology.

Re:Mutual dislike between managers and coders

[tripleevenfall]

There's a lot of truth to what you say. I am one of those. I think I was a "B" grade coder. Not a star, but adequate. I developed the skill, met deadlines and specs. It just wasn't my calling like it is for some.

Frankly, I don't think it's bad to have somewhat mediocre programmers in your management structure. We understand at least a good chunk of what developers are doing, and when we don't you can explain it to us and get understanding. If I'm a McManager who used to be in HR and has never written code, I'm not going to understand your basic needs as an engineering team. You won't be able to explain to me why a certain architecture isn't workable. I understand you and what you need 80% of the time and I can go fight those battles and leave you to code.

I think it's good to have mediocre programmers become managers if they have the management skills necessary (and aren't simply promoted because everyone else is irreplacable). Most of the time those skills are not common to the skillset of the best developers. It's better to have average developers become good managers than having good developers moved out of programming and into management, leaving only mediocre programmers writing mediocre code.

Re:Does anyone know the Happy Medium?

[tripleevenfall]

Where I used to work, (I posted elsewhere here why I left development), part of the problem is that the attitude was always "another small task won't hurt engineering". "Another step in this process is not a big deal"... until there are so many of these tips and checks that you aren't doing anything else but these microtasks.

I think that most places have big problems going cheap on staff. They cheap out on testing staff. They cheap out on training for the support people. They cheap out on resources for environments. All of these things cause more weight to be placed on the developers.

And it's all by design... develppers are replaceable, in many companies' view - more are churned out of college annually and they only have a 2-5 year lifespan on average. Rather than expand budgets to reflect what development really should cost, they simply treat developers as disposable resources on a burnout/replace cycle.

Re:Over my head

[Derkec]

70% Unit Coverage:
        -- You've written code level tests that flex 70% of your code checking for regression failures.

CCN:
        -- Technical term you can look up, but basically it's a measure of how many decision points are in a block of code. Less decision points is simpler. Too many and you may have something difficult to test and difficult for a programmer to understand. Higher complexity generally means more risk and a higher need for testing of various types.

Presprint grooming:
        -- A "sprint" is a time block set aside for development. Usually 2-8 weeks. The goal is declare what you're going to get done in that time and not change the requirements during that time. Between sprints, you can change your processes, "groom" stories (tasks that describe things in a user experience way generally).

Test driven dev:
        -- When writing a new feature, right a test for that feature first and you're close to done when the test passes and you haven't broken other tests.

Distortion: construction is free

[dazedNconfuzed]

In most engineering disciplines, the process of actually building something is long, hard, expensive, and persistent. If the project is build a bridge, you spend a lot of time making sure the design is right; why? because the process of actually building the bridge takes months or years, costs many millions of dollars, and once built is not easily replaced. There is no room for error, so process is taken very seriously as a central part of ensuring timely cost-effective correctness.

In software, the process of actually building something is instant, easy, free, and transient. Type "make all" and go get some coffee; find a bug? tweak a couple lines and do it again. This distorts the development process; "process" gets snubbed as a costly distracting annoyance instead of the means of assuring that what gets delivered is correct, because it's just so dang easy to fix and rebuild in seconds ... losing sight of the long-term cost of not doing it right the first time.

Re:Distortion: construction is free

[Tridus]

A bridge is also a fixed, well-known thing. It's not going to change radically in design between when you start and when you finish building it.

Software on the other hand is written for customers who themselves don't know what they want, in a market that is probably rapidly changing. It WILL change between the start and the end of the project in a lot of cases. Sometimes it changes because the customer changes their mind. Sometimes it changes because the market changes. Sometimes laws change. Sometimes the customers were just flat out wrong in everything they told you and the entire design is wrong as a reuslt.

When you're dealing with that, process does just get in the way.

Re:Distortion: construction is free

[geekoid]

" It's not going to change radically in design between when you start and when you finish building it."

not true.
In the process things gt changed, but it's part of the process. This includes new features added while building.

You don't see it becasue:
A) engineers can attach real world costs to any change and need to get someone to sign off on the costs.

B) You don't build bridges, so you don't see any development issues.

Properly engineered code would mean when a change comes up, you say 'it will take this long, and cost this much'. PLease sign the document stating this. You make the sign off a requirement for the developer. Meaning they get fired or fined. This way you protect them and have someon to take responsibility.

When you go to the customer and say, sure we can do this,. It will take 2 months and cost you 20K extra. they may rethink their 'minor' change.

Since the industry doesn't have that, I do it in email. They few times someone tried to call me out, I just forwarded and email and said 'I told my manager it would take this much longer and cost this much more and he approved it.'
While not a lot of legal protection there, it has worked.
Right now I work with a bunch of civil engineers. They get all the same crap software developers do BUT they have legal requirements for sign off, and protection if someone is trying to build something that isn't structurally safe.

When dealing with that, the process will help you because as you go you will have a base of knowledge about how long something takes and what it costs.

If someone wants a , without changing in parameters you need to call them out. Have a process means you have factual backing you can document.

They say that if software developers built house, civilization would fall when the first woodpecker should up.

I wish that was true, because after the last house fell, we would fix it. Instead the industry just slogs through this miasma of low quality slap it together crap over and over again.

Re:TDD

[tigre]

TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

No, TDD is painful, but it's a long-term investment. If you code once and never have to touch it again, you can get away with making it work right without tests. But if you want to be able to grow your system, you want something to make sure that you don't kill the old functionality in the process of building the new. And sometimes it's just helpful to spell out the expectations of new functionality ahead of time so that you know when you've achieved it.

Process for the sake of process is pointless. And if you have good reason to work around or jettison a given process, go for it. As I told the other programmers at my job, the only piece of our process we would probably consider an absolute necessity is source control. Code review is highly recommended, tests are very strongly pushed, formatting standards are there for good reasons, but there are times when they are not helpful.

The problem always comes down to whether or not your developers are adults with good decision-making capabilities. Process is too often employed as a way to allow you to get stuff done with people of average or less ability, but process also hurts them because they can't figure out when the process is unhelpful. And if process is imposed by fiat, those who could figure that out are frustrated by having to go through the motions. But if you have trustworthy people and you actually trust them, you institute the process for their benefit, and when it's not useful they know not to use it.

Budget

[mrops]

My problem with process has always been budget. Folks higher up budget on the basis of minimal process and expect full process. If you want 70% unit-test coverage, that is twice the time the code would have taken if you did not write unit test. Add 3 times the time if you want good integration tests to go with it.

Unfortunately, this makes a project costly. The problem occurs if the PM then demands full process when the time is not accordingly budgeted for it.

Release cycles also become long.

Re:Budget

[AuMatar]

80% code coverage costing less than 10% development time? Only if you're either ignoring all errors, or have massive try catch blocks that catch everything generically (which means your code coverage numbers are garbage- if you aren't testing each way of generating each type of error, you only have a fraction of that coverage).

I'm not going to deny the usefulness of unit tests, but in my experience writing a good unit test suite takes 50-200% of the time to code the function, depending on the complexity of the code being tested.

Re:Passion isn't important

[LordLucless]

And if you kill the passion, your cost (of hiring new developers because you drove the old ones away) and risk (of losing corporate knowledge in process) increases. So you can say "my money, my rules" all you like, but actually, it's reality that dictates the rules. And reality says if you burden people under a weight of bureaucracy disproportionate to what that bureaucracy is intended to accomplish, they'll leave. And if they don't leave, they're probably afraid of not finding another job, indicating that they're not good enough at their job to be confident in their abilities.

Like pretty much everything, it's a balancing act. You need to provide direction, or the goose is going to go wandering around the yard instead of laying its damn eggs, but if you stifle it too much, the thing's going to croak.