Is Process Killing the Software Industry?

blackbearnh writes "We all know by now that Test Driven Development is a best practice. And so is having 100% of your code reviewed. And 70% unit test coverage. And keeping your CCN complexity numbers below 20. And doing pre-sprint grooming of stories. And a hundred other industry 'best practices' that in isolation seem like a great idea. But at the end of the day, how much time does it leave for developers to be innovative and creative? A piece on O'Reilly Radar argues that excessive process in software development is sucking the life out of passionate developers, all in the name of making sure that 'good code' gets written. 'The underlying feedback loop making this progressively worse is that passionate programmers write great code, but process kills passion. Disaffected programmers write poor code, and poor code makes management add more process in an attempt to "make" their programmers write good code. That just makes morale worse, and so on.'"

Comments

"Creative"

[Anonymous Coward]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

Re:"Creative"

[Entrope]

This is dead on. Every software-developing business needs to decide its own process needs. Even if they are developing safety-critical code that has to pass rigorous certifications (FDA, DO-178B, etc), sane people do not order everything on the process menu. However, an organization does have to look at its market, its code base, its structure/culture, and its history to figure out what kinds of process it should use. Sure, having a defined process means developers spend less time throwing code at the wall, but the code they do throw usually sticks to the wall better and longer, and they usually feel good about that.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

Re:"Creative"

[donscarletti]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

You need creativity to write something accurate, elegant and easily reviewed, tested and verified. You need discipline to actually do the review, testing and verification steps. Both are equally important if you want something solid and delivered on time.

Re:"Creative"

[heathen_01]

This is dead on. Every software-developing business needs to decide its own process needs.

This is just another cargo cult problem. Managers, seeing that another company/team use process X on a successful project, decide to implement the same process for their team. However no process will make a difference if the developers have been directed to build the wrong solution in the first place, even if the code is 100% bug free.

Re:"Creative"

[joebagodonuts]

...and you've exposed the secret - developers can create and deliver good code WITHOUT being audited to death. The whole "lean six-sigma/ITIL/insert-the-name-of-your-favorite-process-methodology-here" is just another way for bean-counters and MBAs to pretend they can do something to add value to an endeavor.

Re:"Creative"

[Omnifarious]

I rated you up, but I notice that most people seem to be missing the most important part of your post.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

I agree that software development can be 'over-processed'. And I think that management frequently decides to 'apply more process!' in lieu of actually carefully thinking through what would help. But usually, when developers complain about process and delivering things on time, I think "Oh, so you just want to slap that code in there without even really knowing if it works. Write code, compile, problem solved!". Good process is what keeps you on-track and disciplined to write quality software. Too much software in our industry is of extremely poor quality, and that's still the case. Developers should feel proud of writing software that works, not how much software they write.

Most of the responses I see to your post seem to be of the "You're darn right, all that process just kills development!" variety. No talking about how to balance things for quality results or anything of that nature. I'm kind of disappointed.

Re:"Creative"

[Archangel Michael]

Was gonna mod you up, but instead, it hit me. The problem isn't the process, it is where the process is engaged. And I think the Article and subsequent posts clearly indicate there is a time and place for process, but not everything needs to be stuck in a process.

In your case, you and your team were building tools for your own use and were doing it the quick and agile way. Dirty code that got stuff done quickly. You bypassed the systemic processes that were in place because they just got in the way.

The successful code base often starts out as UGLY spaghetti code, but eventually needs to be cleaned up.

What needs happen is your skunkworks program should be autonomous from the normal process. Only when something you've created gets noticed does it move from the skunkworks over to the normal development channel. To accomplish this, you have to be willing to hand off your code to someone who is, more than likely, going to "ruin" it by running it through the process of getting it clean and neat.

This way, you can still build fast efficient code that is messy and ugly, and the managment can get code that is functional, and everyone can work towards getting it all nice and pretty by process. Everyone wins.

Just my thought

Re:"Creative"

[Skapare]

And the certifications don't actually check any code, they just check that the processes chosen are the right ones, and can even miss if the processes don't get adhered to. Perfectly developed and certified software can still have a ton of bugs that will crash that 777 or shut off the heart monitor in a hospital.

Less on process and more on coder competency would help. Pick the best coders and double their pay.

Does anyone know the Happy Medium?

[AdmiralXyz]

I'm all-too-aware of this issue and how quickly it sucks the life out of you and prevents anything from getting done, but at the same time obviously having no process doesn't lead to stellar code either. My question is, do any of you work in a place where you think you've struck the right balance? What are you doing?

The one process to rule them all

[Tribaal_ch]

Karma be damned, this is relevant to TFA:

Mutual dislike between managers and coders

[140Mandak262Jamuna]

People who are passionate about coding, rarely make it to the management cadres, partly by choice, partly because they lack the other skills needed to be managers. So the management is filled with folks from sales, marketing and mediocre programmers.

I think may be the guru-disciple system of education practiced (allegedly) in ancient India might bear better fruits. Guru lead teams with disciples learning from them might turn out better quality code, but the system would be expensive in the short run and takes a while to take root. The quarterly bottom line obsessed corporate world is as far away from the system of stoic ascetic guru living in a hut in a jungle accepting princes as students romanticized in Indian mythology.

Re:Mutual dislike between managers and coders

[tripleevenfall]

There's a lot of truth to what you say. I am one of those. I think I was a "B" grade coder. Not a star, but adequate. I developed the skill, met deadlines and specs. It just wasn't my calling like it is for some.

Frankly, I don't think it's bad to have somewhat mediocre programmers in your management structure. We understand at least a good chunk of what developers are doing, and when we don't you can explain it to us and get understanding. If I'm a McManager who used to be in HR and has never written code, I'm not going to understand your basic needs as an engineering team. You won't be able to explain to me why a certain architecture isn't workable. I understand you and what you need 80% of the time and I can go fight those battles and leave you to code.

I think it's good to have mediocre programmers become managers if they have the management skills necessary (and aren't simply promoted because everyone else is irreplacable). Most of the time those skills are not common to the skillset of the best developers. It's better to have average developers become good managers than having good developers moved out of programming and into management, leaving only mediocre programmers writing mediocre code.

TDD

[Anonymous Coward]

"We all know by now that Test Driven Development is a best practice."

No, TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

Re:TDD

[tigre]

TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

No, TDD is painful, but it's a long-term investment. If you code once and never have to touch it again, you can get away with making it work right without tests. But if you want to be able to grow your system, you want something to make sure that you don't kill the old functionality in the process of building the new. And sometimes it's just helpful to spell out the expectations of new functionality ahead of time so that you know when you've achieved it.

Process for the sake of process is pointless. And if you have good reason to work around or jettison a given process, go for it. As I told the other programmers at my job, the only piece of our process we would probably consider an absolute necessity is source control. Code review is highly recommended, tests are very strongly pushed, formatting standards are there for good reasons, but there are times when they are not helpful.

The problem always comes down to whether or not your developers are adults with good decision-making capabilities. Process is too often employed as a way to allow you to get stuff done with people of average or less ability, but process also hurts them because they can't figure out when the process is unhelpful. And if process is imposed by fiat, those who could figure that out are frustrated by having to go through the motions. But if you have trustworthy people and you actually trust them, you institute the process for their benefit, and when it's not useful they know not to use it.

Measuring Code

[rwv]

The main issue is that measuring whether code is good or not is impossible. I promise you that I can write code that has the prescribed level of unit testing and complexity that also doesn't work. Software reliability/dependability was a problem in the 1960s. It's still a problem today. No silver bullet, and all that.

Devs should own the process

[Derkec]

This is really the key insight of most Agile methodologies. Development should own the process and change it to suit their needs during regular retrospectives. The team (not the whining individual) should be able to say, "You know what, I think we're not getting bang for our buck out of this many unit tests, let's shift to 50% coverage." As long as that same team is taking ownership of the regression failures and making an informed trade-off their comfortable with, all is well.

If you get a good team together, you're going to get good code. You'll get better code if you empower them. Experienced and good teams will usually have a lot of these processes and tools in place because noticing things like high code complexity automagically alerts them to "bad smells" that can be examined and either accepted as necessary or invested in to address or test more thouroughly.

Generally, I think development is most fun when you're on a new project and don't give a damn about breaking things. Then it's pure creation. But once an app is older and there's some weird code you're staring at you have to decide, "is this probably a bug, or is this a bug fix for some weird situation or platform?" That's when you wish that the guy having fun three years ago had written some damn tests.

Re:In a nutshell

[Anonymous Coward]

the problem is always the same:

how can manager get bonus lowering staff costs?

they get cheapo developers and throw the top notch methodologies at them in the hope those will make up for the devs inexperience and plain incompetence.

that gives in turn a bad name at methodologies and tools, because cheapo devs cannot understand nor benefit fully from them. what use is a pmd output to a guy who's never got the difference between passing by reference and passing by value?

and there are, lots of them. just check any java forum.

yet, the truth is that some of the methods and tools are a good helper for good and also top notch developers. if you work with the tools, and not by the tools, you can catch way more common errors, be notified of piece of code that may need some attention because grew too much in scope and features and lot of other small things that make your life easier.

but it has to be a cereful choice, based on your experience and knowledge, not something imposed because it's the last buzzword.

Distortion: construction is free

[dazedNconfuzed]

In most engineering disciplines, the process of actually building something is long, hard, expensive, and persistent. If the project is build a bridge, you spend a lot of time making sure the design is right; why? because the process of actually building the bridge takes months or years, costs many millions of dollars, and once built is not easily replaced. There is no room for error, so process is taken very seriously as a central part of ensuring timely cost-effective correctness.

In software, the process of actually building something is instant, easy, free, and transient. Type "make all" and go get some coffee; find a bug? tweak a couple lines and do it again. This distorts the development process; "process" gets snubbed as a costly distracting annoyance instead of the means of assuring that what gets delivered is correct, because it's just so dang easy to fix and rebuild in seconds ... losing sight of the long-term cost of not doing it right the first time.

Re:Distortion: construction is free

[Tridus]

A bridge is also a fixed, well-known thing. It's not going to change radically in design between when you start and when you finish building it.

Software on the other hand is written for customers who themselves don't know what they want, in a market that is probably rapidly changing. It WILL change between the start and the end of the project in a lot of cases. Sometimes it changes because the customer changes their mind. Sometimes it changes because the market changes. Sometimes laws change. Sometimes the customers were just flat out wrong in everything they told you and the entire design is wrong as a reuslt.

When you're dealing with that, process does just get in the way.

Re:Distortion: construction is free

[geekoid]

" It's not going to change radically in design between when you start and when you finish building it."

not true.
In the process things gt changed, but it's part of the process. This includes new features added while building.

You don't see it becasue:
A) engineers can attach real world costs to any change and need to get someone to sign off on the costs.

B) You don't build bridges, so you don't see any development issues.

Properly engineered code would mean when a change comes up, you say 'it will take this long, and cost this much'. PLease sign the document stating this. You make the sign off a requirement for the developer. Meaning they get fired or fined. This way you protect them and have someon to take responsibility.

When you go to the customer and say, sure we can do this,. It will take 2 months and cost you 20K extra. they may rethink their 'minor' change.

Since the industry doesn't have that, I do it in email. They few times someone tried to call me out, I just forwarded and email and said 'I told my manager it would take this much longer and cost this much more and he approved it.'
While not a lot of legal protection there, it has worked.
Right now I work with a bunch of civil engineers. They get all the same crap software developers do BUT they have legal requirements for sign off, and protection if someone is trying to build something that isn't structurally safe.

When dealing with that, the process will help you because as you go you will have a base of knowledge about how long something takes and what it costs.

If someone wants a , without changing in parameters you need to call them out. Have a process means you have factual backing you can document.

They say that if software developers built house, civilization would fall when the first woodpecker should up.

I wish that was true, because after the last house fell, we would fix it. Instead the industry just slogs through this miasma of low quality slap it together crap over and over again.

Re:Passion isn't important

[Derkec]

I firmly disagree. Passion is key. People who don't give a damn and people who don't enjoy coding tend to write crap. We try to hire for passion and smarts (knowing both are hard to interview for).

Passionate people given good processes and tools are ideal.

Budget

[mrops]

My problem with process has always been budget. Folks higher up budget on the basis of minimal process and expect full process. If you want 70% unit-test coverage, that is twice the time the code would have taken if you did not write unit test. Add 3 times the time if you want good integration tests to go with it.

Unfortunately, this makes a project costly. The problem occurs if the PM then demands full process when the time is not accordingly budgeted for it.

Release cycles also become long.

Re:Passion isn't important

[LordLucless]

And if you kill the passion, your cost (of hiring new developers because you drove the old ones away) and risk (of losing corporate knowledge in process) increases. So you can say "my money, my rules" all you like, but actually, it's reality that dictates the rules. And reality says if you burden people under a weight of bureaucracy disproportionate to what that bureaucracy is intended to accomplish, they'll leave. And if they don't leave, they're probably afraid of not finding another job, indicating that they're not good enough at their job to be confident in their abilities.

Like pretty much everything, it's a balancing act. You need to provide direction, or the goose is going to go wandering around the yard instead of laying its damn eggs, but if you stifle it too much, the thing's going to croak.